| Seq | Timestamp | Type | Name | Duration |
|---|---|---|---|---|
| 0/0 | 2026/09/20 01:00 | flow | patch-triage |
11mArgs:null Results: {
"EnableConfigs": [
"TLS",
"TLS_DEVICE",
"INET"
],
"FocusSymbols": [
"tls_setsockopt",
"do_tls_setsockopt",
"do_tls_setsockopt_conf",
"tls_set_sw_offload",
"tls_sw_ctx_init",
"tls_sw_ctx_finalize",
"tls_set_device_offload",
"tls_set_device_offload_rx",
"tls_sw_drain_tx",
"tls_sw_sendmsg_locked",
"tls_validate_xmit_skb_rekey"
],
"KMSANReasoning": "The patch series implements TLS 1.3 hardware offload and key rotation (KeyUpdate) support for kTLS (in net/tls and driver offloads like mlx5).\n\nAnalysis of memory initialization and safety:\n1. Heap/Page Allocations:\n - All newly added context structures (`tls_offload_context_tx`, `tls_offload_context_rx`) continue to be allocated via kzalloc (`kzalloc_obj`), guaranteeing zero-initialization of newly added fields (e.g. `rekey`).\n - The start marker record (`offload_ctx-\u003erekey.start_marker`) is allocated and its fields (`end_seq`, `len = 0`, `num_frags = 0`) are explicitly initialized prior to being queued on `records_list`; its `frags` are not accessed when `num_frags == 0`.\n - The fallback page `dummy_page` in `tls_device_init()` is explicitly allocated with `__GFP_ZERO` and initialized with an identity map specifically to avoid any uninitialized byte leak.\n2. User-Space Info Leaks (getsockopt):\n - In `do_tls_getsockopt_conf()`, reading the rekey crypto info accesses `offload_ctx-\u003erekey.crypto_send.info`. This union is originally populated from user input into a zero-initialized union (`tmp = {}`) and copied into the kzalloc'd `offload_ctx`. When returned via `copy_to_iter()`, strictly `cipher_desc-\u003ecrypto_info` bytes are copied, leaving no uninitialized padding or fields.\n3. sk_buff Bitfield:\n - The new `decrypt_failed` bitfield in `struct sk_buff` resides in the header area zeroed out by `__alloc_skb()` upon allocation, and is maintained via `skb_copy_decrypted()`.\n4. KASAN vs KMSAN Applicability:\n - The risks introduced by this patch series pertain to asynchronous state machine transitions, concurrent RCU dereferences / reader-writer locking (`tx_lock`, `device_offload_lock`, `lock_sock`), and lifetime management of crypto AEAD transforms during fallback / rekeying.\n - Any bugs arising from these changes (e.g. use-after-free of AEAD transforms, double frees, list corruption, or deadlock) are detected by KASAN and LOCKDEP.\n - There are no risks of uninitialized memory reads or information leaks that would require KMSAN detection.\n\nTherefore, a dedicated KMSAN fuzzing session is not justified.",
"NeedsKMSAN": false,
"Reasoning": "The patch modifies both core networking and the kTLS subsystem to support TLS 1.3 KeyUpdate/rekeying. While dedicated hardware offload execution relies on physical SmartNICs, the patch significantly alters reachable software paths that execute in standard virtualized environments (QEMU):\n1. `do_tls_setsockopt` introduces mutex acquisition of `tx_lock` prior to `lock_sock`, altering the locking hierarchy for all kTLS sockets (software or hardware).\n2. `do_tls_setsockopt_conf` invokes `tls_set_device_offload` and `tls_set_device_offload_rx` for initial and rekey configuration checks before falling back to `tls_set_sw_offload`.\n3. `tls_set_sw_offload` has been refactored into `tls_sw_ctx_init` and `tls_sw_ctx_finalize`, changing cipher allocation, setkey, and sequence initialization in software kTLS.\n4. Data path encryption (`tls_do_encryption`, `tls_push_record`) and transmission paths (`tls_sw_sendmsg_locked`, `tls_sw_drain_tx`) now integrate rekey cipher context accessors.\n5. Core networking changes in `net/sched/sch_generic.c` (`dequeue_skb`) and `include/net/tcp.h` (`tcp_write_collapse_fence`) affect socket packet handling.\nThese changes modify reachable logic and synchronization invariants, warranting fuzzing.",
"WorthFuzzing": true
} |
| 1/1 | 2026/09/20 01:00 | action | read-patch-diff |
0mArgs:null Results: {
"PatchDiff": "commit 209c4bda0a621712323d0608683304a02d1a2596\nAuthor: syz-cluster \u003ctriage@syzkaller.com\u003e\nDate: Sun Sep 20 01:00:09 2026 +0000\n\n syz-cluster: applied patch under review\n\ndiff --git a/Documentation/networking/tls-offload.rst b/Documentation/networking/tls-offload.rst\nindex e5802bcd4d22d..cdf84f4b817a7 100644\n--- a/Documentation/networking/tls-offload.rst\n+++ b/Documentation/networking/tls-offload.rst\n@@ -99,9 +99,8 @@ at the end of kernel structures (see :c:member:`driver_state` members\n in ``include/net/tls.h``) to avoid additional allocations and pointer\n dereferences.\n \n-When the offloaded connection is destroyed the core calls\n-the :c:member:`tls_dev_del` callback so the driver can release per-direction\n-state:\n+The core calls the :c:member:`tls_dev_del` callback so the driver can release\n+per-direction state:\n \n .. code-block:: c\n \n@@ -109,7 +108,14 @@ state:\n \t\t\t struct tls_context *ctx,\n \t\t\t enum tls_offload_ctx_dir direction);\n \n-``tls_dev_del`` is mandatory whenever ``tls_dev_add`` is provided.\n+``tls_dev_del`` is called either when the offloaded connection is destroyed or,\n+for a TLS 1.3 connection, when the old key is retired during a rekey (see the\n+`Rekey`_ section). It operates on a single ``direction``, so the driver must\n+release only the state for that direction and must not free state shared\n+between directions or the socket as a whole. After a rekey ``tls_dev_del``,\n+``tls_dev_add`` may be called again for the same socket and direction to\n+install the new key. ``tls_dev_del`` is mandatory whenever ``tls_dev_add`` is\n+provided.\n \n The third TLS device callback is :c:member:`tls_dev_resync`, called by the core\n to synchronize the TCP stream with the record boundaries:\n@@ -205,7 +211,10 @@ Upon reception of a TLS offloaded packet, the driver sets\n the :c:member:`decrypted` mark in :c:type:`struct sk_buff \u003csk_buff\u003e`\n corresponding to the segment. Networking stack makes sure decrypted\n and non-decrypted segments do not get coalesced (e.g. by GRO or socket layer)\n-and takes care of partial decryption.\n+and takes care of partial decryption. A segment the device processed but\n+could not authenticate may instead carry the :c:member:`decrypt_failed`\n+mark; see the `Error handling`_ section for what the mark implies about\n+the payload.\n \n Resync handling\n ===============\n@@ -404,8 +413,121 @@ records, then after 4 records, after 8, after 16... up until every\n Rekey\n =====\n \n-Offload does not currently support TLS 1.3, therefore key rotation\n-is not a concern for offloaded connections at this point.\n+TLS 1.3 allows traffic keys to be updated mid-connection using the\n+KeyUpdate message. Offloaded TLS 1.3 connections must therefore switch\n+keys without tearing down the offload. The device cannot simply be given\n+the new key because records encrypted (TX) or transformed (RX) with the\n+old key may still be in flight. The stack retains the necessary old-key\n+state and bridges the transition in software.\n+\n+TX\n+--\n+\n+On TX, the new key is installed in a temporary software context, and\n+sendmsg is routed through the software path. If no hardware-offloaded\n+records remain unacknowledged, the switch completes inline during\n+setsockopt. Otherwise the rekey is left pending and is completed later,\n+on the sender's next ``sendmsg()`` after all old-key records have been\n+ACKed (see `Completing a deferred rekey`_). Completion calls\n+:c:func:`tls_dev_del` for the old key and reinstalls hardware offload\n+with the new key at the current TCP write sequence. If reinstallation\n+fails, the connection keeps encrypting in software with the new key; the\n+next KeyUpdate re-arms the transition and retries the hardware\n+installation.\n+\n+Unlike the software path, a ``TLS_TX`` setsockopt on an offloaded\n+connection first flushes the open and partially sent hardware records to\n+TCP before installing the new key. It therefore behaves like a blocking\n+``send()`` of that record: it may wait for send buffer space (bounded by\n+``SO_SNDTIMEO``), and on a non-blocking socket it fails with ``-EAGAIN``\n+and must be retried once the socket is writable. The new key is not\n+installed until the call succeeds; the connection keeps using the old key\n+in the meantime.\n+\n+Completing a deferred rekey\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+A deferred rekey is completed by the sender, not by the ACK path. When\n+the last old-key record is acknowledged the stack only marks the rekey\n+as ready; the device is not touched. The switch itself,\n+:c:func:`tls_dev_del` of the old key followed by :c:func:`tls_dev_add`\n+of the new one, runs at the start of the next ``sendmsg()`` on the\n+socket, and that ``sendmsg()`` is the first to be encrypted by hardware\n+again. No other event completes it: ``splice_eof()``, write-space\n+wakeups, retransmissions and pure ACKs all leave the connection on the\n+software path.\n+\n+This is intentional. Completion has to flush the software context's\n+open record to TCP and may sleep for send buffer space, which rules out\n+the ACK and write-space paths. Beyond that, the stack only switches when\n+it has new data to hand to the device: the software path is fully\n+correct with the new key, so deferring the switch costs host CPU but\n+nothing else, and it keeps the device from being programmed for a\n+connection that may never send again.\n+\n+Two consequences follow. A connection that stops sending after a\n+KeyUpdate stays in the deferred state until it is closed: it is\n+encrypted in software with the new key, it is counted in\n+``TlsCurrTxRekey``, and at close it is reported as\n+``TlsTxRekeyAborted``. That counter therefore includes senders that\n+simply had nothing more to send, not only sockets torn down\n+mid-transition, and is not by itself an error indication. And the return\n+to hardware is delayed by at least one ACK round trip after the last\n+old-key record, plus however long the application waits before its next\n+``sendmsg()``. A sender that wants the hardware path back promptly can\n+issue a small ``sendmsg()`` once its old data has been acknowledged.\n+\n+Completion can fail transiently or permanently. If the software flush\n+cannot get send buffer space (``-EAGAIN``, or a signal on a blocking\n+socket) the rekey stays pending, the ``sendmsg()`` proceeds in software,\n+and the next ``sendmsg()`` retries; the ``tls_device_complete_rekey_retry``\n+tracepoint fires. A hard failure (:c:func:`tls_dev_add` rejected, or the\n+netdev gone) is terminal for this KeyUpdate: the connection is pinned to\n+software encryption with the new key, counted in ``TlsTxRekeyFallback``\n+and moved from ``TlsCurrTxDevice`` to ``TlsCurrTxSw``; the\n+``tls_device_complete_rekey_fail`` tracepoint fires. The next ``TLS_TX``\n+setsockopt re-arms the transition and retries.\n+\n+The decision to defer is taken at the start of the ``TLS_TX``\n+setsockopt, before the open hardware record is flushed to TCP. That\n+flush may block for send buffer space, and old-key records acknowledged\n+while it sleeps do not change the decision: the rekey is still deferred\n+and completes on a following ``sendmsg()`` rather than inline. This is\n+conservative, not a correctness issue. The boundary is fixed at the\n+write sequence after the flush, so the acknowledgment of the flushed\n+record itself arms completion; the cost is one more ACK round trip and\n+one more ``sendmsg()``. Applications should not expect an inline switch\n+whenever the socket has unacknowledged data at the time of the\n+setsockopt.\n+\n+RX\n+--\n+\n+On RX, the NIC may already have transformed in-flight records with the\n+old key before the peer's KeyUpdate is parsed. When the KeyUpdate is\n+decoded, the stack removes the old key from the NIC but retains the old\n+AEAD, IV, and record sequence in the software offload context.\n+\n+Each record is classified by the TCP sequence of its first byte relative\n+to the boundary at which the NIC stopped using the old key. Records\n+starting after that boundary carry new-key wire encryption, so the old\n+software AEAD state can be released. Records before the boundary that\n+remain fully encrypted are passed to the software path. Records that\n+were partially transformed by the NIC are re-encrypted with the old key\n+to restore the new-key ciphertext, allowing the software AEAD to decrypt\n+them with the new key.\n+\n+If old-key records are still queued, installation of the new key through\n+:c:func:`tls_dev_add` is deferred until those records have been consumed;\n+otherwise it occurs immediately. When the NIC cannot authenticate a record\n+processed during the transition, the affected fragments are delivered with\n+``skb-\u003edecrypt_failed`` set, following the contract described in the\n+`Error handling`_ section. In a mixed record such a fragment was\n+transformed (XORed) with the old key, and the re-encrypt path uses this to\n+undo the transform on those fragments with the old key while leaving\n+untouched fragments intact. A non-mixed record carrying\n+``skb-\u003edecrypt_failed`` was not transformed; it is still wire ciphertext\n+and is decrypted directly by the software AEAD under the new key.\n \n Error handling\n ==============\n@@ -442,8 +564,43 @@ to the host's stack as it was on the wire (recovering original packet in the\n driver if device provides precise error is sufficient).\n \n The Linux networking stack does not provide a way of reporting per-packet\n-decryption and authentication errors, packets with errors must simply not\n-have the :c:member:`decrypted` mark set.\n+decryption and authentication errors. A packet with errors must not have\n+the :c:member:`decrypted` mark set. In addition, the driver may set the\n+:c:member:`decrypt_failed` mark on a segment the device matched to an\n+offloaded connection and processed but could not authenticate. The two\n+marks are mutually exclusive.\n+\n+The stack interprets :c:member:`decrypt_failed` per record, relative to the\n+:c:member:`decrypted` mark of the other segments making up the same record.\n+Coalescing (GRO, socket layer) and record classification are keyed on\n+:c:member:`decrypted` alone, so :c:member:`decrypt_failed` segments may be\n+merged with unmarked ones. A driver setting the mark must therefore honour\n+the following contract:\n+\n+ * In a record none of whose segments carry :c:member:`decrypted`, every\n+ segment, including one with :c:member:`decrypt_failed` set, must hold\n+ the payload exactly as it was on the wire. This is the general rule\n+ above: if the device did not successfully decrypt any part of a record\n+ it must hand the whole record over untouched. The stack passes such a\n+ record to software decryption directly and does not consult\n+ :c:member:`decrypt_failed`.\n+\n+ * In a record where some segments carry :c:member:`decrypted` (a mixed\n+ record), a segment with :c:member:`decrypt_failed` set must hold payload\n+ the device has already transformed (XORed with the cipher keystream) but\n+ failed to authenticate, and a segment with neither mark must hold the\n+ payload as it was on the wire. The stack re-encrypts the\n+ :c:member:`decrypted` and :c:member:`decrypt_failed` segments to restore\n+ the ciphertext, leaves the unmarked segments intact, and authenticates\n+ the whole record in software.\n+\n+A transformed segment delivered without :c:member:`decrypt_failed`, or an\n+untransformed segment of a mixed record delivered with it, is restored\n+incorrectly and the record fails software authentication. A device which\n+cannot tell the driver whether a failed segment was transformed must\n+recover the original packet before handing it to the stack, as described\n+above, and leave both marks clear. During a TLS 1.3 rekey the mark also\n+tells the stack which key the device applied; see the `Rekey`_ section.\n \n A packet should also not be handled by the TLS offload if it contains\n incorrect checksums.\ndiff --git a/Documentation/networking/tls.rst b/Documentation/networking/tls.rst\nindex 980c442d7161a..cf05543260d85 100644\n--- a/Documentation/networking/tls.rst\n+++ b/Documentation/networking/tls.rst\n@@ -314,6 +314,11 @@ TLS implementation exposes the following per-namespace statistics\n number of TX and RX sessions currently installed where NIC handles\n cryptography\n \n+- ``TlsCurrTxRekey``, ``TlsCurrRxRekey`` -\n+ number of TX and RX sessions currently undergoing a deferred rekey,\n+ i.e. a rekey which could not be applied immediately and is waiting for\n+ in-flight records to drain before the new key is installed in hardware\n+\n - ``TlsTxSw``, ``TlsRxSw`` -\n number of TX and RX sessions opened with host cryptography\n \n@@ -344,3 +349,15 @@ TLS implementation exposes the following per-namespace statistics\n - ``TlsRxRekeyReceived`` -\n number of received KeyUpdate handshake messages, requiring userspace\n to provide a new RX key\n+\n+- ``TlsTxRekeyFallback``, ``TlsRxRekeyFallback`` -\n+ number of rekeys on existing sessions for TX and RX which could not be\n+ offloaded to the NIC and fell back to software cryptography\n+\n+- ``TlsTxRekeyAborted``, ``TlsRxRekeyAborted`` -\n+ number of deferred rekeys for TX and RX which were still pending when\n+ the socket was destroyed, and so never completed. For TX hardware\n+ offload this includes senders that sent nothing further after the\n+ KeyUpdate, since the switch back to hardware only happens on\n+ ``sendmsg()`` (see the Rekey section of\n+ Documentation/networking/tls-offload.rst)\ndiff --git a/MAINTAINERS b/MAINTAINERS\nindex 0e04d92d1b098..4f1645bf2ee5e 100644\n--- a/MAINTAINERS\n+++ b/MAINTAINERS\n@@ -19255,6 +19255,8 @@ F:\tDocumentation/networking/tls*\n F:\tinclude/net/tls.h\n F:\tinclude/uapi/linux/tls.h\n F:\tnet/tls/\n+F:\ttools/testing/selftests/drivers/net/hw/tls_hw_offload.c\n+F:\ttools/testing/selftests/drivers/net/hw/tls_hw_offload.py\n F:\ttools/testing/selftests/net/tls.c\n \n NETWORKING [SOCKETS]\ndiff --git a/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c b/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c\nindex f5acd4be1e69d..29e108ce67645 100644\n--- a/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c\n+++ b/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c\n@@ -431,6 +431,9 @@ static int chcr_ktls_dev_add(struct net_device *netdev, struct sock *sk,\n \tatomic64_inc(\u0026port_stats-\u003ektls_tx_connection_open);\n \tu_ctx = adap-\u003euld[CXGB4_ULD_KTLS].handle;\n \n+\tif (crypto_info-\u003eversion != TLS_1_2_VERSION)\n+\t\tgoto out;\n+\n \tif (direction == TLS_OFFLOAD_CTX_DIR_RX) {\n \t\tpr_err(\"not expecting for RX direction\\n\");\n \t\tgoto out;\ndiff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h\nindex 07a04a142a2ea..0469ca6a0762e 100644\n--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h\n+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h\n@@ -30,7 +30,9 @@ static inline bool mlx5e_is_ktls_device(struct mlx5_core_dev *mdev)\n \t\treturn false;\n \n \treturn (MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_128) ||\n-\t\tMLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_256));\n+\t\tMLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_256) ||\n+\t\tMLX5_CAP_TLS(mdev, tls_1_3_aes_gcm_128) ||\n+\t\tMLX5_CAP_TLS(mdev, tls_1_3_aes_gcm_256));\n }\n \n static inline bool mlx5e_ktls_type_check(struct mlx5_core_dev *mdev,\n@@ -40,10 +42,14 @@ static inline bool mlx5e_ktls_type_check(struct mlx5_core_dev *mdev,\n \tcase TLS_CIPHER_AES_GCM_128:\n \t\tif (crypto_info-\u003eversion == TLS_1_2_VERSION)\n \t\t\treturn MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_128);\n+\t\telse if (crypto_info-\u003eversion == TLS_1_3_VERSION)\n+\t\t\treturn MLX5_CAP_TLS(mdev, tls_1_3_aes_gcm_128);\n \t\tbreak;\n \tcase TLS_CIPHER_AES_GCM_256:\n \t\tif (crypto_info-\u003eversion == TLS_1_2_VERSION)\n \t\t\treturn MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_256);\n+\t\telse if (crypto_info-\u003eversion == TLS_1_3_VERSION)\n+\t\t\treturn MLX5_CAP_TLS(mdev, tls_1_3_aes_gcm_256);\n \t\tbreak;\n \t}\n \ndiff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c\nindex bca45679e2016..8ec40f5fd5b50 100644\n--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c\n+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c\n@@ -602,7 +602,18 @@ void mlx5e_ktls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,\n \t\tstats-\u003etls_resync_req_pkt++;\n \t\tresync_update_sn(rq, skb);\n \t\tbreak;\n-\tdefault: /* CQE_TLS_OFFLOAD_ERROR: */\n+\tcase CQE_TLS_OFFLOAD_ERROR:\n+\t\t/* The device could not authenticate the payload. Depending on\n+\t\t * where the failure occurred the bytes may have been transformed\n+\t\t * (XORed) or left as wire ciphertext. Flag it so that, during a\n+\t\t * TLS 1.3 rekey transition, the re-encrypt path undoes the\n+\t\t * transform on any XORed frag of a mixed record while software\n+\t\t * re-authenticates; a non-mixed record stays wire ciphertext and\n+\t\t * is decrypted directly.\n+\t\t */\n+\t\tskb-\u003edecrypt_failed = 1;\n+\t\tfallthrough;\n+\tdefault: /* CQE_TLS_OFFLOAD_NOT_DECRYPTED: */\n \t\tstats-\u003etls_err++;\n \t\tbreak;\n \t}\ndiff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c\nindex 570a912dd6faf..f3f1be1d40343 100644\n--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c\n+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c\n@@ -6,6 +6,7 @@\n \n enum {\n \tMLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2 = 0x2,\n+\tMLX5E_STATIC_PARAMS_CONTEXT_TLS_1_3 = 0x3,\n };\n \n enum {\n@@ -15,8 +16,10 @@ enum {\n #define EXTRACT_INFO_FIELDS do { \\\n \tsalt = info-\u003esalt; \\\n \trec_seq = info-\u003erec_seq; \\\n+\tiv = info-\u003eiv; \\\n \tsalt_sz = sizeof(info-\u003esalt); \\\n \trec_seq_sz = sizeof(info-\u003erec_seq); \\\n+\tiv_sz = sizeof(info-\u003eiv); \\\n } while (0)\n \n static void\n@@ -24,9 +27,9 @@ fill_static_params(struct mlx5_wqe_tls_static_params_seg *params,\n \t\t union mlx5e_crypto_info *crypto_info,\n \t\t u32 key_id, u32 resync_tcp_sn)\n {\n+\tu16 salt_sz, rec_seq_sz, iv_sz;\n+\tchar *salt, *rec_seq, *iv;\n \tchar *initial_rn, *gcm_iv;\n-\tu16 salt_sz, rec_seq_sz;\n-\tchar *salt, *rec_seq;\n \tu8 tls_version;\n \tu8 *ctx;\n \n@@ -59,7 +62,12 @@ fill_static_params(struct mlx5_wqe_tls_static_params_seg *params,\n \tmemcpy(gcm_iv, salt, salt_sz);\n \tmemcpy(initial_rn, rec_seq, rec_seq_sz);\n \n-\ttls_version = MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2;\n+\tif (crypto_info-\u003ecrypto_info.version == TLS_1_3_VERSION) {\n+\t\tmemcpy(gcm_iv + salt_sz, iv, iv_sz);\n+\t\ttls_version = MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_3;\n+\t} else {\n+\t\ttls_version = MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2;\n+\t}\n \n \tMLX5_SET(tls_static_params, ctx, tls_version, tls_version);\n \tMLX5_SET(tls_static_params, ctx, const_1, 1);\ndiff --git a/drivers/net/ethernet/netronome/nfp/crypto/tls.c b/drivers/net/ethernet/netronome/nfp/crypto/tls.c\nindex 9983d7aa2b9cd..13864c6a55dce 100644\n--- a/drivers/net/ethernet/netronome/nfp/crypto/tls.c\n+++ b/drivers/net/ethernet/netronome/nfp/crypto/tls.c\n@@ -287,6 +287,9 @@ nfp_net_tls_add(struct net_device *netdev, struct sock *sk,\n \tBUILD_BUG_ON(offsetof(struct nfp_net_tls_offload_ctx, rx_end) \u003e\n \t\t TLS_DRIVER_STATE_SIZE_RX);\n \n+\tif (crypto_info-\u003eversion != TLS_1_2_VERSION)\n+\t\treturn -EOPNOTSUPP;\n+\n \tif (!nfp_net_cipher_supported(nn, crypto_info-\u003ecipher_type, direction))\n \t\treturn -EOPNOTSUPP;\n \ndiff --git a/include/linux/skbuff.h b/include/linux/skbuff.h\nindex 421f6fc454511..5da2c1149d982 100644\n--- a/include/linux/skbuff.h\n+++ b/include/linux/skbuff.h\n@@ -851,6 +851,10 @@ enum skb_tstamp_type {\n *\t\tunreadable.\n *\t@dst_pending_confirm: need to confirm neighbour\n *\t@decrypted: Decrypted SKB\n+ *\t@decrypt_failed: hardware could not authenticate this skb's TLS payload.\n+ *\t\tThe payload may have been transformed (XORed) or left as wire\n+ *\t\tciphertext, so software must re-authenticate the record and undo the\n+ *\t\ttransform on any XORed fragment before it can be decrypted\n *\t@slow_gro: state present at GRO time, slower prepare step required\n *\t@tstamp_type: When set, skb-\u003etstamp has the\n *\t\tdelivery_time clock base of skb-\u003etstamp.\n@@ -1025,6 +1029,7 @@ struct sk_buff {\n #endif\n #ifdef CONFIG_SKB_DECRYPTED\n \t__u8\t\t\tdecrypted:1;\n+\t__u8\t\t\tdecrypt_failed:1;\n #endif\n \t__u8\t\t\tslow_gro:1;\n #if IS_ENABLED(CONFIG_IP_SCTP)\n@@ -1716,6 +1721,7 @@ static inline void skb_copy_decrypted(struct sk_buff *to,\n {\n #ifdef CONFIG_SKB_DECRYPTED\n \tto-\u003edecrypted = from-\u003edecrypted;\n+\tto-\u003edecrypt_failed = from-\u003edecrypt_failed;\n #endif\n }\n \ndiff --git a/include/net/tcp.h b/include/net/tcp.h\nindex 5e5f5f9b89a38..8c6d90e962c43 100644\n--- a/include/net/tcp.h\n+++ b/include/net/tcp.h\n@@ -2340,6 +2340,15 @@ static inline void tcp_write_collapse_fence(struct sock *sk)\n {\n \tstruct sk_buff *skb = tcp_write_queue_tail(sk);\n \n+\t/* When nothing is queued for transmit, the last skb of the current\n+\t * state is the rtx queue tail (its end_seq == snd_nxt == write_seq).\n+\t * Fence that instead, otherwise the boundary is left unmarked and a\n+\t * later tcp_retrans_try_collapse()/tcp_shift_skb_data() can merge it\n+\t * with the first skb of the next state across the fence (they only test\n+\t * the tail's EOR, not skb-\u003edecrypted).\n+\t */\n+\tif (!skb)\n+\t\tskb = tcp_rtx_queue_tail(sk);\n \tif (skb)\n \t\tTCP_SKB_CB(skb)-\u003eeor = 1;\n }\ndiff --git a/include/net/tls.h b/include/net/tls.h\nindex e57bef58851ea..6844a685d6e08 100644\n--- a/include/net/tls.h\n+++ b/include/net/tls.h\n@@ -155,6 +155,22 @@ struct tls_record_info {\n \tskb_frag_t frags[MAX_SKB_FRAGS];\n };\n \n+struct cipher_context {\n+\tchar iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE];\n+\tchar rec_seq[TLS_MAX_REC_SEQ_SIZE];\n+};\n+\n+union tls_crypto_context {\n+\tstruct tls_crypto_info info;\n+\tunion {\n+\t\tstruct tls12_crypto_info_aes_gcm_128 aes_gcm_128;\n+\t\tstruct tls12_crypto_info_aes_gcm_256 aes_gcm_256;\n+\t\tstruct tls12_crypto_info_chacha20_poly1305 chacha20_poly1305;\n+\t\tstruct tls12_crypto_info_sm4_gcm sm4_gcm;\n+\t\tstruct tls12_crypto_info_sm4_ccm sm4_ccm;\n+\t};\n+};\n+\n #define TLS_DRIVER_STATE_SIZE_TX\t16\n struct tls_offload_context_tx {\n \tstruct crypto_aead *aead_send;\n@@ -169,6 +185,14 @@ struct tls_offload_context_tx {\n \tvoid (*sk_destruct)(struct sock *sk);\n \tstruct work_struct destruct_work;\n \tstruct tls_context *ctx;\n+\n+\tstruct {\n+\t\tstruct tls_sw_context_tx sw;\t/* SW context for new key */\n+\t\tstruct cipher_context tx;\t/* IV, rec_seq for new key */\n+\t\tunion tls_crypto_context crypto_send; /* Crypto for new key */\n+\t\tstruct tls_record_info *start_marker;\n+\t} rekey;\n+\n \t/* The TLS layer reserves room for driver specific state\n \t * Currently the belief is that there is not enough\n \t * driver specific state to justify another layer of indirection\n@@ -187,28 +211,46 @@ enum tls_context_flags {\n \t * to be atomic.\n \t */\n \tTLS_TX_SYNC_SCHED = 1,\n-\t/* tls_dev_del was called for the RX side, device state was released,\n-\t * but tls_ctx-\u003enetdev might still be kept, because TX-side driver\n-\t * resources might not be released yet. Used to prevent the second\n-\t * tls_dev_del call in tls_device_down if it happens simultaneously.\n+\t/* tls_dev_del was called for the RX side, releasing the NIC's RX\n+\t * offload context, while tls_ctx-\u003enetdev is still kept (TX-side driver\n+\t * resources may not be released yet, or a rekey is about to re-add the\n+\t * context). Set in that case, and during a rekey before re-add, and\n+\t * cleared when tls_dev_add re-establishes the context. Readers use it to\n+\t * avoid a second tls_dev_del and to suppress resync while the NIC has no\n+\t * key. tls_device_down() sets it too, so the rekey paths can test the bit\n+\t * alone.\n \t */\n \tTLS_RX_DEV_CLOSED = 2,\n-};\n-\n-struct cipher_context {\n-\tchar iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE];\n-\tchar rec_seq[TLS_MAX_REC_SEQ_SIZE];\n-};\n-\n-union tls_crypto_context {\n-\tstruct tls_crypto_info info;\n-\tunion {\n-\t\tstruct tls12_crypto_info_aes_gcm_128 aes_gcm_128;\n-\t\tstruct tls12_crypto_info_aes_gcm_256 aes_gcm_256;\n-\t\tstruct tls12_crypto_info_chacha20_poly1305 chacha20_poly1305;\n-\t\tstruct tls12_crypto_info_sm4_gcm sm4_gcm;\n-\t\tstruct tls12_crypto_info_sm4_ccm sm4_ccm;\n-\t};\n+\t/* TX HW context has been tls_dev_del()'d (mid-rekey before the re-add,\n+\t * after a failed re-add, or by tls_device_down()); prevents a second\n+\t * tls_dev_del. Cleared when tls_dev_add re-establishes the context.\n+\t */\n+\tTLS_TX_DEV_CLOSED = 3,\n+\t/* TX rekey is pending, waiting for old-key data to be ACKed.\n+\t * While set, new data uses SW path with new key, HW keeps old key\n+\t * for retransmissions.\n+\t */\n+\tTLS_TX_REKEY_PENDING = 4,\n+\t/* All old-key data has been ACKed, ready to install new key in HW. */\n+\tTLS_TX_REKEY_READY = 5,\n+\t/* HW rekey failed; TX stays on the SW rekey context until the next\n+\t * KeyUpdate re-arms the transition (tls_device_start_rekey()). Also\n+\t * stops tls_tcp_clean_acked() from re-setting TLS_TX_REKEY_READY.\n+\t */\n+\tTLS_TX_REKEY_FAILED = 6,\n+\t/* A rekey has completed on this socket at least once; that arms\n+\t * tls_tx_drop_acked_clone() (see its header for the rationale). WARN\n+\t * avoidance only.\n+\t */\n+\tTLS_TX_REKEY_FLOOR = 7,\n+\t/* The RX side fell back to SW decryption during a rekey (tls_dev_add()\n+\t * failed, or the netdev is gone) and the socket has been moved from the\n+\t * TlsCurrRxDevice to the TlsCurrRxSw gauge while rx_conf stays TLS_HW.\n+\t * Accounting only: the functional state is TLS_RX_DEV_{DEGRADED,CLOSED}.\n+\t * Cleared, moving the socket back, when a later rekey re-adds the NIC\n+\t * context. Mirrors TLS_TX_REKEY_FAILED for the close-time decrement.\n+\t */\n+\tTLS_RX_REKEY_FAILED = 8,\n };\n \n struct tls_prot_info {\n@@ -257,6 +299,20 @@ struct tls_context {\n \t\t\t */\n \tunsigned long flags;\n \n+\tstruct {\n+\t\t/* TCP sequence number boundary for pending rekey.\n+\t\t * Packets with seq \u003c this use old key, \u003e= use new key.\n+\t\t */\n+\t\tu32 boundary_seq;\n+\n+\t\t/* SW encryption contexts for the new key, non-NULL only while\n+\t\t * TLS_TX_REKEY_{PENDING,FAILED}; consulted by tls_sw_ctx_tx() and\n+\t\t * tls_tx_cipher_ctx().\n+\t\t */\n+\t\tstruct tls_sw_context_tx *sw_ctx;\n+\t\tstruct cipher_context *cipher_ctx;\n+\t} rekey;\n+\n \t/* cache cold stuff */\n \tstruct proto *sk_proto;\n \tstruct sock *sk;\n@@ -315,6 +371,14 @@ struct tls_offload_context_rx {\n \tu8 resync_nh_reset:1;\n \t/* CORE_NEXT_HINT-only member, but use the hole here */\n \tu8 resync_nh_do_now:1;\n+\t/* tls_dev_add deferred until old key is freed */\n+\tu8 dev_add_pending:1;\n+\tstruct {\n+\t\tstruct crypto_aead *old_aead_recv; /* old key AEAD cipher */\n+\t\tchar old_iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE]; /* old key IV */\n+\t\tchar old_rec_seq[TLS_MAX_REC_SEQ_SIZE]; /* old key TLS record seq */\n+\t\tu32 old_nic_boundary; /* TCP seq below which the NIC may have used the old key */\n+\t} rekey;\n \tunion {\n \t\t/* TLS_OFFLOAD_SYNC_TYPE_DRIVER_REQ */\n \t\tstruct {\n@@ -356,15 +420,38 @@ tls_validate_xmit_skb(struct sock *sk, struct net_device *dev,\n struct sk_buff *\n tls_validate_xmit_skb_sw(struct sock *sk, struct net_device *dev,\n \t\t\t struct sk_buff *skb);\n+struct sk_buff *\n+tls_validate_xmit_skb_rekey(struct sock *sk, struct net_device *dev,\n+\t\t\t struct sk_buff *skb);\n \n static inline bool tls_is_skb_tx_device_offloaded(const struct sk_buff *skb)\n {\n #ifdef CONFIG_TLS_DEVICE\n \tstruct sock *sk = skb-\u003esk;\n+\ttypeof(sk-\u003esk_validate_xmit_skb) validate;\n+\n+\tif (!sk || !sk_fullsock(sk))\n+\t\treturn false;\n \n-\treturn sk \u0026\u0026 sk_fullsock(sk) \u0026\u0026\n-\t (smp_load_acquire(\u0026sk-\u003esk_validate_xmit_skb) ==\n-\t \u0026tls_validate_xmit_skb);\n+\t/* Pairs with the smp_store_release() that installs or swaps the\n+\t * validator (tls_set_device_offload() / tls_device_start_rekey()): the\n+\t * pointer read here is published together with the offload state it\n+\t * guards, so a non-NULL validator implies that state is visible.\n+\t */\n+\tvalidate = smp_load_acquire(\u0026sk-\u003esk_validate_xmit_skb);\n+\tif (likely(validate == \u0026tls_validate_xmit_skb))\n+\t\treturn true;\n+\n+\t/* A TX rekey (tls_device_start_rekey()) can swap in the rekey validator\n+\t * between this skb's validate_xmit_skb(), where the old validator\n+\t * passed it through as HW-offload plaintext, and here. A skb-\u003edecrypted\n+\t * skb under the rekey validator is therefore that straddler: old-key\n+\t * plaintext whose HW context is still installed (tls_dev_del() runs in\n+\t * tls_device_complete_rekey() only after a synchronize_net() that drains\n+\t * this in-flight xmit), so the NIC must still encrypt it. Everything else\n+\t * the rekey validator emits is ciphertext (skb-\u003edecrypted == 0).\n+\t */\n+\treturn validate == \u0026tls_validate_xmit_skb_rekey \u0026\u0026 skb_is_decrypted(skb);\n #else\n \treturn false;\n #endif\n@@ -389,9 +476,25 @@ static inline struct tls_sw_context_rx *tls_sw_ctx_rx(\n static inline struct tls_sw_context_tx *tls_sw_ctx_tx(\n \t\tconst struct tls_context *tls_ctx)\n {\n+\tstruct tls_sw_context_tx *rekey_ctx = READ_ONCE(tls_ctx-\u003erekey.sw_ctx);\n+\n+\tif (unlikely(rekey_ctx))\n+\t\treturn rekey_ctx;\n+\n \treturn (struct tls_sw_context_tx *)tls_ctx-\u003epriv_ctx_tx;\n }\n \n+static inline struct cipher_context *tls_tx_cipher_ctx(\n+\t\tconst struct tls_context *tls_ctx)\n+{\n+\tstruct cipher_context *rekey_ctx = READ_ONCE(tls_ctx-\u003erekey.cipher_ctx);\n+\n+\tif (unlikely(rekey_ctx))\n+\t\treturn rekey_ctx;\n+\n+\treturn (struct cipher_context *)\u0026tls_ctx-\u003etx;\n+}\n+\n static inline struct tls_offload_context_tx *\n tls_offload_ctx_tx(const struct tls_context *tls_ctx)\n {\ndiff --git a/include/uapi/linux/snmp.h b/include/uapi/linux/snmp.h\nindex 49f5640092a0d..423aec9ae4cac 100644\n--- a/include/uapi/linux/snmp.h\n+++ b/include/uapi/linux/snmp.h\n@@ -369,6 +369,12 @@ enum\n \tLINUX_MIB_TLSTXREKEYOK,\t\t\t/* TlsTxRekeyOk */\n \tLINUX_MIB_TLSTXREKEYERROR,\t\t/* TlsTxRekeyError */\n \tLINUX_MIB_TLSRXREKEYRECEIVED,\t\t/* TlsRxRekeyReceived */\n+\tLINUX_MIB_TLSTXREKEYFALLBACK,\t\t/* TlsTxRekeyFallback */\n+\tLINUX_MIB_TLSRXREKEYFALLBACK,\t\t/* TlsRxRekeyFallback */\n+\tLINUX_MIB_TLSCURRTXREKEY,\t\t/* TlsCurrTxRekey */\n+\tLINUX_MIB_TLSCURRRXREKEY,\t\t/* TlsCurrRxRekey */\n+\tLINUX_MIB_TLSTXREKEYABORTED,\t\t/* TlsTxRekeyAborted */\n+\tLINUX_MIB_TLSRXREKEYABORTED,\t\t/* TlsRxRekeyAborted */\n \t__LINUX_MIB_TLSMAX\n };\n \ndiff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c\nindex 6f6a6f0d5eb0d..fc8ef0d13f5e7 100644\n--- a/net/sched/sch_generic.c\n+++ b/net/sched/sch_generic.c\n@@ -285,6 +285,15 @@ static struct sk_buff *dequeue_skb(struct Qdisc *q, bool *validate,\n \t\t*validate = false;\n \t\tif (xfrm_offload(skb))\n \t\t\t*validate = true;\n+\t\t/* A still-cleartext skb of a crypto-offloaded socket was validated\n+\t\t * against that socket's offload state at the time. That state\n+\t\t * (sk-\u003esk_validate_xmit_skb) can change while the skb is parked here\n+\t\t * e.g. a TLS key update or offload teardown, so re-validate it,\n+\t\t * letting the current callback decide how it reaches the wire instead\n+\t\t * of emitting now-unencrypted plaintext.\n+\t\t */\n+\t\tif (skb_is_decrypted(skb))\n+\t\t\t*validate = true;\n \t\t/* check the reason of requeuing without tx lock first */\n \t\ttxq = skb_get_tx_queue(txq-\u003edev, skb);\n \t\tif (!netif_xmit_frozen_or_stopped(txq)) {\ndiff --git a/net/tls/tls.h b/net/tls/tls.h\nindex 60a37bdaaa250..5d8f4d458df8a 100644\n--- a/net/tls/tls.h\n+++ b/net/tls/tls.h\n@@ -147,13 +147,31 @@ void tls_strp_abort_strp(struct tls_strparser *strp, int err);\n int init_prot_info(struct tls_prot_info *prot,\n \t\t const struct tls_crypto_info *crypto_info,\n \t\t const struct tls_cipher_desc *cipher_desc);\n+/* tls_sw_ctx_init() and tls_sw_ctx_finalize() are two halves of installing\n+ * a SW crypto context, split so the device path can attach the NIC between\n+ * them. finalize() may only be called after an init() that returned 0, and\n+ * both must be called with the same tx and new_crypto_info; on a rekey\n+ * (new_crypto_info != NULL) the two must also see the same\n+ * new_crypto_info-\u003ecipher_type. finalize() commits state and cannot fail,\n+ * so violating this leaves the context inconsistent without any error.\n+ */\n+int tls_sw_ctx_init(struct sock *sk, int tx,\n+\t\t struct tls_crypto_info *new_crypto_info);\n+void tls_sw_ctx_finalize(struct sock *sk, int tx,\n+\t\t\t struct tls_crypto_info *new_crypto_info);\n int tls_set_sw_offload(struct sock *sk, int tx,\n \t\t struct tls_crypto_info *new_crypto_info);\n void tls_update_rx_zc_capable(struct tls_context *tls_ctx);\n void tls_sw_strparser_arm(struct sock *sk, struct tls_context *ctx);\n void tls_sw_strparser_done(struct tls_context *tls_ctx);\n int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size);\n+int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size);\n+void tls_sw_ctx_tx_init(struct sock *sk, struct tls_sw_context_tx *sw_ctx);\n+int tls_sw_drain_tx(struct sock *sk, struct tls_context *ctx, int flags);\n+int tls_encrypt_async_wait(struct tls_sw_context_tx *ctx);\n+int tls_sw_push_pending_record(struct sock *sk, int flags);\n void tls_sw_splice_eof(struct socket *sock);\n+void tls_sw_splice_eof_locked(struct socket *sock);\n void tls_sw_cancel_work_tx(struct tls_context *tls_ctx);\n void tls_sw_release_resources_tx(struct sock *sk);\n void tls_sw_free_ctx_tx(struct tls_context *tls_ctx);\n@@ -230,10 +248,13 @@ static inline bool tls_strp_msg_mixed_decrypted(struct tls_sw_context_rx *ctx)\n #ifdef CONFIG_TLS_DEVICE\n int tls_device_init(void);\n void tls_device_cleanup(void);\n-int tls_set_device_offload(struct sock *sk);\n+int tls_set_device_offload(struct sock *sk,\n+\t\t\t struct tls_crypto_info *crypto_info);\n void tls_device_free_resources_tx(struct sock *sk);\n-int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx);\n+int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,\n+\t\t\t struct tls_crypto_info *crypto_info);\n void tls_device_offload_cleanup_rx(struct sock *sk);\n+void tls_device_rx_del_key(struct sock *sk, struct tls_context *ctx);\n void tls_device_rx_resync_new_rec(struct sock *sk, u32 rcd_len, u32 seq);\n int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx);\n #else\n@@ -241,7 +262,7 @@ static inline int tls_device_init(void) { return 0; }\n static inline void tls_device_cleanup(void) {}\n \n static inline int\n-tls_set_device_offload(struct sock *sk)\n+tls_set_device_offload(struct sock *sk, struct tls_crypto_info *crypto_info)\n {\n \treturn -EOPNOTSUPP;\n }\n@@ -249,13 +270,16 @@ tls_set_device_offload(struct sock *sk)\n static inline void tls_device_free_resources_tx(struct sock *sk) {}\n \n static inline int\n-tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)\n+tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,\n+\t\t\t struct tls_crypto_info *crypto_info)\n {\n \treturn -EOPNOTSUPP;\n }\n \n static inline void tls_device_offload_cleanup_rx(struct sock *sk) {}\n static inline void\n+tls_device_rx_del_key(struct sock *sk, struct tls_context *ctx) {}\n+static inline void\n tls_device_rx_resync_new_rec(struct sock *sk, u32 rcd_len, u32 seq) {}\n \n static inline int\ndiff --git a/net/tls/tls_device.c b/net/tls/tls_device.c\nindex f11d0528fc431..5f45c097bad3c 100644\n--- a/net/tls/tls_device.c\n+++ b/net/tls/tls_device.c\n@@ -57,11 +57,28 @@ static struct page *dummy_page;\n \n static void tls_device_free_ctx(struct tls_context *ctx)\n {\n-\tif (ctx-\u003etx_conf == TLS_HW)\n-\t\tkfree(tls_offload_ctx_tx(ctx));\n+\tif (ctx-\u003etx_conf == TLS_HW) {\n+\t\tstruct tls_offload_context_tx *offload_ctx =\n+\t\t\ttls_offload_ctx_tx(ctx);\n+\n+\t\tkfree(offload_ctx-\u003erekey.start_marker);\n+\t\tmemzero_explicit(\u0026offload_ctx-\u003erekey,\n+\t\t\t\t sizeof(offload_ctx-\u003erekey));\n+\t\tkfree(offload_ctx);\n+\t}\n+\n+\tif (ctx-\u003erx_conf == TLS_HW) {\n+\t\tstruct tls_offload_context_rx *offload_ctx =\n+\t\t\ttls_offload_ctx_rx(ctx);\n \n-\tif (ctx-\u003erx_conf == TLS_HW)\n-\t\tkfree(tls_offload_ctx_rx(ctx));\n+\t\t/* Normally freed and NULLed in tls_device_offload_cleanup_rx();\n+\t\t * free defensively here so a future path can't leak the tfm.\n+\t\t */\n+\t\tcrypto_free_aead(offload_ctx-\u003erekey.old_aead_recv);\n+\t\tmemzero_explicit(\u0026offload_ctx-\u003erekey,\n+\t\t\t\t sizeof(offload_ctx-\u003erekey));\n+\t\tkfree(offload_ctx);\n+\t}\n \n \ttls_ctx_free(NULL, ctx);\n }\n@@ -79,7 +96,9 @@ static void tls_device_tx_del_task(struct work_struct *work)\n \tnetdev = rcu_dereference_protected(ctx-\u003enetdev,\n \t\t\t\t\t !refcount_read(\u0026ctx-\u003erefcount));\n \n-\tnetdev-\u003etlsdev_ops-\u003etls_dev_del(netdev, ctx, TLS_OFFLOAD_CTX_DIR_TX);\n+\tif (!test_bit(TLS_TX_DEV_CLOSED, \u0026ctx-\u003eflags))\n+\t\tnetdev-\u003etlsdev_ops-\u003etls_dev_del(netdev, ctx,\n+\t\t\t\t\t\tTLS_OFFLOAD_CTX_DIR_TX);\n \tdev_put(netdev);\n \tctx-\u003enetdev = NULL;\n \ttls_device_free_ctx(ctx);\n@@ -138,6 +157,174 @@ static struct net_device *get_netdev_for_sock(struct sock *sk)\n \treturn lowest_dev;\n }\n \n+static int tls_device_dev_add_tx(struct sock *sk, struct net_device *netdev,\n+\t\t\t\t struct tls_crypto_info *crypto_info,\n+\t\t\t\t u32 write_seq)\n+{\n+\tconst struct tls_cipher_desc *cipher_desc;\n+\tchar *rec_seq;\n+\tint rc;\n+\n+\tcipher_desc = get_cipher_desc(crypto_info-\u003ecipher_type);\n+\tDEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc-\u003eoffloadable);\n+\n+\trc = netdev-\u003etlsdev_ops-\u003etls_dev_add(netdev, sk, TLS_OFFLOAD_CTX_DIR_TX,\n+\t\t\t\t\t crypto_info, write_seq);\n+\trec_seq = crypto_info_rec_seq(crypto_info, cipher_desc);\n+\ttrace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_TX,\n+\t\t\t\t write_seq, rec_seq, rc);\n+\treturn rc;\n+}\n+\n+/* Caller controls locking: initial-offload path is lock-free (pre-publish);\n+ * rekey path holds offload_ctx-\u003elock.\n+ */\n+static void tls_device_add_start_marker(struct sock *sk,\n+\t\t\t\t\tstruct tls_offload_context_tx *offload_ctx,\n+\t\t\t\t\tstruct tls_record_info *start_marker_record)\n+{\n+\tstart_marker_record-\u003eend_seq = tcp_sk(sk)-\u003ewrite_seq;\n+\tstart_marker_record-\u003elen = 0;\n+\tstart_marker_record-\u003enum_frags = 0;\n+\tlist_add_tail_rcu(\u0026start_marker_record-\u003elist, \u0026offload_ctx-\u003erecords_list);\n+}\n+\n+static void tls_device_commit_start_marker(struct sock *sk,\n+\t\t\t\t\tstruct tls_offload_context_tx *offload_ctx,\n+\t\t\t\t\tstruct tls_record_info *start_marker_record)\n+{\n+\ttls_device_add_start_marker(sk, offload_ctx, start_marker_record);\n+\n+\t/* TLS offload is greatly simplified if we don't send\n+\t * SKBs where only part of the payload needs to be encrypted.\n+\t * So mark the last skb in the write queue as end of record.\n+\t */\n+\ttcp_write_collapse_fence(sk);\n+}\n+\n+/* Account a rekey that could not (re)install the RX key on the NIC. The event\n+ * counter is bumped every time; the gauges move only on the first fallback\n+ * since the socket was last offloaded, so the recurring post-NETDEV_DOWN\n+ * rekeys and repeated failed adds do not drift them. The matching move back is\n+ * in tls_device_dev_add_rx(); the close-time decrement keys off the bit.\n+ */\n+static void tls_device_rx_rekey_fallback(struct sock *sk,\n+\t\t\t\t\t struct tls_context *tls_ctx)\n+{\n+\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYFALLBACK);\n+\tif (!test_and_set_bit(TLS_RX_REKEY_FAILED, \u0026tls_ctx-\u003eflags)) {\n+\t\tTLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);\n+\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXSW);\n+\t}\n+}\n+\n+static int tls_device_dev_add_rx(struct sock *sk, struct tls_context *tls_ctx,\n+\t\t\t\t struct net_device *netdev,\n+\t\t\t\t struct tls_crypto_info *crypto_info,\n+\t\t\t\t u32 cur_seq, bool is_rekey)\n+{\n+\tconst struct tls_cipher_desc *cipher_desc;\n+\tchar *rec_seq;\n+\tint rc;\n+\n+\tcipher_desc = get_cipher_desc(crypto_info-\u003ecipher_type);\n+\tDEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc-\u003eoffloadable);\n+\n+\trc = netdev-\u003etlsdev_ops-\u003etls_dev_add(netdev, sk,\n+\t\t\t\t\t TLS_OFFLOAD_CTX_DIR_RX,\n+\t\t\t\t\t crypto_info, cur_seq);\n+\trec_seq = crypto_info_rec_seq(crypto_info, cipher_desc);\n+\ttrace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_RX,\n+\t\t\t\t cur_seq, rec_seq, rc);\n+\tif (!rc) {\n+\t\tclear_bit(TLS_RX_DEV_DEGRADED, \u0026tls_ctx-\u003eflags);\n+\t\tclear_bit(TLS_RX_DEV_CLOSED, \u0026tls_ctx-\u003eflags);\n+\t\t/* Back on the NIC after an earlier SW fallback: undo its move. */\n+\t\tif (test_and_clear_bit(TLS_RX_REKEY_FAILED, \u0026tls_ctx-\u003eflags)) {\n+\t\t\tTLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXSW);\n+\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);\n+\t\t}\n+\t\tif (is_rekey)\n+\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);\n+\t} else if (is_rekey) {\n+\t\tset_bit(TLS_RX_DEV_DEGRADED, \u0026tls_ctx-\u003eflags);\n+\t\tset_bit(TLS_RX_DEV_CLOSED, \u0026tls_ctx-\u003eflags);\n+\t\ttls_device_rx_rekey_fallback(sk, tls_ctx);\n+\t}\n+\treturn rc;\n+}\n+\n+static void tls_device_deferred_dev_add_rx(struct sock *sk,\n+\t\t\t\t\t struct tls_context *tls_ctx,\n+\t\t\t\t\t struct tls_offload_context_rx *ctx,\n+\t\t\t\t\t u32 rec_start_seq)\n+{\n+\tconst struct tls_cipher_desc *cipher_desc;\n+\tunion tls_crypto_context crypto_ctx;\n+\tstruct net_device *netdev;\n+\n+\tctx-\u003edev_add_pending = 0;\n+\n+\t/* crypto_recv.info.rec_seq is frozen at the value setsockopt() passed\n+\t * in: the new key's first record number. The records that drained\n+\t * between setsockopt() and this boundary crossing were SW-decrypted\n+\t * under the new key and advanced tls_ctx-\u003erx.rec_seq, so the record\n+\t * starting at rec_start_seq, the one being decrypted right now,\n+\t * before tls_rx_one_record() calls tls_advance_record_sn(), is\n+\t * numbered by rx.rec_seq, not by the blob. Hand the NIC the live\n+\t * (TCP seq, record number) pair, as getsockopt(TLS_RX) already does.\n+\t */\n+\tcipher_desc = get_cipher_desc(tls_ctx-\u003ecrypto_recv.info.cipher_type);\n+\tDEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc-\u003eoffloadable);\n+\tcrypto_ctx = tls_ctx-\u003ecrypto_recv;\n+\tmemcpy(crypto_info_rec_seq(\u0026crypto_ctx.info, cipher_desc),\n+\t tls_ctx-\u003erx.rec_seq, cipher_desc-\u003erec_seq);\n+\n+\tdown_read(\u0026device_offload_lock);\n+\tnetdev = rcu_dereference_protected(tls_ctx-\u003enetdev,\n+\t\t\t\t\t lockdep_is_held(\u0026device_offload_lock));\n+\tif (netdev)\n+\t\ttls_device_dev_add_rx(sk, tls_ctx, netdev,\n+\t\t\t\t \u0026crypto_ctx.info,\n+\t\t\t\t rec_start_seq, true);\n+\telse\n+\t\ttls_device_rx_rekey_fallback(sk, tls_ctx);\n+\tup_read(\u0026device_offload_lock);\n+\tmemzero_explicit(\u0026crypto_ctx, sizeof(crypto_ctx));\n+\tTLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXREKEY);\n+}\n+\n+/* Retire the NIC's RX key when a KeyUpdate record is decoded (from\n+ * tls_check_pending_rekey(), lock_sock held). The NIC must lose the old key\n+ * now, before it transforms further post-KeyUpdate records that are new-key on\n+ * the wire. TLS_RX_DEV_CLOSED is re-tested under device_offload_lock because\n+ * tls_device_down() can run in between; synchronize_net() drains the RX path\n+ * before the driver frees its context.\n+ */\n+void tls_device_rx_del_key(struct sock *sk, struct tls_context *ctx)\n+{\n+\tstruct net_device *netdev;\n+\n+\tif (ctx-\u003erx_conf != TLS_HW)\n+\t\treturn;\n+\tif (test_bit(TLS_RX_DEV_CLOSED, \u0026ctx-\u003eflags))\n+\t\treturn;\n+\n+\tdown_read(\u0026device_offload_lock);\n+\tnetdev = rcu_dereference_protected(ctx-\u003enetdev,\n+\t\t\t\t\t lockdep_is_held(\u0026device_offload_lock));\n+\tif (!netdev || test_bit(TLS_RX_DEV_CLOSED, \u0026ctx-\u003eflags)) {\n+\t\tup_read(\u0026device_offload_lock);\n+\t\treturn;\n+\t}\n+\n+\tset_bit(TLS_RX_DEV_CLOSED, \u0026ctx-\u003eflags);\n+\tsynchronize_net();\n+\tnetdev-\u003etlsdev_ops-\u003etls_dev_del(netdev, ctx,\n+\t\t\t\t\tTLS_OFFLOAD_CTX_DIR_RX);\n+\tup_read(\u0026device_offload_lock);\n+}\n+\n static void destroy_record(struct tls_record_info *record)\n {\n \tint i;\n@@ -159,6 +346,57 @@ static void delete_all_records(struct tls_offload_context_tx *offload_ctx)\n \toffload_ctx-\u003eretransmit_hint = NULL;\n }\n \n+static void tls_device_commit_rekey_marker(struct sock *sk,\n+\t\t\t\t\t struct tls_offload_context_tx *offload_ctx,\n+\t\t\t\t\t struct tls_record_info *start_marker_record)\n+{\n+\tstruct tls_record_info *info, *temp;\n+\tunsigned long flags;\n+\t__be64 rcd_sn;\n+\n+\tspin_lock_irqsave(\u0026offload_ctx-\u003elock, flags);\n+\n+\t/* The deferred path reaches here with an empty list; the inline\n+\t * path may still hold the old start marker (never a real record,\n+\t * since tls_has_unacked_records() was false). Only markers are\n+\t * ever at the head, so stop at the first non-marker.\n+\t */\n+\tlist_for_each_entry_safe(info, temp, \u0026offload_ctx-\u003erecords_list, list) {\n+\t\tif (!tls_record_is_start_marker(info))\n+\t\t\tbreak;\n+\t\tlist_del(\u0026info-\u003elist);\n+\t\tdestroy_record(info);\n+\t}\n+\toffload_ctx-\u003eretransmit_hint = NULL;\n+\n+\tmemcpy(\u0026rcd_sn, offload_ctx-\u003erekey.tx.rec_seq, sizeof(rcd_sn));\n+\toffload_ctx-\u003eunacked_record_sn = be64_to_cpu(rcd_sn) - 1;\n+\n+\ttls_device_add_start_marker(sk, offload_ctx, start_marker_record);\n+\n+\tspin_unlock_irqrestore(\u0026offload_ctx-\u003elock, flags);\n+\n+\ttcp_write_collapse_fence(sk);\n+}\n+\n+static bool tls_has_unacked_records(struct tls_offload_context_tx *offload_ctx)\n+{\n+\tstruct tls_record_info *info;\n+\tbool has_unacked = false;\n+\tunsigned long flags;\n+\n+\tspin_lock_irqsave(\u0026offload_ctx-\u003elock, flags);\n+\tlist_for_each_entry(info, \u0026offload_ctx-\u003erecords_list, list) {\n+\t\tif (!tls_record_is_start_marker(info)) {\n+\t\t\thas_unacked = true;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\tspin_unlock_irqrestore(\u0026offload_ctx-\u003elock, flags);\n+\n+\treturn has_unacked;\n+}\n+\n static void tls_tcp_clean_acked(struct sock *sk, u32 acked_seq)\n {\n \tstruct tls_context *tls_ctx = tls_get_ctx(sk);\n@@ -187,6 +425,19 @@ static void tls_tcp_clean_acked(struct sock *sk, u32 acked_seq)\n \t}\n \n \tctx-\u003eunacked_record_sn += deleted_records;\n+\n+\t/* Once all old-key HW records are ACKed, set REKEY_READY to\n+\t * let sendmsg know it can finish the rekey and switch back\n+\t * to HW offload.\n+\t */\n+\tif (test_bit(TLS_TX_REKEY_PENDING, \u0026tls_ctx-\u003eflags) \u0026\u0026\n+\t !test_bit(TLS_TX_REKEY_FAILED, \u0026tls_ctx-\u003eflags)) {\n+\t\tu32 boundary_seq = READ_ONCE(tls_ctx-\u003erekey.boundary_seq);\n+\n+\t\tif (!before(acked_seq, boundary_seq))\n+\t\t\tset_bit(TLS_TX_REKEY_READY, \u0026tls_ctx-\u003eflags);\n+\t}\n+\n \tspin_unlock_irqrestore(\u0026ctx-\u003elock, flags);\n }\n \n@@ -217,7 +468,15 @@ void tls_device_free_resources_tx(struct sock *sk)\n {\n \tstruct tls_context *tls_ctx = tls_get_ctx(sk);\n \n-\ttls_free_partial_record(sk, tls_ctx);\n+\tif (unlikely(tls_ctx-\u003erekey.sw_ctx))\n+\t\ttls_sw_release_resources_tx(sk);\n+\telse\n+\t\ttls_free_partial_record(sk, tls_ctx);\n+\n+\tif (test_bit(TLS_TX_REKEY_PENDING, \u0026tls_ctx-\u003eflags)) {\n+\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYABORTED);\n+\t\tTLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXREKEY);\n+\t}\n }\n \n void tls_offload_tx_resync_request(struct sock *sk, u32 got_seq, u32 exp_seq)\n@@ -317,25 +576,34 @@ static void tls_device_record_close(struct sock *sk,\n \t\t\t\t unsigned char record_type)\n {\n \tstruct tls_prot_info *prot = \u0026ctx-\u003eprot_info;\n-\tstruct page_frag dummy_tag_frag;\n-\n-\t/* append tag\n-\t * device will fill in the tag, we just need to append a placeholder\n-\t * use socket memory to improve coalescing (re-using a single buffer\n-\t * increases frag count)\n-\t * if we can't allocate memory now use the dummy page\n+\tint tail = prot-\u003etag_size + prot-\u003etail_size;\n+\n+\t/* Append tail: tag for TLS 1.2, content_type + tag for TLS 1.3.\n+\t * Device fills in the tag, we just need to append a placeholder.\n+\t * Use socket memory to improve coalescing (re-using a single buffer\n+\t * increases frag count); if allocation fails use dummy_page\n+\t * (offset = record_type gives correct content_type byte via\n+\t * identity mapping)\n \t */\n-\tif (unlikely(pfrag-\u003esize - pfrag-\u003eoffset \u003c prot-\u003etag_size) \u0026\u0026\n-\t !skb_page_frag_refill(prot-\u003etag_size, pfrag, sk-\u003esk_allocation)) {\n-\t\tdummy_tag_frag.page = dummy_page;\n-\t\tdummy_tag_frag.offset = 0;\n-\t\tpfrag = \u0026dummy_tag_frag;\n+\tif (unlikely(!pfrag-\u003epage || pfrag-\u003esize - pfrag-\u003eoffset \u003c tail) \u0026\u0026\n+\t !skb_page_frag_refill(tail, pfrag, sk-\u003esk_allocation)) {\n+\t\tstruct page_frag dummy_pfrag = {\n+\t\t\t.page = dummy_page,\n+\t\t\t.offset = record_type,\n+\t\t};\n+\t\ttls_append_frag(record, \u0026dummy_pfrag, tail);\n+\t} else {\n+\t\tif (prot-\u003etail_size) {\n+\t\t\tchar *content_type_addr = page_address(pfrag-\u003epage) +\n+\t\t\t\t\t\t pfrag-\u003eoffset;\n+\t\t\t*content_type_addr = record_type;\n+\t\t}\n+\t\ttls_append_frag(record, pfrag, tail);\n \t}\n-\ttls_append_frag(record, pfrag, prot-\u003etag_size);\n \n \t/* fill prepend */\n \ttls_fill_prepend(ctx, skb_frag_address(\u0026record-\u003efrags[0]),\n-\t\t\t record-\u003elen - prot-\u003eoverhead_size,\n+\t\t\t record-\u003elen - prot-\u003eoverhead_size + prot-\u003etail_size,\n \t\t\t record_type);\n }\n \n@@ -418,6 +686,9 @@ static int tls_device_copy_data(void *addr, size_t bytes, struct iov_iter *i)\n \treturn 0;\n }\n \n+static int tls_device_complete_rekey(struct sock *sk, struct tls_context *ctx,\n+\t\t\t\t bool deferred, int push_flags);\n+\n static int tls_push_data(struct sock *sk,\n \t\t\t struct iov_iter *iter,\n \t\t\t size_t size, int flags,\n@@ -563,18 +834,54 @@ static int tls_push_data(struct sock *sk,\n \treturn rc;\n }\n \n+/* True while TX is routed through the temporary SW rekey context: a rekey is in\n+ * progress (PENDING) or has failed and the socket stays pinned to SW (FAILED).\n+ */\n+static bool tls_device_tx_uses_sw(const struct tls_context *ctx)\n+{\n+\treturn test_bit(TLS_TX_REKEY_PENDING, \u0026ctx-\u003eflags) ||\n+\t test_bit(TLS_TX_REKEY_FAILED, \u0026ctx-\u003eflags);\n+}\n+\n int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)\n {\n \tunsigned char record_type = TLS_RECORD_TYPE_DATA;\n \tstruct tls_context *tls_ctx = tls_get_ctx(sk);\n \tint rc;\n \n+\t/* Reject unsupported flags up front. tls_push_data() enforces the same\n+\t * set, but during a rekey the send is routed to tls_sw_sendmsg_locked(),\n+\t * which is the _locked variant and does not re-check; without this,\n+\t * MSG_ZEROCOPY / MSG_OOB etc. would reach tcp_sendmsg_locked() on the\n+\t * kernel-owned record pages while PENDING/FAILED.\n+\t */\n+\tif (msg-\u003emsg_flags \u0026 ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL |\n+\t\t\t MSG_SPLICE_PAGES | MSG_EOR))\n+\t\treturn -EOPNOTSUPP;\n+\n \tif (!tls_ctx-\u003ezerocopy_sendfile)\n \t\tmsg-\u003emsg_flags \u0026= ~MSG_SPLICE_PAGES;\n \n \tmutex_lock(\u0026tls_ctx-\u003etx_lock);\n \tlock_sock(sk);\n \n+\t/* Old-key records all ACKed; switch back to HW. */\n+\tif (test_bit(TLS_TX_REKEY_READY, \u0026tls_ctx-\u003eflags)) {\n+\t\trc = tls_device_complete_rekey(sk, tls_ctx, true, msg-\u003emsg_flags);\n+\t\t/* Non-zero here is the transient -EAGAIN retry,\n+\t\t * the next sendmsg retries. Hard failures return 0 after\n+\t\t * falling back to SW and emit tls_device_complete_rekey_fail\n+\t\t * from the fallback path.\n+\t\t */\n+\t\tif (rc)\n+\t\t\ttrace_tls_device_complete_rekey_retry(sk);\n+\t}\n+\n+\tif (tls_device_tx_uses_sw(tls_ctx)) {\n+\t\trc = tls_sw_sendmsg_locked(sk, msg, size);\n+\t\tgoto out;\n+\t}\n+\n \tif (unlikely(msg-\u003emsg_controllen)) {\n \t\trc = tls_process_cmsg(sk, msg, \u0026record_type);\n \t\tif (rc)\n@@ -603,8 +910,10 @@ void tls_device_splice_eof(struct socket *sock)\n \tmutex_lock(\u0026tls_ctx-\u003etx_lock);\n \tlock_sock(sk);\n \n-\tif (tls_is_partially_sent_record(tls_ctx) ||\n-\t tls_is_pending_open_record(tls_ctx)) {\n+\tif (tls_device_tx_uses_sw(tls_ctx)) {\n+\t\ttls_sw_splice_eof_locked(sock);\n+\t} else if (tls_is_partially_sent_record(tls_ctx) ||\n+\t\t tls_is_pending_open_record(tls_ctx)) {\n \t\tiov_iter_bvec(\u0026iter, ITER_SOURCE, NULL, 0, 0);\n \t\ttls_push_data(sk, \u0026iter, 0, 0, TLS_RECORD_TYPE_DATA);\n \t}\n@@ -675,14 +984,30 @@ EXPORT_SYMBOL(tls_get_record);\n \n static int tls_device_push_pending_record(struct sock *sk, int flags)\n {\n+\tstruct tls_context *tls_ctx = tls_get_ctx(sk);\n \tstruct iov_iter iter;\n \n+\tif (tls_device_tx_uses_sw(tls_ctx))\n+\t\treturn tls_sw_push_pending_record(sk, flags);\n+\n \tiov_iter_kvec(\u0026iter, ITER_SOURCE, NULL, 0, 0);\n \treturn tls_push_data(sk, \u0026iter, 0, flags, TLS_RECORD_TYPE_DATA);\n }\n \n void tls_device_write_space(struct sock *sk, struct tls_context *ctx)\n {\n+\tif (tls_device_tx_uses_sw(ctx)) {\n+\t\tstruct tls_offload_context_tx *offload_ctx;\n+\t\tunsigned long flags;\n+\n+\t\toffload_ctx = tls_offload_ctx_tx(ctx);\n+\t\tspin_lock_irqsave(\u0026offload_ctx-\u003elock, flags);\n+\t\tif (tls_device_tx_uses_sw(ctx))\n+\t\t\ttls_sw_write_space(sk, ctx);\n+\t\tspin_unlock_irqrestore(\u0026offload_ctx-\u003elock, flags);\n+\t\treturn;\n+\t}\n+\n \tif (tls_is_partially_sent_record(ctx)) {\n \t\tgfp_t sk_allocation = sk-\u003esk_allocation;\n \n@@ -785,6 +1110,8 @@ void tls_device_rx_resync_new_rec(struct sock *sk, u32 rcd_len, u32 seq)\n \t\treturn;\n \tif (unlikely(test_bit(TLS_RX_DEV_DEGRADED, \u0026tls_ctx-\u003eflags)))\n \t\treturn;\n+\tif (unlikely(test_bit(TLS_RX_DEV_CLOSED, \u0026tls_ctx-\u003eflags)))\n+\t\treturn;\n \n \tprot = \u0026tls_ctx-\u003eprot_info;\n \trx_ctx = tls_offload_ctx_rx(tls_ctx);\n@@ -886,6 +1213,7 @@ static int\n tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)\n {\n \tstruct tls_sw_context_rx *sw_ctx = tls_sw_ctx_rx(tls_ctx);\n+\tstruct tls_prot_info *prot = \u0026tls_ctx-\u003eprot_info;\n \tconst struct tls_cipher_desc *cipher_desc;\n \tint err, offset, copy, data_len, pos;\n \tstruct sk_buff *skb, *skb_iter;\n@@ -897,7 +1225,7 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)\n \tDEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc-\u003eoffloadable);\n \n \trxm = strp_msg(tls_strp_msg(sw_ctx));\n-\torig_buf = kmalloc(rxm-\u003efull_len + TLS_HEADER_SIZE + cipher_desc-\u003eiv,\n+\torig_buf = kmalloc(rxm-\u003efull_len + prot-\u003eprepend_size,\n \t\t\t sk-\u003esk_allocation);\n \tif (!orig_buf)\n \t\treturn -ENOMEM;\n@@ -912,9 +1240,8 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)\n \toffset = rxm-\u003eoffset;\n \n \tsg_init_table(sg, 1);\n-\tsg_set_buf(\u0026sg[0], buf,\n-\t\t rxm-\u003efull_len + TLS_HEADER_SIZE + cipher_desc-\u003eiv);\n-\terr = skb_copy_bits(skb, offset, buf, TLS_HEADER_SIZE + cipher_desc-\u003eiv);\n+\tsg_set_buf(\u0026sg[0], buf, rxm-\u003efull_len + prot-\u003eprepend_size);\n+\terr = skb_copy_bits(skb, offset, buf, prot-\u003eprepend_size);\n \tif (err)\n \t\tgoto free_buf;\n \n@@ -930,7 +1257,7 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)\n \tif (skb_pagelen(skb) \u003e offset) {\n \t\tcopy = min_t(int, skb_pagelen(skb) - offset, data_len);\n \n-\t\tif (skb-\u003edecrypted) {\n+\t\tif (skb-\u003edecrypted || skb-\u003edecrypt_failed) {\n \t\t\terr = skb_store_bits(skb, offset, buf, copy);\n \t\t\tif (err)\n \t\t\t\tgoto free_buf;\n@@ -957,7 +1284,7 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)\n \t\tcopy = min_t(int, skb_iter-\u003elen - frag_pos,\n \t\t\t data_len + rxm-\u003eoffset - offset);\n \n-\t\tif (skb_iter-\u003edecrypted) {\n+\t\tif (skb_iter-\u003edecrypted || skb_iter-\u003edecrypt_failed) {\n \t\t\terr = skb_store_bits(skb_iter, frag_pos, buf, copy);\n \t\t\tif (err)\n \t\t\t\tgoto free_buf;\n@@ -974,6 +1301,77 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)\n \treturn err;\n }\n \n+/*\n+ * Reconstruct a boundary record whose frags the NIC XORed with the old key,\n+ * then hand it to the SW AEAD under the current (new) key.\n+ *\n+ * These are deliberately two different keys: the sender has already done its\n+ * TX KeyUpdate, so the record on the wire is AEAD-encrypted with the new key,\n+ * but the RX NIC still holds the old key and CTR-XORed some frags with the old\n+ * keystream. tls_device_reencrypt() must undo that XOR with the *old* key to\n+ * restore the pristine new-key ciphertext, so swap the old key in only for the\n+ * reconstruction and restore the current key before returning; the SW AEAD\n+ * decrypt that follows then runs under the new key, matching the wire record.\n+ */\n+static int tls_device_reencrypt_old_key(struct sock *sk,\n+\t\t\t\t\tstruct tls_offload_context_rx *ctx,\n+\t\t\t\t\tstruct tls_sw_context_rx *sw_ctx,\n+\t\t\t\t\tstruct tls_context *tls_ctx)\n+{\n+\tstruct crypto_aead *saved_aead = sw_ctx-\u003eaead_recv;\n+\tchar saved_iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE];\n+\tchar saved_rec_seq[TLS_MAX_REC_SEQ_SIZE];\n+\tint ret;\n+\n+\tmemcpy(saved_iv, tls_ctx-\u003erx.iv, sizeof(saved_iv));\n+\tmemcpy(saved_rec_seq, tls_ctx-\u003erx.rec_seq, sizeof(saved_rec_seq));\n+\n+\tsw_ctx-\u003eaead_recv = ctx-\u003erekey.old_aead_recv;\n+\tmemcpy(tls_ctx-\u003erx.iv, ctx-\u003erekey.old_iv, sizeof(ctx-\u003erekey.old_iv));\n+\tmemcpy(tls_ctx-\u003erx.rec_seq, ctx-\u003erekey.old_rec_seq,\n+\t sizeof(ctx-\u003erekey.old_rec_seq));\n+\n+\tret = tls_device_reencrypt(sk, tls_ctx);\n+\n+\tmemcpy(ctx-\u003erekey.old_rec_seq, tls_ctx-\u003erx.rec_seq,\n+\t sizeof(ctx-\u003erekey.old_rec_seq));\n+\n+\tsw_ctx-\u003eaead_recv = saved_aead;\n+\tmemcpy(tls_ctx-\u003erx.iv, saved_iv, sizeof(saved_iv));\n+\tmemcpy(tls_ctx-\u003erx.rec_seq, saved_rec_seq, sizeof(saved_rec_seq));\n+\n+\tif (ret)\n+\t\treturn ret;\n+\n+\ttls_bigint_increment(ctx-\u003erekey.old_rec_seq,\n+\t\t\t tls_ctx-\u003eprot_info.rec_seq_size);\n+\tctx-\u003eresync_nh_reset = 1;\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * TCP sequence of the first byte of the record the strparser currently holds\n+ * or is still collecting. In non-copy mode tcp_sk(sk)-\u003ecopied_seq is left at\n+ * the record start until tls_strp_msg_consume(). In copy mode\n+ * tls_strp_read_copy() zeroes stm.offset and anchor-\u003elen and then\n+ * tls_strp_read_copyin() -\u003e tcp_read_sock() advances copied_seq by every byte\n+ * it appends to the anchor, a complete parsed-ahead record, a partial one\n+ * under rmem pressure, or only header bytes, so subtract anchor-\u003elen to get\n+ * back to the record start. Both the recv path and the setsockopt rekey path\n+ * must classify records against the same start, so share this helper.\n+ */\n+static u32 tls_device_rx_rec_start(struct sock *sk,\n+\t\t\t\t struct tls_sw_context_rx *sw_ctx)\n+{\n+\tu32 copied_seq = tcp_sk(sk)-\u003ecopied_seq;\n+\n+\tif (sw_ctx-\u003estrp.copy_mode)\n+\t\treturn copied_seq - sw_ctx-\u003estrp.anchor-\u003elen;\n+\n+\treturn copied_seq;\n+}\n+\n int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)\n {\n \tstruct tls_offload_context_rx *ctx = tls_offload_ctx_rx(tls_ctx);\n@@ -981,6 +1379,7 @@ int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)\n \tstruct sk_buff *skb = tls_strp_msg(sw_ctx);\n \tstruct strp_msg *rxm = strp_msg(skb);\n \tint is_decrypted, is_encrypted;\n+\tu32 rec_start_seq;\n \n \tif (!tls_strp_msg_mixed_decrypted(sw_ctx)) {\n \t\tis_decrypted = skb-\u003edecrypted;\n@@ -990,10 +1389,77 @@ int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)\n \t\tis_encrypted = 0;\n \t}\n \n-\ttrace_tls_device_decrypted(sk, tcp_sk(sk)-\u003ecopied_seq - rxm-\u003efull_len,\n+\trec_start_seq = tls_device_rx_rec_start(sk, sw_ctx);\n+\n+\ttrace_tls_device_decrypted(sk, rec_start_seq,\n \t\t\t\t tls_ctx-\u003erx.rec_seq, rxm-\u003efull_len,\n \t\t\t\t is_encrypted, is_decrypted);\n \n+\tif (unlikely(ctx-\u003erekey.old_aead_recv)) {\n+\t\tbool nic_touched = !is_encrypted || skb-\u003edecrypt_failed;\n+\t\tbool before_nic_boundary;\n+\n+\t\t/* old_nic_boundary is the TCP stack's view at setsockopt time\n+\t\t * (rcv_nxt plus the out-of-order tail), not the NIC's last\n+\t\t * transformed byte. A segment the NIC transformed with the old\n+\t\t * key before tls_dev_del returned can still be in the RQ/CQ, in\n+\t\t * a GRO list or in the socket backlog when that snapshot is\n+\t\t * taken and reach TCP later, above it. While old_aead_recv is\n+\t\t * held the NIC has no RX context for this socket at all: the\n+\t\t * old one was deleted before old_aead_recv was set and the new\n+\t\t * one is only installed once it is freed below. So a NIC mark\n+\t\t * seen here can only be the old key's transform, wherever the\n+\t\t * record sits relative to the snapshot. Slide the boundary out\n+\t\t * over such a record instead of retiring the old key on it; the\n+\t\t * old key is retired only on a record the NIC never saw.\n+\t\t */\n+\t\tif (nic_touched \u0026\u0026\n+\t\t !before(rec_start_seq, ctx-\u003erekey.old_nic_boundary))\n+\t\t\tctx-\u003erekey.old_nic_boundary = rec_start_seq + rxm-\u003efull_len;\n+\n+\t\tbefore_nic_boundary =\n+\t\t\tbefore(rec_start_seq, ctx-\u003erekey.old_nic_boundary);\n+\n+\t\tif (before_nic_boundary) {\n+\t\t\t/* Non-mixed (skb-\u003edecrypted clear) is untouched wire\n+\t\t\t * ciphertext even if skb-\u003edecrypt_failed is set, so advance\n+\t\t\t * old_rec_seq and let the SW AEAD decrypt it directly.\n+\t\t\t * old_rec_seq tracks the stream's record number, which the\n+\t\t\t * NIC also advances for records it did not transform, so\n+\t\t\t * keeping it in step lets a later NIC-touched record be undone\n+\t\t\t * with the right nonce. A mixed record carries NIC-XORed frags\n+\t\t\t * (skb-\u003edecrypt_failed or skb-\u003edecrypted) and takes the\n+\t\t\t * old-key reencrypt path below, which undoes the transform per\n+\t\t\t * frag before the SW AEAD decrypts.\n+\t\t\t */\n+\t\t\tif (is_encrypted) {\n+\t\t\t\ttls_bigint_increment(ctx-\u003erekey.old_rec_seq,\n+\t\t\t\t\t\t tls_ctx-\u003eprot_info.rec_seq_size);\n+\t\t\t\treturn 0;\n+\t\t\t}\n+\n+\t\t\ttrace_tls_device_rekey_reencrypt(sk, rec_start_seq,\n+\t\t\t\t\t\t\t ctx-\u003erekey.old_nic_boundary);\n+\n+\t\t\treturn tls_device_reencrypt_old_key(sk, ctx,\n+\t\t\t\t\t\t\t sw_ctx, tls_ctx);\n+\t\t}\n+\n+\t\ttrace_tls_device_rekey_done(sk, rec_start_seq,\n+\t\t\t\t\t ctx-\u003erekey.old_nic_boundary);\n+\t\tcrypto_free_aead(ctx-\u003erekey.old_aead_recv);\n+\t\tctx-\u003erekey.old_aead_recv = NULL;\n+\n+\t\t/* Anchor the NIC on the start of this first post-boundary\n+\t\t * record. rec_start_seq already accounts for copy_mode, where\n+\t\t * copied_seq has advanced past the record end; using it keeps\n+\t\t * the (TCP seq, record number) pair consistent in both modes.\n+\t\t */\n+\t\tif (ctx-\u003edev_add_pending)\n+\t\t\ttls_device_deferred_dev_add_rx(sk, tls_ctx, ctx,\n+\t\t\t\t\t\t rec_start_seq);\n+\t}\n+\n \tif (unlikely(test_bit(TLS_RX_DEV_DEGRADED, \u0026tls_ctx-\u003eflags))) {\n \t\tif (likely(is_encrypted || is_decrypted))\n \t\t\treturn is_decrypted;\n@@ -1062,62 +1528,457 @@ static struct tls_offload_context_tx *alloc_offload_ctx_tx(struct tls_context *c\n \treturn offload_ctx;\n }\n \n-int tls_set_device_offload(struct sock *sk)\n+/* Build a fresh AEAD tfm for the rekey with the given key, so it can be\n+ * swapped in only on success. Re-keying a live tfm in place is not atomic:\n+ * a failed crypto_aead_setkey() leaves it with CRYPTO_TFM_NEED_KEY set,\n+ * destroying the previous key. Returns an ERR_PTR() on failure.\n+ */\n+static struct crypto_aead *tls_device_build_rekey_aead(\n+\t\t\t\tconst struct tls_cipher_desc *cipher_desc,\n+\t\t\t\tchar *key, u32 alg_flags)\n {\n-\tstruct tls_record_info *start_marker_record;\n-\tstruct tls_offload_context_tx *offload_ctx;\n+\tstruct crypto_aead *aead;\n+\tint rc;\n+\n+\taead = crypto_alloc_aead(cipher_desc-\u003ecipher_name, 0, alg_flags);\n+\tif (IS_ERR(aead))\n+\t\treturn aead;\n+\n+\trc = crypto_aead_setkey(aead, key, cipher_desc-\u003ekey);\n+\tif (!rc)\n+\t\trc = crypto_aead_setauthsize(aead, cipher_desc-\u003etag);\n+\tif (rc) {\n+\t\tcrypto_free_aead(aead);\n+\t\treturn ERR_PTR(rc);\n+\t}\n+\n+\treturn aead;\n+}\n+\n+static void tls_device_copy_rekey_iv_seq(\n+\t\t\t\tstruct tls_offload_context_tx *offload_ctx,\n+\t\t\t\tconst struct tls_cipher_desc *cipher_desc,\n+\t\t\t\tchar *salt, char *iv, char *rec_seq)\n+{\n+\tmemcpy(offload_ctx-\u003erekey.tx.iv, salt, cipher_desc-\u003esalt);\n+\tmemcpy(offload_ctx-\u003erekey.tx.iv + cipher_desc-\u003esalt, iv,\n+\t cipher_desc-\u003eiv);\n+\tmemcpy(offload_ctx-\u003erekey.tx.rec_seq, rec_seq, cipher_desc-\u003erec_seq);\n+}\n+\n+static int tls_device_init_rekey_sw(struct sock *sk,\n+\t\t\t\t struct tls_context *ctx,\n+\t\t\t\t struct tls_offload_context_tx *offload_ctx,\n+\t\t\t\t struct tls_crypto_info *new_crypto_info)\n+{\n+\tstruct tls_sw_context_tx *sw_ctx = \u0026offload_ctx-\u003erekey.sw;\n+\tconst struct tls_cipher_desc *cipher_desc;\n+\tchar *key;\n+\tint rc;\n+\n+\tcipher_desc = get_cipher_desc(new_crypto_info-\u003ecipher_type);\n+\tDEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc-\u003eoffloadable);\n+\n+\tmemset(sw_ctx, 0, sizeof(*sw_ctx));\n+\ttls_sw_ctx_tx_init(sk, sw_ctx);\n+\n+\tkey = crypto_info_key(new_crypto_info, cipher_desc);\n+\tsw_ctx-\u003eaead_send = tls_device_build_rekey_aead(cipher_desc, key, 0);\n+\tif (IS_ERR(sw_ctx-\u003eaead_send)) {\n+\t\trc = PTR_ERR(sw_ctx-\u003eaead_send);\n+\t\tsw_ctx-\u003eaead_send = NULL;\n+\t\treturn rc;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int tls_device_start_rekey(struct sock *sk,\n+\t\t\t\t struct tls_context *ctx,\n+\t\t\t\t struct tls_offload_context_tx *offload_ctx,\n+\t\t\t\t struct tls_crypto_info *new_crypto_info)\n+{\n+\tbool rekey_pending = test_bit(TLS_TX_REKEY_PENDING, \u0026ctx-\u003eflags);\n+\tbool rekey_failed = test_bit(TLS_TX_REKEY_FAILED, \u0026ctx-\u003eflags);\n+\tconst struct tls_cipher_desc *cipher_desc;\n+\tstruct crypto_aead *new_aead, *old_aead;\n+\tchar *key, *iv, *rec_seq, *salt;\n+\tint push_flags = MSG_NOSIGNAL;\n+\tunsigned long flags;\n+\tint rc;\n+\n+\tcipher_desc = get_cipher_desc(new_crypto_info-\u003ecipher_type);\n+\tDEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc-\u003eoffloadable);\n+\n+\tkey = crypto_info_key(new_crypto_info, cipher_desc);\n+\tiv = crypto_info_iv(new_crypto_info, cipher_desc);\n+\trec_seq = crypto_info_rec_seq(new_crypto_info, cipher_desc);\n+\tsalt = crypto_info_salt(new_crypto_info, cipher_desc);\n+\n+\t/* The record flushes below hand the open/partially sent HW record to\n+\t * TCP and may have to wait for send buffer space. Honour the socket's\n+\t * non-blocking mode so an O_NONBLOCK application is not put to sleep\n+\t * inside setsockopt(): it gets -EAGAIN and retries once the socket is\n+\t * writable. Kernel sockets (no backing file, e.g. nvme-tcp) keep the\n+\t * blocking semantics, matching how they call sendmsg().\n+\t */\n+\tif (sk-\u003esk_socket \u0026\u0026 sk-\u003esk_socket-\u003efile \u0026\u0026\n+\t (sk-\u003esk_socket-\u003efile-\u003ef_flags \u0026 O_NONBLOCK))\n+\t\tpush_flags |= MSG_DONTWAIT;\n+\n+\tif (rekey_pending || rekey_failed) {\n+\t\t/* Flush any SW open_record before swapping the key. -EINPROGRESS\n+\t\t * means an async AEAD accepted the record for encryption; it is a\n+\t\t * success, waited for by tls_encrypt_async_wait() just below (as\n+\t\t * tls_process_cmsg()/tls_sw_drain_tx() also treat it).\n+\t\t */\n+\t\tif (tls_is_pending_open_record(ctx)) {\n+\t\t\trc = ctx-\u003epush_pending_record(sk, push_flags);\n+\t\t\tif (rc \u003c 0 \u0026\u0026 rc != -EINPROGRESS)\n+\t\t\t\treturn rc;\n+\t\t}\n+\n+\t\t/* Wait for in-flight async encryptions submitted to this tfm\n+\t\t * with the previous key before changing it.\n+\t\t */\n+\t\trc = tls_encrypt_async_wait(\u0026offload_ctx-\u003erekey.sw);\n+\t\tif (rc)\n+\t\t\treturn rc;\n+\n+\t\t/* Build the new key into a fresh tfm and swap it in only on\n+\t\t * success; A failed rekey here must leave the SW fallback\n+\t\t * path able to encrypt.\n+\t\t */\n+\t\tnew_aead = tls_device_build_rekey_aead(cipher_desc, key, 0);\n+\t\tif (IS_ERR(new_aead))\n+\t\t\treturn PTR_ERR(new_aead);\n+\n+\t\told_aead = offload_ctx-\u003erekey.sw.aead_send;\n+\t\toffload_ctx-\u003erekey.sw.aead_send = new_aead;\n+\t\tcrypto_free_aead(old_aead);\n+\n+\t\ttls_device_copy_rekey_iv_seq(offload_ctx, cipher_desc,\n+\t\t\t\t\t salt, iv, rec_seq);\n+\n+\t\tif (rekey_failed) {\n+\t\t\t/* Re-arm FAILED -\u003e PENDING under device_offload_lock. The\n+\t\t\t * PENDING set and FAILED clear are two stores to ctx-\u003eflags,\n+\t\t\t * and tls_device_down() tests !PENDING \u0026\u0026 !FAILED as two\n+\t\t\t * separate loads; without the lock those loads could straddle\n+\t\t\t * the flip and see neither bit, letting tls_device_down()\n+\t\t\t * install tls_validate_xmit_skb_sw with PENDING set (dropping\n+\t\t\t * all new-key ciphertext). The lock keeps PENDING || FAILED\n+\t\t\t * observable throughout. Non-blocking, so no NETDEV_DOWN stall.\n+\t\t\t */\n+\t\t\tdown_read(\u0026device_offload_lock);\n+\t\t\tspin_lock_irqsave(\u0026offload_ctx-\u003elock, flags);\n+\t\t\tWRITE_ONCE(ctx-\u003erekey.boundary_seq, tcp_sk(sk)-\u003esnd_una);\n+\t\t\tset_bit(TLS_TX_REKEY_PENDING, \u0026ctx-\u003eflags);\n+\t\t\tspin_unlock_irqrestore(\u0026offload_ctx-\u003elock, flags);\n+\t\t\t/* Release pairs with test_bit_acquire() in the validator:\n+\t\t\t * a TX seeing FAILED clear must see the fresh boundary_seq.\n+\t\t\t */\n+\t\t\tclear_bit_unlock(TLS_TX_REKEY_FAILED, \u0026ctx-\u003eflags);\n+\t\t\tup_read(\u0026device_offload_lock);\n+\t\t\tTLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);\n+\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);\n+\t\t}\n+\t} else {\n+\t\t/* Drain partially sent record and flush open HW record\n+\t\t * before switching to SW.\n+\t\t */\n+\t\tif (tls_is_partially_sent_record(ctx)) {\n+\t\t\trc = tls_push_partial_record(sk, ctx,\n+\t\t\t\t\t\t MSG_SENDPAGE_DECRYPTED |\n+\t\t\t\t\t\t push_flags);\n+\t\t\tif (rc \u003c 0)\n+\t\t\t\treturn rc;\n+\t\t}\n+\t\tif (tls_is_pending_open_record(ctx)) {\n+\t\t\trc = ctx-\u003epush_pending_record(sk, push_flags);\n+\t\t\tif (rc \u003c 0)\n+\t\t\t\treturn rc;\n+\t\t}\n+\n+\t\trc = tls_device_init_rekey_sw(sk, ctx, offload_ctx,\n+\t\t\t\t\t new_crypto_info);\n+\t\tif (rc)\n+\t\t\treturn rc;\n+\n+\t\ttls_device_copy_rekey_iv_seq(offload_ctx, cipher_desc,\n+\t\t\t\t\t salt, iv, rec_seq);\n+\n+\t\t/* Publish the rekey under device_offload_lock so that setting\n+\t\t * TLS_TX_REKEY_PENDING and installing the rekey validator is\n+\t\t * atomic against tls_device_down(), which under down_write() tests\n+\t\t * !PENDING and installs tls_validate_xmit_skb_sw. Otherwise the two\n+\t\t * validator stores could interleave to leave PENDING set with the\n+\t\t * SW validator, and every new-key ciphertext (never on the offload\n+\t\t * records_list) would then be dropped by tls_sw_fallback(). The\n+\t\t * blocking flush and crypto_alloc above deliberately run WITHOUT\n+\t\t * this lock, so a stalled peer cannot hold up NETDEV_DOWN (which\n+\t\t * takes down_write() under RTNL) or any other down_read() user.\n+\t\t */\n+\t\tdown_read(\u0026device_offload_lock);\n+\n+\t\t/* Prevent a partial record straddling the SW/HW boundary. */\n+\t\ttcp_write_collapse_fence(sk);\n+\n+\t\tWRITE_ONCE(ctx-\u003erekey.sw_ctx, \u0026offload_ctx-\u003erekey.sw);\n+\t\tWRITE_ONCE(ctx-\u003erekey.cipher_ctx, \u0026offload_ctx-\u003erekey.tx);\n+\n+\t\tspin_lock_irqsave(\u0026offload_ctx-\u003elock, flags);\n+\t\tWRITE_ONCE(ctx-\u003erekey.boundary_seq, tcp_sk(sk)-\u003ewrite_seq);\n+\t\tset_bit(TLS_TX_REKEY_PENDING, \u0026ctx-\u003eflags);\n+\t\tspin_unlock_irqrestore(\u0026offload_ctx-\u003elock, flags);\n+\n+\t\t/* Switch to rekey validator; new sends won't use HW offload */\n+\t\tsmp_store_release(\u0026sk-\u003esk_validate_xmit_skb,\n+\t\t\t\t tls_validate_xmit_skb_rekey);\n+\n+\t\tup_read(\u0026device_offload_lock);\n+\t}\n+\n+\tunsafe_memcpy(\u0026offload_ctx-\u003erekey.crypto_send.info, new_crypto_info,\n+\t\t cipher_desc-\u003ecrypto_info,\n+\t\t /* checked in do_tls_setsockopt_conf */);\n+\tmemzero_explicit(new_crypto_info, cipher_desc-\u003ecrypto_info);\n+\n+\treturn 0;\n+}\n+\n+static int tls_device_complete_rekey(struct sock *sk, struct tls_context *ctx,\n+\t\t\t\t bool deferred, int push_flags)\n+{\n+\tstruct tls_offload_context_tx *offload_ctx = tls_offload_ctx_tx(ctx);\n+\tstruct crypto_aead *new_aead, *old_aead, *old_sw_aead;\n \tconst struct tls_cipher_desc *cipher_desc;\n-\tstruct tls_crypto_info *crypto_info;\n-\tstruct tls_prot_info *prot;\n \tstruct net_device *netdev;\n-\tstruct tls_context *ctx;\n-\tchar *iv, *rec_seq;\n+\tunsigned long flags;\n+\tchar *key;\n \tint rc;\n \n-\tctx = tls_get_ctx(sk);\n-\tprot = \u0026ctx-\u003eprot_info;\n+\tcipher_desc = get_cipher_desc(offload_ctx-\u003erekey.crypto_send.info.cipher_type);\n+\tDEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc-\u003eoffloadable);\n \n-\tif (ctx-\u003epriv_ctx_tx)\n-\t\treturn -EEXIST;\n+\tDEBUG_NET_WARN_ON_ONCE(!offload_ctx-\u003erekey.start_marker);\n \n-\tnetdev = get_netdev_for_sock(sk);\n+\trc = tls_sw_drain_tx(sk, ctx, push_flags);\n+\t/* -EAGAIN (sndbuf full) and a signal (-EINTR/-ERESTARTSYS from\n+\t * sk_stream_wait_memory()) are transient: leave the rekey PENDING and\n+\t * retry on the next sendmsg rather than permanently dropping HW offload.\n+\t * tls_tx_records() likewise passes these through without aborting.\n+\t */\n+\tif (rc == -EAGAIN || rc == -EINTR || rc == -ERESTARTSYS)\n+\t\treturn rc;\n+\tif (rc)\n+\t\tgoto rekey_fallback;\t/* hard failure: fall back to SW */\n+\n+\tdown_read(\u0026device_offload_lock);\n+\n+\tnetdev = rcu_dereference_protected(ctx-\u003enetdev,\n+\t\t\t\t\t lockdep_is_held(\u0026device_offload_lock));\n \tif (!netdev) {\n-\t\tpr_err_ratelimited(\"%s: netdev not found\\n\", __func__);\n-\t\treturn -EINVAL;\n+\t\trc = -ENODEV;\n+\t\tgoto release_lock;\n \t}\n \n-\tif (!(netdev-\u003efeatures \u0026 NETIF_F_HW_TLS_TX)) {\n-\t\trc = -EOPNOTSUPP;\n-\t\tgoto release_netdev;\n+\t/* Drain in-flight xmit users before tls_dev_del() and before freeing the\n+\t * old fallback aead_send: (1) under the rekey validator a decrypted\n+\t * straddler may still be inside the driver on the HW context (same swap -\u003e\n+\t * synchronize_net -\u003e dev_del order as tls_device_down(), which also keeps a\n+\t * decrypted skb from reaching a torn-down context); (2) pre-boundary\n+\t * retransmits routed to tls_sw_fallback() read aead_send locklessly. No new\n+\t * fallback can start here: every pre-boundary record is ACKed and freed, so\n+\t * fill_sg_in() bails.\n+\t */\n+\tsynchronize_net();\n+\n+\tif (!test_bit(TLS_TX_DEV_CLOSED, \u0026ctx-\u003eflags)) {\n+\t\tnetdev-\u003etlsdev_ops-\u003etls_dev_del(netdev, ctx,\n+\t\t\t\t\t\tTLS_OFFLOAD_CTX_DIR_TX);\n+\t\tset_bit(TLS_TX_DEV_CLOSED, \u0026ctx-\u003eflags);\n \t}\n \n-\tcrypto_info = \u0026ctx-\u003ecrypto_send.info;\n-\tif (crypto_info-\u003eversion != TLS_1_2_VERSION) {\n-\t\trc = -EOPNOTSUPP;\n-\t\tgoto release_netdev;\n+\t/* Build the new SW-fallback key into a fresh tfm and swap it in only\n+\t * on success. Doing this while the HW context is torn down\n+\t * (TLS_TX_DEV_CLOSED set) means a failure falls into rekey_fallback\n+\t * with HW off, so the SW fallback is coherent, same as a dev_add\n+\t * failure.\n+\t */\n+\tkey = crypto_info_key(\u0026offload_ctx-\u003erekey.crypto_send.info, cipher_desc);\n+\tnew_aead = tls_device_build_rekey_aead(cipher_desc, key, CRYPTO_ALG_ASYNC);\n+\tif (IS_ERR(new_aead)) {\n+\t\trc = PTR_ERR(new_aead);\n+\t\tgoto release_lock;\n \t}\n \n-\tcipher_desc = get_cipher_desc(crypto_info-\u003ecipher_type);\n-\tif (!cipher_desc || !cipher_desc-\u003eoffloadable) {\n-\t\trc = -EINVAL;\n-\t\tgoto release_netdev;\n+\t/* crypto_send.info.rec_seq is frozen at setsockopt time; the SW context\n+\t * advanced rekey.tx.rec_seq for every record it sent, so hand the NIC the\n+\t * live record number (mirrors the RX deferred add).\n+\t */\n+\tmemcpy(crypto_info_rec_seq(\u0026offload_ctx-\u003erekey.crypto_send.info, cipher_desc),\n+\t offload_ctx-\u003erekey.tx.rec_seq, cipher_desc-\u003erec_seq);\n+\n+\trc = tls_device_dev_add_tx(sk, netdev, \u0026offload_ctx-\u003erekey.crypto_send.info,\n+\t\t\t\t tcp_sk(sk)-\u003ewrite_seq);\n+\tif (rc) {\n+\t\tcrypto_free_aead(new_aead);\n+\t\tgoto release_lock;\n \t}\n \n-\trc = init_prot_info(prot, crypto_info, cipher_desc);\n+\t/* Point of no return: HW is live with the new key. Swap in the new\n+\t * fallback tfm and drop the old one; the remaining steps cannot fail.\n+\t */\n+\told_aead = offload_ctx-\u003eaead_send;\n+\toffload_ctx-\u003eaead_send = new_aead;\n+\tcrypto_free_aead(old_aead);\n+\tclear_bit(TLS_TX_DEV_CLOSED, \u0026ctx-\u003eflags);\n+\n+\tmemcpy(ctx-\u003etx.iv, offload_ctx-\u003erekey.tx.iv,\n+\t cipher_desc-\u003esalt + cipher_desc-\u003eiv);\n+\tmemcpy(ctx-\u003etx.rec_seq, offload_ctx-\u003erekey.tx.rec_seq,\n+\t cipher_desc-\u003erec_seq);\n+\tunsafe_memcpy(\u0026ctx-\u003ecrypto_send.info,\n+\t\t \u0026offload_ctx-\u003erekey.crypto_send.info,\n+\t\t cipher_desc-\u003ecrypto_info,\n+\t\t /* checked during rekey setup */);\n+\n+\t/* Start marker: the NIC passes through everything before\n+\t * write_seq untouched (it is already SW-encrypted ciphertext),\n+\t * same as during initial offload setup. Also drops the stale\n+\t * marker and rebases unacked_record_sn so the record-sequence\n+\t * bookkeeping stays consistent on the inline path.\n+\t */\n+\ttls_device_commit_rekey_marker(sk, offload_ctx,\n+\t\t\t\t offload_ctx-\u003erekey.start_marker);\n+\n+\told_sw_aead = tls_sw_ctx_tx(ctx)-\u003eaead_send;\n+\n+\tspin_lock_irqsave(\u0026offload_ctx-\u003elock, flags);\n+\tclear_bit(TLS_TX_REKEY_PENDING, \u0026ctx-\u003eflags);\n+\tclear_bit(TLS_TX_REKEY_READY, \u0026ctx-\u003eflags);\n+\tclear_bit(TLS_TX_REKEY_FAILED, \u0026ctx-\u003eflags);\n+\n+\t/* Arm the drop floor before restoring the HW validator: from now on\n+\t * tls_validate_xmit_skb() drops payload retransmits of fully-ACKed data, so\n+\t * a stale clone whose record was purged here does not reach the NIC and trip\n+\t * its WARN on the new start marker. The cleartext leak on that path is closed\n+\t * separately by the skb_is_decrypted() gate in tls_sw_fallback(); this is\n+\t * only WARN avoidance. Set once; stays set for the socket's life.\n+\t */\n+\tset_bit(TLS_TX_REKEY_FLOOR, \u0026ctx-\u003eflags);\n+\n+\t/* Switch back to HW offload validator */\n+\tsmp_store_release(\u0026sk-\u003esk_validate_xmit_skb, tls_validate_xmit_skb);\n+\n+\tWRITE_ONCE(ctx-\u003erekey.sw_ctx, NULL);\n+\tWRITE_ONCE(ctx-\u003erekey.cipher_ctx, NULL);\n+\tspin_unlock_irqrestore(\u0026offload_ctx-\u003elock, flags);\n+\n+\tmemzero_explicit(\u0026offload_ctx-\u003erekey, sizeof(offload_ctx-\u003erekey));\n+\tcrypto_free_aead(old_sw_aead);\n+\n+\tup_read(\u0026device_offload_lock);\n+\n+\tif (deferred)\n+\t\tTLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXREKEY);\n+\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYOK);\n+\treturn 0;\n+\n+release_lock:\n+\tup_read(\u0026device_offload_lock);\n+\n+rekey_fallback:\n+\tkfree(offload_ctx-\u003erekey.start_marker);\n+\toffload_ctx-\u003erekey.start_marker = NULL;\n+\tspin_lock_irqsave(\u0026offload_ctx-\u003elock, flags);\n+\tset_bit(TLS_TX_REKEY_FAILED, \u0026ctx-\u003eflags);\n+\tclear_bit(TLS_TX_REKEY_READY, \u0026ctx-\u003eflags);\n+\tclear_bit(TLS_TX_REKEY_PENDING, \u0026ctx-\u003eflags);\n+\tspin_unlock_irqrestore(\u0026offload_ctx-\u003elock, flags);\n+\tif (deferred)\n+\t\tTLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXREKEY);\n+\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYFALLBACK);\n+\tTLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);\n+\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);\n+\n+\t/* Hard failure: HW rekey gave up and the connection is now pinned to\n+\t * SW encryption. The call site only sees the transient -EAGAIN retry\n+\t * (rc is not propagated here), so emit the trace from the fallback\n+\t * path itself; rc still holds the originating error.\n+\t */\n+\ttrace_tls_device_complete_rekey_fail(sk, rc);\n+\n+\treturn 0;\n+}\n+\n+static int tls_set_device_offload_rekey(struct sock *sk,\n+\t\t\t\t\tstruct tls_context *ctx,\n+\t\t\t\t\tstruct tls_crypto_info *new_crypto_info)\n+{\n+\tstruct tls_offload_context_tx *offload_ctx = tls_offload_ctx_tx(ctx);\n+\tbool rekey_pending = test_bit(TLS_TX_REKEY_PENDING, \u0026ctx-\u003eflags);\n+\tbool rekey_failed = test_bit(TLS_TX_REKEY_FAILED, \u0026ctx-\u003eflags);\n+\tbool defer = true;\n+\tint rc;\n+\n+\t/* Defer the switch back to HW until any in-flight old-key records are\n+\t * ACKed. A partially_sent_record needs no separate check: its record is\n+\t * on records_list before it is sent (tls_push_record()) and stays there\n+\t * until ACKed, so tls_has_unacked_records() already covers it.\n+\t */\n+\tif (!rekey_pending \u0026\u0026 !rekey_failed)\n+\t\tdefer = tls_has_unacked_records(offload_ctx) ||\n+\t\t\ttls_is_pending_open_record(ctx);\n+\n+\tif (!offload_ctx-\u003erekey.start_marker) {\n+\t\toffload_ctx-\u003erekey.start_marker =\n+\t\t\tkmalloc_obj(*offload_ctx-\u003erekey.start_marker);\n+\t\tif (!offload_ctx-\u003erekey.start_marker)\n+\t\t\treturn -ENOMEM;\n+\t}\n+\n+\trc = tls_device_start_rekey(sk, ctx, offload_ctx, new_crypto_info);\n \tif (rc)\n-\t\tgoto release_netdev;\n+\t\treturn rc;\n+\n+\tif (defer) {\n+\t\tif (!rekey_pending)\n+\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXREKEY);\n+\t\telse\n+\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYOK);\n+\t\treturn 0;\n+\t}\n+\n+\treturn tls_device_complete_rekey(sk, ctx, false, 0);\n+}\n+\n+static int tls_set_device_offload_initial(struct sock *sk,\n+\t\t\t\t\t struct tls_context *ctx,\n+\t\t\t\t\t struct net_device *netdev,\n+\t\t\t\t\t struct tls_crypto_info *crypto_info,\n+\t\t\t\t\t const struct tls_cipher_desc *cipher_desc)\n+{\n+\tstruct tls_prot_info *prot = \u0026ctx-\u003eprot_info;\n+\tstruct tls_record_info *start_marker_record;\n+\tstruct tls_offload_context_tx *offload_ctx;\n+\tchar *iv, *rec_seq;\n+\tint rc;\n \n \tiv = crypto_info_iv(crypto_info, cipher_desc);\n \trec_seq = crypto_info_rec_seq(crypto_info, cipher_desc);\n \n+\trc = init_prot_info(prot, crypto_info, cipher_desc);\n+\tif (rc)\n+\t\treturn rc;\n+\n \tmemcpy(ctx-\u003etx.iv + cipher_desc-\u003esalt, iv, cipher_desc-\u003eiv);\n \tmemcpy(ctx-\u003etx.rec_seq, rec_seq, cipher_desc-\u003erec_seq);\n \n \tstart_marker_record = kmalloc_obj(*start_marker_record);\n-\tif (!start_marker_record) {\n-\t\trc = -ENOMEM;\n-\t\tgoto release_netdev;\n-\t}\n+\tif (!start_marker_record)\n+\t\treturn -ENOMEM;\n \n \toffload_ctx = alloc_offload_ctx_tx(ctx);\n \tif (!offload_ctx) {\n@@ -1129,20 +1990,11 @@ int tls_set_device_offload(struct sock *sk)\n \tif (rc)\n \t\tgoto free_offload_ctx;\n \n-\tstart_marker_record-\u003eend_seq = tcp_sk(sk)-\u003ewrite_seq;\n-\tstart_marker_record-\u003elen = 0;\n-\tstart_marker_record-\u003enum_frags = 0;\n-\tlist_add_tail(\u0026start_marker_record-\u003elist, \u0026offload_ctx-\u003erecords_list);\n+\ttls_device_commit_start_marker(sk, offload_ctx, start_marker_record);\n \n \tclean_acked_data_enable(tcp_sk(sk), \u0026tls_tcp_clean_acked);\n \tctx-\u003epush_pending_record = tls_device_push_pending_record;\n \n-\t/* TLS offload is greatly simplified if we don't send\n-\t * SKBs where only part of the payload needs to be encrypted.\n-\t * So mark the last skb in the write queue as end of record.\n-\t */\n-\ttcp_write_collapse_fence(sk);\n-\n \t/* Avoid offloading if the device is down\n \t * We don't want to offload new flows after\n \t * the NETDEV_DOWN event\n@@ -1158,11 +2010,8 @@ int tls_set_device_offload(struct sock *sk)\n \t}\n \n \tctx-\u003epriv_ctx_tx = offload_ctx;\n-\trc = netdev-\u003etlsdev_ops-\u003etls_dev_add(netdev, sk, TLS_OFFLOAD_CTX_DIR_TX,\n-\t\t\t\t\t \u0026ctx-\u003ecrypto_send.info,\n-\t\t\t\t\t tcp_sk(sk)-\u003ewrite_seq);\n-\ttrace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_TX,\n-\t\t\t\t tcp_sk(sk)-\u003ewrite_seq, rec_seq, rc);\n+\trc = tls_device_dev_add_tx(sk, netdev, crypto_info,\n+\t\t\t\t tcp_sk(sk)-\u003ewrite_seq);\n \tif (rc)\n \t\tgoto release_lock;\n \n@@ -1174,7 +2023,6 @@ int tls_set_device_offload(struct sock *sk)\n \t * by the netdev's xmit function.\n \t */\n \tsmp_store_release(\u0026sk-\u003esk_validate_xmit_skb, tls_validate_xmit_skb);\n-\tdev_put(netdev);\n \n \treturn 0;\n \n@@ -1187,20 +2035,44 @@ int tls_set_device_offload(struct sock *sk)\n \tctx-\u003epriv_ctx_tx = NULL;\n free_marker_record:\n \tkfree(start_marker_record);\n-release_netdev:\n-\tdev_put(netdev);\n \treturn rc;\n }\n \n-int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)\n+int tls_set_device_offload(struct sock *sk,\n+\t\t\t struct tls_crypto_info *new_crypto_info)\n {\n-\tstruct tls12_crypto_info_aes_gcm_128 *info;\n-\tstruct tls_offload_context_rx *context;\n+\tstruct tls_crypto_info *crypto_info, *src_crypto_info;\n+\tconst struct tls_cipher_desc *cipher_desc;\n \tstruct net_device *netdev;\n-\tint rc = 0;\n+\tstruct tls_context *ctx;\n+\tint rc;\n \n-\tif (ctx-\u003ecrypto_recv.info.version != TLS_1_2_VERSION)\n-\t\treturn -EOPNOTSUPP;\n+\tctx = tls_get_ctx(sk);\n+\n+\t/* A rekey of a SW-offloaded socket belongs to tls_set_sw_offload(). */\n+\tif (new_crypto_info \u0026\u0026 ctx-\u003etx_conf != TLS_HW)\n+\t\treturn -EINVAL;\n+\n+\tcrypto_info = \u0026ctx-\u003ecrypto_send.info;\n+\tsrc_crypto_info = new_crypto_info ?: crypto_info;\n+\tcipher_desc = get_cipher_desc(src_crypto_info-\u003ecipher_type);\n+\tif (!cipher_desc || !cipher_desc-\u003eoffloadable)\n+\t\treturn -EINVAL;\n+\n+\t/* A rekey targets the device already holding the HW TX context\n+\t * (ctx-\u003enetdev), which can differ from the socket's current route after\n+\t * a route change or bond/team failover; tls_set_device_offload_rekey()\n+\t * and tls_device_complete_rekey() resolve it from ctx-\u003enetdev under\n+\t * device_offload_lock. Only the initial install needs the route device.\n+\t */\n+\tif (new_crypto_info)\n+\t\treturn tls_set_device_offload_rekey(sk, ctx, src_crypto_info);\n+\n+\t/* Initial install: a HW TX context must not already exist, otherwise\n+\t * alloc_offload_ctx_tx() below would silently overwrite it.\n+\t */\n+\tif (ctx-\u003epriv_ctx_tx)\n+\t\treturn -EEXIST;\n \n \tnetdev = get_netdev_for_sock(sk);\n \tif (!netdev) {\n@@ -1208,50 +2080,249 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)\n \t\treturn -EINVAL;\n \t}\n \n-\tif (!(netdev-\u003efeatures \u0026 NETIF_F_HW_TLS_RX)) {\n+\tif (!(netdev-\u003efeatures \u0026 NETIF_F_HW_TLS_TX)) {\n \t\trc = -EOPNOTSUPP;\n \t\tgoto release_netdev;\n \t}\n \n-\t/* Avoid offloading if the device is down\n-\t * We don't want to offload new flows after\n-\t * the NETDEV_DOWN event\n-\t *\n-\t * device_offload_lock is taken in tls_devices's NETDEV_DOWN\n-\t * handler thus protecting from the device going down before\n-\t * ctx was added to tls_device_list.\n-\t */\n-\tdown_read(\u0026device_offload_lock);\n-\tif (!(netdev-\u003eflags \u0026 IFF_UP)) {\n-\t\trc = -EINVAL;\n-\t\tgoto release_lock;\n+\trc = tls_set_device_offload_initial(sk, ctx, netdev, src_crypto_info,\n+\t\t\t\t\t cipher_desc);\n+\n+release_netdev:\n+\tdev_put(netdev);\n+\treturn rc;\n+}\n+\n+int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,\n+\t\t\t struct tls_crypto_info *new_crypto_info)\n+{\n+\tstruct tls_crypto_info *crypto_info, *src_crypto_info;\n+\tconst struct tls_cipher_desc *cipher_desc;\n+\tu32 drain_start = tcp_sk(sk)-\u003ecopied_seq;\n+\tstruct tls_offload_context_rx *context;\n+\tstruct net_device *netdev;\n+\tbool was_dev_add_pending;\n+\tbool moved_aead_recv = false;\n+\tbool retired_pending = false;\n+\tbool put_netdev = false;\n+\tint rc = 0;\n+\n+\t/* A rekey of a SW-offloaded socket belongs to tls_set_sw_offload(). */\n+\tif (new_crypto_info \u0026\u0026 ctx-\u003erx_conf != TLS_HW)\n+\t\treturn -EINVAL;\n+\n+\tcrypto_info = \u0026ctx-\u003ecrypto_recv.info;\n+\tsrc_crypto_info = new_crypto_info ?: crypto_info;\n+\tcipher_desc = get_cipher_desc(src_crypto_info-\u003ecipher_type);\n+\tif (!cipher_desc || !cipher_desc-\u003eoffloadable)\n+\t\treturn -EINVAL;\n+\n+\tif (new_crypto_info) {\n+\t\t/* Rekey targets the device holding the HW RX context, which\n+\t\t * can differ from the socket's route after a route change or\n+\t\t * bond/team failover. Resolve it from ctx-\u003enetdev under\n+\t\t * device_offload_lock, like the other del/add-key paths, not\n+\t\t * via get_netdev_for_sock(). The context owns the reference,\n+\t\t * so don't take an extra one here.\n+\t\t *\n+\t\t * A NULL netdev means tls_device_down() already ran: the HW RX\n+\t\t * context is deleted, TLS_RX_DEV_{DEGRADED,CLOSED} are set and\n+\t\t * every record is decrypted in SW, but rx_conf stays TLS_HW.\n+\t\t * The rekey is still required, the peer's KeyUpdate was parsed\n+\t\t * and recvmsg() returns -EKEYEXPIRED until the new key lands,\n+\t\t * so run the same state machine (queued records may still carry\n+\t\t * the deleted NIC context's old-key XOR) and account the new key\n+\t\t * as a SW fallback in place of the tls_dev_del()/tls_dev_add()\n+\t\t * steps, mirroring the TX side (tls_device_complete_rekey()).\n+\t\t * Do not fail the setsockopt.\n+\t\t */\n+\t\tdown_read(\u0026device_offload_lock);\n+\t\tnetdev = rcu_dereference_protected(ctx-\u003enetdev,\n+\t\t\t\t\t\t lockdep_is_held(\u0026device_offload_lock));\n+\t} else {\n+\t\tnetdev = get_netdev_for_sock(sk);\n+\t\tif (!netdev) {\n+\t\t\tpr_err_ratelimited(\"%s: netdev not found\\n\", __func__);\n+\t\t\treturn -EINVAL;\n+\t\t}\n+\t\tput_netdev = true;\n+\n+\t\tif (!(netdev-\u003efeatures \u0026 NETIF_F_HW_TLS_RX)) {\n+\t\t\trc = -EOPNOTSUPP;\n+\t\t\tgoto release_netdev;\n+\t\t}\n+\n+\t\t/* Avoid offloading if the device is down\n+\t\t * We don't want to offload new flows after\n+\t\t * the NETDEV_DOWN event\n+\t\t *\n+\t\t * device_offload_lock is taken in tls_devices's NETDEV_DOWN\n+\t\t * handler thus protecting from the device going down before\n+\t\t * ctx was added to tls_device_list.\n+\t\t */\n+\t\tdown_read(\u0026device_offload_lock);\n+\t\tif (!(netdev-\u003eflags \u0026 IFF_UP)) {\n+\t\t\trc = -EINVAL;\n+\t\t\tgoto release_lock;\n+\t\t}\n \t}\n \n-\tcontext = kzalloc_obj(*context);\n-\tif (!context) {\n-\t\trc = -ENOMEM;\n-\t\tgoto release_lock;\n+\tif (!new_crypto_info) {\n+\t\tcontext = kzalloc_obj(*context);\n+\t\tif (!context) {\n+\t\t\trc = -ENOMEM;\n+\t\t\tgoto release_lock;\n+\t\t}\n+\t\tctx-\u003epriv_ctx_rx = context;\n+\t} else {\n+\t\tcontext = tls_offload_ctx_rx(ctx);\n \t}\n+\twas_dev_add_pending = context-\u003edev_add_pending;\n \tcontext-\u003eresync_nh_reset = 1;\n \n-\tctx-\u003epriv_ctx_rx = context;\n-\trc = tls_set_sw_offload(sk, 0, NULL);\n+\tif (new_crypto_info) {\n+\t\tstruct tls_sw_context_rx *sw_ctx = tls_sw_ctx_rx(ctx);\n+\n+\t\t/* Classify against the record start, not the raw copied_seq: in\n+\t\t * strparser copy mode tcp_read_sock() has already advanced\n+\t\t * copied_seq past a parsed-ahead (possibly partial) record the\n+\t\t * user has not received, which may still carry the old NIC key's\n+\t\t * XOR. tls_device_decrypted() compensates the same way; keeping\n+\t\t * both in sync is what lets a drained-vs-still-draining decision\n+\t\t * here match the reencrypt-key decision there.\n+\t\t */\n+\t\tdrain_start = tls_device_rx_rec_start(sk, sw_ctx);\n+\n+\t\t/* netdev is NULL only after tls_device_down(), which already\n+\t\t * deleted the HW RX context and set TLS_RX_DEV_CLOSED; the\n+\t\t * netdev check just makes that dependency explicit.\n+\t\t */\n+\t\tif (netdev \u0026\u0026 !test_bit(TLS_RX_DEV_CLOSED, \u0026ctx-\u003eflags)) {\n+\t\t\tset_bit(TLS_RX_DEV_CLOSED, \u0026ctx-\u003eflags);\n+\t\t\tsynchronize_net();\n+\t\t\tnetdev-\u003etlsdev_ops-\u003etls_dev_del(netdev, ctx,\n+\t\t\t\t\t\t\tTLS_OFFLOAD_CTX_DIR_RX);\n+\t\t}\n+\n+\t\tif (context-\u003erekey.old_aead_recv \u0026\u0026\n+\t\t before(drain_start, context-\u003erekey.old_nic_boundary)) {\n+\t\t\t/* Previous rekey still draining. Keep rekey.old_aead_recv,\n+\t\t\t * it is the only key that can undo the NIC-XOR on queued\n+\t\t\t * records. sw_ctx-\u003eaead_recv may be re-setkey'd by\n+\t\t\t * tls_sw_ctx_init(); that intermediate key was never on\n+\t\t\t * the NIC and its wire era is drained, so it is needed\n+\t\t\t * for neither undo nor AEAD. Defer dev_add; the new key\n+\t\t\t * is installed once drain_start crosses rekey.old_nic_boundary.\n+\t\t\t */\n+\t\t\tcontext-\u003edev_add_pending = 1;\n+\t\t\ttrace_tls_device_rekey_start(sk, drain_start,\n+\t\t\t\t\t\t context-\u003erekey.old_nic_boundary,\n+\t\t\t\t\t\t true);\n+\t\t} else {\n+\t\t\tstruct tcp_sock *tp = tcp_sk(sk);\n+\t\t\tu32 nic_end;\n+\n+\t\t\tif (context-\u003erekey.old_aead_recv) {\n+\t\t\t\t/* Prior rekey's era already drained (drain_start is\n+\t\t\t\t * past old_nic_boundary), so retiring its key here\n+\t\t\t\t * is a boundary crossing, same as the free in\n+\t\t\t\t * tls_device_decrypted(); mark it done.\n+\t\t\t\t */\n+\t\t\t\ttrace_tls_device_rekey_done(sk, drain_start,\n+\t\t\t\t\t\t\t context-\u003erekey.old_nic_boundary);\n+\t\t\t\tcrypto_free_aead(context-\u003erekey.old_aead_recv);\n+\t\t\t\tcontext-\u003erekey.old_aead_recv = NULL;\n+\t\t\t}\n+\n+\t\t\t/* Flush the backlog so TCP's view is current, then take the\n+\t\t\t * highest byte TCP holds, including the out-of-order tail:\n+\t\t\t * a NIC-transformed segment behind a host-side drop sits\n+\t\t\t * above rcv_nxt until the retransmit fills the hole and\n+\t\t\t * must still be classified against the old key. This is\n+\t\t\t * still only the stack's view, a transformed segment the\n+\t\t\t * NIC has not delivered yet is caught in-band by\n+\t\t\t * tls_device_decrypted(), which slides the boundary.\n+\t\t\t */\n+\t\t\t__sk_flush_backlog(sk);\n+\t\t\tnic_end = tp-\u003ercv_nxt;\n+\t\t\tif (!RB_EMPTY_ROOT(\u0026tp-\u003eout_of_order_queue) \u0026\u0026\n+\t\t\t after(TCP_SKB_CB(tp-\u003eooo_last_skb)-\u003eend_seq, nic_end))\n+\t\t\t\tnic_end = TCP_SKB_CB(tp-\u003eooo_last_skb)-\u003eend_seq;\n+\n+\t\t\tif (before(drain_start, nic_end)) {\n+\t\t\t\tcontext-\u003erekey.old_aead_recv = sw_ctx-\u003eaead_recv;\n+\t\t\t\t/* NULL so tls_sw_ctx_init() allocates a fresh tfm\n+\t\t\t\t * for the new key instead of re-keying the one we\n+\t\t\t\t * must keep for the drain.\n+\t\t\t\t */\n+\t\t\t\tsw_ctx-\u003eaead_recv = NULL;\n+\t\t\t\tmoved_aead_recv = true;\n+\t\t\t\tmemcpy(context-\u003erekey.old_iv, ctx-\u003erx.iv,\n+\t\t\t\t sizeof(context-\u003erekey.old_iv));\n+\t\t\t\tmemcpy(context-\u003erekey.old_rec_seq, ctx-\u003erx.rec_seq,\n+\t\t\t\t sizeof(context-\u003erekey.old_rec_seq));\n+\t\t\t\tcontext-\u003erekey.old_nic_boundary = nic_end;\n+\t\t\t\tcontext-\u003edev_add_pending = 1;\n+\t\t\t} else if (was_dev_add_pending) {\n+\t\t\t\t/* A prior rekey's deferred dev_add can no longer\n+\t\t\t\t * run: its trigger (old_aead_recv) was just freed\n+\t\t\t\t * above and no new drain replaces it. Its era\n+\t\t\t\t * drained successfully (drain_start is already past\n+\t\t\t\t * old_nic_boundary), so retire it and let the new\n+\t\t\t\t * key install immediately below. retired_pending\n+\t\t\t\t * defers its OK/gauge accounting to the post-init\n+\t\t\t\t * block, past the error goto, so a failed\n+\t\t\t\t * tls_sw_ctx_init() needs no counter undo.\n+\t\t\t\t */\n+\t\t\t\tcontext-\u003edev_add_pending = 0;\n+\t\t\t\tretired_pending = true;\n+\t\t\t}\n+\t\t\ttrace_tls_device_rekey_start(sk, drain_start, nic_end,\n+\t\t\t\t\t\t before(drain_start, nic_end));\n+\t\t}\n+\t}\n+\n+\trc = tls_sw_ctx_init(sk, 0, new_crypto_info);\n \tif (rc)\n \t\tgoto release_ctx;\n \n-\trc = netdev-\u003etlsdev_ops-\u003etls_dev_add(netdev, sk, TLS_OFFLOAD_CTX_DIR_RX,\n-\t\t\t\t\t \u0026ctx-\u003ecrypto_recv.info,\n-\t\t\t\t\t tcp_sk(sk)-\u003ecopied_seq);\n-\tinfo = (void *)\u0026ctx-\u003ecrypto_recv.info;\n-\ttrace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_RX,\n-\t\t\t\t tcp_sk(sk)-\u003ecopied_seq, info-\u003erec_seq, rc);\n-\tif (rc)\n-\t\tgoto free_sw_resources;\n+\tif (!context-\u003edev_add_pending) {\n+\t\tif (retired_pending) {\n+\t\t\t/* Account the superseded rekey that drained OK, mirroring\n+\t\t\t * the deferred-add path: one RXREKEYOK and release its\n+\t\t\t * in-flight gauge. The new key's own OK/FALLBACK is counted\n+\t\t\t * by tls_device_dev_add_rx() just below.\n+\t\t\t */\n+\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);\n+\t\t\tTLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXREKEY);\n+\t\t}\n+\t\tif (netdev) {\n+\t\t\trc = tls_device_dev_add_rx(sk, ctx, netdev,\n+\t\t\t\t\t\t src_crypto_info, drain_start,\n+\t\t\t\t\t\t !!new_crypto_info);\n+\t\t} else {\n+\t\t\t/* No device after tls_device_down(); the SW path keeps\n+\t\t\t * decrypting.\n+\t\t\t */\n+\t\t\ttls_device_rx_rekey_fallback(sk, ctx);\n+\t\t}\n+\t\tif (!new_crypto_info) {\n+\t\t\tif (rc)\n+\t\t\t\tgoto free_sw_resources;\n+\t\t\ttls_device_attach(ctx, sk, netdev);\n+\t\t}\n+\t} else if (!was_dev_add_pending) {\n+\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXREKEY);\n+\t} else {\n+\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);\n+\t}\n+\n+\ttls_sw_ctx_finalize(sk, 0, new_crypto_info);\n \n-\ttls_device_attach(ctx, sk, netdev);\n \tup_read(\u0026device_offload_lock);\n \n-\tdev_put(netdev);\n+\tif (put_netdev)\n+\t\tdev_put(netdev);\n \n \treturn 0;\n \n@@ -1260,17 +2331,39 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)\n \ttls_sw_free_resources_rx(sk);\n \tdown_read(\u0026device_offload_lock);\n release_ctx:\n-\tctx-\u003epriv_ctx_rx = NULL;\n+\tif (!new_crypto_info) {\n+\t\tctx-\u003epriv_ctx_rx = NULL;\n+\t} else {\n+\t\t/* A failed RX rekey is terminal, so there is no HW state to roll\n+\t\t * back to. KeyUpdate is directional and the peer's TX has already\n+\t\t * switched keys, so once the new RX key fails to install the old\n+\t\t * SW key restored below cannot decrypt any further record; the\n+\t\t * socket is dead and the app must close it. The half-torn HW\n+\t\t * context (tls_dev_del already ran) and any dangling\n+\t\t * dev_add_pending / old_aead_recv are reclaimed by\n+\t\t * tls_device_offload_cleanup_rx() on close.\n+\t\t */\n+\t\tcontext-\u003edev_add_pending = was_dev_add_pending;\n+\t\tif (moved_aead_recv) {\n+\t\t\tstruct tls_sw_context_rx *sw_ctx = tls_sw_ctx_rx(ctx);\n+\n+\t\t\tcrypto_free_aead(sw_ctx-\u003eaead_recv);\n+\t\t\tsw_ctx-\u003eaead_recv = context-\u003erekey.old_aead_recv;\n+\t\t\tcontext-\u003erekey.old_aead_recv = NULL;\n+\t\t}\n+\t}\n release_lock:\n \tup_read(\u0026device_offload_lock);\n release_netdev:\n-\tdev_put(netdev);\n+\tif (put_netdev)\n+\t\tdev_put(netdev);\n \treturn rc;\n }\n \n void tls_device_offload_cleanup_rx(struct sock *sk)\n {\n \tstruct tls_context *tls_ctx = tls_get_ctx(sk);\n+\tstruct tls_offload_context_rx *rx_ctx;\n \tstruct net_device *netdev;\n \n \tdown_read(\u0026device_offload_lock);\n@@ -1279,8 +2372,9 @@ void tls_device_offload_cleanup_rx(struct sock *sk)\n \tif (!netdev)\n \t\tgoto out;\n \n-\tnetdev-\u003etlsdev_ops-\u003etls_dev_del(netdev, tls_ctx,\n-\t\t\t\t\tTLS_OFFLOAD_CTX_DIR_RX);\n+\tif (!test_bit(TLS_RX_DEV_CLOSED, \u0026tls_ctx-\u003eflags))\n+\t\tnetdev-\u003etlsdev_ops-\u003etls_dev_del(netdev, tls_ctx,\n+\t\t\t\t\t\tTLS_OFFLOAD_CTX_DIR_RX);\n \n \tif (tls_ctx-\u003etx_conf != TLS_HW) {\n \t\tdev_put(netdev);\n@@ -1290,6 +2384,19 @@ void tls_device_offload_cleanup_rx(struct sock *sk)\n \t}\n out:\n \tup_read(\u0026device_offload_lock);\n+\n+\trx_ctx = tls_offload_ctx_rx(tls_ctx);\n+\tif (rx_ctx \u0026\u0026 rx_ctx-\u003erekey.old_aead_recv) {\n+\t\tcrypto_free_aead(rx_ctx-\u003erekey.old_aead_recv);\n+\t\trx_ctx-\u003erekey.old_aead_recv = NULL;\n+\t}\n+\n+\tif (rx_ctx \u0026\u0026 rx_ctx-\u003edev_add_pending) {\n+\t\trx_ctx-\u003edev_add_pending = 0;\n+\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYABORTED);\n+\t\tTLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXREKEY);\n+\t}\n+\n \ttls_sw_release_resources_rx(sk);\n }\n \n@@ -1317,10 +2424,16 @@ static int tls_device_down(struct net_device *netdev)\n \tspin_unlock_irqrestore(\u0026tls_device_lock, flags);\n \n \tlist_for_each_entry_safe(ctx, tmp, \u0026list, list)\t{\n-\t\t/* Stop offloaded TX and switch to the fallback.\n-\t\t * tls_is_skb_tx_device_offloaded will return false.\n+\t\t/* Stop offloaded TX and switch to the fallback. For a socket not\n+\t\t * mid-rekey, tls_is_skb_tx_device_offloaded() then returns false; a\n+\t\t * PENDING/FAILED socket keeps the rekey validator (under which only a\n+\t\t * decrypted straddler still offloads), and the synchronize_net()\n+\t\t * below drains any such in-flight skb before tls_dev_del().\n \t\t */\n-\t\tWRITE_ONCE(ctx-\u003esk-\u003esk_validate_xmit_skb, tls_validate_xmit_skb_sw);\n+\t\tif (!test_bit(TLS_TX_REKEY_PENDING, \u0026ctx-\u003eflags) \u0026\u0026\n+\t\t !test_bit(TLS_TX_REKEY_FAILED, \u0026ctx-\u003eflags))\n+\t\t\tWRITE_ONCE(ctx-\u003esk-\u003esk_validate_xmit_skb,\n+\t\t\t\t tls_validate_xmit_skb_sw);\n \n \t\t/* Stop the RX and TX resync.\n \t\t * tls_dev_resync must not be called after tls_dev_del.\n@@ -1337,13 +2450,18 @@ static int tls_device_down(struct net_device *netdev)\n \t\tsynchronize_net();\n \n \t\t/* Release the offload context on the driver side. */\n-\t\tif (ctx-\u003etx_conf == TLS_HW)\n+\t\tif (ctx-\u003etx_conf == TLS_HW \u0026\u0026\n+\t\t !test_bit(TLS_TX_DEV_CLOSED, \u0026ctx-\u003eflags)) {\n \t\t\tnetdev-\u003etlsdev_ops-\u003etls_dev_del(netdev, ctx,\n \t\t\t\t\t\t\tTLS_OFFLOAD_CTX_DIR_TX);\n+\t\t\tset_bit(TLS_TX_DEV_CLOSED, \u0026ctx-\u003eflags);\n+\t\t}\n \t\tif (ctx-\u003erx_conf == TLS_HW \u0026\u0026\n-\t\t !test_bit(TLS_RX_DEV_CLOSED, \u0026ctx-\u003eflags))\n+\t\t !test_bit(TLS_RX_DEV_CLOSED, \u0026ctx-\u003eflags)) {\n \t\t\tnetdev-\u003etlsdev_ops-\u003etls_dev_del(netdev, ctx,\n \t\t\t\t\t\t\tTLS_OFFLOAD_CTX_DIR_RX);\n+\t\t\tset_bit(TLS_RX_DEV_CLOSED, \u0026ctx-\u003eflags);\n+\t\t}\n \n \t\tdev_put(netdev);\n \n@@ -1411,12 +2529,27 @@ static struct notifier_block tls_dev_notifier = {\n \n int __init tls_device_init(void)\n {\n-\tint err;\n+\tunsigned char *page_addr;\n+\tint err, i;\n \n-\tdummy_page = alloc_page(GFP_KERNEL);\n+\tdummy_page = alloc_page(GFP_KERNEL | __GFP_ZERO);\n \tif (!dummy_page)\n \t\treturn -ENOMEM;\n \n+\t/* Pre-populate the first 256 bytes with an identity map so that,\n+\t * when this page is used as the tail-frag fallback (allocation\n+\t * failure in tls_device_record_close()), dummy_page[record_type]\n+\t * yields the correct TLS 1.3 content_type byte for any record_type\n+\t * without runtime validation.\n+\t *\n+\t * A high record_type pushes the tag placeholder past the identity\n+\t * map, so __GFP_ZERO is what keeps tag-placeholder bytes defined\n+\t * rather than exposing uninitialized page contents.\n+\t */\n+\tpage_addr = page_address(dummy_page);\n+\tfor (i = 0; i \u003c 256; i++)\n+\t\tpage_addr[i] = (unsigned char)i;\n+\n \tdestruct_wq = alloc_workqueue(\"ktls_device_destruct\", WQ_PERCPU, 0);\n \tif (!destruct_wq) {\n \t\terr = -ENOMEM;\ndiff --git a/net/tls/tls_device_fallback.c b/net/tls/tls_device_fallback.c\nindex 3b7d0ab2bcf17..f2a0ae827bb2a 100644\n--- a/net/tls/tls_device_fallback.c\n+++ b/net/tls/tls_device_fallback.c\n@@ -37,14 +37,15 @@\n \n #include \"tls.h\"\n \n-static int tls_enc_record(struct aead_request *aead_req,\n+static int tls_enc_record(struct tls_context *tls_ctx,\n+\t\t\t struct aead_request *aead_req,\n \t\t\t struct crypto_aead *aead, char *aad,\n \t\t\t char *iv, __be64 rcd_sn,\n \t\t\t struct scatter_walk *in,\n-\t\t\t struct scatter_walk *out, int *in_len,\n-\t\t\t struct tls_prot_info *prot)\n+\t\t\t struct scatter_walk *out, int *in_len)\n {\n \tunsigned char buf[TLS_HEADER_SIZE + TLS_MAX_IV_SIZE];\n+\tstruct tls_prot_info *prot = \u0026tls_ctx-\u003eprot_info;\n \tconst struct tls_cipher_desc *cipher_desc;\n \tstruct scatterlist sg_in[3];\n \tstruct scatterlist sg_out[3];\n@@ -55,7 +56,7 @@ static int tls_enc_record(struct aead_request *aead_req,\n \tcipher_desc = get_cipher_desc(prot-\u003ecipher_type);\n \tDEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc-\u003eoffloadable);\n \n-\tbuf_size = TLS_HEADER_SIZE + cipher_desc-\u003eiv;\n+\tbuf_size = prot-\u003eprepend_size;\n \tlen = min_t(int, *in_len, buf_size);\n \n \tmemcpy_from_scatterwalk(buf, in, len);\n@@ -66,16 +67,27 @@ static int tls_enc_record(struct aead_request *aead_req,\n \t\treturn 0;\n \n \tlen = buf[4] | (buf[3] \u003c\u003c 8);\n-\tlen -= cipher_desc-\u003eiv;\n+\tif (prot-\u003eversion != TLS_1_3_VERSION)\n+\t\tlen -= cipher_desc-\u003eiv;\n \n \ttls_make_aad(aad, len - cipher_desc-\u003etag, (char *)\u0026rcd_sn, buf[0], prot);\n \n-\tmemcpy(iv + cipher_desc-\u003esalt, buf + TLS_HEADER_SIZE, cipher_desc-\u003eiv);\n+\tif (prot-\u003eversion == TLS_1_3_VERSION) {\n+\t\tvoid *iv_src = crypto_info_iv(\u0026tls_ctx-\u003ecrypto_send.info,\n+\t\t\t\t\t cipher_desc);\n+\n+\t\tmemcpy(iv + cipher_desc-\u003esalt, iv_src, cipher_desc-\u003eiv);\n+\t} else {\n+\t\tmemcpy(iv + cipher_desc-\u003esalt, buf + TLS_HEADER_SIZE,\n+\t\t cipher_desc-\u003eiv);\n+\t}\n+\n+\ttls_xor_iv_with_seq(prot, iv, (char *)\u0026rcd_sn);\n \n \tsg_init_table(sg_in, ARRAY_SIZE(sg_in));\n \tsg_init_table(sg_out, ARRAY_SIZE(sg_out));\n-\tsg_set_buf(sg_in, aad, TLS_AAD_SPACE_SIZE);\n-\tsg_set_buf(sg_out, aad, TLS_AAD_SPACE_SIZE);\n+\tsg_set_buf(sg_in, aad, prot-\u003eaad_size);\n+\tsg_set_buf(sg_out, aad, prot-\u003eaad_size);\n \tscatterwalk_get_sglist(in, sg_in + 1);\n \tscatterwalk_get_sglist(out, sg_out + 1);\n \n@@ -108,13 +120,6 @@ static int tls_enc_record(struct aead_request *aead_req,\n \treturn rc;\n }\n \n-static void tls_init_aead_request(struct aead_request *aead_req,\n-\t\t\t\t struct crypto_aead *aead)\n-{\n-\taead_request_set_tfm(aead_req, aead);\n-\taead_request_set_ad(aead_req, TLS_AAD_SPACE_SIZE);\n-}\n-\n static struct aead_request *tls_alloc_aead_request(struct crypto_aead *aead,\n \t\t\t\t\t\t gfp_t flags)\n {\n@@ -124,14 +129,15 @@ static struct aead_request *tls_alloc_aead_request(struct crypto_aead *aead,\n \n \taead_req = kzalloc(req_size, flags);\n \tif (aead_req)\n-\t\ttls_init_aead_request(aead_req, aead);\n+\t\taead_request_set_tfm(aead_req, aead);\n \treturn aead_req;\n }\n \n-static int tls_enc_records(struct aead_request *aead_req,\n+static int tls_enc_records(struct tls_context *tls_ctx,\n+\t\t\t struct aead_request *aead_req,\n \t\t\t struct crypto_aead *aead, struct scatterlist *sg_in,\n \t\t\t struct scatterlist *sg_out, char *aad, char *iv,\n-\t\t\t u64 rcd_sn, int len, struct tls_prot_info *prot)\n+\t\t\t u64 rcd_sn, int len)\n {\n \tstruct scatter_walk out, in;\n \tint rc;\n@@ -140,8 +146,8 @@ static int tls_enc_records(struct aead_request *aead_req,\n \tscatterwalk_start(\u0026out, sg_out);\n \n \tdo {\n-\t\trc = tls_enc_record(aead_req, aead, aad, iv,\n-\t\t\t\t cpu_to_be64(rcd_sn), \u0026in, \u0026out, \u0026len, prot);\n+\t\trc = tls_enc_record(tls_ctx, aead_req, aead, aad, iv,\n+\t\t\t\t cpu_to_be64(rcd_sn), \u0026in, \u0026out, \u0026len);\n \t\trcd_sn++;\n \n \t} while (rc == 0 \u0026\u0026 len);\n@@ -184,6 +190,14 @@ static void complete_skb(struct sk_buff *nskb, struct sk_buff *skb, int headln)\n \n \tskb_copy_header(nskb, skb);\n \n+\t/* nskb now carries ciphertext, but skb_copy_header() inherited\n+\t * skb-\u003edecrypted from the plaintext original. Clear it so the bit keeps\n+\t * meaning \"still-plaintext, needs an encryptor\": otherwise a requeued\n+\t * nskb would be needlessly re-validated (and re-encrypted) and would trip\n+\t * the NIC's decrypted-vs-start-marker WARN.\n+\t */\n+\tnskb-\u003edecrypted = 0;\n+\n \tskb_put(nskb, skb-\u003elen);\n \tmemcpy(nskb-\u003edata, skb-\u003edata, headln);\n \n@@ -314,7 +328,10 @@ static struct sk_buff *tls_enc_skb(struct tls_context *tls_ctx,\n \tcipher_desc = get_cipher_desc(tls_ctx-\u003ecrypto_send.info.cipher_type);\n \tDEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc-\u003eoffloadable);\n \n-\tbuf_len = cipher_desc-\u003esalt + cipher_desc-\u003eiv + TLS_AAD_SPACE_SIZE +\n+\taead_request_set_ad(aead_req, tls_ctx-\u003eprot_info.aad_size);\n+\n+\tbuf_len = cipher_desc-\u003esalt + cipher_desc-\u003eiv +\n+\t\t tls_ctx-\u003eprot_info.aad_size +\n \t\t sync_size + cipher_desc-\u003etag;\n \tbuf = kmalloc(buf_len, GFP_ATOMIC);\n \tif (!buf)\n@@ -324,7 +341,7 @@ static struct sk_buff *tls_enc_skb(struct tls_context *tls_ctx,\n \tsalt = crypto_info_salt(\u0026tls_ctx-\u003ecrypto_send.info, cipher_desc);\n \tmemcpy(iv, salt, cipher_desc-\u003esalt);\n \taad = buf + cipher_desc-\u003esalt + cipher_desc-\u003eiv;\n-\tdummy_buf = aad + TLS_AAD_SPACE_SIZE;\n+\tdummy_buf = aad + tls_ctx-\u003eprot_info.aad_size;\n \n \tnskb = alloc_skb(skb_headroom(skb) + skb-\u003elen, GFP_ATOMIC);\n \tif (!nskb)\n@@ -335,9 +352,8 @@ static struct sk_buff *tls_enc_skb(struct tls_context *tls_ctx,\n \tfill_sg_out(sg_out, buf, tls_ctx, nskb, tcp_payload_offset,\n \t\t payload_len, sync_size, dummy_buf);\n \n-\tif (tls_enc_records(aead_req, ctx-\u003eaead_send, sg_in, sg_out, aad, iv,\n-\t\t\t rcd_sn, sync_size + payload_len,\n-\t\t\t \u0026tls_ctx-\u003eprot_info) \u003c 0)\n+\tif (tls_enc_records(tls_ctx, aead_req, ctx-\u003eaead_send, sg_in, sg_out,\n+\t\t\t aad, iv, rcd_sn, sync_size + payload_len) \u003c 0)\n \t\tgoto free_nskb;\n \n \tcomplete_skb(nskb, skb, tcp_payload_offset);\n@@ -388,8 +404,17 @@ static struct sk_buff *tls_sw_fallback(struct sock *sk, struct sk_buff *skb)\n \tsg_init_table(sg_out, ARRAY_SIZE(sg_out));\n \n \tif (fill_sg_in(sg_in, skb, ctx, \u0026rcd_sn, \u0026sync_size, \u0026resync_sgs)) {\n-\t\t/* bypass packets before kernel TLS socket option was set */\n-\t\tif (sync_size \u003c 0 \u0026\u0026 payload_len \u003c= -sync_size)\n+\t\t/* Below the record range (start marker / already-freed record).\n+\t\t * Pass through only cleartext that was never offload-encrypted\n+\t\t * (skb-\u003edecrypted == 0): genuine pre-TLS bytes sent before the\n+\t\t * socket option was set, or SW-encrypted rekey ciphertext. A\n+\t\t * decrypted=1 skb here is offload-record plaintext whose record was\n+\t\t * purged (e.g. a rekey installed a new start marker above its seq);\n+\t\t * it must never reach the wire in the clear, so continue on and\n+\t\t * drop it (nskb stays NULL).\n+\t\t */\n+\t\tif (sync_size \u003c 0 \u0026\u0026 payload_len \u003c= -sync_size \u0026\u0026\n+\t\t !skb_is_decrypted(skb))\n \t\t\tnskb = skb_get(skb);\n \t\tgoto put_sg;\n \t}\n@@ -408,11 +433,57 @@ static struct sk_buff *tls_sw_fallback(struct sock *sk, struct sk_buff *skb)\n \treturn nskb;\n }\n \n+/* Post-rekey drop floor. Once a rekey has completed (TLS_TX_REKEY_FLOOR set), a\n+ * stale retransmit clone of already-ACKed data may still be dequeued from a\n+ * qdisc; if its offload record was purged at completion it now maps to a rekey\n+ * start marker. The cleartext leak on that path is closed unconditionally by\n+ * the skb_is_decrypted() gate in tls_sw_fallback(); this floor additionally\n+ * drops the clone before it reaches the NIC, avoiding the driver's WARN\n+ * (mlx5e_ktls_handle_tx_skb() SKIP_NO_DATA) on an otherwise-legitimate race.\n+ * Only needed by tls_validate_xmit_skb() (the restored HW-offload validator):\n+ * only there can a purged-record clone reach the NIC and hit the new start\n+ * marker. Under the rekey/SW validators the only skb the NIC offloads is a\n+ * decrypted straddler whose record is still present (no SKIP_NO_DATA), and a\n+ * stale clone is dropped by the skb_is_decrypted() gate in tls_sw_fallback().\n+ * Such a clone is exactly a payload skb whose end_seq \u003c= snd_una: the peer has\n+ * already ACKed that data, so dropping it is always safe. Live/unacked data\n+ * (including a legitimate retransmit, or a straddler ending past snd_una) is\n+ * never touched; pure ACKs and zero-window probes carry no payload and pass.\n+ */\n+static bool tls_tx_drop_acked_clone(struct sock *sk, struct sk_buff *skb)\n+{\n+\tint payload_len = skb-\u003elen - skb_tcp_all_headers(skb);\n+\tu32 end_seq;\n+\n+\tif (likely(!test_bit(TLS_TX_REKEY_FLOOR, \u0026tls_get_ctx(sk)-\u003eflags)))\n+\t\treturn false;\n+\n+\tif (payload_len \u003c= 0)\n+\t\treturn false;\n+\n+\t/* Drop only when the whole payload is already ACKed (end_seq \u003c= snd_una):\n+\t * such a skb is purely a stale retransmit clone the peer already has. A\n+\t * clone straddling snd_una still carries unacked bytes, so leave it to the\n+\t * normal paths (a live record is re-encrypted; a marker/freed-record hit is\n+\t * dropped there too). Both the leak (skb_is_decrypted() gate) and the mlx5\n+\t * WARN only concern the fully-ACKed case handled here.\n+\t */\n+\tend_seq = ntohl(tcp_hdr(skb)-\u003eseq) + payload_len;\n+\treturn !after(end_seq, READ_ONCE(tcp_sk(sk)-\u003esnd_una));\n+}\n+\n struct sk_buff *tls_validate_xmit_skb(struct sock *sk,\n \t\t\t\t struct net_device *dev,\n \t\t\t\t struct sk_buff *skb)\n {\n-\tif (dev == rcu_dereference_bh(tls_get_ctx(sk)-\u003enetdev) ||\n+\tstruct tls_context *tls_ctx = tls_get_ctx(sk);\n+\n+\tif (unlikely(tls_tx_drop_acked_clone(sk, skb))) {\n+\t\tkfree_skb(skb);\n+\t\treturn NULL;\n+\t}\n+\n+\tif (dev == rcu_dereference_bh(tls_ctx-\u003enetdev) ||\n \t netif_is_bond_master(dev))\n \t\treturn skb;\n \n@@ -427,6 +498,65 @@ struct sk_buff *tls_validate_xmit_skb_sw(struct sock *sk,\n \treturn tls_sw_fallback(sk, skb);\n }\n \n+struct sk_buff *tls_validate_xmit_skb_rekey(struct sock *sk,\n+\t\t\t\t\t struct net_device *dev,\n+\t\t\t\t\t struct sk_buff *skb)\n+{\n+\tstruct tls_context *tls_ctx = tls_get_ctx(sk);\n+\tu32 tcp_seq = ntohl(tcp_hdr(skb)-\u003eseq);\n+\tu32 pivot_seq;\n+\n+\t/* acquire pairs with clear_bit_unlock() on re-arm; makes the refreshed\n+\t * boundary_seq visible in the else branch below.\n+\t */\n+\tif (test_bit_acquire(TLS_TX_REKEY_FAILED, \u0026tls_ctx-\u003eflags)) {\n+\t\tint payload_len = skb-\u003elen - skb_tcp_all_headers(skb);\n+\t\tu32 snd_una = READ_ONCE(tcp_sk(sk)-\u003esnd_una);\n+\n+\t\t/* FAILED: HW context gone and all old-key plaintext ACKed\n+\t\t * (snd_una \u003e= boundary_seq). seq \u003c boundary_seq is old-key data\n+\t\t * whose records are freed, so tls_sw_fallback() drops it. seq \u003e=\n+\t\t * boundary_seq is SW ciphertext with no record. A retransmit is\n+\t\t * built at seq == snd_una (tcp_trim_head()), so an ACK landing\n+\t\t * before we run can move snd_una past seq while the tail is\n+\t\t * unacked; pivoting on snd_una alone would drop that live data\n+\t\t * and force an RTO. Pass through any non-decrypted skb ending\n+\t\t * past snd_una (mirrors tls_tx_drop_acked_clone()); fully-ACKed\n+\t\t * clones fall to the pivot and are dropped.\n+\t\t */\n+\t\tif (payload_len \u003e 0 \u0026\u0026 !skb_is_decrypted(skb) \u0026\u0026\n+\t\t after(tcp_seq + payload_len, snd_una))\n+\t\t\treturn skb;\n+\n+\t\tpivot_seq = snd_una;\n+\t} else {\n+\t\t/* PENDING: new-key data is SW-encrypted at seq \u003e= boundary_seq;\n+\t\t * old-key data below it is still unacked.\n+\t\t *\n+\t\t * On the first arm, boundary_seq is published by the\n+\t\t * smp_store_release() of sk_validate_xmit_skb in\n+\t\t * tls_device_start_rekey(); the xmit path loads that pointer with a\n+\t\t * plain read (net/core/dev.c), so pair it here with an smp_rmb()\n+\t\t * before reading boundary_seq. A stale boundary_seq (0) would pass an\n+\t\t * unacked old-key plaintext skb through; tls_is_skb_tx_device_offloaded()\n+\t\t * would still HW-encrypt it with the installed old key, so not a leak,\n+\t\t * but the barrier keeps the pivot accurate.\n+\t\t */\n+\t\tsmp_rmb();\n+\t\tpivot_seq = READ_ONCE(tls_ctx-\u003erekey.boundary_seq);\n+\t}\n+\n+\t/* At or after the pivot: already correctly encrypted, pass through */\n+\tif (!before(tcp_seq, pivot_seq))\n+\t\treturn skb;\n+\n+\t/* Below the pivot: retransmit of old data, SW fallback with old key */\n+\treturn tls_sw_fallback(sk, skb);\n+}\n+\n+/* Address taken by tls_is_skb_tx_device_offloaded() in the offload drivers. */\n+EXPORT_SYMBOL_GPL(tls_validate_xmit_skb_rekey);\n+\n struct sk_buff *tls_encrypt_skb(struct sk_buff *skb)\n {\n \treturn tls_sw_fallback(skb-\u003esk, skb);\ndiff --git a/net/tls/tls_main.c b/net/tls/tls_main.c\nindex fbb274287aa5f..0a9e7d15fa95b 100644\n--- a/net/tls/tls_main.c\n+++ b/net/tls/tls_main.c\n@@ -347,16 +347,28 @@ static void tls_sk_proto_cleanup(struct sock *sk,\n \t\ttls_sw_release_resources_tx(sk);\n \t\tTLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);\n \t} else if (ctx-\u003etx_conf == TLS_HW) {\n+\t\tbool rekey_failed = test_bit(TLS_TX_REKEY_FAILED, \u0026ctx-\u003eflags);\n+\n \t\ttls_device_free_resources_tx(sk);\n-\t\tTLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);\n+\n+\t\tif (rekey_failed)\n+\t\t\tTLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);\n+\t\telse\n+\t\t\tTLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);\n \t}\n \n \tif (ctx-\u003erx_conf == TLS_SW) {\n \t\ttls_sw_release_resources_rx(sk);\n \t\tTLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXSW);\n \t} else if (ctx-\u003erx_conf == TLS_HW) {\n+\t\tbool rekey_failed = test_bit(TLS_RX_REKEY_FAILED, \u0026ctx-\u003eflags);\n+\n \t\ttls_device_offload_cleanup_rx(sk);\n-\t\tTLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);\n+\n+\t\tif (rekey_failed)\n+\t\t\tTLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXSW);\n+\t\telse\n+\t\t\tTLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);\n \t}\n }\n \n@@ -369,6 +381,8 @@ static void tls_sk_proto_close(struct sock *sk, long timeout)\n \n \tif (ctx-\u003etx_conf == TLS_SW)\n \t\ttls_sw_cancel_work_tx(ctx);\n+\telse if (ctx-\u003etx_conf == TLS_HW \u0026\u0026 ctx-\u003erekey.sw_ctx)\n+\t\ttls_sw_cancel_work_tx(ctx);\n \n \tlock_sock(sk);\n \tfree_ctx = ctx-\u003etx_conf != TLS_HW \u0026\u0026 ctx-\u003erx_conf != TLS_HW;\n@@ -445,8 +459,17 @@ static int do_tls_getsockopt_conf(struct sock *sk, sockopt_t *opt, int tx)\n \n \t/* get user crypto info */\n \tif (tx) {\n-\t\tcrypto_info = \u0026ctx-\u003ecrypto_send.info;\n-\t\tcctx = \u0026ctx-\u003etx;\n+\t\t/* Select the cipher context via the same accessor the data path\n+\t\t * uses, so getsockopt reports the IV/rec_seq that sendmsg encrypts\n+\t\t * with (the pending rekey's while one is in flight, else the\n+\t\t * active key). crypto_info has no accessor; select it the same way.\n+\t\t * lock_sock is held, so rekey.cipher_ctx cannot change under us.\n+\t\t */\n+\t\tcctx = tls_tx_cipher_ctx(ctx);\n+\t\tif (ctx-\u003erekey.cipher_ctx)\n+\t\t\tcrypto_info = \u0026tls_offload_ctx_tx(ctx)-\u003erekey.crypto_send.info;\n+\t\telse\n+\t\t\tcrypto_info = \u0026ctx-\u003ecrypto_send.info;\n \t} else {\n \t\tcrypto_info = \u0026ctx-\u003ecrypto_recv.info;\n \t\tcctx = \u0026ctx-\u003erx;\n@@ -710,11 +733,18 @@ static int do_tls_setsockopt_conf(struct sock *sk, sockptr_t optval,\n \t}\n \n \tif (tx) {\n-\t\trc = tls_set_device_offload(sk);\n+\t\trc = tls_set_device_offload(sk, update ? crypto_info : NULL);\n \t\tconf = TLS_HW;\n \t\tif (!rc) {\n-\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXDEVICE);\n-\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);\n+\t\t\tif (!update) {\n+\t\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXDEVICE);\n+\t\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);\n+\t\t\t}\n+\t\t} else if (update \u0026\u0026 ctx-\u003etx_conf == TLS_HW) {\n+\t\t\t/* HW rekey failed - return the actual error.\n+\t\t\t * Cannot fall back to SW for an existing HW connection.\n+\t\t\t */\n+\t\t\tgoto err_crypto_info;\n \t\t} else {\n \t\t\trc = tls_set_sw_offload(sk, 1,\n \t\t\t\t\t\tupdate ? crypto_info : NULL);\n@@ -730,11 +760,19 @@ static int do_tls_setsockopt_conf(struct sock *sk, sockptr_t optval,\n \t\t\tconf = TLS_SW;\n \t\t}\n \t} else {\n-\t\trc = tls_set_device_offload_rx(sk, ctx);\n+\t\trc = tls_set_device_offload_rx(sk, ctx,\n+\t\t\t\t\t update ? crypto_info : NULL);\n \t\tconf = TLS_HW;\n \t\tif (!rc) {\n-\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXDEVICE);\n-\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);\n+\t\t\tif (!update) {\n+\t\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXDEVICE);\n+\t\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);\n+\t\t\t}\n+\t\t} else if (update \u0026\u0026 ctx-\u003erx_conf == TLS_HW) {\n+\t\t\t/* HW rekey failed - return the actual error.\n+\t\t\t * Cannot fall back to SW for an existing HW connection.\n+\t\t\t */\n+\t\t\tgoto err_crypto_info;\n \t\t} else {\n \t\t\trc = tls_set_sw_offload(sk, 0,\n \t\t\t\t\t\tupdate ? crypto_info : NULL);\n@@ -773,7 +811,11 @@ static int do_tls_setsockopt_conf(struct sock *sk, sockptr_t optval,\n \treturn 0;\n \n err_crypto_info:\n-\tif (update) {\n+\t/* -EAGAIN is a transient sndbuf-full condition on a non-blocking rekey,\n+\t * not a failed KeyUpdate: the old key stays installed and userspace\n+\t * retries once the socket is writable, so don't count it as an error.\n+\t */\n+\tif (update \u0026\u0026 rc != -EAGAIN) {\n \t\tTLS_INC_STATS(sock_net(sk), tx ? LINUX_MIB_TLSTXREKEYERROR\n \t\t\t\t\t : LINUX_MIB_TLSRXREKEYERROR);\n \t}\n@@ -866,12 +908,29 @@ static int do_tls_setsockopt(struct sock *sk, int optname, sockptr_t optval,\n \n \tswitch (optname) {\n \tcase TLS_TX:\n-\tcase TLS_RX:\n+\tcase TLS_RX: {\n+\t\t/* tls_device_sendmsg() holds tx_lock across the lock_sock drop\n+\t\t * in sk_stream_wait_memory() with a half-built open_record\n+\t\t * exposed. A concurrent HW-offload rekey (tls_device_start_rekey())\n+\t\t * would flush that record and swap the key under the sender,\n+\t\t * corrupting record framing. Serialize TX setsockopt against\n+\t\t * the data path with tx_lock, unconditionally for TLS_TX,\n+\t\t * since during initial setup there is no sender contending it.\n+\t\t */\n+\t\tbool tx = optname == TLS_TX;\n+\n+\t\tif (tx) {\n+\t\t\trc = mutex_lock_interruptible(\u0026tls_get_ctx(sk)-\u003etx_lock);\n+\t\t\tif (rc)\n+\t\t\t\tbreak;\n+\t\t}\n \t\tlock_sock(sk);\n-\t\trc = do_tls_setsockopt_conf(sk, optval, optlen,\n-\t\t\t\t\t optname == TLS_TX);\n+\t\trc = do_tls_setsockopt_conf(sk, optval, optlen, tx);\n \t\trelease_sock(sk);\n+\t\tif (tx)\n+\t\t\tmutex_unlock(\u0026tls_get_ctx(sk)-\u003etx_lock);\n \t\tbreak;\n+\t}\n \tcase TLS_TX_ZEROCOPY_RO:\n \t\tlock_sock(sk);\n \t\trc = do_tls_setsockopt_tx_zc(sk, optval, optlen);\ndiff --git a/net/tls/tls_proc.c b/net/tls/tls_proc.c\nindex 4012c4372d4c0..6255f7b07eb76 100644\n--- a/net/tls/tls_proc.c\n+++ b/net/tls/tls_proc.c\n@@ -27,6 +27,12 @@ static const struct snmp_mib tls_mib_list[] = {\n \tSNMP_MIB_ITEM(\"TlsTxRekeyOk\", LINUX_MIB_TLSTXREKEYOK),\n \tSNMP_MIB_ITEM(\"TlsTxRekeyError\", LINUX_MIB_TLSTXREKEYERROR),\n \tSNMP_MIB_ITEM(\"TlsRxRekeyReceived\", LINUX_MIB_TLSRXREKEYRECEIVED),\n+\tSNMP_MIB_ITEM(\"TlsTxRekeyFallback\", LINUX_MIB_TLSTXREKEYFALLBACK),\n+\tSNMP_MIB_ITEM(\"TlsRxRekeyFallback\", LINUX_MIB_TLSRXREKEYFALLBACK),\n+\tSNMP_MIB_ITEM(\"TlsCurrTxRekey\", LINUX_MIB_TLSCURRTXREKEY),\n+\tSNMP_MIB_ITEM(\"TlsCurrRxRekey\", LINUX_MIB_TLSCURRRXREKEY),\n+\tSNMP_MIB_ITEM(\"TlsTxRekeyAborted\", LINUX_MIB_TLSTXREKEYABORTED),\n+\tSNMP_MIB_ITEM(\"TlsRxRekeyAborted\", LINUX_MIB_TLSRXREKEYABORTED),\n };\n \n static int tls_statistics_seq_show(struct seq_file *seq, void *v)\ndiff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c\nindex d1ad31986cf2c..d546091dd5240 100644\n--- a/net/tls/tls_sw.c\n+++ b/net/tls/tls_sw.c\n@@ -522,7 +522,7 @@ static void tls_encrypt_done(void *data, int err)\n \t\tcomplete(\u0026ctx-\u003easync_wait.completion);\n }\n \n-static int tls_encrypt_async_wait(struct tls_sw_context_tx *ctx)\n+int tls_encrypt_async_wait(struct tls_sw_context_tx *ctx)\n {\n \tif (!atomic_dec_and_test(\u0026ctx-\u003eencrypt_pending))\n \t\tcrypto_wait_req(-EINPROGRESS, \u0026ctx-\u003easync_wait);\n@@ -555,11 +555,11 @@ static int tls_do_encryption(struct sock *sk,\n \t\tbreak;\n \t}\n \n-\tmemcpy(\u0026rec-\u003eiv_data[iv_offset], tls_ctx-\u003etx.iv,\n+\tmemcpy(\u0026rec-\u003eiv_data[iv_offset], tls_tx_cipher_ctx(tls_ctx)-\u003eiv,\n \t prot-\u003eiv_size + prot-\u003esalt_size);\n \n \ttls_xor_iv_with_seq(prot, rec-\u003eiv_data + iv_offset,\n-\t\t\t tls_ctx-\u003etx.rec_seq);\n+\t\t\t tls_tx_cipher_ctx(tls_ctx)-\u003erec_seq);\n \n \tsge-\u003eoffset += prot-\u003eprepend_size;\n \tsge-\u003elength -= prot-\u003eprepend_size;\n@@ -610,7 +610,7 @@ static int tls_do_encryption(struct sock *sk,\n \n \t/* Unhook the record from context if encryption is not failure */\n \tctx-\u003eopen_rec = NULL;\n-\ttls_advance_record_sn(sk, prot, \u0026tls_ctx-\u003etx);\n+\ttls_advance_record_sn(sk, prot, tls_tx_cipher_ctx(tls_ctx));\n \treturn rc;\n }\n \n@@ -676,7 +676,7 @@ static int tls_push_record(struct sock *sk, int flags,\n \tsg_chain(rec-\u003esg_aead_out, 2, \u0026msg_en-\u003esg.data[i]);\n \n \ttls_make_aad(rec-\u003eaad_space, msg_pl-\u003esg.size + prot-\u003etail_size,\n-\t\t tls_ctx-\u003etx.rec_seq, record_type, prot);\n+\t\t tls_tx_cipher_ctx(tls_ctx)-\u003erec_seq, record_type, prot);\n \n \ttls_fill_prepend(tls_ctx,\n \t\t\t page_address(sg_page(\u0026msg_en-\u003esg.data[i])) +\n@@ -712,7 +712,7 @@ static int bpf_exec_tx_verdict(struct sk_msg *msg, struct sock *sk,\n \treturn err;\n }\n \n-static int tls_sw_push_pending_record(struct sock *sk, int flags)\n+int tls_sw_push_pending_record(struct sock *sk, int flags)\n {\n \tstruct tls_context *tls_ctx = tls_get_ctx(sk);\n \tstruct tls_sw_context_tx *ctx = tls_sw_ctx_tx(tls_ctx);\n@@ -763,8 +763,7 @@ static int tls_sw_sendmsg_splice(struct sock *sk, struct msghdr *msg,\n \treturn 0;\n }\n \n-static int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg,\n-\t\t\t\t size_t size)\n+int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)\n {\n \tlong timeo = sock_sndtimeo(sk, msg-\u003emsg_flags \u0026 MSG_DONTWAIT);\n \tstruct tls_context *tls_ctx = tls_get_ctx(sk);\n@@ -1027,8 +1026,13 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)\n \n /*\n * Handle unexpected EOF during splice without SPLICE_F_MORE set.\n+ *\n+ * Inner logic of tls_sw_splice_eof(), factored out so the device\n+ * TX path can reuse it with tls_ctx-\u003etx_lock and the socket lock\n+ * already held. Callers not already holding both locks must use the\n+ * tls_sw_splice_eof() wrapper instead.\n */\n-void tls_sw_splice_eof(struct socket *sock)\n+void tls_sw_splice_eof_locked(struct socket *sock)\n {\n \tstruct sock *sk = sock-\u003esk;\n \tstruct tls_context *tls_ctx = tls_get_ctx(sk);\n@@ -1039,21 +1043,15 @@ void tls_sw_splice_eof(struct socket *sock)\n \tbool retrying = false;\n \tint ret = 0;\n \n-\tif (!ctx-\u003eopen_rec)\n-\t\treturn;\n-\n-\tmutex_lock(\u0026tls_ctx-\u003etx_lock);\n-\tlock_sock(sk);\n-\n retry:\n-\t/* same checks as in tls_sw_push_pending_record() */\n+\t/* same open_rec / empty-record checks as tls_sw_push_pending_record() */\n \trec = ctx-\u003eopen_rec;\n \tif (!rec)\n-\t\tgoto unlock;\n+\t\treturn;\n \n \tmsg_pl = \u0026rec-\u003emsg_plaintext;\n \tif (msg_pl-\u003esg.size == 0)\n-\t\tgoto unlock;\n+\t\treturn;\n \n \t/* Perform transmission. */\n \tret = bpf_exec_tx_verdict(msg_pl, sk, TLS_RECORD_TYPE_DATA,\n@@ -1062,26 +1060,38 @@ void tls_sw_splice_eof(struct socket *sock)\n \tcase 0:\n \tcase -EAGAIN:\n \t\tif (retrying)\n-\t\t\tgoto unlock;\n+\t\t\treturn;\n \t\tretrying = true;\n \t\tgoto retry;\n \tcase -EINPROGRESS:\n \t\tbreak;\n \tdefault:\n-\t\tgoto unlock;\n+\t\treturn;\n \t}\n \n \t/* Wait for pending encryptions to get completed */\n \tif (tls_encrypt_async_wait(ctx))\n-\t\tgoto unlock;\n+\t\treturn;\n \n \t/* Transmit if any encryptions have completed */\n \tif (test_and_clear_bit(BIT_TX_SCHEDULED, \u0026ctx-\u003etx_bitmask)) {\n \t\tcancel_delayed_work(\u0026ctx-\u003etx_work.work);\n \t\ttls_tx_records(sk, 0);\n \t}\n+}\n+\n+void tls_sw_splice_eof(struct socket *sock)\n+{\n+\tstruct sock *sk = sock-\u003esk;\n+\tstruct tls_context *tls_ctx = tls_get_ctx(sk);\n+\tstruct tls_sw_context_tx *ctx = tls_sw_ctx_tx(tls_ctx);\n \n-unlock:\n+\tif (!ctx-\u003eopen_rec)\n+\t\treturn;\n+\n+\tmutex_lock(\u0026tls_ctx-\u003etx_lock);\n+\tlock_sock(sk);\n+\ttls_sw_splice_eof_locked(sock);\n \trelease_sock(sk);\n \tmutex_unlock(\u0026tls_ctx-\u003etx_lock);\n }\n@@ -1551,6 +1561,7 @@ static int tls_check_pending_rekey(struct sock *sk, struct tls_context *ctx,\n \tif (hs_type == TLS_HANDSHAKE_KEYUPDATE) {\n \t\tstruct tls_sw_context_rx *rx_ctx = ctx-\u003epriv_ctx_rx;\n \n+\t\ttls_device_rx_del_key(sk, ctx);\n \t\tWRITE_ONCE(rx_ctx-\u003ekey_update_pending, true);\n \t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYRECEIVED);\n \t}\n@@ -2401,6 +2412,40 @@ static void tx_work_handler(struct work_struct *work)\n \t}\n }\n \n+void tls_sw_ctx_tx_init(struct sock *sk, struct tls_sw_context_tx *sw_ctx)\n+{\n+\tcrypto_init_wait(\u0026sw_ctx-\u003easync_wait);\n+\tatomic_set(\u0026sw_ctx-\u003eencrypt_pending, 1);\n+\tINIT_LIST_HEAD(\u0026sw_ctx-\u003etx_list);\n+\tINIT_DELAYED_WORK(\u0026sw_ctx-\u003etx_work.work, tx_work_handler);\n+\tsw_ctx-\u003etx_work.sk = sk;\n+}\n+\n+int tls_sw_drain_tx(struct sock *sk, struct tls_context *ctx, int flags)\n+{\n+\tstruct tls_sw_context_tx *sw_ctx = tls_sw_ctx_tx(ctx);\n+\tint rc;\n+\n+\tflags = (flags \u0026 MSG_DONTWAIT) | MSG_NOSIGNAL;\n+\n+\tif (sw_ctx-\u003eopen_rec)\n+\t\ttls_sw_push_pending_record(sk, flags);\n+\trc = tls_encrypt_async_wait(sw_ctx);\n+\tif (rc)\n+\t\treturn rc;\n+\trc = tls_tx_records(sk, flags);\n+\tif (rc \u003c 0 || tls_is_partially_sent_record(ctx) ||\n+\t tls_is_pending_open_record(ctx) ||\n+\t !list_empty(\u0026sw_ctx-\u003etx_list))\n+\t\treturn rc \u003c 0 ? rc : -EAGAIN;\n+\n+\ttls_free_open_rec(sk);\n+\n+\tcancel_delayed_work_sync(\u0026sw_ctx-\u003etx_work.work);\n+\tclear_bit(BIT_TX_SCHEDULED, \u0026sw_ctx-\u003etx_bitmask);\n+\treturn 0;\n+}\n+\n static bool tls_is_tx_ready(struct tls_sw_context_tx *ctx)\n {\n \tstruct tls_rec *rec;\n@@ -2452,11 +2497,7 @@ static struct tls_sw_context_tx *init_ctx_tx(struct tls_context *ctx, struct soc\n \t\tsw_ctx_tx = ctx-\u003epriv_ctx_tx;\n \t}\n \n-\tcrypto_init_wait(\u0026sw_ctx_tx-\u003easync_wait);\n-\tatomic_set(\u0026sw_ctx_tx-\u003eencrypt_pending, 1);\n-\tINIT_LIST_HEAD(\u0026sw_ctx_tx-\u003etx_list);\n-\tINIT_DELAYED_WORK(\u0026sw_ctx_tx-\u003etx_work.work, tx_work_handler);\n-\tsw_ctx_tx-\u003etx_work.sk = sk;\n+\ttls_sw_ctx_tx_init(sk, sw_ctx_tx);\n \n \treturn sw_ctx_tx;\n }\n@@ -2522,20 +2563,19 @@ static void tls_finish_key_update(struct sock *sk, struct tls_context *tls_ctx)\n \tctx-\u003esaved_data_ready(sk);\n }\n \n-int tls_set_sw_offload(struct sock *sk, int tx,\n-\t\t struct tls_crypto_info *new_crypto_info)\n+int tls_sw_ctx_init(struct sock *sk, int tx,\n+\t\t struct tls_crypto_info *new_crypto_info)\n {\n \tstruct tls_crypto_info *crypto_info, *src_crypto_info;\n \tstruct tls_sw_context_tx *sw_ctx_tx = NULL;\n \tstruct tls_sw_context_rx *sw_ctx_rx = NULL;\n \tconst struct tls_cipher_desc *cipher_desc;\n-\tchar *iv, *rec_seq, *key, *salt;\n-\tstruct cipher_context *cctx;\n \tstruct tls_prot_info *prot;\n \tstruct crypto_aead **aead;\n \tstruct tls_context *ctx;\n \tstruct crypto_tfm *tfm;\n \tint rc = 0;\n+\tchar *key;\n \n \tctx = tls_get_ctx(sk);\n \tprot = \u0026ctx-\u003eprot_info;\n@@ -2556,12 +2596,10 @@ int tls_set_sw_offload(struct sock *sk, int tx,\n \tif (tx) {\n \t\tsw_ctx_tx = ctx-\u003epriv_ctx_tx;\n \t\tcrypto_info = \u0026ctx-\u003ecrypto_send.info;\n-\t\tcctx = \u0026ctx-\u003etx;\n \t\taead = \u0026sw_ctx_tx-\u003eaead_send;\n \t} else {\n \t\tsw_ctx_rx = ctx-\u003epriv_ctx_rx;\n \t\tcrypto_info = \u0026ctx-\u003ecrypto_recv.info;\n-\t\tcctx = \u0026ctx-\u003erx;\n \t\taead = \u0026sw_ctx_rx-\u003eaead_recv;\n \t}\n \n@@ -2577,11 +2615,12 @@ int tls_set_sw_offload(struct sock *sk, int tx,\n \tif (rc)\n \t\tgoto free_priv;\n \n-\tiv = crypto_info_iv(src_crypto_info, cipher_desc);\n \tkey = crypto_info_key(src_crypto_info, cipher_desc);\n-\tsalt = crypto_info_salt(src_crypto_info, cipher_desc);\n-\trec_seq = crypto_info_rec_seq(src_crypto_info, cipher_desc);\n \n+\t/* A rekey normally reuses the existing tfm; the RX HW rekey hands over a\n+\t * NULL aead (the old one is retained for the drain), so allocate and\n+\t * configure authsize only when a fresh tfm is created here.\n+\t */\n \tif (!*aead) {\n \t\t*aead = crypto_alloc_aead(cipher_desc-\u003ecipher_name, 0, 0);\n \t\tif (IS_ERR(*aead)) {\n@@ -2589,9 +2628,14 @@ int tls_set_sw_offload(struct sock *sk, int tx,\n \t\t\t*aead = NULL;\n \t\t\tgoto free_priv;\n \t\t}\n+\n+\t\trc = crypto_aead_setauthsize(*aead, prot-\u003etag_size);\n+\t\tif (rc)\n+\t\t\tgoto free_aead;\n \t}\n \n-\tctx-\u003epush_pending_record = tls_sw_push_pending_record;\n+\tif (tx)\n+\t\tctx-\u003epush_pending_record = tls_sw_push_pending_record;\n \n \t/* setkey is the last operation that could fail during a\n \t * rekey. if it succeeds, we can start modifying the\n@@ -2605,12 +2649,6 @@ int tls_set_sw_offload(struct sock *sk, int tx,\n \t\t\tgoto free_aead;\n \t}\n \n-\tif (!new_crypto_info) {\n-\t\trc = crypto_aead_setauthsize(*aead, prot-\u003etag_size);\n-\t\tif (rc)\n-\t\t\tgoto free_aead;\n-\t}\n-\n \tif (!tx \u0026\u0026 !new_crypto_info) {\n \t\ttfm = crypto_aead_tfm(sw_ctx_rx-\u003eaead_recv);\n \n@@ -2624,19 +2662,6 @@ int tls_set_sw_offload(struct sock *sk, int tx,\n \t\t\tgoto free_aead;\n \t}\n \n-\tmemcpy(cctx-\u003eiv, salt, cipher_desc-\u003esalt);\n-\tmemcpy(cctx-\u003eiv + cipher_desc-\u003esalt, iv, cipher_desc-\u003eiv);\n-\tmemcpy(cctx-\u003erec_seq, rec_seq, cipher_desc-\u003erec_seq);\n-\n-\tif (new_crypto_info) {\n-\t\tunsafe_memcpy(crypto_info, new_crypto_info,\n-\t\t\t cipher_desc-\u003ecrypto_info,\n-\t\t\t /* size was checked in do_tls_setsockopt_conf */);\n-\t\tmemzero_explicit(new_crypto_info, cipher_desc-\u003ecrypto_info);\n-\t\tif (!tx)\n-\t\t\ttls_finish_key_update(sk, ctx);\n-\t}\n-\n \tgoto out;\n \n free_aead:\n@@ -2655,3 +2680,57 @@ int tls_set_sw_offload(struct sock *sk, int tx,\n out:\n \treturn rc;\n }\n+\n+void tls_sw_ctx_finalize(struct sock *sk, int tx,\n+\t\t\t struct tls_crypto_info *new_crypto_info)\n+{\n+\tstruct tls_crypto_info *crypto_info, *src_crypto_info;\n+\tconst struct tls_cipher_desc *cipher_desc;\n+\tstruct tls_context *ctx = tls_get_ctx(sk);\n+\tstruct cipher_context *cctx;\n+\tchar *iv, *salt, *rec_seq;\n+\n+\tif (tx) {\n+\t\tcrypto_info = \u0026ctx-\u003ecrypto_send.info;\n+\t\tcctx = \u0026ctx-\u003etx;\n+\t} else {\n+\t\tcrypto_info = \u0026ctx-\u003ecrypto_recv.info;\n+\t\tcctx = \u0026ctx-\u003erx;\n+\t}\n+\n+\tsrc_crypto_info = new_crypto_info ?: crypto_info;\n+\n+\t/* Infallible: tls_sw_ctx_init() already validated cipher_type. */\n+\tcipher_desc = get_cipher_desc(src_crypto_info-\u003ecipher_type);\n+\n+\tiv = crypto_info_iv(src_crypto_info, cipher_desc);\n+\tsalt = crypto_info_salt(src_crypto_info, cipher_desc);\n+\trec_seq = crypto_info_rec_seq(src_crypto_info, cipher_desc);\n+\n+\tmemcpy(cctx-\u003eiv, salt, cipher_desc-\u003esalt);\n+\tmemcpy(cctx-\u003eiv + cipher_desc-\u003esalt, iv, cipher_desc-\u003eiv);\n+\tmemcpy(cctx-\u003erec_seq, rec_seq, cipher_desc-\u003erec_seq);\n+\n+\tif (new_crypto_info) {\n+\t\tunsafe_memcpy(crypto_info, new_crypto_info,\n+\t\t\t cipher_desc-\u003ecrypto_info,\n+\t\t\t /* size was checked in do_tls_setsockopt_conf */);\n+\t\tmemzero_explicit(new_crypto_info, cipher_desc-\u003ecrypto_info);\n+\n+\t\tif (!tx)\n+\t\t\ttls_finish_key_update(sk, ctx);\n+\t}\n+}\n+\n+int tls_set_sw_offload(struct sock *sk, int tx,\n+\t\t struct tls_crypto_info *new_crypto_info)\n+{\n+\tint rc;\n+\n+\trc = tls_sw_ctx_init(sk, tx, new_crypto_info);\n+\tif (rc)\n+\t\treturn rc;\n+\n+\ttls_sw_ctx_finalize(sk, tx, new_crypto_info);\n+\treturn 0;\n+}\ndiff --git a/net/tls/trace.h b/net/tls/trace.h\nindex 2d8ce4ff3265b..5b9c1f86d82df 100644\n--- a/net/tls/trace.h\n+++ b/net/tls/trace.h\n@@ -192,6 +192,124 @@ TRACE_EVENT(tls_device_tx_resync_send,\n \t)\n );\n \n+TRACE_EVENT(tls_device_rekey_start,\n+\n+\tTP_PROTO(struct sock *sk, u32 copied_seq, u32 nic_boundary,\n+\t\t bool inflight),\n+\n+\tTP_ARGS(sk, copied_seq, nic_boundary, inflight),\n+\n+\tTP_STRUCT__entry(\n+\t\t__field(\tstruct sock *,\tsk\t\t)\n+\t\t__field(\tu32,\t\tcopied_seq\t)\n+\t\t__field(\tu32,\t\tnic_boundary\t)\n+\t\t__field(\tbool,\t\tinflight\t)\n+\t),\n+\n+\tTP_fast_assign(\n+\t\t__entry-\u003esk = sk;\n+\t\t__entry-\u003ecopied_seq = copied_seq;\n+\t\t__entry-\u003enic_boundary = nic_boundary;\n+\t\t__entry-\u003einflight = inflight;\n+\t),\n+\n+\tTP_printk(\n+\t\t\"sk=%p copied_seq=%u nic_boundary=%u inflight=%d\",\n+\t\t__entry-\u003esk, __entry-\u003ecopied_seq, __entry-\u003enic_boundary,\n+\t\t__entry-\u003einflight\n+\t)\n+);\n+\n+TRACE_EVENT(tls_device_rekey_reencrypt,\n+\n+\tTP_PROTO(struct sock *sk, u32 tcp_seq, u32 nic_boundary),\n+\n+\tTP_ARGS(sk, tcp_seq, nic_boundary),\n+\n+\tTP_STRUCT__entry(\n+\t\t__field(\tstruct sock *,\tsk\t\t)\n+\t\t__field(\tu32,\t\ttcp_seq\t\t)\n+\t\t__field(\tu32,\t\tnic_boundary\t)\n+\t),\n+\n+\tTP_fast_assign(\n+\t\t__entry-\u003esk = sk;\n+\t\t__entry-\u003etcp_seq = tcp_seq;\n+\t\t__entry-\u003enic_boundary = nic_boundary;\n+\t),\n+\n+\tTP_printk(\n+\t\t\"sk=%p tcp_seq=%u nic_boundary=%u\",\n+\t\t__entry-\u003esk, __entry-\u003etcp_seq, __entry-\u003enic_boundary\n+\t)\n+);\n+\n+TRACE_EVENT(tls_device_rekey_done,\n+\n+\tTP_PROTO(struct sock *sk, u32 tcp_seq, u32 nic_boundary),\n+\n+\tTP_ARGS(sk, tcp_seq, nic_boundary),\n+\n+\tTP_STRUCT__entry(\n+\t\t__field(\tstruct sock *,\tsk\t\t)\n+\t\t__field(\tu32,\t\ttcp_seq\t\t)\n+\t\t__field(\tu32,\t\tnic_boundary\t)\n+\t),\n+\n+\tTP_fast_assign(\n+\t\t__entry-\u003esk = sk;\n+\t\t__entry-\u003etcp_seq = tcp_seq;\n+\t\t__entry-\u003enic_boundary = nic_boundary;\n+\t),\n+\n+\tTP_printk(\n+\t\t\"sk=%p tcp_seq=%u nic_boundary=%u\",\n+\t\t__entry-\u003esk, __entry-\u003etcp_seq, __entry-\u003enic_boundary\n+\t)\n+);\n+\n+TRACE_EVENT(tls_device_complete_rekey_fail,\n+\n+\tTP_PROTO(struct sock *sk, int rc),\n+\n+\tTP_ARGS(sk, rc),\n+\n+\tTP_STRUCT__entry(\n+\t\t__field(\tstruct sock *,\tsk\t)\n+\t\t__field(\tint,\t\trc\t)\n+\t),\n+\n+\tTP_fast_assign(\n+\t\t__entry-\u003esk = sk;\n+\t\t__entry-\u003erc = rc;\n+\t),\n+\n+\tTP_printk(\n+\t\t\"sk=%p rc=%d\",\n+\t\t__entry-\u003esk, __entry-\u003erc\n+\t)\n+);\n+\n+TRACE_EVENT(tls_device_complete_rekey_retry,\n+\n+\tTP_PROTO(struct sock *sk),\n+\n+\tTP_ARGS(sk),\n+\n+\tTP_STRUCT__entry(\n+\t\t__field(\tstruct sock *,\tsk\t)\n+\t),\n+\n+\tTP_fast_assign(\n+\t\t__entry-\u003esk = sk;\n+\t),\n+\n+\tTP_printk(\n+\t\t\"sk=%p\",\n+\t\t__entry-\u003esk\n+\t)\n+);\n+\n #endif /* _TLS_TRACE_H_ */\n \n #undef TRACE_INCLUDE_PATH\ndiff --git a/tools/testing/selftests/drivers/net/hw/.gitignore b/tools/testing/selftests/drivers/net/hw/.gitignore\nindex 46540468a7753..911a9bfeaf41e 100644\n--- a/tools/testing/selftests/drivers/net/hw/.gitignore\n+++ b/tools/testing/selftests/drivers/net/hw/.gitignore\n@@ -1,4 +1,5 @@\n # SPDX-License-Identifier: GPL-2.0-only\n iou-zcrx\n ncdevmem\n+tls_hw_offload\n toeplitz\ndiff --git a/tools/testing/selftests/drivers/net/hw/Makefile b/tools/testing/selftests/drivers/net/hw/Makefile\nindex 8aebdc6feb177..b3831c2d09ea5 100644\n--- a/tools/testing/selftests/drivers/net/hw/Makefile\n+++ b/tools/testing/selftests/drivers/net/hw/Makefile\n@@ -46,6 +46,7 @@ TEST_PROGS = \\\n \trss_drv.py \\\n \trss_flow_label.py \\\n \trss_input_xfrm.py \\\n+\ttls_hw_offload.py \\\n \ttoeplitz.py \\\n \ttso.py \\\n \tuserns_devmem.py \\\n@@ -80,6 +81,7 @@ YNL_GEN_FILES := \\\n # end of YNL_GEN_FILES\n TEST_GEN_FILES += $(YNL_GEN_FILES)\n TEST_GEN_FILES += $(patsubst %.c,%.o,$(wildcard *.bpf.c))\n+TEST_GEN_FILES += tls_hw_offload\n \n include ../../../lib.mk\n \ndiff --git a/tools/testing/selftests/drivers/net/hw/config b/tools/testing/selftests/drivers/net/hw/config\nindex d89a9ba176558..169e608516bd5 100644\n--- a/tools/testing/selftests/drivers/net/hw/config\n+++ b/tools/testing/selftests/drivers/net/hw/config\n@@ -22,6 +22,8 @@ CONFIG_NET_IPIP=y\n CONFIG_NETKIT=y\n CONFIG_NET_SCH_INGRESS=y\n CONFIG_SYNC_FILE=y\n+CONFIG_TLS=y\n+CONFIG_TLS_DEVICE=y\n CONFIG_UDMABUF=y\n CONFIG_USER_NS=y\n CONFIG_VXLAN=y\ndiff --git a/tools/testing/selftests/drivers/net/hw/tls_hw_offload.c b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.c\nnew file mode 100644\nindex 0000000000000..303c6752ace27\n--- /dev/null\n+++ b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.c\n@@ -0,0 +1,1132 @@\n+// SPDX-License-Identifier: GPL-2.0\n+/*\n+ * TLS Hardware Offload Two-Node Test\n+ *\n+ * Tests kTLS hardware offload between two physical nodes using\n+ * hardcoded keys. Supports TLS 1.2/1.3, AES-GCM-128/256, and rekey.\n+ */\n+\n+#include \u003cstdio.h\u003e\n+#include \u003cstdlib.h\u003e\n+#include \u003cstring.h\u003e\n+#include \u003cunistd.h\u003e\n+#include \u003cerrno.h\u003e\n+#include \u003climits.h\u003e\n+#include \u003ctime.h\u003e\n+#include \u003csys/time.h\u003e\n+#include \u003csignal.h\u003e\n+#include \u003csys/types.h\u003e\n+#include \u003csys/socket.h\u003e\n+#include \u003cnetinet/in.h\u003e\n+#include \u003cnetinet/tcp.h\u003e\n+#include \u003cnetdb.h\u003e\n+#include \u003clinux/tls.h\u003e\n+\n+#define TLS_RECORD_TYPE_HANDSHAKE\t\t22\n+#define TLS_HANDSHAKE_KEY_UPDATE\t\t0x18\n+\n+/* Large enough for a TLS 1.3 KeyUpdate handshake record's plaintext. */\n+#define MIN_BUF_SIZE 16\n+\n+/* Initial key material */\n+static struct tls12_crypto_info_aes_gcm_128 tls_info_key0_128 = {\n+\t.info = {\n+\t\t.version = TLS_1_3_VERSION,\n+\t\t.cipher_type = TLS_CIPHER_AES_GCM_128,\n+\t},\n+\t.iv = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08 },\n+\t.key = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08,\n+\t\t 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10 },\n+\t.salt = { 0x01, 0x02, 0x03, 0x04 },\n+\t.rec_seq = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },\n+};\n+\n+static struct tls12_crypto_info_aes_gcm_256 tls_info_key0_256 = {\n+\t.info = {\n+\t\t.version = TLS_1_3_VERSION,\n+\t\t.cipher_type = TLS_CIPHER_AES_GCM_256,\n+\t},\n+\t.iv = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08 },\n+\t.key = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08,\n+\t\t 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10,\n+\t\t 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18,\n+\t\t 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, 0x20 },\n+\t.salt = { 0x01, 0x02, 0x03, 0x04 },\n+\t.rec_seq = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },\n+};\n+\n+static int num_rekeys;\n+static int num_iterations = 100;\n+static int cipher_type = TLS_CIPHER_AES_GCM_128;\n+static int tls_version = TLS_1_3_VERSION;\n+static int server_port = 4433;\n+static char *server_ip;\n+/* Address family to force: AF_UNSPEC (any), AF_INET (-4), AF_INET6 (-6). */\n+static int force_family = AF_UNSPEC;\n+\n+static int send_size = 16384;\n+static int random_size_max;\n+/* Burst mode: sender keeps pushing records without reading from the peer;\n+ * receiver drains without echoing back. Only the client initiates rekey.\n+ */\n+static int burst_mode;\n+static int zc_rx;\n+\n+/* XOR each byte with the generation so both endpoints derive the\n+ * same per-generation key without a real KDF. Generation 0 leaves\n+ * the base key unchanged.\n+ */\n+static void derive_key_fields(unsigned char *key, int key_size,\n+\t\t\t unsigned char *iv, int iv_size,\n+\t\t\t unsigned char *salt, int salt_size,\n+\t\t\t unsigned char *rec_seq, int rec_seq_size,\n+\t\t\t int generation)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i \u003c key_size; i++)\n+\t\tkey[i] ^= generation;\n+\tfor (i = 0; i \u003c iv_size; i++)\n+\t\tiv[i] ^= generation;\n+\tfor (i = 0; i \u003c salt_size; i++)\n+\t\tsalt[i] ^= generation;\n+\tmemset(rec_seq, 0, rec_seq_size);\n+}\n+\n+static void derive_key_128(struct tls12_crypto_info_aes_gcm_128 *key,\n+\t\t\t int generation)\n+{\n+\tmemcpy(key, \u0026tls_info_key0_128, sizeof(*key));\n+\tkey-\u003einfo.version = tls_version;\n+\tderive_key_fields(key-\u003ekey, TLS_CIPHER_AES_GCM_128_KEY_SIZE,\n+\t\t\t key-\u003eiv, TLS_CIPHER_AES_GCM_128_IV_SIZE,\n+\t\t\t key-\u003esalt, TLS_CIPHER_AES_GCM_128_SALT_SIZE,\n+\t\t\t key-\u003erec_seq, TLS_CIPHER_AES_GCM_128_REC_SEQ_SIZE,\n+\t\t\t generation);\n+}\n+\n+static void derive_key_256(struct tls12_crypto_info_aes_gcm_256 *key,\n+\t\t\t int generation)\n+{\n+\tmemcpy(key, \u0026tls_info_key0_256, sizeof(*key));\n+\tkey-\u003einfo.version = tls_version;\n+\tderive_key_fields(key-\u003ekey, TLS_CIPHER_AES_GCM_256_KEY_SIZE,\n+\t\t\t key-\u003eiv, TLS_CIPHER_AES_GCM_256_IV_SIZE,\n+\t\t\t key-\u003esalt, TLS_CIPHER_AES_GCM_256_SALT_SIZE,\n+\t\t\t key-\u003erec_seq, TLS_CIPHER_AES_GCM_256_REC_SEQ_SIZE,\n+\t\t\t generation);\n+}\n+\n+static const char *cipher_name(int cipher)\n+{\n+\tswitch (cipher) {\n+\tcase TLS_CIPHER_AES_GCM_128: return \"AES-GCM-128\";\n+\tcase TLS_CIPHER_AES_GCM_256: return \"AES-GCM-256\";\n+\tdefault: return \"unknown\";\n+\t}\n+}\n+\n+static const char *version_name(int version)\n+{\n+\tswitch (version) {\n+\tcase TLS_1_2_VERSION: return \"TLS 1.2\";\n+\tcase TLS_1_3_VERSION: return \"TLS 1.3\";\n+\tdefault: return \"unknown\";\n+\t}\n+}\n+\n+static int setup_tls_ulp(int fd)\n+{\n+\tint ret;\n+\n+\tret = setsockopt(fd, IPPROTO_TCP, TCP_ULP, \"tls\", sizeof(\"tls\"));\n+\tif (ret \u003c 0) {\n+\t\tprintf(\"SETUP ERROR: TCP_ULP failed: %s\\n\", strerror(errno));\n+\t\treturn -1;\n+\t}\n+\treturn 0;\n+}\n+\n+/* Echo (non-burst) mode drives both directions from a single thread: the\n+ * client pushes a whole payload with one blocking send() and only reads the\n+ * echo afterwards, while the server blocks in send() mid-echo. If a payload\n+ * exceeds the peer's receive window the two sides deadlock - client stuck in\n+ * send(), server stuck echoing, neither draining the other. Size the socket\n+ * buffers so a full payload always fits in the peer's window (the forward\n+ * send() then completes without needing the peer to read concurrently); the\n+ * send/recv timeouts armed by set_io_timeouts() turn any residual stall into a\n+ * loud EAGAIN instead of a hang.\n+ */\n+static void configure_echo_socket(int fd, int payload)\n+{\n+\tint want = payload;\n+\n+\tif (want \u003c MIN_BUF_SIZE)\n+\t\twant = MIN_BUF_SIZE;\n+\n+\t/* SO_*BUFFORCE bypasses the rmem_max/wmem_max sysctl caps (needs\n+\t * CAP_NET_ADMIN); fall back to the best-effort, cap-limited option\n+\t * when unprivileged - the timeouts below still turn any resulting\n+\t * stall into a loud failure rather than a hang.\n+\t */\n+\tif (setsockopt(fd, SOL_SOCKET, SO_RCVBUFFORCE, \u0026want, sizeof(want)) \u003c 0)\n+\t\tsetsockopt(fd, SOL_SOCKET, SO_RCVBUF, \u0026want, sizeof(want));\n+\tif (setsockopt(fd, SOL_SOCKET, SO_SNDBUFFORCE, \u0026want, sizeof(want)) \u003c 0)\n+\t\tsetsockopt(fd, SOL_SOCKET, SO_SNDBUF, \u0026want, sizeof(want));\n+}\n+\n+/* Arm send/recv timeouts so any unexpected stall fails loudly with EAGAIN\n+ * instead of hanging until the harness SIGKILLs us. Wanted in both echo and\n+ * burst modes - burst mode has no other stall guard.\n+ */\n+static void set_io_timeouts(int fd)\n+{\n+\tstruct timeval tv = { .tv_sec = 8, .tv_usec = 0 };\n+\n+\tsetsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, \u0026tv, sizeof(tv));\n+\tsetsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, \u0026tv, sizeof(tv));\n+}\n+\n+/* Send the whole buffer, looping over short counts. A blocking SOCK_STREAM\n+ * send() may return fewer bytes than requested (e.g. when SO_SNDTIMEO fires\n+ * after partial progress) without setting errno, so a short count is not an\n+ * error - only a negative return is. Looping also sends each iteration as one\n+ * uninterrupted run of bytes, which the peer's userspace reassembly in burst\n+ * mode counts on to keep iterations aligned.\n+ */\n+static int send_all(int fd, const char *buf, ssize_t len)\n+{\n+\tssize_t sent = 0;\n+\tssize_t ret;\n+\n+\twhile (sent \u003c len) {\n+\t\tret = send(fd, buf + sent, len - sent, 0);\n+\t\tif (ret \u003c 0) {\n+\t\t\tprintf(\"FAIL: send failed: %s\\n\", strerror(errno));\n+\t\t\treturn -1;\n+\t\t}\n+\t\tsent += ret;\n+\t}\n+\treturn 0;\n+}\n+\n+static int set_zc_rx(int fd)\n+{\n+\tint val = 1;\n+\n+\tif (setsockopt(fd, SOL_TLS, TLS_RX_EXPECT_NO_PAD, \u0026val,\n+\t\t sizeof(val)) \u003c 0) {\n+\t\tprintf(\"SETUP ERROR: TLS_RX_EXPECT_NO_PAD failed: %s\\n\",\n+\t\t strerror(errno));\n+\t\treturn -1;\n+\t}\n+\treturn 0;\n+}\n+\n+/* Send a TLS 1.3 KeyUpdate handshake record. The kernel only\n+ * inspects the HandshakeType byte to detect KeyUpdate, so don't\n+ * bother with the 3-byte length or request_update fields.\n+ */\n+static int send_tls_key_update(int fd)\n+{\n+\tchar cmsg_buf[CMSG_SPACE(sizeof(unsigned char))];\n+\tunsigned char key_update_msg = TLS_HANDSHAKE_KEY_UPDATE;\n+\tstruct msghdr msg = {0};\n+\tstruct cmsghdr *cmsg;\n+\tstruct iovec iov;\n+\n+\tiov.iov_base = \u0026key_update_msg;\n+\tiov.iov_len = sizeof(key_update_msg);\n+\n+\tmsg.msg_iov = \u0026iov;\n+\tmsg.msg_iovlen = 1;\n+\tmsg.msg_control = cmsg_buf;\n+\tmsg.msg_controllen = sizeof(cmsg_buf);\n+\n+\tcmsg = CMSG_FIRSTHDR(\u0026msg);\n+\tcmsg-\u003ecmsg_level = SOL_TLS;\n+\tcmsg-\u003ecmsg_type = TLS_SET_RECORD_TYPE;\n+\tcmsg-\u003ecmsg_len = CMSG_LEN(sizeof(unsigned char));\n+\t*CMSG_DATA(cmsg) = TLS_RECORD_TYPE_HANDSHAKE;\n+\tmsg.msg_controllen = cmsg-\u003ecmsg_len;\n+\n+\tif (sendmsg(fd, \u0026msg, 0) \u003c 0) {\n+\t\tprintf(\"sendmsg KeyUpdate failed: %s\\n\", strerror(errno));\n+\t\treturn -1;\n+\t}\n+\n+\tprintf(\"Sent TLS KeyUpdate handshake message\\n\");\n+\treturn 0;\n+}\n+\n+static int recv_tls_message(int fd, char *buf, size_t buflen, int *record_type,\n+\t\t\t int flags)\n+{\n+\tchar cmsg_buf[CMSG_SPACE(sizeof(unsigned char))];\n+\tstruct msghdr msg = {0};\n+\tstruct cmsghdr *cmsg;\n+\tstruct iovec iov;\n+\tint ret;\n+\n+\tiov.iov_base = buf;\n+\tiov.iov_len = buflen;\n+\n+\tmsg.msg_iov = \u0026iov;\n+\tmsg.msg_iovlen = 1;\n+\tmsg.msg_control = cmsg_buf;\n+\tmsg.msg_controllen = sizeof(cmsg_buf);\n+\n+\tret = recvmsg(fd, \u0026msg, flags);\n+\tif (ret \u003c= 0)\n+\t\treturn ret;\n+\n+\tcmsg = CMSG_FIRSTHDR(\u0026msg);\n+\tif (cmsg \u0026\u0026 cmsg-\u003ecmsg_level == SOL_TLS \u0026\u0026\n+\t cmsg-\u003ecmsg_type == TLS_GET_RECORD_TYPE)\n+\t\t*record_type = *((unsigned char *)CMSG_DATA(cmsg));\n+\n+\treturn ret;\n+}\n+\n+/* Confirm a handshake record starting with HandshakeType KeyUpdate. */\n+static int check_keyupdate(const char *buf, int len, int record_type)\n+{\n+\tif (record_type != TLS_RECORD_TYPE_HANDSHAKE) {\n+\t\tprintf(\"Expected handshake record (0x%02x), got 0x%02x\\n\",\n+\t\t TLS_RECORD_TYPE_HANDSHAKE, record_type);\n+\t\treturn -1;\n+\t}\n+\tif (len \u003c 1 || (unsigned char)buf[0] != TLS_HANDSHAKE_KEY_UPDATE) {\n+\t\tprintf(\"Expected KeyUpdate (0x%02x), got 0x%02x\\n\",\n+\t\t TLS_HANDSHAKE_KEY_UPDATE,\n+\t\t len ? (unsigned char)buf[0] : 0);\n+\t\treturn -1;\n+\t}\n+\tprintf(\"Received TLS KeyUpdate\\n\");\n+\treturn 0;\n+}\n+\n+static int recv_tls_keyupdate(int fd)\n+{\n+\tchar buf[MIN_BUF_SIZE];\n+\tint record_type = 0;\n+\tint ret;\n+\n+\tret = recv_tls_message(fd, buf, sizeof(buf), \u0026record_type, 0);\n+\tif (ret \u003c 0) {\n+\t\tprintf(\"recv_tls_message failed: %s\\n\", strerror(errno));\n+\t\treturn -1;\n+\t}\n+\n+\treturn check_keyupdate(buf, ret, record_type);\n+}\n+\n+static int check_ekeyexpired(int fd)\n+{\n+\tchar buf[MIN_BUF_SIZE];\n+\tint ret;\n+\n+\tret = recv(fd, buf, sizeof(buf), MSG_DONTWAIT);\n+\tif (ret == -1 \u0026\u0026 errno == EKEYEXPIRED) {\n+\t\tprintf(\"recv() returned EKEYEXPIRED as expected\\n\");\n+\t\treturn 0;\n+\t}\n+\tif (ret \u003e 0) {\n+\t\tprintf(\"FAIL: recv() returned %d bytes, expected EKEYEXPIRED\\n\",\n+\t\t ret);\n+\t\treturn -1;\n+\t}\n+\tif (ret == 0) {\n+\t\tprintf(\"FAIL: connection closed during rekey\\n\");\n+\t\treturn -1;\n+\t}\n+\tprintf(\"FAIL: recv() returned unexpected error: %s\\n\",\n+\t strerror(errno));\n+\treturn -1;\n+}\n+\n+static int do_tls_rekey(int fd, int direction, int generation, int cipher)\n+{\n+\tconst char *dir = direction == TLS_TX ? \"TX\" : \"RX\";\n+\tint ret;\n+\n+\tprintf(\"%s TLS_%s %s gen %d...\\n\",\n+\t generation ? \"Rekeying\" : \"Installing\",\n+\t dir, cipher_name(cipher), generation);\n+\n+\tif (cipher == TLS_CIPHER_AES_GCM_256) {\n+\t\tstruct tls12_crypto_info_aes_gcm_256 key;\n+\n+\t\tderive_key_256(\u0026key, generation);\n+\t\tret = setsockopt(fd, SOL_TLS, direction, \u0026key, sizeof(key));\n+\t} else {\n+\t\tstruct tls12_crypto_info_aes_gcm_128 key;\n+\n+\t\tderive_key_128(\u0026key, generation);\n+\t\tret = setsockopt(fd, SOL_TLS, direction, \u0026key, sizeof(key));\n+\t}\n+\n+\tif (ret \u003c 0) {\n+\t\tprintf(\"%sTLS_%s %s gen %d failed: %s\\n\",\n+\t\t generation ? \"\" : \"SETUP ERROR: \", dir,\n+\t\t cipher_name(cipher), generation, strerror(errno));\n+\t\treturn -1;\n+\t}\n+\tprintf(\"TLS_%s %s gen %d installed\\n\",\n+\t dir, cipher_name(cipher), generation);\n+\treturn 0;\n+}\n+\n+/* Open a TCP connection to server_ip:server_port, switch to the TLS\n+ * ULP, and install initial generation-0 TX/RX keys. Works over IPv4 or\n+ * IPv6: getaddrinfo() resolves server_ip (honouring any -4/-6 forced\n+ * family and %zone scope IDs in link-local addresses). Returns the fd on\n+ * success, -1 on error (with the fd already closed).\n+ */\n+static int client_connect_tls(void)\n+{\n+\tstruct addrinfo hints = {0}, *res, *rp;\n+\tchar port_str[16];\n+\tint csk = -1;\n+\tint ret;\n+\n+\thints.ai_family = force_family;\n+\thints.ai_socktype = SOCK_STREAM;\n+\thints.ai_protocol = IPPROTO_TCP;\n+\tsnprintf(port_str, sizeof(port_str), \"%d\", server_port);\n+\n+\tret = getaddrinfo(server_ip, port_str, \u0026hints, \u0026res);\n+\tif (ret) {\n+\t\tprintf(\"SETUP ERROR: getaddrinfo(%s): %s\\n\", server_ip,\n+\t\t gai_strerror(ret));\n+\t\treturn -1;\n+\t}\n+\n+\tprintf(\"Connecting to %s:%d...\\n\", server_ip, server_port);\n+\tfor (rp = res; rp; rp = rp-\u003eai_next) {\n+\t\tcsk = socket(rp-\u003eai_family, rp-\u003eai_socktype, rp-\u003eai_protocol);\n+\t\tif (csk \u003c 0)\n+\t\t\tcontinue;\n+\t\tif (connect(csk, rp-\u003eai_addr, rp-\u003eai_addrlen) == 0)\n+\t\t\tbreak;\n+\t\tclose(csk);\n+\t\tcsk = -1;\n+\t}\n+\tfreeaddrinfo(res);\n+\n+\tif (csk \u003c 0) {\n+\t\tprintf(\"SETUP ERROR: connect to %s:%d failed: %s\\n\",\n+\t\t server_ip, server_port, strerror(errno));\n+\t\treturn -1;\n+\t}\n+\tprintf(\"Connected!\\n\");\n+\n+\tif (setup_tls_ulp(csk) \u003c 0)\n+\t\tgoto err;\n+\n+\tif (do_tls_rekey(csk, TLS_TX, 0, cipher_type) \u003c 0 ||\n+\t do_tls_rekey(csk, TLS_RX, 0, cipher_type) \u003c 0)\n+\t\tgoto err;\n+\n+\tset_io_timeouts(csk);\n+\tif (!burst_mode)\n+\t\tconfigure_echo_socket(csk, random_size_max \u003e 0 ?\n+\t\t\t\t\t random_size_max : send_size);\n+\n+\treturn csk;\n+err:\n+\tclose(csk);\n+\treturn -1;\n+}\n+\n+/* Drain `len` echoed bytes from the server and verify they match the\n+ * payload we just sent.\n+ */\n+static int client_recv_echo(int fd, const char *sent, char *echo_buf,\n+\t\t\t ssize_t len)\n+{\n+\tssize_t total = 0;\n+\tssize_t n;\n+\n+\twhile (total \u003c len) {\n+\t\tn = recv(fd, echo_buf + total, len - total, 0);\n+\t\tif (n \u003c 0) {\n+\t\t\tprintf(\"FAIL: Echo recv failed: %s\\n\", strerror(errno));\n+\t\t\treturn -1;\n+\t\t}\n+\t\tif (n == 0) {\n+\t\t\tprintf(\"FAIL: Connection closed during echo\\n\");\n+\t\t\treturn -1;\n+\t\t}\n+\t\ttotal += n;\n+\t}\n+\n+\tif (memcmp(sent, echo_buf, len) != 0) {\n+\t\tprintf(\"FAIL: Echo data mismatch!\\n\");\n+\t\treturn -1;\n+\t}\n+\tprintf(\"Received echo %zd bytes (ok)\\n\", total);\n+\treturn 0;\n+}\n+\n+/* Client side of a rekey: send KeyUpdate and rotate TX. In echo mode\n+ * also wait for the peer's KeyUpdate and rotate RX.\n+ */\n+static int client_rekey(int fd, int generation)\n+{\n+\tif (send_tls_key_update(fd) \u003c 0) {\n+\t\tprintf(\"FAIL: send KeyUpdate\\n\");\n+\t\treturn -1;\n+\t}\n+\n+\tif (do_tls_rekey(fd, TLS_TX, generation, cipher_type) \u003c 0)\n+\t\treturn -1;\n+\n+\tif (burst_mode)\n+\t\treturn 0;\n+\n+\tif (recv_tls_keyupdate(fd) \u003c 0) {\n+\t\tprintf(\"FAIL: recv KeyUpdate from server\\n\");\n+\t\treturn -1;\n+\t}\n+\n+\tif (check_ekeyexpired(fd) \u003c 0)\n+\t\treturn -1;\n+\n+\treturn do_tls_rekey(fd, TLS_RX, generation, cipher_type);\n+}\n+\n+static int do_client(void)\n+{\n+\tchar *buf = NULL, *echo_buf = NULL;\n+\tint max_size, rekey_interval;\n+\tint csk = -1, i;\n+\tint test_result = -1;\n+\tint current_gen = 0;\n+\tint next_rekey_at;\n+\tssize_t n;\n+\n+\tmax_size = random_size_max \u003e 0 ? random_size_max : send_size;\n+\tif (max_size \u003c MIN_BUF_SIZE)\n+\t\tmax_size = MIN_BUF_SIZE;\n+\tbuf = malloc(max_size);\n+\tif (!burst_mode)\n+\t\techo_buf = malloc(max_size);\n+\tif (!buf || (!burst_mode \u0026\u0026 !echo_buf)) {\n+\t\tprintf(\"SETUP ERROR: failed to allocate buffers\\n\");\n+\t\tgoto out;\n+\t}\n+\n+\tcsk = client_connect_tls();\n+\tif (csk \u003c 0)\n+\t\tgoto out;\n+\n+\tif (num_rekeys)\n+\t\tprintf(\"TLS %s setup complete. Will perform %d rekey(s).\\n\",\n+\t\t cipher_name(cipher_type), num_rekeys);\n+\telse\n+\t\tprintf(\"TLS setup complete.\\n\");\n+\n+\tif (random_size_max \u003e 0)\n+\t\tprintf(\"Sending %d messages of random size (1..%d bytes)...\\n\",\n+\t\t num_iterations, random_size_max);\n+\telse\n+\t\tprintf(\"Sending %d messages of %d bytes...\\n\",\n+\t\t num_iterations, send_size);\n+\n+\trekey_interval = num_iterations / (num_rekeys + 1);\n+\tnext_rekey_at = rekey_interval;\n+\n+\tfor (i = 1; i \u003c= num_iterations; i++) {\n+\t\tint this_size;\n+\n+\t\tif (random_size_max \u003e 0)\n+\t\t\tthis_size = (rand() % random_size_max) + 1;\n+\t\telse\n+\t\t\tthis_size = send_size;\n+\n+\t\t/* In burst mode, use a per-iteration fill pattern so the\n+\t\t * receiver can detect any plaintext corruption without a\n+\t\t * round-trip echo.\n+\t\t */\n+\t\tif (burst_mode) {\n+\t\t\tmemset(buf, i \u0026 0xFF, this_size);\n+\t\t} else {\n+\t\t\tint j;\n+\n+\t\t\tfor (j = 0; j \u003c this_size; j++)\n+\t\t\t\tbuf[j] = rand() \u0026 0xFF;\n+\t\t}\n+\n+\t\tif (send_all(csk, buf, this_size) \u003c 0)\n+\t\t\tgoto out;\n+\t\tn = this_size;\n+\n+\t\tif (!burst_mode) {\n+\t\t\tprintf(\"Sent %zd bytes (iteration %d)\\n\", n, i);\n+\t\t\tif (client_recv_echo(csk, buf, echo_buf, n) \u003c 0)\n+\t\t\t\tgoto out;\n+\t\t}\n+\n+\t\t/* Rekey at intervals. In echo mode this is a full bidirectional\n+\t\t * exchange; in burst mode the client only rotates its TX key\n+\t\t * and sends KeyUpdate - the peer is expected to follow.\n+\t\t */\n+\t\tif (num_rekeys \u0026\u0026 current_gen \u003c num_rekeys \u0026\u0026\n+\t\t i == next_rekey_at) {\n+\t\t\tcurrent_gen++;\n+\t\t\tprintf(\"\\n=== Client Rekey gen %d ===\\n\", current_gen);\n+\n+\t\t\tif (client_rekey(csk, current_gen) \u003c 0)\n+\t\t\t\tgoto out;\n+\n+\t\t\tnext_rekey_at += rekey_interval;\n+\t\t\tprintf(\"=== Client Rekey gen %d Complete ===\\n\\n\",\n+\t\t\t current_gen);\n+\t\t}\n+\t}\n+\n+\ttest_result = 0;\n+out:\n+\tif (num_rekeys)\n+\t\tprintf(\"Rekeys completed: %d/%d\\n\", current_gen, num_rekeys);\n+\tif (csk \u003e= 0)\n+\t\tclose(csk);\n+\tfree(buf);\n+\tfree(echo_buf);\n+\treturn test_result;\n+}\n+\n+/* Bind/listen on server_port, accept one client, switch to the TLS ULP\n+ * and install initial generation-0 keys (plus zc_rx if requested).\n+ * Returns the connected fd on success and writes the listener fd to\n+ * *lsk_out so the caller can close it. Returns -1 on error, with all\n+ * intermediate fds already closed and *lsk_out left at -1.\n+ */\n+static int server_accept_tls(int *lsk_out)\n+{\n+\tstruct addrinfo hints = {0}, *res, *rp;\n+\tint lsk = -1, csk, one = 1;\n+\tchar port_str[16];\n+\tint ret;\n+\n+\t*lsk_out = -1;\n+\n+\t/* AI_PASSIVE gives a wildcard bind address for the chosen family\n+\t * (0.0.0.0 / ::). The family is forced by -4/-6; when unspecified,\n+\t * bind the first entry that works.\n+\t */\n+\thints.ai_family = force_family;\n+\thints.ai_socktype = SOCK_STREAM;\n+\thints.ai_protocol = IPPROTO_TCP;\n+\thints.ai_flags = AI_PASSIVE;\n+\tsnprintf(port_str, sizeof(port_str), \"%d\", server_port);\n+\n+\tret = getaddrinfo(NULL, port_str, \u0026hints, \u0026res);\n+\tif (ret) {\n+\t\tprintf(\"SETUP ERROR: getaddrinfo(port %d): %s\\n\", server_port,\n+\t\t gai_strerror(ret));\n+\t\treturn -1;\n+\t}\n+\n+\tfor (rp = res; rp; rp = rp-\u003eai_next) {\n+\t\tlsk = socket(rp-\u003eai_family, rp-\u003eai_socktype, rp-\u003eai_protocol);\n+\t\tif (lsk \u003c 0)\n+\t\t\tcontinue;\n+\t\tsetsockopt(lsk, SOL_SOCKET, SO_REUSEADDR, \u0026one, sizeof(one));\n+\t\tif (bind(lsk, rp-\u003eai_addr, rp-\u003eai_addrlen) == 0)\n+\t\t\tbreak;\n+\t\tclose(lsk);\n+\t\tlsk = -1;\n+\t}\n+\tfreeaddrinfo(res);\n+\n+\tif (lsk \u003c 0) {\n+\t\tprintf(\"SETUP ERROR: failed to bind port %d: %s\\n\",\n+\t\t server_port, strerror(errno));\n+\t\treturn -1;\n+\t}\n+\n+\tif (listen(lsk, 1) \u003c 0) {\n+\t\tprintf(\"SETUP ERROR: listen failed: %s\\n\", strerror(errno));\n+\t\tclose(lsk);\n+\t\treturn -1;\n+\t}\n+\n+\tprintf(\"Server listening on port %d\\n\", server_port);\n+\tprintf(\"Waiting for client connection...\\n\");\n+\n+\t/* Bound accept() so a client that never connects (a deploy or connect\n+\t * failure on the peer) does not block the server forever and leak the\n+\t * process past the harness timeout. accept() honours SO_RCVTIMEO on the\n+\t * listening socket; the client connects right after wait_port_listen(),\n+\t * so 30s is generous.\n+\t */\n+\t{\n+\t\tstruct timeval tv = { .tv_sec = 30, .tv_usec = 0 };\n+\n+\t\tsetsockopt(lsk, SOL_SOCKET, SO_RCVTIMEO, \u0026tv, sizeof(tv));\n+\t}\n+\n+\tcsk = accept(lsk, (struct sockaddr *)NULL, (socklen_t *)NULL);\n+\tif (csk \u003c 0) {\n+\t\tif (errno == EAGAIN || errno == EWOULDBLOCK)\n+\t\t\tprintf(\"SETUP ERROR: accept timed out; client never connected\\n\");\n+\t\telse\n+\t\t\tprintf(\"SETUP ERROR: accept failed: %s\\n\", strerror(errno));\n+\t\tclose(lsk);\n+\t\treturn -1;\n+\t}\n+\tprintf(\"Client connected!\\n\");\n+\n+\tif (setup_tls_ulp(csk) \u003c 0)\n+\t\tgoto err;\n+\n+\tif (do_tls_rekey(csk, TLS_TX, 0, cipher_type) \u003c 0 ||\n+\t do_tls_rekey(csk, TLS_RX, 0, cipher_type) \u003c 0)\n+\t\tgoto err;\n+\n+\tif (zc_rx \u0026\u0026 set_zc_rx(csk) \u003c 0)\n+\t\tgoto err;\n+\n+\tset_io_timeouts(csk);\n+\tif (!burst_mode)\n+\t\tconfigure_echo_socket(csk, random_size_max \u003e 0 ?\n+\t\t\t\t\t random_size_max : send_size);\n+\n+\t*lsk_out = lsk;\n+\treturn csk;\n+err:\n+\tclose(csk);\n+\tclose(lsk);\n+\treturn -1;\n+}\n+\n+/* Server side of a rekey: confirm recv() reports EKEYEXPIRED, then rotate RX.\n+ * In echo mode also send a KeyUpdate back and rotate TX.\n+ */\n+static int server_rekey(int fd, int generation)\n+{\n+\tif (check_ekeyexpired(fd) \u003c 0)\n+\t\treturn -1;\n+\n+\tif (do_tls_rekey(fd, TLS_RX, generation, cipher_type) \u003c 0)\n+\t\treturn -1;\n+\n+\tif (burst_mode)\n+\t\treturn 0;\n+\n+\tif (send_tls_key_update(fd) \u003c 0) {\n+\t\tprintf(\"FAIL: send KeyUpdate\\n\");\n+\t\treturn -1;\n+\t}\n+\n+\treturn do_tls_rekey(fd, TLS_TX, generation, cipher_type);\n+}\n+\n+/* Burst mode: verify one reassembled iteration of send_size plaintext bytes,\n+ * each filled with (send_iter \u0026 0xff). Catches decrypt-succeeded-but-\n+ * plaintext-corrupt bugs that AEAD counters alone would miss.\n+ */\n+static int server_verify_burst(const char *buf, int send_iter)\n+{\n+\tunsigned char expect = send_iter \u0026 0xFF;\n+\tint j;\n+\n+\tfor (j = 0; j \u003c send_size; j++) {\n+\t\tif ((unsigned char)buf[j] != expect) {\n+\t\t\tprintf(\"FAIL: data mismatch iter %d off %d: exp 0x%02x got 0x%02x\\n\",\n+\t\t\t send_iter, j, expect, (unsigned char)buf[j]);\n+\t\t\treturn -1;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int do_server(void)\n+{\n+\tint lsk = -1, csk = -1;\n+\tssize_t n, total = 0;\n+\tint test_result = -1;\n+\tint current_gen = 0;\n+\tint recv_count = 0;\n+\tint send_iter = 1;\n+\tchar *buf = NULL;\n+\tint record_type = 0;\n+\tint filled = 0;\n+\tint buf_size;\n+\n+\tbuf_size = send_size;\n+\tif (buf_size \u003c MIN_BUF_SIZE)\n+\t\tbuf_size = MIN_BUF_SIZE;\n+\tbuf = malloc(buf_size);\n+\tif (!buf) {\n+\t\tprintf(\"SETUP ERROR: failed to allocate buffer\\n\");\n+\t\tgoto out;\n+\t}\n+\n+\tcsk = server_accept_tls(\u0026lsk);\n+\tif (csk \u003c 0)\n+\t\tgoto out;\n+\n+\tprintf(\"TLS %s setup complete. Receiving...\\n\",\n+\t cipher_name(cipher_type));\n+\n+\t/* Burst mode: reassemble one iteration (send_size bytes) in userspace\n+\t * from however much each recv returns, rather than demanding a full\n+\t * send_size batch in a single MSG_WAITALL call. A blocking MSG_WAITALL\n+\t * of send_size deadlocks when the client's last record of an iteration\n+\t * is still partly in flight as its socket buffer fills: the server\n+\t * waits for bytes the client cannot send until the server reads, and\n+\t * the server will not read until it has the whole batch. Draining\n+\t * whatever is available keeps the receive window open and breaks that\n+\t * cycle. kTLS never splits a record and returns data and control\n+\t * (KeyUpdate) records separately, and each iteration is a whole number\n+\t * of records, so capping each recv at the iteration boundary keeps the\n+\t * reassembly aligned and delivers a KeyUpdate on its own.\n+\t */\n+\n+\t/* Main receive loop */\n+\twhile (1) {\n+\t\tchar *dst = burst_mode ? buf + filled : buf;\n+\t\tsize_t want = burst_mode ? (size_t)(send_size - filled)\n+\t\t\t\t\t : (size_t)buf_size;\n+\n+\t\tn = recv_tls_message(csk, dst, want, \u0026record_type, 0);\n+\t\tif (n == 0) {\n+\t\t\t/* A clean close on an iteration boundary is success;\n+\t\t\t * one with a partial iteration still buffered means the\n+\t\t\t * peer dropped the tail - the truncated-data case this\n+\t\t\t * test exists to catch, so fail loudly.\n+\t\t\t */\n+\t\t\tif (burst_mode \u0026\u0026 filled) {\n+\t\t\t\tprintf(\"FAIL: closed mid-iteration (%d/%d bytes buffered)\\n\",\n+\t\t\t\t filled, send_size);\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\t\t\tprintf(\"Connection closed by client\\n\");\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (n \u003c 0) {\n+\t\t\tprintf(\"FAIL: recv failed: %s\\n\", strerror(errno));\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\t/* Handle KeyUpdate. In echo mode the server mirrors the\n+\t\t * rekey back to the peer; in burst mode it only rotates its\n+\t\t * RX key and keeps draining. A KeyUpdate always lands on a\n+\t\t * send_size boundary, so no partial iteration must be buffered\n+\t\t * when one arrives.\n+\t\t */\n+\t\tif (record_type == TLS_RECORD_TYPE_HANDSHAKE) {\n+\t\t\t/* Check for a partial iteration before validating the\n+\t\t\t * KeyUpdate, so a mid-iteration arrival fails with this\n+\t\t\t * message rather than a misleading KeyUpdate-OK line.\n+\t\t\t */\n+\t\t\tif (burst_mode \u0026\u0026 filled) {\n+\t\t\t\tprintf(\"FAIL: KeyUpdate mid-iteration (%d/%d bytes buffered)\\n\",\n+\t\t\t\t filled, send_size);\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\t\t\tif (check_keyupdate(dst, n, record_type) \u003c 0)\n+\t\t\t\tgoto out;\n+\t\t\tcurrent_gen++;\n+\t\t\tprintf(\"\\n=== Server Rekey gen %d ===\\n\", current_gen);\n+\n+\t\t\tif (server_rekey(csk, current_gen) \u003c 0)\n+\t\t\t\tgoto out;\n+\n+\t\t\tprintf(\"=== Server Rekey gen %d Complete ===\\n\\n\",\n+\t\t\t current_gen);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\ttotal += n;\n+\n+\t\tif (burst_mode) {\n+\t\t\tfilled += n;\n+\t\t\tif (filled \u003c send_size)\n+\t\t\t\tcontinue;\n+\t\t\tif (server_verify_burst(buf, send_iter) \u003c 0)\n+\t\t\t\tgoto out;\n+\t\t\trecv_count++;\n+\t\t\tsend_iter++;\n+\t\t\tfilled = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\trecv_count++;\n+\t\tprintf(\"Received %zd bytes (total: %zd, count: %d)\\n\",\n+\t\t n, total, recv_count);\n+\n+\t\tif (send_all(csk, buf, n) \u003c 0)\n+\t\t\tgoto out;\n+\t\tprintf(\"Echoed %zd bytes back to client\\n\", n);\n+\t}\n+\n+\ttest_result = 0;\n+out:\n+\tprintf(\"Connection closed. Total received: %zd bytes\\n\", total);\n+\tif (num_rekeys)\n+\t\tprintf(\"Rekeys completed: %d\\n\", current_gen);\n+\n+\tif (csk \u003e= 0)\n+\t\tclose(csk);\n+\tif (lsk \u003e= 0)\n+\t\tclose(lsk);\n+\tfree(buf);\n+\treturn test_result;\n+}\n+\n+static int parse_int_arg(const char *arg, int min, int max,\n+\t\t\t const char *name, int *out)\n+{\n+\tchar *endp;\n+\tlong val;\n+\n+\terrno = 0;\n+\tval = strtol(arg, \u0026endp, 10);\n+\tif (errno || endp == arg || *endp != '\\0' || val \u003c min || val \u003e max) {\n+\t\tif (max == INT_MAX)\n+\t\t\tprintf(\"ERROR: Invalid %s '%s'. Must be \u003e= %d.\\n\",\n+\t\t\t name, arg, min);\n+\t\telse\n+\t\t\tprintf(\"ERROR: Invalid %s '%s'. Must be %d..%d.\\n\",\n+\t\t\t name, arg, min, max);\n+\t\treturn -1;\n+\t}\n+\t*out = (int)val;\n+\treturn 0;\n+}\n+\n+static int parse_cipher_option(const char *arg)\n+{\n+\tif (strcmp(arg, \"128\") == 0) {\n+\t\tcipher_type = TLS_CIPHER_AES_GCM_128;\n+\t\treturn 0;\n+\t} else if (strcmp(arg, \"256\") == 0) {\n+\t\tcipher_type = TLS_CIPHER_AES_GCM_256;\n+\t\treturn 0;\n+\t}\n+\tprintf(\"ERROR: Invalid cipher '%s'. Must be 128 or 256.\\n\", arg);\n+\treturn -1;\n+}\n+\n+static int parse_version_option(const char *arg)\n+{\n+\tif (strcmp(arg, \"1.2\") == 0) {\n+\t\ttls_version = TLS_1_2_VERSION;\n+\t\treturn 0;\n+\t} else if (strcmp(arg, \"1.3\") == 0) {\n+\t\ttls_version = TLS_1_3_VERSION;\n+\t\treturn 0;\n+\t}\n+\tprintf(\"ERROR: Invalid TLS version '%s'. Must be 1.2 or 1.3.\\n\", arg);\n+\treturn -1;\n+}\n+\n+static void print_usage(const char *prog)\n+{\n+\tprintf(\"TLS Hardware Offload Two-Node Test\\n\\n\");\n+\tprintf(\"Usage:\\n\");\n+\tprintf(\" %s server [OPTIONS]\\n\", prog);\n+\tprintf(\" %s client -s \u003cip\u003e [OPTIONS]\\n\", prog);\n+\tprintf(\"\\nOptions:\\n\");\n+\tprintf(\" -s \u003cip\u003e Server IP address, v4 or v6 (client, required)\\n\");\n+\tprintf(\" -p \u003cport\u003e Server port (default: 4433)\\n\");\n+\tprintf(\" -4 Force IPv4 (default: auto/either)\\n\");\n+\tprintf(\" -6 Force IPv6 (default: auto/either)\\n\");\n+\tprintf(\" -b \u003csize\u003e Send buffer size in bytes (default: 16384)\\n\");\n+\tprintf(\" -r \u003cmax\u003e Use random send buffer sizes (1..\u003cmax\u003e)\\n\");\n+\tprintf(\" -v \u003cversion\u003e TLS version: 1.2 or 1.3 (default: 1.3)\\n\");\n+\tprintf(\" -c \u003ccipher\u003e Cipher: 128 or 256 (default: 128)\\n\");\n+\tprintf(\" -n \u003cN\u003e Number of send/echo iterations (default: 100)\\n\");\n+\tprintf(\" -k \u003cN\u003e Perform N rekeys (client only, TLS 1.3; N \u003c iterations)\\n\");\n+\tprintf(\" -B Burst mode: client sends continuously without echo;\\n\");\n+\tprintf(\" server drains and handles KeyUpdate without responding.\\n\");\n+\tprintf(\" -Z Set TLS_RX_EXPECT_NO_PAD on the server: TLS 1.3\\n\");\n+\tprintf(\" opt-in to the zero-copy RX fast path. Not needed\\n\");\n+\tprintf(\" for TLS 1.2 (always eligible). Server only.\\n\");\n+\tprintf(\" -h Show this help message\\n\");\n+\tprintf(\"\\nExample:\\n\");\n+\tprintf(\" Node A: %s server\\n\", prog);\n+\tprintf(\" Node B: %s client -s 192.168.20.2\\n\", prog);\n+\tprintf(\"\\nRekey Example (3 rekeys, TLS 1.3 only):\\n\");\n+\tprintf(\" Node A: %s server\\n\", prog);\n+\tprintf(\" Node B: %s client -s 192.168.20.2 -k 3\\n\", prog);\n+\tprintf(\"\\nBurst Mode Example (client stresses TX rekey under load):\\n\");\n+\tprintf(\" Node A: %s server -B\\n\", prog);\n+\tprintf(\" Node B: %s client -s 192.168.20.2 -B -k 3\\n\", prog);\n+\tprintf(\"\\nIPv6 Example:\\n\");\n+\tprintf(\" Node A: %s server -6\\n\", prog);\n+\tprintf(\" Node B: %s client -6 -s fd00::2\\n\", prog);\n+}\n+\n+int main(int argc, char *argv[])\n+{\n+\tint send_size_set = 0;\n+\tint is_server;\n+\tint opt;\n+\n+\t/* When the peer aborts a TLS connection (e.g. tls_err_abort() on a\n+\t * failed decrypt), a send() here would raise SIGPIPE and kill us by\n+\t * signal, so the harness sees only a bare non-zero exit with no\n+\t * \"FAIL:\" line. Ignore it and let send()/sendmsg() return EPIPE, which\n+\t * send_all()/send_tls_key_update() report.\n+\t */\n+\tsignal(SIGPIPE, SIG_IGN);\n+\n+\tif (argc \u003c 2 ||\n+\t (strcmp(argv[1], \"server\") \u0026\u0026 strcmp(argv[1], \"client\"))) {\n+\t\tprint_usage(argv[0]);\n+\t\treturn 1;\n+\t}\n+\tis_server = !strcmp(argv[1], \"server\");\n+\n+\toptind = 2; /* skip subcommand */\n+\twhile ((opt = getopt(argc, argv, \"s:p:b:r:c:v:k:n:BZ46h\")) != -1) {\n+\t\tswitch (opt) {\n+\t\tcase 's':\n+\t\t\tserver_ip = optarg;\n+\t\t\tbreak;\n+\t\tcase '4':\n+\t\t\tif (force_family == AF_INET6) {\n+\t\t\t\tprintf(\"ERROR: -4 and -6 are mutually exclusive\\n\");\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tforce_family = AF_INET;\n+\t\t\tbreak;\n+\t\tcase '6':\n+\t\t\tif (force_family == AF_INET) {\n+\t\t\t\tprintf(\"ERROR: -4 and -6 are mutually exclusive\\n\");\n+\t\t\t\treturn 1;\n+\t\t\t}\n+\t\t\tforce_family = AF_INET6;\n+\t\t\tbreak;\n+\t\tcase 'B':\n+\t\t\tburst_mode = 1;\n+\t\t\tbreak;\n+\t\tcase 'Z':\n+\t\t\tzc_rx = 1;\n+\t\t\tbreak;\n+\t\tcase 'p':\n+\t\t\tif (parse_int_arg(optarg, 1, 65535, \"port\",\n+\t\t\t\t\t \u0026server_port) \u003c 0)\n+\t\t\t\treturn 1;\n+\t\t\tbreak;\n+\t\tcase 'b':\n+\t\t\tif (parse_int_arg(optarg, 1, INT_MAX, \"buffer size\",\n+\t\t\t\t\t \u0026send_size) \u003c 0)\n+\t\t\t\treturn 1;\n+\t\t\tsend_size_set = 1;\n+\t\t\tbreak;\n+\t\tcase 'r':\n+\t\t\tif (parse_int_arg(optarg, 1, INT_MAX, \"random size\",\n+\t\t\t\t\t \u0026random_size_max) \u003c 0)\n+\t\t\t\treturn 1;\n+\t\t\tbreak;\n+\t\tcase 'c':\n+\t\t\tif (parse_cipher_option(optarg) \u003c 0)\n+\t\t\t\treturn 1;\n+\t\t\tbreak;\n+\t\tcase 'v':\n+\t\t\tif (parse_version_option(optarg) \u003c 0)\n+\t\t\t\treturn 1;\n+\t\t\tbreak;\n+\t\tcase 'k':\n+\t\t\tif (parse_int_arg(optarg, 1, 255, \"rekey count\",\n+\t\t\t\t\t \u0026num_rekeys) \u003c 0)\n+\t\t\t\treturn 1;\n+\t\t\tbreak;\n+\t\tcase 'n':\n+\t\t\tif (parse_int_arg(optarg, 1, INT_MAX, \"iteration count\",\n+\t\t\t\t\t \u0026num_iterations) \u003c 0)\n+\t\t\t\treturn 1;\n+\t\t\tbreak;\n+\t\tcase 'h':\n+\t\t\tprint_usage(argv[0]);\n+\t\t\treturn 0;\n+\t\tdefault:\n+\t\t\tprint_usage(argv[0]);\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\tif (send_size_set \u0026\u0026 random_size_max \u003e 0) {\n+\t\tprintf(\"ERROR: -b and -r are mutually exclusive\\n\");\n+\t\treturn 1;\n+\t}\n+\n+\tif (zc_rx \u0026\u0026 tls_version != TLS_1_3_VERSION) {\n+\t\tprintf(\"ERROR: -Z (TLS_RX_EXPECT_NO_PAD) requires TLS 1.3\\n\");\n+\t\treturn 1;\n+\t}\n+\n+\tif (burst_mode \u0026\u0026 random_size_max \u003e 0) {\n+\t\tprintf(\"ERROR: -B and -r are mutually exclusive\\n\");\n+\t\treturn 1;\n+\t}\n+\n+\tif (burst_mode \u0026\u0026 send_size \u003c MIN_BUF_SIZE) {\n+\t\tprintf(\"ERROR: -b must be \u003e= %d in burst mode (-B)\\n\",\n+\t\t MIN_BUF_SIZE);\n+\t\treturn 1;\n+\t}\n+\n+\tif (is_server) {\n+\t\tif (server_ip) {\n+\t\t\tprintf(\"warning: -s is ignored in server mode\\n\");\n+\t\t\tserver_ip = NULL;\n+\t\t}\n+\t\tif (random_size_max \u003e 0) {\n+\t\t\tprintf(\"warning: -r is ignored in server mode\\n\");\n+\t\t\trandom_size_max = 0;\n+\t\t}\n+\t\tif (num_rekeys) {\n+\t\t\tprintf(\"warning: -k is ignored in server mode\\n\");\n+\t\t\tnum_rekeys = 0;\n+\t\t}\n+\t} else {\n+\t\tif (!server_ip) {\n+\t\t\tprintf(\"ERROR: Client requires -s \u003cip\u003e option\\n\");\n+\t\t\treturn 1;\n+\t\t}\n+\t\tif (tls_version == TLS_1_2_VERSION \u0026\u0026 num_rekeys) {\n+\t\t\tprintf(\"ERROR: TLS 1.2 does not support rekey\\n\");\n+\t\t\treturn 1;\n+\t\t}\n+\t\tif (num_rekeys \u003e= num_iterations) {\n+\t\t\tprintf(\"ERROR: num_rekeys (%d) must be \u003c num_iterations (%d)\\n\",\n+\t\t\t num_rekeys, num_iterations);\n+\t\t\treturn 1;\n+\t\t}\n+\t\tif (zc_rx) {\n+\t\t\tprintf(\"ERROR: -Z applies to the server (receiver) only\\n\");\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\tprintf(\"TLS Version: %s\\n\", version_name(tls_version));\n+\tprintf(\"Cipher: %s\\n\", cipher_name(cipher_type));\n+\tprintf(\"Address family: %s\\n\",\n+\t force_family == AF_INET ? \"IPv4\" :\n+\t force_family == AF_INET6 ? \"IPv6\" : \"auto\");\n+\tif (random_size_max \u003e 0)\n+\t\tprintf(\"Buffer size: random (1..%d)\\n\", random_size_max);\n+\telse\n+\t\tprintf(\"Buffer size: %d\\n\", send_size);\n+\n+\tif (num_rekeys)\n+\t\tprintf(\"Rekey testing ENABLED: %d rekey(s)\\n\", num_rekeys);\n+\tif (burst_mode)\n+\t\tprintf(\"Burst mode ENABLED\\n\");\n+\tif (zc_rx)\n+\t\tprintf(\"TLS_RX_EXPECT_NO_PAD ENABLED\\n\");\n+\n+\tsrand(time(NULL));\n+\n+\tif (is_server)\n+\t\treturn do_server() ? 1 : 0;\n+\n+\treturn do_client() ? 1 : 0;\n+}\ndiff --git a/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py\nnew file mode 100755\nindex 0000000000000..99ae5b3b8996a\n--- /dev/null\n+++ b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py\n@@ -0,0 +1,446 @@\n+#!/usr/bin/env python3\n+# SPDX-License-Identifier: GPL-2.0\n+\n+\"\"\"Test kTLS hardware offload using a C helper binary.\"\"\"\n+\n+from collections import defaultdict\n+\n+from lib.py import ksft_run, ksft_exit, ksft_pr, KsftSkipEx\n+from lib.py import ksft_ge, ksft_eq\n+from lib.py import ksft_variants, KsftNamedVariant\n+from lib.py import NetDrvEpEnv\n+from lib.py import cmd, bkg, wait_port_listen, rand_port\n+from lib.py import CmdExitFailure\n+\n+# Burst variants push hundreds of MB and perform many rekeys, so they\n+# need far longer than the default cmd() timeout.\n+BURST_TIMEOUT_S = 180\n+REKEY_TIMEOUT_S = 90\n+\n+# Reading /proc/net/tls_stat is trivial locally, but on the remote it runs\n+# over ssh, where connection setup can occasionally spike past the short\n+# default cmd() timeout. Give these tiny reads plenty of headroom so a slow\n+# ssh round-trip doesn't fail an otherwise-good variant.\n+STATS_TIMEOUT_S = 30\n+\n+# Per-packet HW crypto counters exposed via `ethtool -S` on the DUT NIC,\n+# keyed by the `ethtool -i` driver name. TlsTxDevice/TlsRxDevice in\n+# /proc/net/tls_stat only prove tls_dev_add() accepted the offload; these\n+# increment once per packet the NIC actually encrypted/decrypted (mlx5 counts\n+# gso_segs, not records), so they prove the HW crypto path was exercised.\n+# Names are driver-specific, so the\n+# check only runs on drivers listed here and is skipped (not failed) on\n+# others, keeping the test portable across NICs.\n+HW_CRYPTO_COUNTERS = {\n+ 'mlx5_core': {'Tx': 'tx_tls_encrypted_packets',\n+ 'Rx': 'rx_tls_decrypted_packets'},\n+}\n+\n+\n+def check_tls_support(cfg):\n+ \"\"\"Skip the suite unless both hosts have kTLS and the DUT HW offload.\"\"\"\n+ # The tls module is autoloaded lazily on the first TCP_ULP=\"tls\"\n+ # setsockopt, so /proc/net/tls_stat (created from the module's pernet\n+ # init) may not exist yet on a freshly booted host. Load the module\n+ # explicitly before probing for it.\n+ try:\n+ cmd(\"modprobe tls\")\n+ cmd(\"modprobe tls\", host=cfg.remote)\n+ cmd(\"test -f /proc/net/tls_stat\")\n+ cmd(\"test -f /proc/net/tls_stat\", host=cfg.remote)\n+ except CmdExitFailure as e:\n+ raise KsftSkipEx(f\"kTLS not supported: {e}\") from e\n+\n+ try:\n+ features = cmd(f\"ethtool -k {cfg.ifname}\").stdout\n+ if 'tls-hw-tx-offload: on' not in features:\n+ raise KsftSkipEx(\"Device does not support TLS HW TX offload\")\n+ if 'tls-hw-rx-offload: on' not in features:\n+ raise KsftSkipEx(\"Device does not support TLS HW RX offload\")\n+ except CmdExitFailure as e:\n+ raise KsftSkipEx(f\"Cannot determine TLS HW offload support: {e}\") from e\n+\n+\n+def read_tls_stats(host=None):\n+ \"\"\"Snapshot the per-netns TLS MIB from /proc/net/tls_stat as a dict.\"\"\"\n+ # /proc/net/tls_stat exposes the per-netns TLS MIB (TLS_INC_STATS on\n+ # sock_net(sk)). The test runs on a real NIC in the host namespace, so\n+ # these counters are shared with anything else doing kTLS there. The\n+ # strict before/after delta checks (exact rekey-outcome sums, zero error\n+ # counters) assume no other kTLS activity in this namespace during a\n+ # variant's window; concurrent kTLS users would perturb the deltas and\n+ # cause spurious failures. Don't run other kTLS workloads alongside this\n+ # test.\n+ stats = defaultdict(int)\n+ output = cmd(\"cat /proc/net/tls_stat\", host=host, timeout=STATS_TIMEOUT_S)\n+ for line in output.stdout.strip().split('\\n'):\n+ parts = line.split()\n+ if len(parts) == 2:\n+ stats[parts[0]] = int(parts[1])\n+ return stats\n+\n+\n+def nic_driver(cfg):\n+ \"\"\"DUT NIC driver name from `ethtool -i`, or None if undetermined.\"\"\"\n+ try:\n+ output = cmd(f\"ethtool -i {cfg.ifname}\").stdout\n+ except CmdExitFailure:\n+ return None\n+ for line in output.splitlines():\n+ if line.startswith('driver:'):\n+ return line.split(':', 1)[1].strip()\n+ return None\n+\n+\n+def read_nic_stats(cfg):\n+ \"\"\"Snapshot the DUT NIC's `ethtool -S` counters as a dict.\"\"\"\n+ # Driver per-record TLS counters from `ethtool -S` on the DUT NIC. Same\n+ # before/after-delta caveat as read_tls_stats(): these are device-wide,\n+ # so concurrent kTLS traffic on this NIC would perturb the deltas.\n+ stats = defaultdict(int)\n+ output = cmd(f\"ethtool -S {cfg.ifname}\").stdout\n+ for line in output.strip().split('\\n'):\n+ key, sep, val = line.partition(':')\n+ if sep and val.strip().isdigit():\n+ stats[key.strip()] = int(val.strip())\n+ return stats\n+\n+\n+def stat_diff(before, after, key):\n+ \"\"\"Return the delta of counter `key` between two stat snapshots.\"\"\"\n+ return after[key] - before[key]\n+\n+\n+def check_hw_crypto(cfg, before, after, with_tx, with_rx):\n+ \"\"\"DUT-side ethtool -S check: the NIC actually crypto'd records in HW.\n+\n+ Complements the TlsTxDevice/TlsRxDevice MIBs, which only confirm the\n+ offload was installed, not that any record was processed in hardware.\n+ Driver-specific; skipped (without failing) on drivers not in\n+ HW_CRYPTO_COUNTERS so the test stays portable.\n+ \"\"\"\n+ counters = HW_CRYPTO_COUNTERS.get(cfg.nic_driver)\n+ if not counters:\n+ ksft_pr(f\"NOTE: DUT driver '{cfg.nic_driver}' has no known per-record \"\n+ f\"HW crypto counters, skipping ethtool -S check\")\n+ return\n+\n+ for direction, active in (('Tx', with_tx), ('Rx', with_rx)):\n+ if not active:\n+ continue\n+ key = counters[direction]\n+ if key not in after:\n+ ksft_pr(f\"NOTE: DUT {direction}: counter '{key}' not exposed by \"\n+ f\"{cfg.nic_driver}, skipping\")\n+ continue\n+ got = stat_diff(before, after, key)\n+ ksft_ge(got, 1,\n+ comment=f\"DUT {direction}: NIC reported no HW crypto \"\n+ f\"({key}={got})\")\n+\n+\n+def check_path(before, after, direction, role, require_hw):\n+ \"\"\"On the DUT, require HW offload; on the remote, HW or SW is fine.\"\"\"\n+ dev = stat_diff(before, after, f'Tls{direction}Device')\n+ sw = stat_diff(before, after, f'Tls{direction}Sw')\n+ if require_hw:\n+ ksft_ge(dev, 1,\n+ comment=f\"{role} {direction}: HW offload not engaged \"\n+ f\"(Device={dev}, Sw={sw})\")\n+ else:\n+ ksft_ge(dev + sw, 1,\n+ comment=f\"{role} {direction}: no TLS activity \"\n+ f\"(Device={dev}, Sw={sw})\")\n+\n+\n+def verify_tls_counters(stats_before, stats_after, expected_rekeys,\n+ tls_role, is_dut, burst=False, allow_fallback=False):\n+ \"\"\"Verify TLS counters on one side of the connection.\n+\n+ tls_role: 'client' or 'server' (TLS role this side played).\n+ is_dut: True for the local DUT; requires HW offload counters.\n+ burst: burst mode - only the TLS client rotates its TX key; the TLS\n+ server only follows with an RX rotation on KeyUpdate receipt.\n+ allow_fallback: tolerate rekeys completing in SW (TlsRx/TxRekeyFallback).\n+ Default False: a rekey on an up, offload-capable device must stay\n+ in HW, so any fallback is a regression. Set True only where SW\n+ fallback is expected (e.g. a mid-connection link-flap variant, or\n+ the peer, whose offload state is not under test).\n+ \"\"\"\n+ role = 'DUT' if is_dut else 'Peer'\n+\n+ def diff(key):\n+ return stat_diff(stats_before, stats_after, key)\n+\n+ # In burst mode the TLS client only TXs and the TLS server only RXs.\n+ # In echo mode both sides drive both directions.\n+ with_tx = not burst or tls_role == 'client'\n+ with_rx = not burst or tls_role != 'client'\n+\n+ if with_tx:\n+ check_path(stats_before, stats_after, 'Tx', role, require_hw=is_dut)\n+ if with_rx:\n+ check_path(stats_before, stats_after, 'Rx', role, require_hw=is_dut)\n+\n+ if expected_rekeys \u003e 0:\n+ if with_tx:\n+ # Each KeyUpdate yields exactly one terminal outcome, so\n+ # TlsTxRekeyOk + TlsTxRekeyAborted + TlsTxRekeyFallback == N.\n+ # At most one rekey can be PENDING at socket close (single\n+ # TLS_TX_REKEY_PENDING bit), so at most one lands in\n+ # TlsTxRekeyAborted. TlsTxRekeyFallback is a legitimate, graceful\n+ # degradation: the device did not (re)install the HW context for\n+ # that rekey (device gone, dev_add rejected, or a transient\n+ # crypto/alloc error) so it completed in SW while the kernel\n+ # returned success. It is recoverable - the next KeyUpdate\n+ # re-attempts HW offload (tls_device_start_rekey() clears\n+ # TLS_TX_REKEY_FAILED). It is folded into the outcome sum below; on\n+ # the DUT it must be 0 (allow_fallback=False), on the peer it is\n+ # only NOTEd. A genuine rekey bug still surfaces as TlsTxRekeyError.\n+ ksft_ge(1, diff('TlsTxRekeyAborted'),\n+ comment=f\"{role} Tx: TlsTxRekeyAborted expected \u003c= 1\")\n+ ksft_eq(diff('TlsTxRekeyOk') + diff('TlsTxRekeyAborted') +\n+ diff('TlsTxRekeyFallback'), expected_rekeys,\n+ comment=f\"{role} Tx: rekey outcomes must sum to \"\n+ f\"{expected_rekeys}\")\n+ fallback = diff('TlsTxRekeyFallback')\n+ if allow_fallback:\n+ if fallback:\n+ ksft_pr(f\"NOTE: {role} Tx: {fallback} rekey(s) completed \"\n+ f\"in SW (TlsTxRekeyFallback); HW not re-installed\")\n+ else:\n+ ksft_eq(fallback, 0,\n+ comment=f\"{role} Tx: TlsTxRekeyFallback expected 0 \"\n+ f\"(rekey must stay in HW offload)\")\n+ ksft_eq(diff('TlsTxRekeyError'), 0,\n+ comment=f\"{role} Tx: TlsTxRekeyError expected 0\")\n+ ksft_eq(diff('TlsCurrTxRekey'), 0,\n+ comment=f\"{role} Tx: TlsCurrTxRekey expected 0\")\n+ if with_rx:\n+ # As on TX, each received KeyUpdate yields one terminal outcome:\n+ # TlsRxRekeyOk + TlsRxRekeyAborted + TlsRxRekeyFallback == N.\n+ # At most one rekey can be deferred (single dev_add_pending) at\n+ # socket close, landing in TlsRxRekeyAborted. TlsRxRekeyFallback\n+ # is a recoverable, graceful degradation (dev_add failed or the\n+ # device was gone, so RX temporarily dropped to SW; the next\n+ # KeyUpdate re-adds the HW context and clears TLS_RX_DEV_DEGRADED).\n+ # It is folded into the outcome sum below; on the DUT it must be 0\n+ # (allow_fallback=False), on the peer it is only NOTEd. A genuine\n+ # rekey bug still surfaces as TlsRxRekeyError.\n+ ksft_ge(1, diff('TlsRxRekeyAborted'),\n+ comment=f\"{role} Rx: TlsRxRekeyAborted expected \u003c= 1\")\n+ ksft_eq(diff('TlsRxRekeyOk') + diff('TlsRxRekeyAborted') +\n+ diff('TlsRxRekeyFallback'), expected_rekeys,\n+ comment=f\"{role} Rx: rekey outcomes must sum to \"\n+ f\"{expected_rekeys}\")\n+ ksft_eq(diff('TlsRxRekeyReceived'), expected_rekeys,\n+ comment=f\"{role} Rx: TlsRxRekeyReceived expected \"\n+ f\"{expected_rekeys}\")\n+ fallback = diff('TlsRxRekeyFallback')\n+ if allow_fallback:\n+ if fallback:\n+ ksft_pr(f\"NOTE: {role} Rx: {fallback} rekey(s) completed \"\n+ f\"in SW (TlsRxRekeyFallback); HW not re-installed\")\n+ else:\n+ ksft_eq(fallback, 0,\n+ comment=f\"{role} Rx: TlsRxRekeyFallback expected 0 \"\n+ f\"(rekey must stay in HW offload)\")\n+ ksft_eq(diff('TlsRxRekeyError'), 0,\n+ comment=f\"{role} Rx: TlsRxRekeyError expected 0\")\n+ ksft_eq(diff('TlsCurrRxRekey'), 0,\n+ comment=f\"{role} Rx: TlsCurrRxRekey expected 0\")\n+\n+ ksft_eq(diff('TlsDecryptError'), 0,\n+ comment=f\"{role}: TlsDecryptError expected 0\")\n+\n+\n+def run_tls_test(cfg, cipher=\"128\", tls_version=\"1.3\", rekey=0,\n+ buffer_size=None, random_max=None, burst=False, zc=False,\n+ dut_role=\"client\", num_iterations=None, ipver=\"4\"):\n+ \"\"\"Run the TLS offload test.\n+\n+ dut_role: 'client' (default) - DUT runs the TLS client, remote the server.\n+ 'server' - swap: DUT listens, remote connects. Used for burst_rx\n+ so the DUT's RX path is the one under rekey pressure.\n+\n+ ipver: '4' or '6' - IP version to run over. The C helper is forced to the\n+ matching family with -4/-6 and connects to the peer's v4/v6 address.\n+ Variants requesting '6' skip cleanly when the environment lacks IPv6\n+ connectivity (require_ipver()).\n+\n+ The DUT (local) is the kernel under test; the remote is just a traffic\n+ source/sink and may run any kernel without HW offload. Both sides run\n+ kTLS because TLS is pairwise, but verify_tls_counters() requires HW\n+ offload only on the DUT (is_dut=True); the peer may use SW kTLS.\n+\n+ Rekey/burst variants additionally require the peer to support TLS 1.3\n+ KeyUpdate (as the RX or TX side of the rotation). SW KeyUpdate and its\n+ MIB counters landed together in v6.14; an older peer cannot follow the\n+ rotation, so those variants are skipped rather than failed when the peer\n+ lacks the rekey counters (see the probe below).\n+ \"\"\"\n+ cfg.require_ipver(ipver)\n+\n+ port = rand_port()\n+ send_size = random_max or buffer_size\n+\n+ if dut_role == \"client\":\n+ server_bin, server_host = cfg.bin_remote, cfg.remote\n+ client_bin, client_host = cfg.bin_local, None\n+ client_target = cfg.remote_addr_v[ipver]\n+ else:\n+ server_bin, server_host = cfg.bin_local, None\n+ client_bin, client_host = cfg.bin_remote, cfg.remote\n+ client_target = cfg.addr_v[ipver]\n+\n+ server_parts = [f\"{server_bin} server -p {port} -c {cipher}\",\n+ f\"-v {tls_version}\", f\"-{ipver}\"]\n+ if burst:\n+ server_parts.append(\"-B\")\n+ if zc:\n+ server_parts.append(\"-Z\")\n+ if send_size:\n+ server_parts.append(f\"-b {send_size}\")\n+ server_cmd = \" \".join(server_parts)\n+\n+ client_parts = [f\"{client_bin} client -s {client_target}\",\n+ f\"-p {port} -c {cipher} -v {tls_version} -{ipver}\"]\n+ if rekey:\n+ client_parts.append(f\"-k {rekey}\")\n+ if burst:\n+ client_parts.append(\"-B\")\n+ if num_iterations:\n+ client_parts.append(f\"-n {num_iterations}\")\n+ if random_max:\n+ client_parts.append(f\"-r {random_max}\")\n+ elif buffer_size:\n+ client_parts.append(f\"-b {buffer_size}\")\n+ client_cmd = \" \".join(client_parts)\n+\n+ if burst:\n+ cmd_timeout = BURST_TIMEOUT_S\n+ elif rekey:\n+ cmd_timeout = REKEY_TIMEOUT_S\n+ else:\n+ cmd_timeout = 20\n+\n+ stats_before_local = read_tls_stats()\n+ stats_before_remote = read_tls_stats(host=cfg.remote)\n+ nic_before = read_nic_stats(cfg)\n+\n+ # /proc/net/tls_stat lists every MIB the running kernel knows (0 or not),\n+ # so a missing name means the peer predates that counter. The base rekey\n+ # counters (TlsRxRekeyReceived, Tls{Rx,Tx}RekeyOk, Tls{Rx,Tx}RekeyError)\n+ # shipped with SW KeyUpdate in v6.14; a peer without them cannot follow a\n+ # KeyUpdate, so the rekey/burst variants can't run against it. Skip cleanly\n+ # here rather than letting the peer-side rekey-sum / RxRekeyReceived checks\n+ # report a confusing \"expected N, got 0\" later. TlsRxRekeyReceived is a\n+ # reliable probe: the peer must bump it to have processed the rotation at all.\n+ #\n+ # Only a base v6.14 counter is probed. The newer HW-path MIBs (Aborted,\n+ # Fallback, CurrRekey) are structurally 0 on a SW-only peer and defaultdict\n+ # returns 0 for absent names, so the peer-side checks hold either way.\n+ if rekey and 'TlsRxRekeyReceived' not in stats_before_remote:\n+ raise KsftSkipEx(\"Peer kernel lacks TLS 1.3 KeyUpdate support \"\n+ \"(no rekey MIB counters); required for rekey tests\")\n+\n+ with bkg(server_cmd, host=server_host, exit_wait=True):\n+ wait_port_listen(port, host=server_host)\n+ # Start the client in the background so we keep a handle to it. A\n+ # foreground cmd() raises TimeoutExpired from inside its constructor\n+ # if the client hangs, and since the child is not killed on timeout\n+ # it would be left running with no handle to reap it. A leaked\n+ # client keeps bumping the per-netns TLS counters (TlsTxRekeyAborted,\n+ # TlsDecryptError, ...) and would corrupt the before/after\n+ # measurement window of a later variant. The finally clause reaps it\n+ # within this variant's window instead.\n+ client = cmd(client_cmd, host=client_host, background=True)\n+ try:\n+ client.process(terminate=False, fail=True, timeout=cmd_timeout)\n+ finally:\n+ if client.proc.poll() is None:\n+ client.process(terminate=True, fail=False, timeout=5)\n+\n+ stats_after_local = read_tls_stats()\n+ stats_after_remote = read_tls_stats(host=cfg.remote)\n+ nic_after = read_nic_stats(cfg)\n+\n+ peer_tls_role = 'server' if dut_role == 'client' else 'client'\n+\n+ # Which directions the DUT drives (mirrors verify_tls_counters()): in\n+ # burst mode the TLS client only TXs and the server only RXs; echo mode\n+ # drives both.\n+ dut_with_tx = not burst or dut_role == 'client'\n+ dut_with_rx = not burst or dut_role != 'client'\n+\n+ verify_tls_counters(stats_before_local, stats_after_local,\n+ rekey, dut_role, is_dut=True, burst=burst)\n+ check_hw_crypto(cfg, nic_before, nic_after, dut_with_tx, dut_with_rx)\n+ verify_tls_counters(stats_before_remote, stats_after_remote,\n+ rekey, peer_tls_role, is_dut=False, burst=burst,\n+ allow_fallback=True)\n+\n+\n+# The cipher/version matrix runs over IPv4; the socket setup is the only\n+# IP-version-specific code path, so a single representative variant over\n+# IPv6 is enough to cover it (it skips cleanly without v6 connectivity).\n+# The rekey and burst suites below likewise stay on IPv4 to bound runtime.\n+@ksft_variants([\n+ KsftNamedVariant(\"tls13_aes128\", \"128\", \"1.3\", \"4\"),\n+ KsftNamedVariant(\"tls13_aes256\", \"256\", \"1.3\", \"4\"),\n+ KsftNamedVariant(\"tls12_aes128\", \"128\", \"1.2\", \"4\"),\n+ KsftNamedVariant(\"tls12_aes256\", \"256\", \"1.2\", \"4\"),\n+ KsftNamedVariant(\"tls13_aes128_ip6\", \"128\", \"1.3\", \"6\"),\n+])\n+def test_tls_offload(cfg, cipher, tls_version, ipver):\n+ \"\"\"Cipher/version matrix over the HW offload data path, no rekey.\"\"\"\n+ run_tls_test(cfg, cipher=cipher, tls_version=tls_version, ipver=ipver)\n+\n+\n+@ksft_variants([\n+ KsftNamedVariant(\"single\", 1),\n+ KsftNamedVariant(\"multiple\", 99),\n+ KsftNamedVariant(\"small_buf\", 30, 512),\n+ KsftNamedVariant(\"large_buf\", 10, 2097152),\n+ KsftNamedVariant(\"random_buf\", 20, None, 8192),\n+])\n+def test_tls_offload_rekey(cfg, rekey, buffer_size=None, random_max=None):\n+ \"\"\"Echo-mode TLS 1.3 KeyUpdate rekeys across a range of buffer sizes.\"\"\"\n+ run_tls_test(cfg, cipher=\"128\", tls_version=\"1.3\", rekey=rekey,\n+ buffer_size=buffer_size, random_max=random_max)\n+\n+\n+# Columns: dut_role zc interval rekeys buffer_size\n+@ksft_variants([\n+ KsftNamedVariant(\"burst_tx_rekey_every_1\", \"client\", False, 1, 50, 65536),\n+ KsftNamedVariant(\"burst_tx_rekey_every_1000\", \"client\", False, 1000, 3, 65536),\n+ KsftNamedVariant(\"burst_rx_rekey_every_10\", \"server\", False, 10, 20, 65536),\n+ KsftNamedVariant(\"burst_rx_rekey_every_10000\", \"server\", False, 10000, 1, 32768),\n+ KsftNamedVariant(\"burst_rx_zc_rekey_every_100\", \"server\", True, 100, 10, 65536),\n+ KsftNamedVariant(\"burst_rx_zc_rekey_every_20000\", \"server\", True, 20000, 1, 16384),\n+])\n+def test_tls_offload_burst(cfg, dut_role, zc, interval, rekeys, buffer_size):\n+ \"\"\"High-volume one-directional traffic with frequent rekeys.\"\"\"\n+ run_tls_test(cfg, cipher=\"128\", tls_version=\"1.3\", rekey=rekeys,\n+ buffer_size=buffer_size, burst=True, zc=zc, dut_role=dut_role,\n+ num_iterations=interval * (rekeys + 1))\n+\n+\n+def main() -\u003e None:\n+ \"\"\"Set up the DUT/peer environment and run the offload test suites.\"\"\"\n+ with NetDrvEpEnv(__file__, nsim_test=False) as cfg:\n+ cfg.bin_local = cfg.test_dir / \"tls_hw_offload\"\n+ if not cfg.bin_local.exists():\n+ raise KsftSkipEx(f\"tls_hw_offload binary not found at {cfg.bin_local}\")\n+ cfg.bin_remote = cfg.remote.deploy(cfg.bin_local)\n+ cfg.require_ipver(\"4\")\n+ check_tls_support(cfg)\n+ cfg.nic_driver = nic_driver(cfg)\n+\n+ ksft_run([test_tls_offload, test_tls_offload_rekey,\n+ test_tls_offload_burst], args=(cfg, ))\n+ ksft_exit()\n+\n+\n+if __name__ == \"__main__\":\n+ main()\n"
} |
| 2/1 | 2026/09/20 01:00 | agent | patch-evaluator |
6mModel:core Args: null Results: {
"EnableConfigs": [
"TLS",
"TLS_DEVICE",
"INET"
],
"FocusSymbols": [
"tls_setsockopt",
"do_tls_setsockopt",
"do_tls_setsockopt_conf",
"tls_set_sw_offload",
"tls_sw_ctx_init",
"tls_sw_ctx_finalize",
"tls_set_device_offload",
"tls_set_device_offload_rx",
"tls_sw_drain_tx",
"tls_sw_sendmsg_locked",
"tls_validate_xmit_skb_rekey"
],
"Reasoning": "The patch modifies both core networking and the kTLS subsystem to support TLS 1.3 KeyUpdate/rekeying. While dedicated hardware offload execution relies on physical SmartNICs, the patch significantly alters reachable software paths that execute in standard virtualized environments (QEMU):\n1. `do_tls_setsockopt` introduces mutex acquisition of `tx_lock` prior to `lock_sock`, altering the locking hierarchy for all kTLS sockets (software or hardware).\n2. `do_tls_setsockopt_conf` invokes `tls_set_device_offload` and `tls_set_device_offload_rx` for initial and rekey configuration checks before falling back to `tls_set_sw_offload`.\n3. `tls_set_sw_offload` has been refactored into `tls_sw_ctx_init` and `tls_sw_ctx_finalize`, changing cipher allocation, setkey, and sequence initialization in software kTLS.\n4. Data path encryption (`tls_do_encryption`, `tls_push_record`) and transmission paths (`tls_sw_sendmsg_locked`, `tls_sw_drain_tx`) now integrate rekey cipher context accessors.\n5. Core networking changes in `net/sched/sch_generic.c` (`dequeue_skb`) and `include/net/tcp.h` (`tcp_write_collapse_fence`) affect socket packet handling.\nThese changes modify reachable logic and synchronization invariants, warranting fuzzing.",
"WorthFuzzing": true
}Instruction: You are an expert Linux kernel maintainer and security engineer.
Your job is to review a provided patch series and evaluate whether it warrants fuzzing with syzkaller.
IMPORTANT: The changes have ALREADY been applied and committed as the HEAD commit in
your workspace. Do NOT rely on internal assumptions. You must actively use your code access
tools to inspect the actual source code, callers, and surrounding context.
================================================================================
1. CORE TRIAGE PHILOSOPHY
================================================================================
The goal of patch fuzzing is to discover crashes, regressions, exposed latent bugs,
and newly triggered assertions introduced by the patch series.
- REACHABILITY IS THE PRIMARY GATE:
Fuzzing can only discover bugs in code that can actually execute in standard virtualized
environments (GCE or QEMU, utilizing software-emulated devices like USB gadgets, netdev, tun/tap).
If the modified code is structurally unreachable (see Section 2), it MUST NOT be fuzzed,
regardless of whether it adds assertions or complex logic.
- DO NOT BLINDLY TRUST "NO FUNCTIONAL CHANGE" (NFCI) OR "REFACTORING" CLAIMS:
Patch authors routinely label changes as "cleanups", "refactorings", or state
"No functional change intended". Do NOT take these claims at face value.
Code refactorings that rearrange logic, introduce helper functions, or alter state management
in core subsystems frequently introduce subtle semantic shifts or uncover latent kernel bugs.
If reachable executable code is modified or refactored, it MUST be fuzzed.
- NEW OR MODIFIED ASSERTIONS IN REACHABLE CODE MUST BE FUZZED:
When a patch introduces or modifies runtime checks or assertions (e.g., WARN_ON*, VM_WARN_ON*,
BUG_ON*, lockdep_assert*) in reachable code paths, it enforces new or stricter invariants.
Even if the author believes the invariant always holds, fuzzing is essential to verify whether
an unusual sequence of operations can violate it.
================================================================================
2. WHEN TO RETURN WorthFuzzing=false (NEGATIVE CRITERIA)
================================================================================
Return WorthFuzzing=false ONLY IF all modified code falls strictly into one or more of these categories:
- Non-kernel and non-executable changes:
* Modifications to Documentation/, comments, or spelling fixes.
* User-space directories, self-tests, samples, or scripts (e.g., tools/, samples/, scripts/, usr/)
that do not affect the compiled kernel image (vmlinux) or kernel modules.
* Purely decorative logging (e.g., message strings in pr_err, printk, dev_info) or tracepoints
that do not alter control flow or data structures.
* Build system or Kconfig changes that do not alter compiled C logic.
- Structurally unreachable hardware:
* Vendor-specific PCIe switches, SmartNICs, or GPU drivers (e.g., mlxsw, pds_core, qed,
ionic, amdgpu) requiring physical ASIC/PCIe cards not emulated in standard QEMU.
- Unreachable execution paths:
* Driver teardown callbacks (.remove, .shutdown, pci_unregister_driver) executed only during
physical PCI hot-unplug or manual sysfs driver unbinding.
* Code paths exclusive to architectures other than the target architecture.
================================================================================
3. WHEN TO RETURN WorthFuzzing=true (POSITIVE CRITERIA)
================================================================================
Return WorthFuzzing=true whenever the patch touches reachable executable code, including:
- Core Subsystems:
* Any logic modifications in memory management (mm/), synchronization/locking (kernel/locking/),
BPF, scheduler, core networking, VFS, or syscall handling.
- Refactorings and Code Cleanups:
* Any restructuring of reachable data structures, helper abstractions, or algorithm flows.
- Runtime Assertions and Defensive Checks:
* Any introduction or alteration of assertions (WARN_ON*, VM_WARN_ON*, BUG_ON*, etc.) in reachable paths.
- Reachable Drivers and Protocols:
* Drivers accessible via virtual buses (virtio, USB gadget, loopback, netlink, binder, sockets, etc.).
================================================================================
4. EXTRACTING FocusSymbols (PREVENTING DILUTION)
================================================================================
When WorthFuzzing=true, you must extract specific kernel functions into FocusSymbols to guide the fuzzer:
- AVOID UBIQUITOUS LIFECYCLE HOT-PATHS:
Do NOT list generic, ubiquitous functions called by almost every program in the corpus
(including, but not limited to: general memory allocators and deallocators, page fault
and trap handlers, or core synchronization primitives; this is not an exhaustive list).
Listing ubiquitous functions causes the fuzzer to classify thousands of unrelated tests as "focused",
which severely dilutes fuzzing effort away from the actual changes.
- TARGET SPECIFIC FEATURE LOGIC AND ENTRYPOINTS:
List functions that specifically implement the logic being added or altered, or direct API entrypoints
for the subsystem feature under review.
- HANDLING STATIC INLINE FUNCTIONS IN HEADERS (.h):
Compiler-inlined static functions (such as static inlines in mm/*.h or include/linux/*.h) lack
distinct symbol addresses in vmlinux and cannot be targeted directly by symbol coverage filters.
If the changes are primarily in static inline helpers, identify non-static, feature-specific caller
functions in .c files that exercise them (avoiding ubiquitous lifecycle wrappers).
================================================================================
5. IDENTIFYING EnableConfigs
================================================================================
Identify any specific CONFIG_ options required to properly compile and reach the modified code:
- Inspect Kconfig files and #ifdef guards; do not make assumptions.
- Check "depends on" lines in Kconfig to include any non-standard parent subsystem configs needed.
- Strip any 'CONFIG_' prefix (e.g., return "NET_IPV4" instead of "CONFIG_NET_IPV4").
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt: Target architecture: amd64
For your convenience, here is the diff of the changes:
commit 209c4bda0a621712323d0608683304a02d1a2596
Author: syz-cluster <triage@syzkaller.com>
Date: Sun Sep 20 01:00:09 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/networking/tls-offload.rst b/Documentation/networking/tls-offload.rst
index e5802bcd4d22d..cdf84f4b817a7 100644
--- a/Documentation/networking/tls-offload.rst
+++ b/Documentation/networking/tls-offload.rst
@@ -99,9 +99,8 @@ at the end of kernel structures (see :c:member:`driver_state` members
in ``include/net/tls.h``) to avoid additional allocations and pointer
dereferences.
-When the offloaded connection is destroyed the core calls
-the :c:member:`tls_dev_del` callback so the driver can release per-direction
-state:
+The core calls the :c:member:`tls_dev_del` callback so the driver can release
+per-direction state:
.. code-block:: c
@@ -109,7 +108,14 @@ state:
struct tls_context *ctx,
enum tls_offload_ctx_dir direction);
-``tls_dev_del`` is mandatory whenever ``tls_dev_add`` is provided.
+``tls_dev_del`` is called either when the offloaded connection is destroyed or,
+for a TLS 1.3 connection, when the old key is retired during a rekey (see the
+`Rekey`_ section). It operates on a single ``direction``, so the driver must
+release only the state for that direction and must not free state shared
+between directions or the socket as a whole. After a rekey ``tls_dev_del``,
+``tls_dev_add`` may be called again for the same socket and direction to
+install the new key. ``tls_dev_del`` is mandatory whenever ``tls_dev_add`` is
+provided.
The third TLS device callback is :c:member:`tls_dev_resync`, called by the core
to synchronize the TCP stream with the record boundaries:
@@ -205,7 +211,10 @@ Upon reception of a TLS offloaded packet, the driver sets
the :c:member:`decrypted` mark in :c:type:`struct sk_buff <sk_buff>`
corresponding to the segment. Networking stack makes sure decrypted
and non-decrypted segments do not get coalesced (e.g. by GRO or socket layer)
-and takes care of partial decryption.
+and takes care of partial decryption. A segment the device processed but
+could not authenticate may instead carry the :c:member:`decrypt_failed`
+mark; see the `Error handling`_ section for what the mark implies about
+the payload.
Resync handling
===============
@@ -404,8 +413,121 @@ records, then after 4 records, after 8, after 16... up until every
Rekey
=====
-Offload does not currently support TLS 1.3, therefore key rotation
-is not a concern for offloaded connections at this point.
+TLS 1.3 allows traffic keys to be updated mid-connection using the
+KeyUpdate message. Offloaded TLS 1.3 connections must therefore switch
+keys without tearing down the offload. The device cannot simply be given
+the new key because records encrypted (TX) or transformed (RX) with the
+old key may still be in flight. The stack retains the necessary old-key
+state and bridges the transition in software.
+
+TX
+--
+
+On TX, the new key is installed in a temporary software context, and
+sendmsg is routed through the software path. If no hardware-offloaded
+records remain unacknowledged, the switch completes inline during
+setsockopt. Otherwise the rekey is left pending and is completed later,
+on the sender's next ``sendmsg()`` after all old-key records have been
+ACKed (see `Completing a deferred rekey`_). Completion calls
+:c:func:`tls_dev_del` for the old key and reinstalls hardware offload
+with the new key at the current TCP write sequence. If reinstallation
+fails, the connection keeps encrypting in software with the new key; the
+next KeyUpdate re-arms the transition and retries the hardware
+installation.
+
+Unlike the software path, a ``TLS_TX`` setsockopt on an offloaded
+connection first flushes the open and partially sent hardware records to
+TCP before installing the new key. It therefore behaves like a blocking
+``send()`` of that record: it may wait for send buffer space (bounded by
+``SO_SNDTIMEO``), and on a non-blocking socket it fails with ``-EAGAIN``
+and must be retried once the socket is writable. The new key is not
+installed until the call succeeds; the connection keeps using the old key
+in the meantime.
+
+Completing a deferred rekey
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A deferred rekey is completed by the sender, not by the ACK path. When
+the last old-key record is acknowledged the stack only marks the rekey
+as ready; the device is not touched. The switch itself,
+:c:func:`tls_dev_del` of the old key followed by :c:func:`tls_dev_add`
+of the new one, runs at the start of the next ``sendmsg()`` on the
+socket, and that ``sendmsg()`` is the first to be encrypted by hardware
+again. No other event completes it: ``splice_eof()``, write-space
+wakeups, retransmissions and pure ACKs all leave the connection on the
+software path.
+
+This is intentional. Completion has to flush the software context's
+open record to TCP and may sleep for send buffer space, which rules out
+the ACK and write-space paths. Beyond that, the stack only switches when
+it has new data to hand to the device: the software path is fully
+correct with the new key, so deferring the switch costs host CPU but
+nothing else, and it keeps the device from being programmed for a
+connection that may never send again.
+
+Two consequences follow. A connection that stops sending after a
+KeyUpdate stays in the deferred state until it is closed: it is
+encrypted in software with the new key, it is counted in
+``TlsCurrTxRekey``, and at close it is reported as
+``TlsTxRekeyAborted``. That counter therefore includes senders that
+simply had nothing more to send, not only sockets torn down
+mid-transition, and is not by itself an error indication. And the return
+to hardware is delayed by at least one ACK round trip after the last
+old-key record, plus however long the application waits before its next
+``sendmsg()``. A sender that wants the hardware path back promptly can
+issue a small ``sendmsg()`` once its old data has been acknowledged.
+
+Completion can fail transiently or permanently. If the software flush
+cannot get send buffer space (``-EAGAIN``, or a signal on a blocking
+socket) the rekey stays pending, the ``sendmsg()`` proceeds in software,
+and the next ``sendmsg()`` retries; the ``tls_device_complete_rekey_retry``
+tracepoint fires. A hard failure (:c:func:`tls_dev_add` rejected, or the
+netdev gone) is terminal for this KeyUpdate: the connection is pinned to
+software encryption with the new key, counted in ``TlsTxRekeyFallback``
+and moved from ``TlsCurrTxDevice`` to ``TlsCurrTxSw``; the
+``tls_device_complete_rekey_fail`` tracepoint fires. The next ``TLS_TX``
+setsockopt re-arms the transition and retries.
+
+The decision to defer is taken at the start of the ``TLS_TX``
+setsockopt, before the open hardware record is flushed to TCP. That
+flush may block for send buffer space, and old-key records acknowledged
+while it sleeps do not change the decision: the rekey is still deferred
+and completes on a following ``sendmsg()`` rather than inline. This is
+conservative, not a correctness issue. The boundary is fixed at the
+write sequence after the flush, so the acknowledgment of the flushed
+record itself arms completion; the cost is one more ACK round trip and
+one more ``sendmsg()``. Applications should not expect an inline switch
+whenever the socket has unacknowledged data at the time of the
+setsockopt.
+
+RX
+--
+
+On RX, the NIC may already have transformed in-flight records with the
+old key before the peer's KeyUpdate is parsed. When the KeyUpdate is
+decoded, the stack removes the old key from the NIC but retains the old
+AEAD, IV, and record sequence in the software offload context.
+
+Each record is classified by the TCP sequence of its first byte relative
+to the boundary at which the NIC stopped using the old key. Records
+starting after that boundary carry new-key wire encryption, so the old
+software AEAD state can be released. Records before the boundary that
+remain fully encrypted are passed to the software path. Records that
+were partially transformed by the NIC are re-encrypted with the old key
+to restore the new-key ciphertext, allowing the software AEAD to decrypt
+them with the new key.
+
+If old-key records are still queued, installation of the new key through
+:c:func:`tls_dev_add` is deferred until those records have been consumed;
+otherwise it occurs immediately. When the NIC cannot authenticate a record
+processed during the transition, the affected fragments are delivered with
+``skb->decrypt_failed`` set, following the contract described in the
+`Error handling`_ section. In a mixed record such a fragment was
+transformed (XORed) with the old key, and the re-encrypt path uses this to
+undo the transform on those fragments with the old key while leaving
+untouched fragments intact. A non-mixed record carrying
+``skb->decrypt_failed`` was not transformed; it is still wire ciphertext
+and is decrypted directly by the software AEAD under the new key.
Error handling
==============
@@ -442,8 +564,43 @@ to the host's stack as it was on the wire (recovering original packet in the
driver if device provides precise error is sufficient).
The Linux networking stack does not provide a way of reporting per-packet
-decryption and authentication errors, packets with errors must simply not
-have the :c:member:`decrypted` mark set.
+decryption and authentication errors. A packet with errors must not have
+the :c:member:`decrypted` mark set. In addition, the driver may set the
+:c:member:`decrypt_failed` mark on a segment the device matched to an
+offloaded connection and processed but could not authenticate. The two
+marks are mutually exclusive.
+
+The stack interprets :c:member:`decrypt_failed` per record, relative to the
+:c:member:`decrypted` mark of the other segments making up the same record.
+Coalescing (GRO, socket layer) and record classification are keyed on
+:c:member:`decrypted` alone, so :c:member:`decrypt_failed` segments may be
+merged with unmarked ones. A driver setting the mark must therefore honour
+the following contract:
+
+ * In a record none of whose segments carry :c:member:`decrypted`, every
+ segment, including one with :c:member:`decrypt_failed` set, must hold
+ the payload exactly as it was on the wire. This is the general rule
+ above: if the device did not successfully decrypt any part of a record
+ it must hand the whole record over untouched. The stack passes such a
+ record to software decryption directly and does not consult
+ :c:member:`decrypt_failed`.
+
+ * In a record where some segments carry :c:member:`decrypted` (a mixed
+ record), a segment with :c:member:`decrypt_failed` set must hold payload
+ the device has already transformed (XORed with the cipher keystream) but
+ failed to authenticate, and a segment with neither mark must hold the
+ payload as it was on the wire. The stack re-encrypts the
+ :c:member:`decrypted` and :c:member:`decrypt_failed` segments to restore
+ the ciphertext, leaves the unmarked segments intact, and authenticates
+ the whole record in software.
+
+A transformed segment delivered without :c:member:`decrypt_failed`, or an
+untransformed segment of a mixed record delivered with it, is restored
+incorrectly and the record fails software authentication. A device which
+cannot tell the driver whether a failed segment was transformed must
+recover the original packet before handing it to the stack, as described
+above, and leave both marks clear. During a TLS 1.3 rekey the mark also
+tells the stack which key the device applied; see the `Rekey`_ section.
A packet should also not be handled by the TLS offload if it contains
incorrect checksums.
diff --git a/Documentation/networking/tls.rst b/Documentation/networking/tls.rst
index 980c442d7161a..cf05543260d85 100644
--- a/Documentation/networking/tls.rst
+++ b/Documentation/networking/tls.rst
@@ -314,6 +314,11 @@ TLS implementation exposes the following per-namespace statistics
number of TX and RX sessions currently installed where NIC handles
cryptography
+- ``TlsCurrTxRekey``, ``TlsCurrRxRekey`` -
+ number of TX and RX sessions currently undergoing a deferred rekey,
+ i.e. a rekey which could not be applied immediately and is waiting for
+ in-flight records to drain before the new key is installed in hardware
+
- ``TlsTxSw``, ``TlsRxSw`` -
number of TX and RX sessions opened with host cryptography
@@ -344,3 +349,15 @@ TLS implementation exposes the following per-namespace statistics
- ``TlsRxRekeyReceived`` -
number of received KeyUpdate handshake messages, requiring userspace
to provide a new RX key
+
+- ``TlsTxRekeyFallback``, ``TlsRxRekeyFallback`` -
+ number of rekeys on existing sessions for TX and RX which could not be
+ offloaded to the NIC and fell back to software cryptography
+
+- ``TlsTxRekeyAborted``, ``TlsRxRekeyAborted`` -
+ number of deferred rekeys for TX and RX which were still pending when
+ the socket was destroyed, and so never completed. For TX hardware
+ offload this includes senders that sent nothing further after the
+ KeyUpdate, since the switch back to hardware only happens on
+ ``sendmsg()`` (see the Rekey section of
+ Documentation/networking/tls-offload.rst)
diff --git a/MAINTAINERS b/MAINTAINERS
index 0e04d92d1b098..4f1645bf2ee5e 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -19255,6 +19255,8 @@ F: Documentation/networking/tls*
F: include/net/tls.h
F: include/uapi/linux/tls.h
F: net/tls/
+F: tools/testing/selftests/drivers/net/hw/tls_hw_offload.c
+F: tools/testing/selftests/drivers/net/hw/tls_hw_offload.py
F: tools/testing/selftests/net/tls.c
NETWORKING [SOCKETS]
diff --git a/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c b/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c
index f5acd4be1e69d..29e108ce67645 100644
--- a/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c
+++ b/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c
@@ -431,6 +431,9 @@ static int chcr_ktls_dev_add(struct net_device *netdev, struct sock *sk,
atomic64_inc(&port_stats->ktls_tx_connection_open);
u_ctx = adap->uld[CXGB4_ULD_KTLS].handle;
+ if (crypto_info->version != TLS_1_2_VERSION)
+ goto out;
+
if (direction == TLS_OFFLOAD_CTX_DIR_RX) {
pr_err("not expecting for RX direction\n");
goto out;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
index 07a04a142a2ea..0469ca6a0762e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
@@ -30,7 +30,9 @@ static inline bool mlx5e_is_ktls_device(struct mlx5_core_dev *mdev)
return false;
return (MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_128) ||
- MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_256));
+ MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_256) ||
+ MLX5_CAP_TLS(mdev, tls_1_3_aes_gcm_128) ||
+ MLX5_CAP_TLS(mdev, tls_1_3_aes_gcm_256));
}
static inline bool mlx5e_ktls_type_check(struct mlx5_core_dev *mdev,
@@ -40,10 +42,14 @@ static inline bool mlx5e_ktls_type_check(struct mlx5_core_dev *mdev,
case TLS_CIPHER_AES_GCM_128:
if (crypto_info->version == TLS_1_2_VERSION)
return MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_128);
+ else if (crypto_info->version == TLS_1_3_VERSION)
+ return MLX5_CAP_TLS(mdev, tls_1_3_aes_gcm_128);
break;
case TLS_CIPHER_AES_GCM_256:
if (crypto_info->version == TLS_1_2_VERSION)
return MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_256);
+ else if (crypto_info->version == TLS_1_3_VERSION)
+ return MLX5_CAP_TLS(mdev, tls_1_3_aes_gcm_256);
break;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
index bca45679e2016..8ec40f5fd5b50 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
@@ -602,7 +602,18 @@ void mlx5e_ktls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
stats->tls_resync_req_pkt++;
resync_update_sn(rq, skb);
break;
- default: /* CQE_TLS_OFFLOAD_ERROR: */
+ case CQE_TLS_OFFLOAD_ERROR:
+ /* The device could not authenticate the payload. Depending on
+ * where the failure occurred the bytes may have been transformed
+ * (XORed) or left as wire ciphertext. Flag it so that, during a
+ * TLS 1.3 rekey transition, the re-encrypt path undoes the
+ * transform on any XORed frag of a mixed record while software
+ * re-authenticates; a non-mixed record stays wire ciphertext and
+ * is decrypted directly.
+ */
+ skb->decrypt_failed = 1;
+ fallthrough;
+ default: /* CQE_TLS_OFFLOAD_NOT_DECRYPTED: */
stats->tls_err++;
break;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
index 570a912dd6faf..f3f1be1d40343 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
@@ -6,6 +6,7 @@
enum {
MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2 = 0x2,
+ MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_3 = 0x3,
};
enum {
@@ -15,8 +16,10 @@ enum {
#define EXTRACT_INFO_FIELDS do { \
salt = info->salt; \
rec_seq = info->rec_seq; \
+ iv = info->iv; \
salt_sz = sizeof(info->salt); \
rec_seq_sz = sizeof(info->rec_seq); \
+ iv_sz = sizeof(info->iv); \
} while (0)
static void
@@ -24,9 +27,9 @@ fill_static_params(struct mlx5_wqe_tls_static_params_seg *params,
union mlx5e_crypto_info *crypto_info,
u32 key_id, u32 resync_tcp_sn)
{
+ u16 salt_sz, rec_seq_sz, iv_sz;
+ char *salt, *rec_seq, *iv;
char *initial_rn, *gcm_iv;
- u16 salt_sz, rec_seq_sz;
- char *salt, *rec_seq;
u8 tls_version;
u8 *ctx;
@@ -59,7 +62,12 @@ fill_static_params(struct mlx5_wqe_tls_static_params_seg *params,
memcpy(gcm_iv, salt, salt_sz);
memcpy(initial_rn, rec_seq, rec_seq_sz);
- tls_version = MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2;
+ if (crypto_info->crypto_info.version == TLS_1_3_VERSION) {
+ memcpy(gcm_iv + salt_sz, iv, iv_sz);
+ tls_version = MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_3;
+ } else {
+ tls_version = MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2;
+ }
MLX5_SET(tls_static_params, ctx, tls_version, tls_version);
MLX5_SET(tls_static_params, ctx, const_1, 1);
diff --git a/drivers/net/ethernet/netronome/nfp/crypto/tls.c b/drivers/net/ethernet/netronome/nfp/crypto/tls.c
index 9983d7aa2b9cd..13864c6a55dce 100644
--- a/drivers/net/ethernet/netronome/nfp/crypto/tls.c
+++ b/drivers/net/ethernet/netronome/nfp/crypto/tls.c
@@ -287,6 +287,9 @@ nfp_net_tls_add(struct net_device *netdev, struct sock *sk,
BUILD_BUG_ON(offsetof(struct nfp_net_tls_offload_ctx, rx_end) >
TLS_DRIVER_STATE_SIZE_RX);
+ if (crypto_info->version != TLS_1_2_VERSION)
+ return -EOPNOTSUPP;
+
if (!nfp_net_cipher_supported(nn, crypto_info->cipher_type, direction))
return -EOPNOTSUPP;
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 421f6fc454511..5da2c1149d982 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -851,6 +851,10 @@ enum skb_tstamp_type {
* unreadable.
* @dst_pending_confirm: need to confirm neighbour
* @decrypted: Decrypted SKB
+ * @decrypt_failed: hardware could not authenticate this skb's TLS payload.
+ * The payload may have been transformed (XORed) or left as wire
+ * ciphertext, so software must re-authenticate the record and undo the
+ * transform on any XORed fragment before it can be decrypted
* @slow_gro: state present at GRO time, slower prepare step required
* @tstamp_type: When set, skb->tstamp has the
* delivery_time clock base of skb->tstamp.
@@ -1025,6 +1029,7 @@ struct sk_buff {
#endif
#ifdef CONFIG_SKB_DECRYPTED
__u8 decrypted:1;
+ __u8 decrypt_failed:1;
#endif
__u8 slow_gro:1;
#if IS_ENABLED(CONFIG_IP_SCTP)
@@ -1716,6 +1721,7 @@ static inline void skb_copy_decrypted(struct sk_buff *to,
{
#ifdef CONFIG_SKB_DECRYPTED
to->decrypted = from->decrypted;
+ to->decrypt_failed = from->decrypt_failed;
#endif
}
diff --git a/include/net/tcp.h b/include/net/tcp.h
index 5e5f5f9b89a38..8c6d90e962c43 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -2340,6 +2340,15 @@ static inline void tcp_write_collapse_fence(struct sock *sk)
{
struct sk_buff *skb = tcp_write_queue_tail(sk);
+ /* When nothing is queued for transmit, the last skb of the current
+ * state is the rtx queue tail (its end_seq == snd_nxt == write_seq).
+ * Fence that instead, otherwise the boundary is left unmarked and a
+ * later tcp_retrans_try_collapse()/tcp_shift_skb_data() can merge it
+ * with the first skb of the next state across the fence (they only test
+ * the tail's EOR, not skb->decrypted).
+ */
+ if (!skb)
+ skb = tcp_rtx_queue_tail(sk);
if (skb)
TCP_SKB_CB(skb)->eor = 1;
}
diff --git a/include/net/tls.h b/include/net/tls.h
index e57bef58851ea..6844a685d6e08 100644
--- a/include/net/tls.h
+++ b/include/net/tls.h
@@ -155,6 +155,22 @@ struct tls_record_info {
skb_frag_t frags[MAX_SKB_FRAGS];
};
+struct cipher_context {
+ char iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE];
+ char rec_seq[TLS_MAX_REC_SEQ_SIZE];
+};
+
+union tls_crypto_context {
+ struct tls_crypto_info info;
+ union {
+ struct tls12_crypto_info_aes_gcm_128 aes_gcm_128;
+ struct tls12_crypto_info_aes_gcm_256 aes_gcm_256;
+ struct tls12_crypto_info_chacha20_poly1305 chacha20_poly1305;
+ struct tls12_crypto_info_sm4_gcm sm4_gcm;
+ struct tls12_crypto_info_sm4_ccm sm4_ccm;
+ };
+};
+
#define TLS_DRIVER_STATE_SIZE_TX 16
struct tls_offload_context_tx {
struct crypto_aead *aead_send;
@@ -169,6 +185,14 @@ struct tls_offload_context_tx {
void (*sk_destruct)(struct sock *sk);
struct work_struct destruct_work;
struct tls_context *ctx;
+
+ struct {
+ struct tls_sw_context_tx sw; /* SW context for new key */
+ struct cipher_context tx; /* IV, rec_seq for new key */
+ union tls_crypto_context crypto_send; /* Crypto for new key */
+ struct tls_record_info *start_marker;
+ } rekey;
+
/* The TLS layer reserves room for driver specific state
* Currently the belief is that there is not enough
* driver specific state to justify another layer of indirection
@@ -187,28 +211,46 @@ enum tls_context_flags {
* to be atomic.
*/
TLS_TX_SYNC_SCHED = 1,
- /* tls_dev_del was called for the RX side, device state was released,
- * but tls_ctx->netdev might still be kept, because TX-side driver
- * resources might not be released yet. Used to prevent the second
- * tls_dev_del call in tls_device_down if it happens simultaneously.
+ /* tls_dev_del was called for the RX side, releasing the NIC's RX
+ * offload context, while tls_ctx->netdev is still kept (TX-side driver
+ * resources may not be released yet, or a rekey is about to re-add the
+ * context). Set in that case, and during a rekey before re-add, and
+ * cleared when tls_dev_add re-establishes the context. Readers use it to
+ * avoid a second tls_dev_del and to suppress resync while the NIC has no
+ * key. tls_device_down() sets it too, so the rekey paths can test the bit
+ * alone.
*/
TLS_RX_DEV_CLOSED = 2,
-};
-
-struct cipher_context {
- char iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE];
- char rec_seq[TLS_MAX_REC_SEQ_SIZE];
-};
-
-union tls_crypto_context {
- struct tls_crypto_info info;
- union {
- struct tls12_crypto_info_aes_gcm_128 aes_gcm_128;
- struct tls12_crypto_info_aes_gcm_256 aes_gcm_256;
- struct tls12_crypto_info_chacha20_poly1305 chacha20_poly1305;
- struct tls12_crypto_info_sm4_gcm sm4_gcm;
- struct tls12_crypto_info_sm4_ccm sm4_ccm;
- };
+ /* TX HW context has been tls_dev_del()'d (mid-rekey before the re-add,
+ * after a failed re-add, or by tls_device_down()); prevents a second
+ * tls_dev_del. Cleared when tls_dev_add re-establishes the context.
+ */
+ TLS_TX_DEV_CLOSED = 3,
+ /* TX rekey is pending, waiting for old-key data to be ACKed.
+ * While set, new data uses SW path with new key, HW keeps old key
+ * for retransmissions.
+ */
+ TLS_TX_REKEY_PENDING = 4,
+ /* All old-key data has been ACKed, ready to install new key in HW. */
+ TLS_TX_REKEY_READY = 5,
+ /* HW rekey failed; TX stays on the SW rekey context until the next
+ * KeyUpdate re-arms the transition (tls_device_start_rekey()). Also
+ * stops tls_tcp_clean_acked() from re-setting TLS_TX_REKEY_READY.
+ */
+ TLS_TX_REKEY_FAILED = 6,
+ /* A rekey has completed on this socket at least once; that arms
+ * tls_tx_drop_acked_clone() (see its header for the rationale). WARN
+ * avoidance only.
+ */
+ TLS_TX_REKEY_FLOOR = 7,
+ /* The RX side fell back to SW decryption during a rekey (tls_dev_add()
+ * failed, or the netdev is gone) and the socket has been moved from the
+ * TlsCurrRxDevice to the TlsCurrRxSw gauge while rx_conf stays TLS_HW.
+ * Accounting only: the functional state is TLS_RX_DEV_{DEGRADED,CLOSED}.
+ * Cleared, moving the socket back, when a later rekey re-adds the NIC
+ * context. Mirrors TLS_TX_REKEY_FAILED for the close-time decrement.
+ */
+ TLS_RX_REKEY_FAILED = 8,
};
struct tls_prot_info {
@@ -257,6 +299,20 @@ struct tls_context {
*/
unsigned long flags;
+ struct {
+ /* TCP sequence number boundary for pending rekey.
+ * Packets with seq < this use old key, >= use new key.
+ */
+ u32 boundary_seq;
+
+ /* SW encryption contexts for the new key, non-NULL only while
+ * TLS_TX_REKEY_{PENDING,FAILED}; consulted by tls_sw_ctx_tx() and
+ * tls_tx_cipher_ctx().
+ */
+ struct tls_sw_context_tx *sw_ctx;
+ struct cipher_context *cipher_ctx;
+ } rekey;
+
/* cache cold stuff */
struct proto *sk_proto;
struct sock *sk;
@@ -315,6 +371,14 @@ struct tls_offload_context_rx {
u8 resync_nh_reset:1;
/* CORE_NEXT_HINT-only member, but use the hole here */
u8 resync_nh_do_now:1;
+ /* tls_dev_add deferred until old key is freed */
+ u8 dev_add_pending:1;
+ struct {
+ struct crypto_aead *old_aead_recv; /* old key AEAD cipher */
+ char old_iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE]; /* old key IV */
+ char old_rec_seq[TLS_MAX_REC_SEQ_SIZE]; /* old key TLS record seq */
+ u32 old_nic_boundary; /* TCP seq below which the NIC may have used the old key */
+ } rekey;
union {
/* TLS_OFFLOAD_SYNC_TYPE_DRIVER_REQ */
struct {
@@ -356,15 +420,38 @@ tls_validate_xmit_skb(struct sock *sk, struct net_device *dev,
struct sk_buff *
tls_validate_xmit_skb_sw(struct sock *sk, struct net_device *dev,
struct sk_buff *skb);
+struct sk_buff *
+tls_validate_xmit_skb_rekey(struct sock *sk, struct net_device *dev,
+ struct sk_buff *skb);
static inline bool tls_is_skb_tx_device_offloaded(const struct sk_buff *skb)
{
#ifdef CONFIG_TLS_DEVICE
struct sock *sk = skb->sk;
+ typeof(sk->sk_validate_xmit_skb) validate;
+
+ if (!sk || !sk_fullsock(sk))
+ return false;
- return sk && sk_fullsock(sk) &&
- (smp_load_acquire(&sk->sk_validate_xmit_skb) ==
- &tls_validate_xmit_skb);
+ /* Pairs with the smp_store_release() that installs or swaps the
+ * validator (tls_set_device_offload() / tls_device_start_rekey()): the
+ * pointer read here is published together with the offload state it
+ * guards, so a non-NULL validator implies that state is visible.
+ */
+ validate = smp_load_acquire(&sk->sk_validate_xmit_skb);
+ if (likely(validate == &tls_validate_xmit_skb))
+ return true;
+
+ /* A TX rekey (tls_device_start_rekey()) can swap in the rekey validator
+ * between this skb's validate_xmit_skb(), where the old validator
+ * passed it through as HW-offload plaintext, and here. A skb->decrypted
+ * skb under the rekey validator is therefore that straddler: old-key
+ * plaintext whose HW context is still installed (tls_dev_del() runs in
+ * tls_device_complete_rekey() only after a synchronize_net() that drains
+ * this in-flight xmit), so the NIC must still encrypt it. Everything else
+ * the rekey validator emits is ciphertext (skb->decrypted == 0).
+ */
+ return validate == &tls_validate_xmit_skb_rekey && skb_is_decrypted(skb);
#else
return false;
#endif
@@ -389,9 +476,25 @@ static inline struct tls_sw_context_rx *tls_sw_ctx_rx(
static inline struct tls_sw_context_tx *tls_sw_ctx_tx(
const struct tls_context *tls_ctx)
{
+ struct tls_sw_context_tx *rekey_ctx = READ_ONCE(tls_ctx->rekey.sw_ctx);
+
+ if (unlikely(rekey_ctx))
+ return rekey_ctx;
+
return (struct tls_sw_context_tx *)tls_ctx->priv_ctx_tx;
}
+static inline struct cipher_context *tls_tx_cipher_ctx(
+ const struct tls_context *tls_ctx)
+{
+ struct cipher_context *rekey_ctx = READ_ONCE(tls_ctx->rekey.cipher_ctx);
+
+ if (unlikely(rekey_ctx))
+ return rekey_ctx;
+
+ return (struct cipher_context *)&tls_ctx->tx;
+}
+
static inline struct tls_offload_context_tx *
tls_offload_ctx_tx(const struct tls_context *tls_ctx)
{
diff --git a/include/uapi/linux/snmp.h b/include/uapi/linux/snmp.h
index 49f5640092a0d..423aec9ae4cac 100644
--- a/include/uapi/linux/snmp.h
+++ b/include/uapi/linux/snmp.h
@@ -369,6 +369,12 @@ enum
LINUX_MIB_TLSTXREKEYOK, /* TlsTxRekeyOk */
LINUX_MIB_TLSTXREKEYERROR, /* TlsTxRekeyError */
LINUX_MIB_TLSRXREKEYRECEIVED, /* TlsRxRekeyReceived */
+ LINUX_MIB_TLSTXREKEYFALLBACK, /* TlsTxRekeyFallback */
+ LINUX_MIB_TLSRXREKEYFALLBACK, /* TlsRxRekeyFallback */
+ LINUX_MIB_TLSCURRTXREKEY, /* TlsCurrTxRekey */
+ LINUX_MIB_TLSCURRRXREKEY, /* TlsCurrRxRekey */
+ LINUX_MIB_TLSTXREKEYABORTED, /* TlsTxRekeyAborted */
+ LINUX_MIB_TLSRXREKEYABORTED, /* TlsRxRekeyAborted */
__LINUX_MIB_TLSMAX
};
diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index 6f6a6f0d5eb0d..fc8ef0d13f5e7 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -285,6 +285,15 @@ static struct sk_buff *dequeue_skb(struct Qdisc *q, bool *validate,
*validate = false;
if (xfrm_offload(skb))
*validate = true;
+ /* A still-cleartext skb of a crypto-offloaded socket was validated
+ * against that socket's offload state at the time. That state
+ * (sk->sk_validate_xmit_skb) can change while the skb is parked here
+ * e.g. a TLS key update or offload teardown, so re-validate it,
+ * letting the current callback decide how it reaches the wire instead
+ * of emitting now-unencrypted plaintext.
+ */
+ if (skb_is_decrypted(skb))
+ *validate = true;
/* check the reason of requeuing without tx lock first */
txq = skb_get_tx_queue(txq->dev, skb);
if (!netif_xmit_frozen_or_stopped(txq)) {
diff --git a/net/tls/tls.h b/net/tls/tls.h
index 60a37bdaaa250..5d8f4d458df8a 100644
--- a/net/tls/tls.h
+++ b/net/tls/tls.h
@@ -147,13 +147,31 @@ void tls_strp_abort_strp(struct tls_strparser *strp, int err);
int init_prot_info(struct tls_prot_info *prot,
const struct tls_crypto_info *crypto_info,
const struct tls_cipher_desc *cipher_desc);
+/* tls_sw_ctx_init() and tls_sw_ctx_finalize() are two halves of installing
+ * a SW crypto context, split so the device path can attach the NIC between
+ * them. finalize() may only be called after an init() that returned 0, and
+ * both must be called with the same tx and new_crypto_info; on a rekey
+ * (new_crypto_info != NULL) the two must also see the same
+ * new_crypto_info->cipher_type. finalize() commits state and cannot fail,
+ * so violating this leaves the context inconsistent without any error.
+ */
+int tls_sw_ctx_init(struct sock *sk, int tx,
+ struct tls_crypto_info *new_crypto_info);
+void tls_sw_ctx_finalize(struct sock *sk, int tx,
+ struct tls_crypto_info *new_crypto_info);
int tls_set_sw_offload(struct sock *sk, int tx,
struct tls_crypto_info *new_crypto_info);
void tls_update_rx_zc_capable(struct tls_context *tls_ctx);
void tls_sw_strparser_arm(struct sock *sk, struct tls_context *ctx);
void tls_sw_strparser_done(struct tls_context *tls_ctx);
int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size);
+int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size);
+void tls_sw_ctx_tx_init(struct sock *sk, struct tls_sw_context_tx *sw_ctx);
+int tls_sw_drain_tx(struct sock *sk, struct tls_context *ctx, int flags);
+int tls_encrypt_async_wait(struct tls_sw_context_tx *ctx);
+int tls_sw_push_pending_record(struct sock *sk, int flags);
void tls_sw_splice_eof(struct socket *sock);
+void tls_sw_splice_eof_locked(struct socket *sock);
void tls_sw_cancel_work_tx(struct tls_context *tls_ctx);
void tls_sw_release_resources_tx(struct sock *sk);
void tls_sw_free_ctx_tx(struct tls_context *tls_ctx);
@@ -230,10 +248,13 @@ static inline bool tls_strp_msg_mixed_decrypted(struct tls_sw_context_rx *ctx)
#ifdef CONFIG_TLS_DEVICE
int tls_device_init(void);
void tls_device_cleanup(void);
-int tls_set_device_offload(struct sock *sk);
+int tls_set_device_offload(struct sock *sk,
+ struct tls_crypto_info *crypto_info);
void tls_device_free_resources_tx(struct sock *sk);
-int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx);
+int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,
+ struct tls_crypto_info *crypto_info);
void tls_device_offload_cleanup_rx(struct sock *sk);
+void tls_device_rx_del_key(struct sock *sk, struct tls_context *ctx);
void tls_device_rx_resync_new_rec(struct sock *sk, u32 rcd_len, u32 seq);
int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx);
#else
@@ -241,7 +262,7 @@ static inline int tls_device_init(void) { return 0; }
static inline void tls_device_cleanup(void) {}
static inline int
-tls_set_device_offload(struct sock *sk)
+tls_set_device_offload(struct sock *sk, struct tls_crypto_info *crypto_info)
{
return -EOPNOTSUPP;
}
@@ -249,13 +270,16 @@ tls_set_device_offload(struct sock *sk)
static inline void tls_device_free_resources_tx(struct sock *sk) {}
static inline int
-tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
+tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,
+ struct tls_crypto_info *crypto_info)
{
return -EOPNOTSUPP;
}
static inline void tls_device_offload_cleanup_rx(struct sock *sk) {}
static inline void
+tls_device_rx_del_key(struct sock *sk, struct tls_context *ctx) {}
+static inline void
tls_device_rx_resync_new_rec(struct sock *sk, u32 rcd_len, u32 seq) {}
static inline int
diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
index f11d0528fc431..5f45c097bad3c 100644
--- a/net/tls/tls_device.c
+++ b/net/tls/tls_device.c
@@ -57,11 +57,28 @@ static struct page *dummy_page;
static void tls_device_free_ctx(struct tls_context *ctx)
{
- if (ctx->tx_conf == TLS_HW)
- kfree(tls_offload_ctx_tx(ctx));
+ if (ctx->tx_conf == TLS_HW) {
+ struct tls_offload_context_tx *offload_ctx =
+ tls_offload_ctx_tx(ctx);
+
+ kfree(offload_ctx->rekey.start_marker);
+ memzero_explicit(&offload_ctx->rekey,
+ sizeof(offload_ctx->rekey));
+ kfree(offload_ctx);
+ }
+
+ if (ctx->rx_conf == TLS_HW) {
+ struct tls_offload_context_rx *offload_ctx =
+ tls_offload_ctx_rx(ctx);
- if (ctx->rx_conf == TLS_HW)
- kfree(tls_offload_ctx_rx(ctx));
+ /* Normally freed and NULLed in tls_device_offload_cleanup_rx();
+ * free defensively here so a future path can't leak the tfm.
+ */
+ crypto_free_aead(offload_ctx->rekey.old_aead_recv);
+ memzero_explicit(&offload_ctx->rekey,
+ sizeof(offload_ctx->rekey));
+ kfree(offload_ctx);
+ }
tls_ctx_free(NULL, ctx);
}
@@ -79,7 +96,9 @@ static void tls_device_tx_del_task(struct work_struct *work)
netdev = rcu_dereference_protected(ctx->netdev,
!refcount_read(&ctx->refcount));
- netdev->tlsdev_ops->tls_dev_del(netdev, ctx, TLS_OFFLOAD_CTX_DIR_TX);
+ if (!test_bit(TLS_TX_DEV_CLOSED, &ctx->flags))
+ netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
+ TLS_OFFLOAD_CTX_DIR_TX);
dev_put(netdev);
ctx->netdev = NULL;
tls_device_free_ctx(ctx);
@@ -138,6 +157,174 @@ static struct net_device *get_netdev_for_sock(struct sock *sk)
return lowest_dev;
}
+static int tls_device_dev_add_tx(struct sock *sk, struct net_device *netdev,
+ struct tls_crypto_info *crypto_info,
+ u32 write_seq)
+{
+ const struct tls_cipher_desc *cipher_desc;
+ char *rec_seq;
+ int rc;
+
+ cipher_desc = get_cipher_desc(crypto_info->cipher_type);
+ DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
+
+ rc = netdev->tlsdev_ops->tls_dev_add(netdev, sk, TLS_OFFLOAD_CTX_DIR_TX,
+ crypto_info, write_seq);
+ rec_seq = crypto_info_rec_seq(crypto_info, cipher_desc);
+ trace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_TX,
+ write_seq, rec_seq, rc);
+ return rc;
+}
+
+/* Caller controls locking: initial-offload path is lock-free (pre-publish);
+ * rekey path holds offload_ctx->lock.
+ */
+static void tls_device_add_start_marker(struct sock *sk,
+ struct tls_offload_context_tx *offload_ctx,
+ struct tls_record_info *start_marker_record)
+{
+ start_marker_record->end_seq = tcp_sk(sk)->write_seq;
+ start_marker_record->len = 0;
+ start_marker_record->num_frags = 0;
+ list_add_tail_rcu(&start_marker_record->list, &offload_ctx->records_list);
+}
+
+static void tls_device_commit_start_marker(struct sock *sk,
+ struct tls_offload_context_tx *offload_ctx,
+ struct tls_record_info *start_marker_record)
+{
+ tls_device_add_start_marker(sk, offload_ctx, start_marker_record);
+
+ /* TLS offload is greatly simplified if we don't send
+ * SKBs where only part of the payload needs to be encrypted.
+ * So mark the last skb in the write queue as end of record.
+ */
+ tcp_write_collapse_fence(sk);
+}
+
+/* Account a rekey that could not (re)install the RX key on the NIC. The event
+ * counter is bumped every time; the gauges move only on the first fallback
+ * since the socket was last offloaded, so the recurring post-NETDEV_DOWN
+ * rekeys and repeated failed adds do not drift them. The matching move back is
+ * in tls_device_dev_add_rx(); the close-time decrement keys off the bit.
+ */
+static void tls_device_rx_rekey_fallback(struct sock *sk,
+ struct tls_context *tls_ctx)
+{
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYFALLBACK);
+ if (!test_and_set_bit(TLS_RX_REKEY_FAILED, &tls_ctx->flags)) {
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXSW);
+ }
+}
+
+static int tls_device_dev_add_rx(struct sock *sk, struct tls_context *tls_ctx,
+ struct net_device *netdev,
+ struct tls_crypto_info *crypto_info,
+ u32 cur_seq, bool is_rekey)
+{
+ const struct tls_cipher_desc *cipher_desc;
+ char *rec_seq;
+ int rc;
+
+ cipher_desc = get_cipher_desc(crypto_info->cipher_type);
+ DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
+
+ rc = netdev->tlsdev_ops->tls_dev_add(netdev, sk,
+ TLS_OFFLOAD_CTX_DIR_RX,
+ crypto_info, cur_seq);
+ rec_seq = crypto_info_rec_seq(crypto_info, cipher_desc);
+ trace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_RX,
+ cur_seq, rec_seq, rc);
+ if (!rc) {
+ clear_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags);
+ clear_bit(TLS_RX_DEV_CLOSED, &tls_ctx->flags);
+ /* Back on the NIC after an earlier SW fallback: undo its move. */
+ if (test_and_clear_bit(TLS_RX_REKEY_FAILED, &tls_ctx->flags)) {
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXSW);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);
+ }
+ if (is_rekey)
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);
+ } else if (is_rekey) {
+ set_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags);
+ set_bit(TLS_RX_DEV_CLOSED, &tls_ctx->flags);
+ tls_device_rx_rekey_fallback(sk, tls_ctx);
+ }
+ return rc;
+}
+
+static void tls_device_deferred_dev_add_rx(struct sock *sk,
+ struct tls_context *tls_ctx,
+ struct tls_offload_context_rx *ctx,
+ u32 rec_start_seq)
+{
+ const struct tls_cipher_desc *cipher_desc;
+ union tls_crypto_context crypto_ctx;
+ struct net_device *netdev;
+
+ ctx->dev_add_pending = 0;
+
+ /* crypto_recv.info.rec_seq is frozen at the value setsockopt() passed
+ * in: the new key's first record number. The records that drained
+ * between setsockopt() and this boundary crossing were SW-decrypted
+ * under the new key and advanced tls_ctx->rx.rec_seq, so the record
+ * starting at rec_start_seq, the one being decrypted right now,
+ * before tls_rx_one_record() calls tls_advance_record_sn(), is
+ * numbered by rx.rec_seq, not by the blob. Hand the NIC the live
+ * (TCP seq, record number) pair, as getsockopt(TLS_RX) already does.
+ */
+ cipher_desc = get_cipher_desc(tls_ctx->crypto_recv.info.cipher_type);
+ DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
+ crypto_ctx = tls_ctx->crypto_recv;
+ memcpy(crypto_info_rec_seq(&crypto_ctx.info, cipher_desc),
+ tls_ctx->rx.rec_seq, cipher_desc->rec_seq);
+
+ down_read(&device_offload_lock);
+ netdev = rcu_dereference_protected(tls_ctx->netdev,
+ lockdep_is_held(&device_offload_lock));
+ if (netdev)
+ tls_device_dev_add_rx(sk, tls_ctx, netdev,
+ &crypto_ctx.info,
+ rec_start_seq, true);
+ else
+ tls_device_rx_rekey_fallback(sk, tls_ctx);
+ up_read(&device_offload_lock);
+ memzero_explicit(&crypto_ctx, sizeof(crypto_ctx));
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXREKEY);
+}
+
+/* Retire the NIC's RX key when a KeyUpdate record is decoded (from
+ * tls_check_pending_rekey(), lock_sock held). The NIC must lose the old key
+ * now, before it transforms further post-KeyUpdate records that are new-key on
+ * the wire. TLS_RX_DEV_CLOSED is re-tested under device_offload_lock because
+ * tls_device_down() can run in between; synchronize_net() drains the RX path
+ * before the driver frees its context.
+ */
+void tls_device_rx_del_key(struct sock *sk, struct tls_context *ctx)
+{
+ struct net_device *netdev;
+
+ if (ctx->rx_conf != TLS_HW)
+ return;
+ if (test_bit(TLS_RX_DEV_CLOSED, &ctx->flags))
+ return;
+
+ down_read(&device_offload_lock);
+ netdev = rcu_dereference_protected(ctx->netdev,
+ lockdep_is_held(&device_offload_lock));
+ if (!netdev || test_bit(TLS_RX_DEV_CLOSED, &ctx->flags)) {
+ up_read(&device_offload_lock);
+ return;
+ }
+
+ set_bit(TLS_RX_DEV_CLOSED, &ctx->flags);
+ synchronize_net();
+ netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
+ TLS_OFFLOAD_CTX_DIR_RX);
+ up_read(&device_offload_lock);
+}
+
static void destroy_record(struct tls_record_info *record)
{
int i;
@@ -159,6 +346,57 @@ static void delete_all_records(struct tls_offload_context_tx *offload_ctx)
offload_ctx->retransmit_hint = NULL;
}
+static void tls_device_commit_rekey_marker(struct sock *sk,
+ struct tls_offload_context_tx *offload_ctx,
+ struct tls_record_info *start_marker_record)
+{
+ struct tls_record_info *info, *temp;
+ unsigned long flags;
+ __be64 rcd_sn;
+
+ spin_lock_irqsave(&offload_ctx->lock, flags);
+
+ /* The deferred path reaches here with an empty list; the inline
+ * path may still hold the old start marker (never a real record,
+ * since tls_has_unacked_records() was false). Only markers are
+ * ever at the head, so stop at the first non-marker.
+ */
+ list_for_each_entry_safe(info, temp, &offload_ctx->records_list, list) {
+ if (!tls_record_is_start_marker(info))
+ break;
+ list_del(&info->list);
+ destroy_record(info);
+ }
+ offload_ctx->retransmit_hint = NULL;
+
+ memcpy(&rcd_sn, offload_ctx->rekey.tx.rec_seq, sizeof(rcd_sn));
+ offload_ctx->unacked_record_sn = be64_to_cpu(rcd_sn) - 1;
+
+ tls_device_add_start_marker(sk, offload_ctx, start_marker_record);
+
+ spin_unlock_irqrestore(&offload_ctx->lock, flags);
+
+ tcp_write_collapse_fence(sk);
+}
+
+static bool tls_has_unacked_records(struct tls_offload_context_tx *offload_ctx)
+{
+ struct tls_record_info *info;
+ bool has_unacked = false;
+ unsigned long flags;
+
+ spin_lock_irqsave(&offload_ctx->lock, flags);
+ list_for_each_entry(info, &offload_ctx->records_list, list) {
+ if (!tls_record_is_start_marker(info)) {
+ has_unacked = true;
+ break;
+ }
+ }
+ spin_unlock_irqrestore(&offload_ctx->lock, flags);
+
+ return has_unacked;
+}
+
static void tls_tcp_clean_acked(struct sock *sk, u32 acked_seq)
{
struct tls_context *tls_ctx = tls_get_ctx(sk);
@@ -187,6 +425,19 @@ static void tls_tcp_clean_acked(struct sock *sk, u32 acked_seq)
}
ctx->unacked_record_sn += deleted_records;
+
+ /* Once all old-key HW records are ACKed, set REKEY_READY to
+ * let sendmsg know it can finish the rekey and switch back
+ * to HW offload.
+ */
+ if (test_bit(TLS_TX_REKEY_PENDING, &tls_ctx->flags) &&
+ !test_bit(TLS_TX_REKEY_FAILED, &tls_ctx->flags)) {
+ u32 boundary_seq = READ_ONCE(tls_ctx->rekey.boundary_seq);
+
+ if (!before(acked_seq, boundary_seq))
+ set_bit(TLS_TX_REKEY_READY, &tls_ctx->flags);
+ }
+
spin_unlock_irqrestore(&ctx->lock, flags);
}
@@ -217,7 +468,15 @@ void tls_device_free_resources_tx(struct sock *sk)
{
struct tls_context *tls_ctx = tls_get_ctx(sk);
- tls_free_partial_record(sk, tls_ctx);
+ if (unlikely(tls_ctx->rekey.sw_ctx))
+ tls_sw_release_resources_tx(sk);
+ else
+ tls_free_partial_record(sk, tls_ctx);
+
+ if (test_bit(TLS_TX_REKEY_PENDING, &tls_ctx->flags)) {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYABORTED);
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXREKEY);
+ }
}
void tls_offload_tx_resync_request(struct sock *sk, u32 got_seq, u32 exp_seq)
@@ -317,25 +576,34 @@ static void tls_device_record_close(struct sock *sk,
unsigned char record_type)
{
struct tls_prot_info *prot = &ctx->prot_info;
- struct page_frag dummy_tag_frag;
-
- /* append tag
- * device will fill in the tag, we just need to append a placeholder
- * use socket memory to improve coalescing (re-using a single buffer
- * increases frag count)
- * if we can't allocate memory now use the dummy page
+ int tail = prot->tag_size + prot->tail_size;
+
+ /* Append tail: tag for TLS 1.2, content_type + tag for TLS 1.3.
+ * Device fills in the tag, we just need to append a placeholder.
+ * Use socket memory to improve coalescing (re-using a single buffer
+ * increases frag count); if allocation fails use dummy_page
+ * (offset = record_type gives correct content_type byte via
+ * identity mapping)
*/
- if (unlikely(pfrag->size - pfrag->offset < prot->tag_size) &&
- !skb_page_frag_refill(prot->tag_size, pfrag, sk->sk_allocation)) {
- dummy_tag_frag.page = dummy_page;
- dummy_tag_frag.offset = 0;
- pfrag = &dummy_tag_frag;
+ if (unlikely(!pfrag->page || pfrag->size - pfrag->offset < tail) &&
+ !skb_page_frag_refill(tail, pfrag, sk->sk_allocation)) {
+ struct page_frag dummy_pfrag = {
+ .page = dummy_page,
+ .offset = record_type,
+ };
+ tls_append_frag(record, &dummy_pfrag, tail);
+ } else {
+ if (prot->tail_size) {
+ char *content_type_addr = page_address(pfrag->page) +
+ pfrag->offset;
+ *content_type_addr = record_type;
+ }
+ tls_append_frag(record, pfrag, tail);
}
- tls_append_frag(record, pfrag, prot->tag_size);
/* fill prepend */
tls_fill_prepend(ctx, skb_frag_address(&record->frags[0]),
- record->len - prot->overhead_size,
+ record->len - prot->overhead_size + prot->tail_size,
record_type);
}
@@ -418,6 +686,9 @@ static int tls_device_copy_data(void *addr, size_t bytes, struct iov_iter *i)
return 0;
}
+static int tls_device_complete_rekey(struct sock *sk, struct tls_context *ctx,
+ bool deferred, int push_flags);
+
static int tls_push_data(struct sock *sk,
struct iov_iter *iter,
size_t size, int flags,
@@ -563,18 +834,54 @@ static int tls_push_data(struct sock *sk,
return rc;
}
+/* True while TX is routed through the temporary SW rekey context: a rekey is in
+ * progress (PENDING) or has failed and the socket stays pinned to SW (FAILED).
+ */
+static bool tls_device_tx_uses_sw(const struct tls_context *ctx)
+{
+ return test_bit(TLS_TX_REKEY_PENDING, &ctx->flags) ||
+ test_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+}
+
int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)
{
unsigned char record_type = TLS_RECORD_TYPE_DATA;
struct tls_context *tls_ctx = tls_get_ctx(sk);
int rc;
+ /* Reject unsupported flags up front. tls_push_data() enforces the same
+ * set, but during a rekey the send is routed to tls_sw_sendmsg_locked(),
+ * which is the _locked variant and does not re-check; without this,
+ * MSG_ZEROCOPY / MSG_OOB etc. would reach tcp_sendmsg_locked() on the
+ * kernel-owned record pages while PENDING/FAILED.
+ */
+ if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL |
+ MSG_SPLICE_PAGES | MSG_EOR))
+ return -EOPNOTSUPP;
+
if (!tls_ctx->zerocopy_sendfile)
msg->msg_flags &= ~MSG_SPLICE_PAGES;
mutex_lock(&tls_ctx->tx_lock);
lock_sock(sk);
+ /* Old-key records all ACKed; switch back to HW. */
+ if (test_bit(TLS_TX_REKEY_READY, &tls_ctx->flags)) {
+ rc = tls_device_complete_rekey(sk, tls_ctx, true, msg->msg_flags);
+ /* Non-zero here is the transient -EAGAIN retry,
+ * the next sendmsg retries. Hard failures return 0 after
+ * falling back to SW and emit tls_device_complete_rekey_fail
+ * from the fallback path.
+ */
+ if (rc)
+ trace_tls_device_complete_rekey_retry(sk);
+ }
+
+ if (tls_device_tx_uses_sw(tls_ctx)) {
+ rc = tls_sw_sendmsg_locked(sk, msg, size);
+ goto out;
+ }
+
if (unlikely(msg->msg_controllen)) {
rc = tls_process_cmsg(sk, msg, &record_type);
if (rc)
@@ -603,8 +910,10 @@ void tls_device_splice_eof(struct socket *sock)
mutex_lock(&tls_ctx->tx_lock);
lock_sock(sk);
- if (tls_is_partially_sent_record(tls_ctx) ||
- tls_is_pending_open_record(tls_ctx)) {
+ if (tls_device_tx_uses_sw(tls_ctx)) {
+ tls_sw_splice_eof_locked(sock);
+ } else if (tls_is_partially_sent_record(tls_ctx) ||
+ tls_is_pending_open_record(tls_ctx)) {
iov_iter_bvec(&iter, ITER_SOURCE, NULL, 0, 0);
tls_push_data(sk, &iter, 0, 0, TLS_RECORD_TYPE_DATA);
}
@@ -675,14 +984,30 @@ EXPORT_SYMBOL(tls_get_record);
static int tls_device_push_pending_record(struct sock *sk, int flags)
{
+ struct tls_context *tls_ctx = tls_get_ctx(sk);
struct iov_iter iter;
+ if (tls_device_tx_uses_sw(tls_ctx))
+ return tls_sw_push_pending_record(sk, flags);
+
iov_iter_kvec(&iter, ITER_SOURCE, NULL, 0, 0);
return tls_push_data(sk, &iter, 0, flags, TLS_RECORD_TYPE_DATA);
}
void tls_device_write_space(struct sock *sk, struct tls_context *ctx)
{
+ if (tls_device_tx_uses_sw(ctx)) {
+ struct tls_offload_context_tx *offload_ctx;
+ unsigned long flags;
+
+ offload_ctx = tls_offload_ctx_tx(ctx);
+ spin_lock_irqsave(&offload_ctx->lock, flags);
+ if (tls_device_tx_uses_sw(ctx))
+ tls_sw_write_space(sk, ctx);
+ spin_unlock_irqrestore(&offload_ctx->lock, flags);
+ return;
+ }
+
if (tls_is_partially_sent_record(ctx)) {
gfp_t sk_allocation = sk->sk_allocation;
@@ -785,6 +1110,8 @@ void tls_device_rx_resync_new_rec(struct sock *sk, u32 rcd_len, u32 seq)
return;
if (unlikely(test_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags)))
return;
+ if (unlikely(test_bit(TLS_RX_DEV_CLOSED, &tls_ctx->flags)))
+ return;
prot = &tls_ctx->prot_info;
rx_ctx = tls_offload_ctx_rx(tls_ctx);
@@ -886,6 +1213,7 @@ static int
tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
{
struct tls_sw_context_rx *sw_ctx = tls_sw_ctx_rx(tls_ctx);
+ struct tls_prot_info *prot = &tls_ctx->prot_info;
const struct tls_cipher_desc *cipher_desc;
int err, offset, copy, data_len, pos;
struct sk_buff *skb, *skb_iter;
@@ -897,7 +1225,7 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
rxm = strp_msg(tls_strp_msg(sw_ctx));
- orig_buf = kmalloc(rxm->full_len + TLS_HEADER_SIZE + cipher_desc->iv,
+ orig_buf = kmalloc(rxm->full_len + prot->prepend_size,
sk->sk_allocation);
if (!orig_buf)
return -ENOMEM;
@@ -912,9 +1240,8 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
offset = rxm->offset;
sg_init_table(sg, 1);
- sg_set_buf(&sg[0], buf,
- rxm->full_len + TLS_HEADER_SIZE + cipher_desc->iv);
- err = skb_copy_bits(skb, offset, buf, TLS_HEADER_SIZE + cipher_desc->iv);
+ sg_set_buf(&sg[0], buf, rxm->full_len + prot->prepend_size);
+ err = skb_copy_bits(skb, offset, buf, prot->prepend_size);
if (err)
goto free_buf;
@@ -930,7 +1257,7 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
if (skb_pagelen(skb) > offset) {
copy = min_t(int, skb_pagelen(skb) - offset, data_len);
- if (skb->decrypted) {
+ if (skb->decrypted || skb->decrypt_failed) {
err = skb_store_bits(skb, offset, buf, copy);
if (err)
goto free_buf;
@@ -957,7 +1284,7 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
copy = min_t(int, skb_iter->len - frag_pos,
data_len + rxm->offset - offset);
- if (skb_iter->decrypted) {
+ if (skb_iter->decrypted || skb_iter->decrypt_failed) {
err = skb_store_bits(skb_iter, frag_pos, buf, copy);
if (err)
goto free_buf;
@@ -974,6 +1301,77 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
return err;
}
+/*
+ * Reconstruct a boundary record whose frags the NIC XORed with the old key,
+ * then hand it to the SW AEAD under the current (new) key.
+ *
+ * These are deliberately two different keys: the sender has already done its
+ * TX KeyUpdate, so the record on the wire is AEAD-encrypted with the new key,
+ * but the RX NIC still holds the old key and CTR-XORed some frags with the old
+ * keystream. tls_device_reencrypt() must undo that XOR with the *old* key to
+ * restore the pristine new-key ciphertext, so swap the old key in only for the
+ * reconstruction and restore the current key before returning; the SW AEAD
+ * decrypt that follows then runs under the new key, matching the wire record.
+ */
+static int tls_device_reencrypt_old_key(struct sock *sk,
+ struct tls_offload_context_rx *ctx,
+ struct tls_sw_context_rx *sw_ctx,
+ struct tls_context *tls_ctx)
+{
+ struct crypto_aead *saved_aead = sw_ctx->aead_recv;
+ char saved_iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE];
+ char saved_rec_seq[TLS_MAX_REC_SEQ_SIZE];
+ int ret;
+
+ memcpy(saved_iv, tls_ctx->rx.iv, sizeof(saved_iv));
+ memcpy(saved_rec_seq, tls_ctx->rx.rec_seq, sizeof(saved_rec_seq));
+
+ sw_ctx->aead_recv = ctx->rekey.old_aead_recv;
+ memcpy(tls_ctx->rx.iv, ctx->rekey.old_iv, sizeof(ctx->rekey.old_iv));
+ memcpy(tls_ctx->rx.rec_seq, ctx->rekey.old_rec_seq,
+ sizeof(ctx->rekey.old_rec_seq));
+
+ ret = tls_device_reencrypt(sk, tls_ctx);
+
+ memcpy(ctx->rekey.old_rec_seq, tls_ctx->rx.rec_seq,
+ sizeof(ctx->rekey.old_rec_seq));
+
+ sw_ctx->aead_recv = saved_aead;
+ memcpy(tls_ctx->rx.iv, saved_iv, sizeof(saved_iv));
+ memcpy(tls_ctx->rx.rec_seq, saved_rec_seq, sizeof(saved_rec_seq));
+
+ if (ret)
+ return ret;
+
+ tls_bigint_increment(ctx->rekey.old_rec_seq,
+ tls_ctx->prot_info.rec_seq_size);
+ ctx->resync_nh_reset = 1;
+
+ return 0;
+}
+
+/*
+ * TCP sequence of the first byte of the record the strparser currently holds
+ * or is still collecting. In non-copy mode tcp_sk(sk)->copied_seq is left at
+ * the record start until tls_strp_msg_consume(). In copy mode
+ * tls_strp_read_copy() zeroes stm.offset and anchor->len and then
+ * tls_strp_read_copyin() -> tcp_read_sock() advances copied_seq by every byte
+ * it appends to the anchor, a complete parsed-ahead record, a partial one
+ * under rmem pressure, or only header bytes, so subtract anchor->len to get
+ * back to the record start. Both the recv path and the setsockopt rekey path
+ * must classify records against the same start, so share this helper.
+ */
+static u32 tls_device_rx_rec_start(struct sock *sk,
+ struct tls_sw_context_rx *sw_ctx)
+{
+ u32 copied_seq = tcp_sk(sk)->copied_seq;
+
+ if (sw_ctx->strp.copy_mode)
+ return copied_seq - sw_ctx->strp.anchor->len;
+
+ return copied_seq;
+}
+
int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)
{
struct tls_offload_context_rx *ctx = tls_offload_ctx_rx(tls_ctx);
@@ -981,6 +1379,7 @@ int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)
struct sk_buff *skb = tls_strp_msg(sw_ctx);
struct strp_msg *rxm = strp_msg(skb);
int is_decrypted, is_encrypted;
+ u32 rec_start_seq;
if (!tls_strp_msg_mixed_decrypted(sw_ctx)) {
is_decrypted = skb->decrypted;
@@ -990,10 +1389,77 @@ int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)
is_encrypted = 0;
}
- trace_tls_device_decrypted(sk, tcp_sk(sk)->copied_seq - rxm->full_len,
+ rec_start_seq = tls_device_rx_rec_start(sk, sw_ctx);
+
+ trace_tls_device_decrypted(sk, rec_start_seq,
tls_ctx->rx.rec_seq, rxm->full_len,
is_encrypted, is_decrypted);
+ if (unlikely(ctx->rekey.old_aead_recv)) {
+ bool nic_touched = !is_encrypted || skb->decrypt_failed;
+ bool before_nic_boundary;
+
+ /* old_nic_boundary is the TCP stack's view at setsockopt time
+ * (rcv_nxt plus the out-of-order tail), not the NIC's last
+ * transformed byte. A segment the NIC transformed with the old
+ * key before tls_dev_del returned can still be in the RQ/CQ, in
+ * a GRO list or in the socket backlog when that snapshot is
+ * taken and reach TCP later, above it. While old_aead_recv is
+ * held the NIC has no RX context for this socket at all: the
+ * old one was deleted before old_aead_recv was set and the new
+ * one is only installed once it is freed below. So a NIC mark
+ * seen here can only be the old key's transform, wherever the
+ * record sits relative to the snapshot. Slide the boundary out
+ * over such a record instead of retiring the old key on it; the
+ * old key is retired only on a record the NIC never saw.
+ */
+ if (nic_touched &&
+ !before(rec_start_seq, ctx->rekey.old_nic_boundary))
+ ctx->rekey.old_nic_boundary = rec_start_seq + rxm->full_len;
+
+ before_nic_boundary =
+ before(rec_start_seq, ctx->rekey.old_nic_boundary);
+
+ if (before_nic_boundary) {
+ /* Non-mixed (skb->decrypted clear) is untouched wire
+ * ciphertext even if skb->decrypt_failed is set, so advance
+ * old_rec_seq and let the SW AEAD decrypt it directly.
+ * old_rec_seq tracks the stream's record number, which the
+ * NIC also advances for records it did not transform, so
+ * keeping it in step lets a later NIC-touched record be undone
+ * with the right nonce. A mixed record carries NIC-XORed frags
+ * (skb->decrypt_failed or skb->decrypted) and takes the
+ * old-key reencrypt path below, which undoes the transform per
+ * frag before the SW AEAD decrypts.
+ */
+ if (is_encrypted) {
+ tls_bigint_increment(ctx->rekey.old_rec_seq,
+ tls_ctx->prot_info.rec_seq_size);
+ return 0;
+ }
+
+ trace_tls_device_rekey_reencrypt(sk, rec_start_seq,
+ ctx->rekey.old_nic_boundary);
+
+ return tls_device_reencrypt_old_key(sk, ctx,
+ sw_ctx, tls_ctx);
+ }
+
+ trace_tls_device_rekey_done(sk, rec_start_seq,
+ ctx->rekey.old_nic_boundary);
+ crypto_free_aead(ctx->rekey.old_aead_recv);
+ ctx->rekey.old_aead_recv = NULL;
+
+ /* Anchor the NIC on the start of this first post-boundary
+ * record. rec_start_seq already accounts for copy_mode, where
+ * copied_seq has advanced past the record end; using it keeps
+ * the (TCP seq, record number) pair consistent in both modes.
+ */
+ if (ctx->dev_add_pending)
+ tls_device_deferred_dev_add_rx(sk, tls_ctx, ctx,
+ rec_start_seq);
+ }
+
if (unlikely(test_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags))) {
if (likely(is_encrypted || is_decrypted))
return is_decrypted;
@@ -1062,62 +1528,457 @@ static struct tls_offload_context_tx *alloc_offload_ctx_tx(struct tls_context *c
return offload_ctx;
}
-int tls_set_device_offload(struct sock *sk)
+/* Build a fresh AEAD tfm for the rekey with the given key, so it can be
+ * swapped in only on success. Re-keying a live tfm in place is not atomic:
+ * a failed crypto_aead_setkey() leaves it with CRYPTO_TFM_NEED_KEY set,
+ * destroying the previous key. Returns an ERR_PTR() on failure.
+ */
+static struct crypto_aead *tls_device_build_rekey_aead(
+ const struct tls_cipher_desc *cipher_desc,
+ char *key, u32 alg_flags)
{
- struct tls_record_info *start_marker_record;
- struct tls_offload_context_tx *offload_ctx;
+ struct crypto_aead *aead;
+ int rc;
+
+ aead = crypto_alloc_aead(cipher_desc->cipher_name, 0, alg_flags);
+ if (IS_ERR(aead))
+ return aead;
+
+ rc = crypto_aead_setkey(aead, key, cipher_desc->key);
+ if (!rc)
+ rc = crypto_aead_setauthsize(aead, cipher_desc->tag);
+ if (rc) {
+ crypto_free_aead(aead);
+ return ERR_PTR(rc);
+ }
+
+ return aead;
+}
+
+static void tls_device_copy_rekey_iv_seq(
+ struct tls_offload_context_tx *offload_ctx,
+ const struct tls_cipher_desc *cipher_desc,
+ char *salt, char *iv, char *rec_seq)
+{
+ memcpy(offload_ctx->rekey.tx.iv, salt, cipher_desc->salt);
+ memcpy(offload_ctx->rekey.tx.iv + cipher_desc->salt, iv,
+ cipher_desc->iv);
+ memcpy(offload_ctx->rekey.tx.rec_seq, rec_seq, cipher_desc->rec_seq);
+}
+
+static int tls_device_init_rekey_sw(struct sock *sk,
+ struct tls_context *ctx,
+ struct tls_offload_context_tx *offload_ctx,
+ struct tls_crypto_info *new_crypto_info)
+{
+ struct tls_sw_context_tx *sw_ctx = &offload_ctx->rekey.sw;
+ const struct tls_cipher_desc *cipher_desc;
+ char *key;
+ int rc;
+
+ cipher_desc = get_cipher_desc(new_crypto_info->cipher_type);
+ DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
+
+ memset(sw_ctx, 0, sizeof(*sw_ctx));
+ tls_sw_ctx_tx_init(sk, sw_ctx);
+
+ key = crypto_info_key(new_crypto_info, cipher_desc);
+ sw_ctx->aead_send = tls_device_build_rekey_aead(cipher_desc, key, 0);
+ if (IS_ERR(sw_ctx->aead_send)) {
+ rc = PTR_ERR(sw_ctx->aead_send);
+ sw_ctx->aead_send = NULL;
+ return rc;
+ }
+
+ return 0;
+}
+
+static int tls_device_start_rekey(struct sock *sk,
+ struct tls_context *ctx,
+ struct tls_offload_context_tx *offload_ctx,
+ struct tls_crypto_info *new_crypto_info)
+{
+ bool rekey_pending = test_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+ bool rekey_failed = test_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+ const struct tls_cipher_desc *cipher_desc;
+ struct crypto_aead *new_aead, *old_aead;
+ char *key, *iv, *rec_seq, *salt;
+ int push_flags = MSG_NOSIGNAL;
+ unsigned long flags;
+ int rc;
+
+ cipher_desc = get_cipher_desc(new_crypto_info->cipher_type);
+ DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
+
+ key = crypto_info_key(new_crypto_info, cipher_desc);
+ iv = crypto_info_iv(new_crypto_info, cipher_desc);
+ rec_seq = crypto_info_rec_seq(new_crypto_info, cipher_desc);
+ salt = crypto_info_salt(new_crypto_info, cipher_desc);
+
+ /* The record flushes below hand the open/partially sent HW record to
+ * TCP and may have to wait for send buffer space. Honour the socket's
+ * non-blocking mode so an O_NONBLOCK application is not put to sleep
+ * inside setsockopt(): it gets -EAGAIN and retries once the socket is
+ * writable. Kernel sockets (no backing file, e.g. nvme-tcp) keep the
+ * blocking semantics, matching how they call sendmsg().
+ */
+ if (sk->sk_socket && sk->sk_socket->file &&
+ (sk->sk_socket->file->f_flags & O_NONBLOCK))
+ push_flags |= MSG_DONTWAIT;
+
+ if (rekey_pending || rekey_failed) {
+ /* Flush any SW open_record before swapping the key. -EINPROGRESS
+ * means an async AEAD accepted the record for encryption; it is a
+ * success, waited for by tls_encrypt_async_wait() just below (as
+ * tls_process_cmsg()/tls_sw_drain_tx() also treat it).
+ */
+ if (tls_is_pending_open_record(ctx)) {
+ rc = ctx->push_pending_record(sk, push_flags);
+ if (rc < 0 && rc != -EINPROGRESS)
+ return rc;
+ }
+
+ /* Wait for in-flight async encryptions submitted to this tfm
+ * with the previous key before changing it.
+ */
+ rc = tls_encrypt_async_wait(&offload_ctx->rekey.sw);
+ if (rc)
+ return rc;
+
+ /* Build the new key into a fresh tfm and swap it in only on
+ * success; A failed rekey here must leave the SW fallback
+ * path able to encrypt.
+ */
+ new_aead = tls_device_build_rekey_aead(cipher_desc, key, 0);
+ if (IS_ERR(new_aead))
+ return PTR_ERR(new_aead);
+
+ old_aead = offload_ctx->rekey.sw.aead_send;
+ offload_ctx->rekey.sw.aead_send = new_aead;
+ crypto_free_aead(old_aead);
+
+ tls_device_copy_rekey_iv_seq(offload_ctx, cipher_desc,
+ salt, iv, rec_seq);
+
+ if (rekey_failed) {
+ /* Re-arm FAILED -> PENDING under device_offload_lock. The
+ * PENDING set and FAILED clear are two stores to ctx->flags,
+ * and tls_device_down() tests !PENDING && !FAILED as two
+ * separate loads; without the lock those loads could straddle
+ * the flip and see neither bit, letting tls_device_down()
+ * install tls_validate_xmit_skb_sw with PENDING set (dropping
+ * all new-key ciphertext). The lock keeps PENDING || FAILED
+ * observable throughout. Non-blocking, so no NETDEV_DOWN stall.
+ */
+ down_read(&device_offload_lock);
+ spin_lock_irqsave(&offload_ctx->lock, flags);
+ WRITE_ONCE(ctx->rekey.boundary_seq, tcp_sk(sk)->snd_una);
+ set_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+ spin_unlock_irqrestore(&offload_ctx->lock, flags);
+ /* Release pairs with test_bit_acquire() in the validator:
+ * a TX seeing FAILED clear must see the fresh boundary_seq.
+ */
+ clear_bit_unlock(TLS_TX_REKEY_FAILED, &ctx->flags);
+ up_read(&device_offload_lock);
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
+ }
+ } else {
+ /* Drain partially sent record and flush open HW record
+ * before switching to SW.
+ */
+ if (tls_is_partially_sent_record(ctx)) {
+ rc = tls_push_partial_record(sk, ctx,
+ MSG_SENDPAGE_DECRYPTED |
+ push_flags);
+ if (rc < 0)
+ return rc;
+ }
+ if (tls_is_pending_open_record(ctx)) {
+ rc = ctx->push_pending_record(sk, push_flags);
+ if (rc < 0)
+ return rc;
+ }
+
+ rc = tls_device_init_rekey_sw(sk, ctx, offload_ctx,
+ new_crypto_info);
+ if (rc)
+ return rc;
+
+ tls_device_copy_rekey_iv_seq(offload_ctx, cipher_desc,
+ salt, iv, rec_seq);
+
+ /* Publish the rekey under device_offload_lock so that setting
+ * TLS_TX_REKEY_PENDING and installing the rekey validator is
+ * atomic against tls_device_down(), which under down_write() tests
+ * !PENDING and installs tls_validate_xmit_skb_sw. Otherwise the two
+ * validator stores could interleave to leave PENDING set with the
+ * SW validator, and every new-key ciphertext (never on the offload
+ * records_list) would then be dropped by tls_sw_fallback(). The
+ * blocking flush and crypto_alloc above deliberately run WITHOUT
+ * this lock, so a stalled peer cannot hold up NETDEV_DOWN (which
+ * takes down_write() under RTNL) or any other down_read() user.
+ */
+ down_read(&device_offload_lock);
+
+ /* Prevent a partial record straddling the SW/HW boundary. */
+ tcp_write_collapse_fence(sk);
+
+ WRITE_ONCE(ctx->rekey.sw_ctx, &offload_ctx->rekey.sw);
+ WRITE_ONCE(ctx->rekey.cipher_ctx, &offload_ctx->rekey.tx);
+
+ spin_lock_irqsave(&offload_ctx->lock, flags);
+ WRITE_ONCE(ctx->rekey.boundary_seq, tcp_sk(sk)->write_seq);
+ set_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+ spin_unlock_irqrestore(&offload_ctx->lock, flags);
+
+ /* Switch to rekey validator; new sends won't use HW offload */
+ smp_store_release(&sk->sk_validate_xmit_skb,
+ tls_validate_xmit_skb_rekey);
+
+ up_read(&device_offload_lock);
+ }
+
+ unsafe_memcpy(&offload_ctx->rekey.crypto_send.info, new_crypto_info,
+ cipher_desc->crypto_info,
+ /* checked in do_tls_setsockopt_conf */);
+ memzero_explicit(new_crypto_info, cipher_desc->crypto_info);
+
+ return 0;
+}
+
+static int tls_device_complete_rekey(struct sock *sk, struct tls_context *ctx,
+ bool deferred, int push_flags)
+{
+ struct tls_offload_context_tx *offload_ctx = tls_offload_ctx_tx(ctx);
+ struct crypto_aead *new_aead, *old_aead, *old_sw_aead;
const struct tls_cipher_desc *cipher_desc;
- struct tls_crypto_info *crypto_info;
- struct tls_prot_info *prot;
struct net_device *netdev;
- struct tls_context *ctx;
- char *iv, *rec_seq;
+ unsigned long flags;
+ char *key;
int rc;
- ctx = tls_get_ctx(sk);
- prot = &ctx->prot_info;
+ cipher_desc = get_cipher_desc(offload_ctx->rekey.crypto_send.info.cipher_type);
+ DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
- if (ctx->priv_ctx_tx)
- return -EEXIST;
+ DEBUG_NET_WARN_ON_ONCE(!offload_ctx->rekey.start_marker);
- netdev = get_netdev_for_sock(sk);
+ rc = tls_sw_drain_tx(sk, ctx, push_flags);
+ /* -EAGAIN (sndbuf full) and a signal (-EINTR/-ERESTARTSYS from
+ * sk_stream_wait_memory()) are transient: leave the rekey PENDING and
+ * retry on the next sendmsg rather than permanently dropping HW offload.
+ * tls_tx_records() likewise passes these through without aborting.
+ */
+ if (rc == -EAGAIN || rc == -EINTR || rc == -ERESTARTSYS)
+ return rc;
+ if (rc)
+ goto rekey_fallback; /* hard failure: fall back to SW */
+
+ down_read(&device_offload_lock);
+
+ netdev = rcu_dereference_protected(ctx->netdev,
+ lockdep_is_held(&device_offload_lock));
if (!netdev) {
- pr_err_ratelimited("%s: netdev not found\n", __func__);
- return -EINVAL;
+ rc = -ENODEV;
+ goto release_lock;
}
- if (!(netdev->features & NETIF_F_HW_TLS_TX)) {
- rc = -EOPNOTSUPP;
- goto release_netdev;
+ /* Drain in-flight xmit users before tls_dev_del() and before freeing the
+ * old fallback aead_send: (1) under the rekey validator a decrypted
+ * straddler may still be inside the driver on the HW context (same swap ->
+ * synchronize_net -> dev_del order as tls_device_down(), which also keeps a
+ * decrypted skb from reaching a torn-down context); (2) pre-boundary
+ * retransmits routed to tls_sw_fallback() read aead_send locklessly. No new
+ * fallback can start here: every pre-boundary record is ACKed and freed, so
+ * fill_sg_in() bails.
+ */
+ synchronize_net();
+
+ if (!test_bit(TLS_TX_DEV_CLOSED, &ctx->flags)) {
+ netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
+ TLS_OFFLOAD_CTX_DIR_TX);
+ set_bit(TLS_TX_DEV_CLOSED, &ctx->flags);
}
- crypto_info = &ctx->crypto_send.info;
- if (crypto_info->version != TLS_1_2_VERSION) {
- rc = -EOPNOTSUPP;
- goto release_netdev;
+ /* Build the new SW-fallback key into a fresh tfm and swap it in only
+ * on success. Doing this while the HW context is torn down
+ * (TLS_TX_DEV_CLOSED set) means a failure falls into rekey_fallback
+ * with HW off, so the SW fallback is coherent, same as a dev_add
+ * failure.
+ */
+ key = crypto_info_key(&offload_ctx->rekey.crypto_send.info, cipher_desc);
+ new_aead = tls_device_build_rekey_aead(cipher_desc, key, CRYPTO_ALG_ASYNC);
+ if (IS_ERR(new_aead)) {
+ rc = PTR_ERR(new_aead);
+ goto release_lock;
}
- cipher_desc = get_cipher_desc(crypto_info->cipher_type);
- if (!cipher_desc || !cipher_desc->offloadable) {
- rc = -EINVAL;
- goto release_netdev;
+ /* crypto_send.info.rec_seq is frozen at setsockopt time; the SW context
+ * advanced rekey.tx.rec_seq for every record it sent, so hand the NIC the
+ * live record number (mirrors the RX deferred add).
+ */
+ memcpy(crypto_info_rec_seq(&offload_ctx->rekey.crypto_send.info, cipher_desc),
+ offload_ctx->rekey.tx.rec_seq, cipher_desc->rec_seq);
+
+ rc = tls_device_dev_add_tx(sk, netdev, &offload_ctx->rekey.crypto_send.info,
+ tcp_sk(sk)->write_seq);
+ if (rc) {
+ crypto_free_aead(new_aead);
+ goto release_lock;
}
- rc = init_prot_info(prot, crypto_info, cipher_desc);
+ /* Point of no return: HW is live with the new key. Swap in the new
+ * fallback tfm and drop the old one; the remaining steps cannot fail.
+ */
+ old_aead = offload_ctx->aead_send;
+ offload_ctx->aead_send = new_aead;
+ crypto_free_aead(old_aead);
+ clear_bit(TLS_TX_DEV_CLOSED, &ctx->flags);
+
+ memcpy(ctx->tx.iv, offload_ctx->rekey.tx.iv,
+ cipher_desc->salt + cipher_desc->iv);
+ memcpy(ctx->tx.rec_seq, offload_ctx->rekey.tx.rec_seq,
+ cipher_desc->rec_seq);
+ unsafe_memcpy(&ctx->crypto_send.info,
+ &offload_ctx->rekey.crypto_send.info,
+ cipher_desc->crypto_info,
+ /* checked during rekey setup */);
+
+ /* Start marker: the NIC passes through everything before
+ * write_seq untouched (it is already SW-encrypted ciphertext),
+ * same as during initial offload setup. Also drops the stale
+ * marker and rebases unacked_record_sn so the record-sequence
+ * bookkeeping stays consistent on the inline path.
+ */
+ tls_device_commit_rekey_marker(sk, offload_ctx,
+ offload_ctx->rekey.start_marker);
+
+ old_sw_aead = tls_sw_ctx_tx(ctx)->aead_send;
+
+ spin_lock_irqsave(&offload_ctx->lock, flags);
+ clear_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+ clear_bit(TLS_TX_REKEY_READY, &ctx->flags);
+ clear_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+
+ /* Arm the drop floor before restoring the HW validator: from now on
+ * tls_validate_xmit_skb() drops payload retransmits of fully-ACKed data, so
+ * a stale clone whose record was purged here does not reach the NIC and trip
+ * its WARN on the new start marker. The cleartext leak on that path is closed
+ * separately by the skb_is_decrypted() gate in tls_sw_fallback(); this is
+ * only WARN avoidance. Set once; stays set for the socket's life.
+ */
+ set_bit(TLS_TX_REKEY_FLOOR, &ctx->flags);
+
+ /* Switch back to HW offload validator */
+ smp_store_release(&sk->sk_validate_xmit_skb, tls_validate_xmit_skb);
+
+ WRITE_ONCE(ctx->rekey.sw_ctx, NULL);
+ WRITE_ONCE(ctx->rekey.cipher_ctx, NULL);
+ spin_unlock_irqrestore(&offload_ctx->lock, flags);
+
+ memzero_explicit(&offload_ctx->rekey, sizeof(offload_ctx->rekey));
+ crypto_free_aead(old_sw_aead);
+
+ up_read(&device_offload_lock);
+
+ if (deferred)
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXREKEY);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYOK);
+ return 0;
+
+release_lock:
+ up_read(&device_offload_lock);
+
+rekey_fallback:
+ kfree(offload_ctx->rekey.start_marker);
+ offload_ctx->rekey.start_marker = NULL;
+ spin_lock_irqsave(&offload_ctx->lock, flags);
+ set_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+ clear_bit(TLS_TX_REKEY_READY, &ctx->flags);
+ clear_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+ spin_unlock_irqrestore(&offload_ctx->lock, flags);
+ if (deferred)
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXREKEY);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYFALLBACK);
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
+
+ /* Hard failure: HW rekey gave up and the connection is now pinned to
+ * SW encryption. The call site only sees the transient -EAGAIN retry
+ * (rc is not propagated here), so emit the trace from the fallback
+ * path itself; rc still holds the originating error.
+ */
+ trace_tls_device_complete_rekey_fail(sk, rc);
+
+ return 0;
+}
+
+static int tls_set_device_offload_rekey(struct sock *sk,
+ struct tls_context *ctx,
+ struct tls_crypto_info *new_crypto_info)
+{
+ struct tls_offload_context_tx *offload_ctx = tls_offload_ctx_tx(ctx);
+ bool rekey_pending = test_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+ bool rekey_failed = test_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+ bool defer = true;
+ int rc;
+
+ /* Defer the switch back to HW until any in-flight old-key records are
+ * ACKed. A partially_sent_record needs no separate check: its record is
+ * on records_list before it is sent (tls_push_record()) and stays there
+ * until ACKed, so tls_has_unacked_records() already covers it.
+ */
+ if (!rekey_pending && !rekey_failed)
+ defer = tls_has_unacked_records(offload_ctx) ||
+ tls_is_pending_open_record(ctx);
+
+ if (!offload_ctx->rekey.start_marker) {
+ offload_ctx->rekey.start_marker =
+ kmalloc_obj(*offload_ctx->rekey.start_marker);
+ if (!offload_ctx->rekey.start_marker)
+ return -ENOMEM;
+ }
+
+ rc = tls_device_start_rekey(sk, ctx, offload_ctx, new_crypto_info);
if (rc)
- goto release_netdev;
+ return rc;
+
+ if (defer) {
+ if (!rekey_pending)
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXREKEY);
+ else
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYOK);
+ return 0;
+ }
+
+ return tls_device_complete_rekey(sk, ctx, false, 0);
+}
+
+static int tls_set_device_offload_initial(struct sock *sk,
+ struct tls_context *ctx,
+ struct net_device *netdev,
+ struct tls_crypto_info *crypto_info,
+ const struct tls_cipher_desc *cipher_desc)
+{
+ struct tls_prot_info *prot = &ctx->prot_info;
+ struct tls_record_info *start_marker_record;
+ struct tls_offload_context_tx *offload_ctx;
+ char *iv, *rec_seq;
+ int rc;
iv = crypto_info_iv(crypto_info, cipher_desc);
rec_seq = crypto_info_rec_seq(crypto_info, cipher_desc);
+ rc = init_prot_info(prot, crypto_info, cipher_desc);
+ if (rc)
+ return rc;
+
memcpy(ctx->tx.iv + cipher_desc->salt, iv, cipher_desc->iv);
memcpy(ctx->tx.rec_seq, rec_seq, cipher_desc->rec_seq);
start_marker_record = kmalloc_obj(*start_marker_record);
- if (!start_marker_record) {
- rc = -ENOMEM;
- goto release_netdev;
- }
+ if (!start_marker_record)
+ return -ENOMEM;
offload_ctx = alloc_offload_ctx_tx(ctx);
if (!offload_ctx) {
@@ -1129,20 +1990,11 @@ int tls_set_device_offload(struct sock *sk)
if (rc)
goto free_offload_ctx;
- start_marker_record->end_seq = tcp_sk(sk)->write_seq;
- start_marker_record->len = 0;
- start_marker_record->num_frags = 0;
- list_add_tail(&start_marker_record->list, &offload_ctx->records_list);
+ tls_device_commit_start_marker(sk, offload_ctx, start_marker_record);
clean_acked_data_enable(tcp_sk(sk), &tls_tcp_clean_acked);
ctx->push_pending_record = tls_device_push_pending_record;
- /* TLS offload is greatly simplified if we don't send
- * SKBs where only part of the payload needs to be encrypted.
- * So mark the last skb in the write queue as end of record.
- */
- tcp_write_collapse_fence(sk);
-
/* Avoid offloading if the device is down
* We don't want to offload new flows after
* the NETDEV_DOWN event
@@ -1158,11 +2010,8 @@ int tls_set_device_offload(struct sock *sk)
}
ctx->priv_ctx_tx = offload_ctx;
- rc = netdev->tlsdev_ops->tls_dev_add(netdev, sk, TLS_OFFLOAD_CTX_DIR_TX,
- &ctx->crypto_send.info,
- tcp_sk(sk)->write_seq);
- trace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_TX,
- tcp_sk(sk)->write_seq, rec_seq, rc);
+ rc = tls_device_dev_add_tx(sk, netdev, crypto_info,
+ tcp_sk(sk)->write_seq);
if (rc)
goto release_lock;
@@ -1174,7 +2023,6 @@ int tls_set_device_offload(struct sock *sk)
* by the netdev's xmit function.
*/
smp_store_release(&sk->sk_validate_xmit_skb, tls_validate_xmit_skb);
- dev_put(netdev);
return 0;
@@ -1187,20 +2035,44 @@ int tls_set_device_offload(struct sock *sk)
ctx->priv_ctx_tx = NULL;
free_marker_record:
kfree(start_marker_record);
-release_netdev:
- dev_put(netdev);
return rc;
}
-int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
+int tls_set_device_offload(struct sock *sk,
+ struct tls_crypto_info *new_crypto_info)
{
- struct tls12_crypto_info_aes_gcm_128 *info;
- struct tls_offload_context_rx *context;
+ struct tls_crypto_info *crypto_info, *src_crypto_info;
+ const struct tls_cipher_desc *cipher_desc;
struct net_device *netdev;
- int rc = 0;
+ struct tls_context *ctx;
+ int rc;
- if (ctx->crypto_recv.info.version != TLS_1_2_VERSION)
- return -EOPNOTSUPP;
+ ctx = tls_get_ctx(sk);
+
+ /* A rekey of a SW-offloaded socket belongs to tls_set_sw_offload(). */
+ if (new_crypto_info && ctx->tx_conf != TLS_HW)
+ return -EINVAL;
+
+ crypto_info = &ctx->crypto_send.info;
+ src_crypto_info = new_crypto_info ?: crypto_info;
+ cipher_desc = get_cipher_desc(src_crypto_info->cipher_type);
+ if (!cipher_desc || !cipher_desc->offloadable)
+ return -EINVAL;
+
+ /* A rekey targets the device already holding the HW TX context
+ * (ctx->netdev), which can differ from the socket's current route after
+ * a route change or bond/team failover; tls_set_device_offload_rekey()
+ * and tls_device_complete_rekey() resolve it from ctx->netdev under
+ * device_offload_lock. Only the initial install needs the route device.
+ */
+ if (new_crypto_info)
+ return tls_set_device_offload_rekey(sk, ctx, src_crypto_info);
+
+ /* Initial install: a HW TX context must not already exist, otherwise
+ * alloc_offload_ctx_tx() below would silently overwrite it.
+ */
+ if (ctx->priv_ctx_tx)
+ return -EEXIST;
netdev = get_netdev_for_sock(sk);
if (!netdev) {
@@ -1208,50 +2080,249 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
return -EINVAL;
}
- if (!(netdev->features & NETIF_F_HW_TLS_RX)) {
+ if (!(netdev->features & NETIF_F_HW_TLS_TX)) {
rc = -EOPNOTSUPP;
goto release_netdev;
}
- /* Avoid offloading if the device is down
- * We don't want to offload new flows after
- * the NETDEV_DOWN event
- *
- * device_offload_lock is taken in tls_devices's NETDEV_DOWN
- * handler thus protecting from the device going down before
- * ctx was added to tls_device_list.
- */
- down_read(&device_offload_lock);
- if (!(netdev->flags & IFF_UP)) {
- rc = -EINVAL;
- goto release_lock;
+ rc = tls_set_device_offload_initial(sk, ctx, netdev, src_crypto_info,
+ cipher_desc);
+
+release_netdev:
+ dev_put(netdev);
+ return rc;
+}
+
+int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,
+ struct tls_crypto_info *new_crypto_info)
+{
+ struct tls_crypto_info *crypto_info, *src_crypto_info;
+ const struct tls_cipher_desc *cipher_desc;
+ u32 drain_start = tcp_sk(sk)->copied_seq;
+ struct tls_offload_context_rx *context;
+ struct net_device *netdev;
+ bool was_dev_add_pending;
+ bool moved_aead_recv = false;
+ bool retired_pending = false;
+ bool put_netdev = false;
+ int rc = 0;
+
+ /* A rekey of a SW-offloaded socket belongs to tls_set_sw_offload(). */
+ if (new_crypto_info && ctx->rx_conf != TLS_HW)
+ return -EINVAL;
+
+ crypto_info = &ctx->crypto_recv.info;
+ src_crypto_info = new_crypto_info ?: crypto_info;
+ cipher_desc = get_cipher_desc(src_crypto_info->cipher_type);
+ if (!cipher_desc || !cipher_desc->offloadable)
+ return -EINVAL;
+
+ if (new_crypto_info) {
+ /* Rekey targets the device holding the HW RX context, which
+ * can differ from the socket's route after a route change or
+ * bond/team failover. Resolve it from ctx->netdev under
+ * device_offload_lock, like the other del/add-key paths, not
+ * via get_netdev_for_sock(). The context owns the reference,
+ * so don't take an extra one here.
+ *
+ * A NULL netdev means tls_device_down() already ran: the HW RX
+ * context is deleted, TLS_RX_DEV_{DEGRADED,CLOSED} are set and
+ * every record is decrypted in SW, but rx_conf stays TLS_HW.
+ * The rekey is still required, the peer's KeyUpdate was parsed
+ * and recvmsg() returns -EKEYEXPIRED until the new key lands,
+ * so run the same state machine (queued records may still carry
+ * the deleted NIC context's old-key XOR) and account the new key
+ * as a SW fallback in place of the tls_dev_del()/tls_dev_add()
+ * steps, mirroring the TX side (tls_device_complete_rekey()).
+ * Do not fail the setsockopt.
+ */
+ down_read(&device_offload_lock);
+ netdev = rcu_dereference_protected(ctx->netdev,
+ lockdep_is_held(&device_offload_lock));
+ } else {
+ netdev = get_netdev_for_sock(sk);
+ if (!netdev) {
+ pr_err_ratelimited("%s: netdev not found\n", __func__);
+ return -EINVAL;
+ }
+ put_netdev = true;
+
+ if (!(netdev->features & NETIF_F_HW_TLS_RX)) {
+ rc = -EOPNOTSUPP;
+ goto release_netdev;
+ }
+
+ /* Avoid offloading if the device is down
+ * We don't want to offload new flows after
+ * the NETDEV_DOWN event
+ *
+ * device_offload_lock is taken in tls_devices's NETDEV_DOWN
+ * handler thus protecting from the device going down before
+ * ctx was added to tls_device_list.
+ */
+ down_read(&device_offload_lock);
+ if (!(netdev->flags & IFF_UP)) {
+ rc = -EINVAL;
+ goto release_lock;
+ }
}
- context = kzalloc_obj(*context);
- if (!context) {
- rc = -ENOMEM;
- goto release_lock;
+ if (!new_crypto_info) {
+ context = kzalloc_obj(*context);
+ if (!context) {
+ rc = -ENOMEM;
+ goto release_lock;
+ }
+ ctx->priv_ctx_rx = context;
+ } else {
+ context = tls_offload_ctx_rx(ctx);
}
+ was_dev_add_pending = context->dev_add_pending;
context->resync_nh_reset = 1;
- ctx->priv_ctx_rx = context;
- rc = tls_set_sw_offload(sk, 0, NULL);
+ if (new_crypto_info) {
+ struct tls_sw_context_rx *sw_ctx = tls_sw_ctx_rx(ctx);
+
+ /* Classify against the record start, not the raw copied_seq: in
+ * strparser copy mode tcp_read_sock() has already advanced
+ * copied_seq past a parsed-ahead (possibly partial) record the
+ * user has not received, which may still carry the old NIC key's
+ * XOR. tls_device_decrypted() compensates the same way; keeping
+ * both in sync is what lets a drained-vs-still-draining decision
+ * here match the reencrypt-key decision there.
+ */
+ drain_start = tls_device_rx_rec_start(sk, sw_ctx);
+
+ /* netdev is NULL only after tls_device_down(), which already
+ * deleted the HW RX context and set TLS_RX_DEV_CLOSED; the
+ * netdev check just makes that dependency explicit.
+ */
+ if (netdev && !test_bit(TLS_RX_DEV_CLOSED, &ctx->flags)) {
+ set_bit(TLS_RX_DEV_CLOSED, &ctx->flags);
+ synchronize_net();
+ netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
+ TLS_OFFLOAD_CTX_DIR_RX);
+ }
+
+ if (context->rekey.old_aead_recv &&
+ before(drain_start, context->rekey.old_nic_boundary)) {
+ /* Previous rekey still draining. Keep rekey.old_aead_recv,
+ * it is the only key that can undo the NIC-XOR on queued
+ * records. sw_ctx->aead_recv may be re-setkey'd by
+ * tls_sw_ctx_init(); that intermediate key was never on
+ * the NIC and its wire era is drained, so it is needed
+ * for neither undo nor AEAD. Defer dev_add; the new key
+ * is installed once drain_start crosses rekey.old_nic_boundary.
+ */
+ context->dev_add_pending = 1;
+ trace_tls_device_rekey_start(sk, drain_start,
+ context->rekey.old_nic_boundary,
+ true);
+ } else {
+ struct tcp_sock *tp = tcp_sk(sk);
+ u32 nic_end;
+
+ if (context->rekey.old_aead_recv) {
+ /* Prior rekey's era already drained (drain_start is
+ * past old_nic_boundary), so retiring its key here
+ * is a boundary crossing, same as the free in
+ * tls_device_decrypted(); mark it done.
+ */
+ trace_tls_device_rekey_done(sk, drain_start,
+ context->rekey.old_nic_boundary);
+ crypto_free_aead(context->rekey.old_aead_recv);
+ context->rekey.old_aead_recv = NULL;
+ }
+
+ /* Flush the backlog so TCP's view is current, then take the
+ * highest byte TCP holds, including the out-of-order tail:
+ * a NIC-transformed segment behind a host-side drop sits
+ * above rcv_nxt until the retransmit fills the hole and
+ * must still be classified against the old key. This is
+ * still only the stack's view, a transformed segment the
+ * NIC has not delivered yet is caught in-band by
+ * tls_device_decrypted(), which slides the boundary.
+ */
+ __sk_flush_backlog(sk);
+ nic_end = tp->rcv_nxt;
+ if (!RB_EMPTY_ROOT(&tp->out_of_order_queue) &&
+ after(TCP_SKB_CB(tp->ooo_last_skb)->end_seq, nic_end))
+ nic_end = TCP_SKB_CB(tp->ooo_last_skb)->end_seq;
+
+ if (before(drain_start, nic_end)) {
+ context->rekey.old_aead_recv = sw_ctx->aead_recv;
+ /* NULL so tls_sw_ctx_init() allocates a fresh tfm
+ * for the new key instead of re-keying the one we
+ * must keep for the drain.
+ */
+ sw_ctx->aead_recv = NULL;
+ moved_aead_recv = true;
+ memcpy(context->rekey.old_iv, ctx->rx.iv,
+ sizeof(context->rekey.old_iv));
+ memcpy(context->rekey.old_rec_seq, ctx->rx.rec_seq,
+ sizeof(context->rekey.old_rec_seq));
+ context->rekey.old_nic_boundary = nic_end;
+ context->dev_add_pending = 1;
+ } else if (was_dev_add_pending) {
+ /* A prior rekey's deferred dev_add can no longer
+ * run: its trigger (old_aead_recv) was just freed
+ * above and no new drain replaces it. Its era
+ * drained successfully (drain_start is already past
+ * old_nic_boundary), so retire it and let the new
+ * key install immediately below. retired_pending
+ * defers its OK/gauge accounting to the post-init
+ * block, past the error goto, so a failed
+ * tls_sw_ctx_init() needs no counter undo.
+ */
+ context->dev_add_pending = 0;
+ retired_pending = true;
+ }
+ trace_tls_device_rekey_start(sk, drain_start, nic_end,
+ before(drain_start, nic_end));
+ }
+ }
+
+ rc = tls_sw_ctx_init(sk, 0, new_crypto_info);
if (rc)
goto release_ctx;
- rc = netdev->tlsdev_ops->tls_dev_add(netdev, sk, TLS_OFFLOAD_CTX_DIR_RX,
- &ctx->crypto_recv.info,
- tcp_sk(sk)->copied_seq);
- info = (void *)&ctx->crypto_recv.info;
- trace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_RX,
- tcp_sk(sk)->copied_seq, info->rec_seq, rc);
- if (rc)
- goto free_sw_resources;
+ if (!context->dev_add_pending) {
+ if (retired_pending) {
+ /* Account the superseded rekey that drained OK, mirroring
+ * the deferred-add path: one RXREKEYOK and release its
+ * in-flight gauge. The new key's own OK/FALLBACK is counted
+ * by tls_device_dev_add_rx() just below.
+ */
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXREKEY);
+ }
+ if (netdev) {
+ rc = tls_device_dev_add_rx(sk, ctx, netdev,
+ src_crypto_info, drain_start,
+ !!new_crypto_info);
+ } else {
+ /* No device after tls_device_down(); the SW path keeps
+ * decrypting.
+ */
+ tls_device_rx_rekey_fallback(sk, ctx);
+ }
+ if (!new_crypto_info) {
+ if (rc)
+ goto free_sw_resources;
+ tls_device_attach(ctx, sk, netdev);
+ }
+ } else if (!was_dev_add_pending) {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXREKEY);
+ } else {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);
+ }
+
+ tls_sw_ctx_finalize(sk, 0, new_crypto_info);
- tls_device_attach(ctx, sk, netdev);
up_read(&device_offload_lock);
- dev_put(netdev);
+ if (put_netdev)
+ dev_put(netdev);
return 0;
@@ -1260,17 +2331,39 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
tls_sw_free_resources_rx(sk);
down_read(&device_offload_lock);
release_ctx:
- ctx->priv_ctx_rx = NULL;
+ if (!new_crypto_info) {
+ ctx->priv_ctx_rx = NULL;
+ } else {
+ /* A failed RX rekey is terminal, so there is no HW state to roll
+ * back to. KeyUpdate is directional and the peer's TX has already
+ * switched keys, so once the new RX key fails to install the old
+ * SW key restored below cannot decrypt any further record; the
+ * socket is dead and the app must close it. The half-torn HW
+ * context (tls_dev_del already ran) and any dangling
+ * dev_add_pending / old_aead_recv are reclaimed by
+ * tls_device_offload_cleanup_rx() on close.
+ */
+ context->dev_add_pending = was_dev_add_pending;
+ if (moved_aead_recv) {
+ struct tls_sw_context_rx *sw_ctx = tls_sw_ctx_rx(ctx);
+
+ crypto_free_aead(sw_ctx->aead_recv);
+ sw_ctx->aead_recv = context->rekey.old_aead_recv;
+ context->rekey.old_aead_recv = NULL;
+ }
+ }
release_lock:
up_read(&device_offload_lock);
release_netdev:
- dev_put(netdev);
+ if (put_netdev)
+ dev_put(netdev);
return rc;
}
void tls_device_offload_cleanup_rx(struct sock *sk)
{
struct tls_context *tls_ctx = tls_get_ctx(sk);
+ struct tls_offload_context_rx *rx_ctx;
struct net_device *netdev;
down_read(&device_offload_lock);
@@ -1279,8 +2372,9 @@ void tls_device_offload_cleanup_rx(struct sock *sk)
if (!netdev)
goto out;
- netdev->tlsdev_ops->tls_dev_del(netdev, tls_ctx,
- TLS_OFFLOAD_CTX_DIR_RX);
+ if (!test_bit(TLS_RX_DEV_CLOSED, &tls_ctx->flags))
+ netdev->tlsdev_ops->tls_dev_del(netdev, tls_ctx,
+ TLS_OFFLOAD_CTX_DIR_RX);
if (tls_ctx->tx_conf != TLS_HW) {
dev_put(netdev);
@@ -1290,6 +2384,19 @@ void tls_device_offload_cleanup_rx(struct sock *sk)
}
out:
up_read(&device_offload_lock);
+
+ rx_ctx = tls_offload_ctx_rx(tls_ctx);
+ if (rx_ctx && rx_ctx->rekey.old_aead_recv) {
+ crypto_free_aead(rx_ctx->rekey.old_aead_recv);
+ rx_ctx->rekey.old_aead_recv = NULL;
+ }
+
+ if (rx_ctx && rx_ctx->dev_add_pending) {
+ rx_ctx->dev_add_pending = 0;
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYABORTED);
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXREKEY);
+ }
+
tls_sw_release_resources_rx(sk);
}
@@ -1317,10 +2424,16 @@ static int tls_device_down(struct net_device *netdev)
spin_unlock_irqrestore(&tls_device_lock, flags);
list_for_each_entry_safe(ctx, tmp, &list, list) {
- /* Stop offloaded TX and switch to the fallback.
- * tls_is_skb_tx_device_offloaded will return false.
+ /* Stop offloaded TX and switch to the fallback. For a socket not
+ * mid-rekey, tls_is_skb_tx_device_offloaded() then returns false; a
+ * PENDING/FAILED socket keeps the rekey validator (under which only a
+ * decrypted straddler still offloads), and the synchronize_net()
+ * below drains any such in-flight skb before tls_dev_del().
*/
- WRITE_ONCE(ctx->sk->sk_validate_xmit_skb, tls_validate_xmit_skb_sw);
+ if (!test_bit(TLS_TX_REKEY_PENDING, &ctx->flags) &&
+ !test_bit(TLS_TX_REKEY_FAILED, &ctx->flags))
+ WRITE_ONCE(ctx->sk->sk_validate_xmit_skb,
+ tls_validate_xmit_skb_sw);
/* Stop the RX and TX resync.
* tls_dev_resync must not be called after tls_dev_del.
@@ -1337,13 +2450,18 @@ static int tls_device_down(struct net_device *netdev)
synchronize_net();
/* Release the offload context on the driver side. */
- if (ctx->tx_conf == TLS_HW)
+ if (ctx->tx_conf == TLS_HW &&
+ !test_bit(TLS_TX_DEV_CLOSED, &ctx->flags)) {
netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
TLS_OFFLOAD_CTX_DIR_TX);
+ set_bit(TLS_TX_DEV_CLOSED, &ctx->flags);
+ }
if (ctx->rx_conf == TLS_HW &&
- !test_bit(TLS_RX_DEV_CLOSED, &ctx->flags))
+ !test_bit(TLS_RX_DEV_CLOSED, &ctx->flags)) {
netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
TLS_OFFLOAD_CTX_DIR_RX);
+ set_bit(TLS_RX_DEV_CLOSED, &ctx->flags);
+ }
dev_put(netdev);
@@ -1411,12 +2529,27 @@ static struct notifier_block tls_dev_notifier = {
int __init tls_device_init(void)
{
- int err;
+ unsigned char *page_addr;
+ int err, i;
- dummy_page = alloc_page(GFP_KERNEL);
+ dummy_page = alloc_page(GFP_KERNEL | __GFP_ZERO);
if (!dummy_page)
return -ENOMEM;
+ /* Pre-populate the first 256 bytes with an identity map so that,
+ * when this page is used as the tail-frag fallback (allocation
+ * failure in tls_device_record_close()), dummy_page[record_type]
+ * yields the correct TLS 1.3 content_type byte for any record_type
+ * without runtime validation.
+ *
+ * A high record_type pushes the tag placeholder past the identity
+ * map, so __GFP_ZERO is what keeps tag-placeholder bytes defined
+ * rather than exposing uninitialized page contents.
+ */
+ page_addr = page_address(dummy_page);
+ for (i = 0; i < 256; i++)
+ page_addr[i] = (unsigned char)i;
+
destruct_wq = alloc_workqueue("ktls_device_destruct", WQ_PERCPU, 0);
if (!destruct_wq) {
err = -ENOMEM;
diff --git a/net/tls/tls_device_fallback.c b/net/tls/tls_device_fallback.c
index 3b7d0ab2bcf17..f2a0ae827bb2a 100644
--- a/net/tls/tls_device_fallback.c
+++ b/net/tls/tls_device_fallback.c
@@ -37,14 +37,15 @@
#include "tls.h"
-static int tls_enc_record(struct aead_request *aead_req,
+static int tls_enc_record(struct tls_context *tls_ctx,
+ struct aead_request *aead_req,
struct crypto_aead *aead, char *aad,
char *iv, __be64 rcd_sn,
struct scatter_walk *in,
- struct scatter_walk *out, int *in_len,
- struct tls_prot_info *prot)
+ struct scatter_walk *out, int *in_len)
{
unsigned char buf[TLS_HEADER_SIZE + TLS_MAX_IV_SIZE];
+ struct tls_prot_info *prot = &tls_ctx->prot_info;
const struct tls_cipher_desc *cipher_desc;
struct scatterlist sg_in[3];
struct scatterlist sg_out[3];
@@ -55,7 +56,7 @@ static int tls_enc_record(struct aead_request *aead_req,
cipher_desc = get_cipher_desc(prot->cipher_type);
DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
- buf_size = TLS_HEADER_SIZE + cipher_desc->iv;
+ buf_size = prot->prepend_size;
len = min_t(int, *in_len, buf_size);
memcpy_from_scatterwalk(buf, in, len);
@@ -66,16 +67,27 @@ static int tls_enc_record(struct aead_request *aead_req,
return 0;
len = buf[4] | (buf[3] << 8);
- len -= cipher_desc->iv;
+ if (prot->version != TLS_1_3_VERSION)
+ len -= cipher_desc->iv;
tls_make_aad(aad, len - cipher_desc->tag, (char *)&rcd_sn, buf[0], prot);
- memcpy(iv + cipher_desc->salt, buf + TLS_HEADER_SIZE, cipher_desc->iv);
+ if (prot->version == TLS_1_3_VERSION) {
+ void *iv_src = crypto_info_iv(&tls_ctx->crypto_send.info,
+ cipher_desc);
+
+ memcpy(iv + cipher_desc->salt, iv_src, cipher_desc->iv);
+ } else {
+ memcpy(iv + cipher_desc->salt, buf + TLS_HEADER_SIZE,
+ cipher_desc->iv);
+ }
+
+ tls_xor_iv_with_seq(prot, iv, (char *)&rcd_sn);
sg_init_table(sg_in, ARRAY_SIZE(sg_in));
sg_init_table(sg_out, ARRAY_SIZE(sg_out));
- sg_set_buf(sg_in, aad, TLS_AAD_SPACE_SIZE);
- sg_set_buf(sg_out, aad, TLS_AAD_SPACE_SIZE);
+ sg_set_buf(sg_in, aad, prot->aad_size);
+ sg_set_buf(sg_out, aad, prot->aad_size);
scatterwalk_get_sglist(in, sg_in + 1);
scatterwalk_get_sglist(out, sg_out + 1);
@@ -108,13 +120,6 @@ static int tls_enc_record(struct aead_request *aead_req,
return rc;
}
-static void tls_init_aead_request(struct aead_request *aead_req,
- struct crypto_aead *aead)
-{
- aead_request_set_tfm(aead_req, aead);
- aead_request_set_ad(aead_req, TLS_AAD_SPACE_SIZE);
-}
-
static struct aead_request *tls_alloc_aead_request(struct crypto_aead *aead,
gfp_t flags)
{
@@ -124,14 +129,15 @@ static struct aead_request *tls_alloc_aead_request(struct crypto_aead *aead,
aead_req = kzalloc(req_size, flags);
if (aead_req)
- tls_init_aead_request(aead_req, aead);
+ aead_request_set_tfm(aead_req, aead);
return aead_req;
}
-static int tls_enc_records(struct aead_request *aead_req,
+static int tls_enc_records(struct tls_context *tls_ctx,
+ struct aead_request *aead_req,
struct crypto_aead *aead, struct scatterlist *sg_in,
struct scatterlist *sg_out, char *aad, char *iv,
- u64 rcd_sn, int len, struct tls_prot_info *prot)
+ u64 rcd_sn, int len)
{
struct scatter_walk out, in;
int rc;
@@ -140,8 +146,8 @@ static int tls_enc_records(struct aead_request *aead_req,
scatterwalk_start(&out, sg_out);
do {
- rc = tls_enc_record(aead_req, aead, aad, iv,
- cpu_to_be64(rcd_sn), &in, &out, &len, prot);
+ rc = tls_enc_record(tls_ctx, aead_req, aead, aad, iv,
+ cpu_to_be64(rcd_sn), &in, &out, &len);
rcd_sn++;
} while (rc == 0 && len);
@@ -184,6 +190,14 @@ static void complete_skb(struct sk_buff *nskb, struct sk_buff *skb, int headln)
skb_copy_header(nskb, skb);
+ /* nskb now carries ciphertext, but skb_copy_header() inherited
+ * skb->decrypted from the plaintext original. Clear it so the bit keeps
+ * meaning "still-plaintext, needs an encryptor": otherwise a requeued
+ * nskb would be needlessly re-validated (and re-encrypted) and would trip
+ * the NIC's decrypted-vs-start-marker WARN.
+ */
+ nskb->decrypted = 0;
+
skb_put(nskb, skb->len);
memcpy(nskb->data, skb->data, headln);
@@ -314,7 +328,10 @@ static struct sk_buff *tls_enc_skb(struct tls_context *tls_ctx,
cipher_desc = get_cipher_desc(tls_ctx->crypto_send.info.cipher_type);
DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
- buf_len = cipher_desc->salt + cipher_desc->iv + TLS_AAD_SPACE_SIZE +
+ aead_request_set_ad(aead_req, tls_ctx->prot_info.aad_size);
+
+ buf_len = cipher_desc->salt + cipher_desc->iv +
+ tls_ctx->prot_info.aad_size +
sync_size + cipher_desc->tag;
buf = kmalloc(buf_len, GFP_ATOMIC);
if (!buf)
@@ -324,7 +341,7 @@ static struct sk_buff *tls_enc_skb(struct tls_context *tls_ctx,
salt = crypto_info_salt(&tls_ctx->crypto_send.info, cipher_desc);
memcpy(iv, salt, cipher_desc->salt);
aad = buf + cipher_desc->salt + cipher_desc->iv;
- dummy_buf = aad + TLS_AAD_SPACE_SIZE;
+ dummy_buf = aad + tls_ctx->prot_info.aad_size;
nskb = alloc_skb(skb_headroom(skb) + skb->len, GFP_ATOMIC);
if (!nskb)
@@ -335,9 +352,8 @@ static struct sk_buff *tls_enc_skb(struct tls_context *tls_ctx,
fill_sg_out(sg_out, buf, tls_ctx, nskb, tcp_payload_offset,
payload_len, sync_size, dummy_buf);
- if (tls_enc_records(aead_req, ctx->aead_send, sg_in, sg_out, aad, iv,
- rcd_sn, sync_size + payload_len,
- &tls_ctx->prot_info) < 0)
+ if (tls_enc_records(tls_ctx, aead_req, ctx->aead_send, sg_in, sg_out,
+ aad, iv, rcd_sn, sync_size + payload_len) < 0)
goto free_nskb;
complete_skb(nskb, skb, tcp_payload_offset);
@@ -388,8 +404,17 @@ static struct sk_buff *tls_sw_fallback(struct sock *sk, struct sk_buff *skb)
sg_init_table(sg_out, ARRAY_SIZE(sg_out));
if (fill_sg_in(sg_in, skb, ctx, &rcd_sn, &sync_size, &resync_sgs)) {
- /* bypass packets before kernel TLS socket option was set */
- if (sync_size < 0 && payload_len <= -sync_size)
+ /* Below the record range (start marker / already-freed record).
+ * Pass through only cleartext that was never offload-encrypted
+ * (skb->decrypted == 0): genuine pre-TLS bytes sent before the
+ * socket option was set, or SW-encrypted rekey ciphertext. A
+ * decrypted=1 skb here is offload-record plaintext whose record was
+ * purged (e.g. a rekey installed a new start marker above its seq);
+ * it must never reach the wire in the clear, so continue on and
+ * drop it (nskb stays NULL).
+ */
+ if (sync_size < 0 && payload_len <= -sync_size &&
+ !skb_is_decrypted(skb))
nskb = skb_get(skb);
goto put_sg;
}
@@ -408,11 +433,57 @@ static struct sk_buff *tls_sw_fallback(struct sock *sk, struct sk_buff *skb)
return nskb;
}
+/* Post-rekey drop floor. Once a rekey has completed (TLS_TX_REKEY_FLOOR set), a
+ * stale retransmit clone of already-ACKed data may still be dequeued from a
+ * qdisc; if its offload record was purged at completion it now maps to a rekey
+ * start marker. The cleartext leak on that path is closed unconditionally by
+ * the skb_is_decrypted() gate in tls_sw_fallback(); this floor additionally
+ * drops the clone before it reaches the NIC, avoiding the driver's WARN
+ * (mlx5e_ktls_handle_tx_skb() SKIP_NO_DATA) on an otherwise-legitimate race.
+ * Only needed by tls_validate_xmit_skb() (the restored HW-offload validator):
+ * only there can a purged-record clone reach the NIC and hit the new start
+ * marker. Under the rekey/SW validators the only skb the NIC offloads is a
+ * decrypted straddler whose record is still present (no SKIP_NO_DATA), and a
+ * stale clone is dropped by the skb_is_decrypted() gate in tls_sw_fallback().
+ * Such a clone is exactly a payload skb whose end_seq <= snd_una: the peer has
+ * already ACKed that data, so dropping it is always safe. Live/unacked data
+ * (including a legitimate retransmit, or a straddler ending past snd_una) is
+ * never touched; pure ACKs and zero-window probes carry no payload and pass.
+ */
+static bool tls_tx_drop_acked_clone(struct sock *sk, struct sk_buff *skb)
+{
+ int payload_len = skb->len - skb_tcp_all_headers(skb);
+ u32 end_seq;
+
+ if (likely(!test_bit(TLS_TX_REKEY_FLOOR, &tls_get_ctx(sk)->flags)))
+ return false;
+
+ if (payload_len <= 0)
+ return false;
+
+ /* Drop only when the whole payload is already ACKed (end_seq <= snd_una):
+ * such a skb is purely a stale retransmit clone the peer already has. A
+ * clone straddling snd_una still carries unacked bytes, so leave it to the
+ * normal paths (a live record is re-encrypted; a marker/freed-record hit is
+ * dropped there too). Both the leak (skb_is_decrypted() gate) and the mlx5
+ * WARN only concern the fully-ACKed case handled here.
+ */
+ end_seq = ntohl(tcp_hdr(skb)->seq) + payload_len;
+ return !after(end_seq, READ_ONCE(tcp_sk(sk)->snd_una));
+}
+
struct sk_buff *tls_validate_xmit_skb(struct sock *sk,
struct net_device *dev,
struct sk_buff *skb)
{
- if (dev == rcu_dereference_bh(tls_get_ctx(sk)->netdev) ||
+ struct tls_context *tls_ctx = tls_get_ctx(sk);
+
+ if (unlikely(tls_tx_drop_acked_clone(sk, skb))) {
+ kfree_skb(skb);
+ return NULL;
+ }
+
+ if (dev == rcu_dereference_bh(tls_ctx->netdev) ||
netif_is_bond_master(dev))
return skb;
@@ -427,6 +498,65 @@ struct sk_buff *tls_validate_xmit_skb_sw(struct sock *sk,
return tls_sw_fallback(sk, skb);
}
+struct sk_buff *tls_validate_xmit_skb_rekey(struct sock *sk,
+ struct net_device *dev,
+ struct sk_buff *skb)
+{
+ struct tls_context *tls_ctx = tls_get_ctx(sk);
+ u32 tcp_seq = ntohl(tcp_hdr(skb)->seq);
+ u32 pivot_seq;
+
+ /* acquire pairs with clear_bit_unlock() on re-arm; makes the refreshed
+ * boundary_seq visible in the else branch below.
+ */
+ if (test_bit_acquire(TLS_TX_REKEY_FAILED, &tls_ctx->flags)) {
+ int payload_len = skb->len - skb_tcp_all_headers(skb);
+ u32 snd_una = READ_ONCE(tcp_sk(sk)->snd_una);
+
+ /* FAILED: HW context gone and all old-key plaintext ACKed
+ * (snd_una >= boundary_seq). seq < boundary_seq is old-key data
+ * whose records are freed, so tls_sw_fallback() drops it. seq >=
+ * boundary_seq is SW ciphertext with no record. A retransmit is
+ * built at seq == snd_una (tcp_trim_head()), so an ACK landing
+ * before we run can move snd_una past seq while the tail is
+ * unacked; pivoting on snd_una alone would drop that live data
+ * and force an RTO. Pass through any non-decrypted skb ending
+ * past snd_una (mirrors tls_tx_drop_acked_clone()); fully-ACKed
+ * clones fall to the pivot and are dropped.
+ */
+ if (payload_len > 0 && !skb_is_decrypted(skb) &&
+ after(tcp_seq + payload_len, snd_una))
+ return skb;
+
+ pivot_seq = snd_una;
+ } else {
+ /* PENDING: new-key data is SW-encrypted at seq >= boundary_seq;
+ * old-key data below it is still unacked.
+ *
+ * On the first arm, boundary_seq is published by the
+ * smp_store_release() of sk_validate_xmit_skb in
+ * tls_device_start_rekey(); the xmit path loads that pointer with a
+ * plain read (net/core/dev.c), so pair it here with an smp_rmb()
+ * before reading boundary_seq. A stale boundary_seq (0) would pass an
+ * unacked old-key plaintext skb through; tls_is_skb_tx_device_offloaded()
+ * would still HW-encrypt it with the installed old key, so not a leak,
+ * but the barrier keeps the pivot accurate.
+ */
+ smp_rmb();
+ pivot_seq = READ_ONCE(tls_ctx->rekey.boundary_seq);
+ }
+
+ /* At or after the pivot: already correctly encrypted, pass through */
+ if (!before(tcp_seq, pivot_seq))
+ return skb;
+
+ /* Below the pivot: retransmit of old data, SW fallback with old key */
+ return tls_sw_fallback(sk, skb);
+}
+
+/* Address taken by tls_is_skb_tx_device_offloaded() in the offload drivers. */
+EXPORT_SYMBOL_GPL(tls_validate_xmit_skb_rekey);
+
struct sk_buff *tls_encrypt_skb(struct sk_buff *skb)
{
return tls_sw_fallback(skb->sk, skb);
diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c
index fbb274287aa5f..0a9e7d15fa95b 100644
--- a/net/tls/tls_main.c
+++ b/net/tls/tls_main.c
@@ -347,16 +347,28 @@ static void tls_sk_proto_cleanup(struct sock *sk,
tls_sw_release_resources_tx(sk);
TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
} else if (ctx->tx_conf == TLS_HW) {
+ bool rekey_failed = test_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+
tls_device_free_resources_tx(sk);
- TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
+
+ if (rekey_failed)
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
+ else
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
}
if (ctx->rx_conf == TLS_SW) {
tls_sw_release_resources_rx(sk);
TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXSW);
} else if (ctx->rx_conf == TLS_HW) {
+ bool rekey_failed = test_bit(TLS_RX_REKEY_FAILED, &ctx->flags);
+
tls_device_offload_cleanup_rx(sk);
- TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);
+
+ if (rekey_failed)
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXSW);
+ else
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);
}
}
@@ -369,6 +381,8 @@ static void tls_sk_proto_close(struct sock *sk, long timeout)
if (ctx->tx_conf == TLS_SW)
tls_sw_cancel_work_tx(ctx);
+ else if (ctx->tx_conf == TLS_HW && ctx->rekey.sw_ctx)
+ tls_sw_cancel_work_tx(ctx);
lock_sock(sk);
free_ctx = ctx->tx_conf != TLS_HW && ctx->rx_conf != TLS_HW;
@@ -445,8 +459,17 @@ static int do_tls_getsockopt_conf(struct sock *sk, sockopt_t *opt, int tx)
/* get user crypto info */
if (tx) {
- crypto_info = &ctx->crypto_send.info;
- cctx = &ctx->tx;
+ /* Select the cipher context via the same accessor the data path
+ * uses, so getsockopt reports the IV/rec_seq that sendmsg encrypts
+ * with (the pending rekey's while one is in flight, else the
+ * active key). crypto_info has no accessor; select it the same way.
+ * lock_sock is held, so rekey.cipher_ctx cannot change under us.
+ */
+ cctx = tls_tx_cipher_ctx(ctx);
+ if (ctx->rekey.cipher_ctx)
+ crypto_info = &tls_offload_ctx_tx(ctx)->rekey.crypto_send.info;
+ else
+ crypto_info = &ctx->crypto_send.info;
} else {
crypto_info = &ctx->crypto_recv.info;
cctx = &ctx->rx;
@@ -710,11 +733,18 @@ static int do_tls_setsockopt_conf(struct sock *sk, sockptr_t optval,
}
if (tx) {
- rc = tls_set_device_offload(sk);
+ rc = tls_set_device_offload(sk, update ? crypto_info : NULL);
conf = TLS_HW;
if (!rc) {
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXDEVICE);
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
+ if (!update) {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXDEVICE);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
+ }
+ } else if (update && ctx->tx_conf == TLS_HW) {
+ /* HW rekey failed - return the actual error.
+ * Cannot fall back to SW for an existing HW connection.
+ */
+ goto err_crypto_info;
} else {
rc = tls_set_sw_offload(sk, 1,
update ? crypto_info : NULL);
@@ -730,11 +760,19 @@ static int do_tls_setsockopt_conf(struct sock *sk, sockptr_t optval,
conf = TLS_SW;
}
} else {
- rc = tls_set_device_offload_rx(sk, ctx);
+ rc = tls_set_device_offload_rx(sk, ctx,
+ update ? crypto_info : NULL);
conf = TLS_HW;
if (!rc) {
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXDEVICE);
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);
+ if (!update) {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXDEVICE);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);
+ }
+ } else if (update && ctx->rx_conf == TLS_HW) {
+ /* HW rekey failed - return the actual error.
+ * Cannot fall back to SW for an existing HW connection.
+ */
+ goto err_crypto_info;
} else {
rc = tls_set_sw_offload(sk, 0,
update ? crypto_info : NULL);
@@ -773,7 +811,11 @@ static int do_tls_setsockopt_conf(struct sock *sk, sockptr_t optval,
return 0;
err_crypto_info:
- if (update) {
+ /* -EAGAIN is a transient sndbuf-full condition on a non-blocking rekey,
+ * not a failed KeyUpdate: the old key stays installed and userspace
+ * retries once the socket is writable, so don't count it as an error.
+ */
+ if (update && rc != -EAGAIN) {
TLS_INC_STATS(sock_net(sk), tx ? LINUX_MIB_TLSTXREKEYERROR
: LINUX_MIB_TLSRXREKEYERROR);
}
@@ -866,12 +908,29 @@ static int do_tls_setsockopt(struct sock *sk, int optname, sockptr_t optval,
switch (optname) {
case TLS_TX:
- case TLS_RX:
+ case TLS_RX: {
+ /* tls_device_sendmsg() holds tx_lock across the lock_sock drop
+ * in sk_stream_wait_memory() with a half-built open_record
+ * exposed. A concurrent HW-offload rekey (tls_device_start_rekey())
+ * would flush that record and swap the key under the sender,
+ * corrupting record framing. Serialize TX setsockopt against
+ * the data path with tx_lock, unconditionally for TLS_TX,
+ * since during initial setup there is no sender contending it.
+ */
+ bool tx = optname == TLS_TX;
+
+ if (tx) {
+ rc = mutex_lock_interruptible(&tls_get_ctx(sk)->tx_lock);
+ if (rc)
+ break;
+ }
lock_sock(sk);
- rc = do_tls_setsockopt_conf(sk, optval, optlen,
- optname == TLS_TX);
+ rc = do_tls_setsockopt_conf(sk, optval, optlen, tx);
release_sock(sk);
+ if (tx)
+ mutex_unlock(&tls_get_ctx(sk)->tx_lock);
break;
+ }
case TLS_TX_ZEROCOPY_RO:
lock_sock(sk);
rc = do_tls_setsockopt_tx_zc(sk, optval, optlen);
diff --git a/net/tls/tls_proc.c b/net/tls/tls_proc.c
index 4012c4372d4c0..6255f7b07eb76 100644
--- a/net/tls/tls_proc.c
+++ b/net/tls/tls_proc.c
@@ -27,6 +27,12 @@ static const struct snmp_mib tls_mib_list[] = {
SNMP_MIB_ITEM("TlsTxRekeyOk", LINUX_MIB_TLSTXREKEYOK),
SNMP_MIB_ITEM("TlsTxRekeyError", LINUX_MIB_TLSTXREKEYERROR),
SNMP_MIB_ITEM("TlsRxRekeyReceived", LINUX_MIB_TLSRXREKEYRECEIVED),
+ SNMP_MIB_ITEM("TlsTxRekeyFallback", LINUX_MIB_TLSTXREKEYFALLBACK),
+ SNMP_MIB_ITEM("TlsRxRekeyFallback", LINUX_MIB_TLSRXREKEYFALLBACK),
+ SNMP_MIB_ITEM("TlsCurrTxRekey", LINUX_MIB_TLSCURRTXREKEY),
+ SNMP_MIB_ITEM("TlsCurrRxRekey", LINUX_MIB_TLSCURRRXREKEY),
+ SNMP_MIB_ITEM("TlsTxRekeyAborted", LINUX_MIB_TLSTXREKEYABORTED),
+ SNMP_MIB_ITEM("TlsRxRekeyAborted", LINUX_MIB_TLSRXREKEYABORTED),
};
static int tls_statistics_seq_show(struct seq_file *seq, void *v)
diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index d1ad31986cf2c..d546091dd5240 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -522,7 +522,7 @@ static void tls_encrypt_done(void *data, int err)
complete(&ctx->async_wait.completion);
}
-static int tls_encrypt_async_wait(struct tls_sw_context_tx *ctx)
+int tls_encrypt_async_wait(struct tls_sw_context_tx *ctx)
{
if (!atomic_dec_and_test(&ctx->encrypt_pending))
crypto_wait_req(-EINPROGRESS, &ctx->async_wait);
@@ -555,11 +555,11 @@ static int tls_do_encryption(struct sock *sk,
break;
}
- memcpy(&rec->iv_data[iv_offset], tls_ctx->tx.iv,
+ memcpy(&rec->iv_data[iv_offset], tls_tx_cipher_ctx(tls_ctx)->iv,
prot->iv_size + prot->salt_size);
tls_xor_iv_with_seq(prot, rec->iv_data + iv_offset,
- tls_ctx->tx.rec_seq);
+ tls_tx_cipher_ctx(tls_ctx)->rec_seq);
sge->offset += prot->prepend_size;
sge->length -= prot->prepend_size;
@@ -610,7 +610,7 @@ static int tls_do_encryption(struct sock *sk,
/* Unhook the record from context if encryption is not failure */
ctx->open_rec = NULL;
- tls_advance_record_sn(sk, prot, &tls_ctx->tx);
+ tls_advance_record_sn(sk, prot, tls_tx_cipher_ctx(tls_ctx));
return rc;
}
@@ -676,7 +676,7 @@ static int tls_push_record(struct sock *sk, int flags,
sg_chain(rec->sg_aead_out, 2, &msg_en->sg.data[i]);
tls_make_aad(rec->aad_space, msg_pl->sg.size + prot->tail_size,
- tls_ctx->tx.rec_seq, record_type, prot);
+ tls_tx_cipher_ctx(tls_ctx)->rec_seq, record_type, prot);
tls_fill_prepend(tls_ctx,
page_address(sg_page(&msg_en->sg.data[i])) +
@@ -712,7 +712,7 @@ static int bpf_exec_tx_verdict(struct sk_msg *msg, struct sock *sk,
return err;
}
-static int tls_sw_push_pending_record(struct sock *sk, int flags)
+int tls_sw_push_pending_record(struct sock *sk, int flags)
{
struct tls_context *tls_ctx = tls_get_ctx(sk);
struct tls_sw_context_tx *ctx = tls_sw_ctx_tx(tls_ctx);
@@ -763,8 +763,7 @@ static int tls_sw_sendmsg_splice(struct sock *sk, struct msghdr *msg,
return 0;
}
-static int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg,
- size_t size)
+int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
{
long timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT);
struct tls_context *tls_ctx = tls_get_ctx(sk);
@@ -1027,8 +1026,13 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)
/*
* Handle unexpected EOF during splice without SPLICE_F_MORE set.
+ *
+ * Inner logic of tls_sw_splice_eof(), factored out so the device
+ * TX path can reuse it with tls_ctx->tx_lock and the socket lock
+ * already held. Callers not already holding both locks must use the
+ * tls_sw_splice_eof() wrapper instead.
*/
-void tls_sw_splice_eof(struct socket *sock)
+void tls_sw_splice_eof_locked(struct socket *sock)
{
struct sock *sk = sock->sk;
struct tls_context *tls_ctx = tls_get_ctx(sk);
@@ -1039,21 +1043,15 @@ void tls_sw_splice_eof(struct socket *sock)
bool retrying = false;
int ret = 0;
- if (!ctx->open_rec)
- return;
-
- mutex_lock(&tls_ctx->tx_lock);
- lock_sock(sk);
-
retry:
- /* same checks as in tls_sw_push_pending_record() */
+ /* same open_rec / empty-record checks as tls_sw_push_pending_record() */
rec = ctx->open_rec;
if (!rec)
- goto unlock;
+ return;
msg_pl = &rec->msg_plaintext;
if (msg_pl->sg.size == 0)
- goto unlock;
+ return;
/* Perform transmission. */
ret = bpf_exec_tx_verdict(msg_pl, sk, TLS_RECORD_TYPE_DATA,
@@ -1062,26 +1060,38 @@ void tls_sw_splice_eof(struct socket *sock)
case 0:
case -EAGAIN:
if (retrying)
- goto unlock;
+ return;
retrying = true;
goto retry;
case -EINPROGRESS:
break;
default:
- goto unlock;
+ return;
}
/* Wait for pending encryptions to get completed */
if (tls_encrypt_async_wait(ctx))
- goto unlock;
+ return;
/* Transmit if any encryptions have completed */
if (test_and_clear_bit(BIT_TX_SCHEDULED, &ctx->tx_bitmask)) {
cancel_delayed_work(&ctx->tx_work.work);
tls_tx_records(sk, 0);
}
+}
+
+void tls_sw_splice_eof(struct socket *sock)
+{
+ struct sock *sk = sock->sk;
+ struct tls_context *tls_ctx = tls_get_ctx(sk);
+ struct tls_sw_context_tx *ctx = tls_sw_ctx_tx(tls_ctx);
-unlock:
+ if (!ctx->open_rec)
+ return;
+
+ mutex_lock(&tls_ctx->tx_lock);
+ lock_sock(sk);
+ tls_sw_splice_eof_locked(sock);
release_sock(sk);
mutex_unlock(&tls_ctx->tx_lock);
}
@@ -1551,6 +1561,7 @@ static int tls_check_pending_rekey(struct sock *sk, struct tls_context *ctx,
if (hs_type == TLS_HANDSHAKE_KEYUPDATE) {
struct tls_sw_context_rx *rx_ctx = ctx->priv_ctx_rx;
+ tls_device_rx_del_key(sk, ctx);
WRITE_ONCE(rx_ctx->key_update_pending, true);
TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYRECEIVED);
}
@@ -2401,6 +2412,40 @@ static void tx_work_handler(struct work_struct *work)
}
}
+void tls_sw_ctx_tx_init(struct sock *sk, struct tls_sw_context_tx *sw_ctx)
+{
+ crypto_init_wait(&sw_ctx->async_wait);
+ atomic_set(&sw_ctx->encrypt_pending, 1);
+ INIT_LIST_HEAD(&sw_ctx->tx_list);
+ INIT_DELAYED_WORK(&sw_ctx->tx_work.work, tx_work_handler);
+ sw_ctx->tx_work.sk = sk;
+}
+
+int tls_sw_drain_tx(struct sock *sk, struct tls_context *ctx, int flags)
+{
+ struct tls_sw_context_tx *sw_ctx = tls_sw_ctx_tx(ctx);
+ int rc;
+
+ flags = (flags & MSG_DONTWAIT) | MSG_NOSIGNAL;
+
+ if (sw_ctx->open_rec)
+ tls_sw_push_pending_record(sk, flags);
+ rc = tls_encrypt_async_wait(sw_ctx);
+ if (rc)
+ return rc;
+ rc = tls_tx_records(sk, flags);
+ if (rc < 0 || tls_is_partially_sent_record(ctx) ||
+ tls_is_pending_open_record(ctx) ||
+ !list_empty(&sw_ctx->tx_list))
+ return rc < 0 ? rc : -EAGAIN;
+
+ tls_free_open_rec(sk);
+
+ cancel_delayed_work_sync(&sw_ctx->tx_work.work);
+ clear_bit(BIT_TX_SCHEDULED, &sw_ctx->tx_bitmask);
+ return 0;
+}
+
static bool tls_is_tx_ready(struct tls_sw_context_tx *ctx)
{
struct tls_rec *rec;
@@ -2452,11 +2497,7 @@ static struct tls_sw_context_tx *init_ctx_tx(struct tls_context *ctx, struct soc
sw_ctx_tx = ctx->priv_ctx_tx;
}
- crypto_init_wait(&sw_ctx_tx->async_wait);
- atomic_set(&sw_ctx_tx->encrypt_pending, 1);
- INIT_LIST_HEAD(&sw_ctx_tx->tx_list);
- INIT_DELAYED_WORK(&sw_ctx_tx->tx_work.work, tx_work_handler);
- sw_ctx_tx->tx_work.sk = sk;
+ tls_sw_ctx_tx_init(sk, sw_ctx_tx);
return sw_ctx_tx;
}
@@ -2522,20 +2563,19 @@ static void tls_finish_key_update(struct sock *sk, struct tls_context *tls_ctx)
ctx->saved_data_ready(sk);
}
-int tls_set_sw_offload(struct sock *sk, int tx,
- struct tls_crypto_info *new_crypto_info)
+int tls_sw_ctx_init(struct sock *sk, int tx,
+ struct tls_crypto_info *new_crypto_info)
{
struct tls_crypto_info *crypto_info, *src_crypto_info;
struct tls_sw_context_tx *sw_ctx_tx = NULL;
struct tls_sw_context_rx *sw_ctx_rx = NULL;
const struct tls_cipher_desc *cipher_desc;
- char *iv, *rec_seq, *key, *salt;
- struct cipher_context *cctx;
struct tls_prot_info *prot;
struct crypto_aead **aead;
struct tls_context *ctx;
struct crypto_tfm *tfm;
int rc = 0;
+ char *key;
ctx = tls_get_ctx(sk);
prot = &ctx->prot_info;
@@ -2556,12 +2596,10 @@ int tls_set_sw_offload(struct sock *sk, int tx,
if (tx) {
sw_ctx_tx = ctx->priv_ctx_tx;
crypto_info = &ctx->crypto_send.info;
- cctx = &ctx->tx;
aead = &sw_ctx_tx->aead_send;
} else {
sw_ctx_rx = ctx->priv_ctx_rx;
crypto_info = &ctx->crypto_recv.info;
- cctx = &ctx->rx;
aead = &sw_ctx_rx->aead_recv;
}
@@ -2577,11 +2615,12 @@ int tls_set_sw_offload(struct sock *sk, int tx,
if (rc)
goto free_priv;
- iv = crypto_info_iv(src_crypto_info, cipher_desc);
key = crypto_info_key(src_crypto_info, cipher_desc);
- salt = crypto_info_salt(src_crypto_info, cipher_desc);
- rec_seq = crypto_info_rec_seq(src_crypto_info, cipher_desc);
+ /* A rekey normally reuses the existing tfm; the RX HW rekey hands over a
+ * NULL aead (the old one is retained for the drain), so allocate and
+ * configure authsize only when a fresh tfm is created here.
+ */
if (!*aead) {
*aead = crypto_alloc_aead(cipher_desc->cipher_name, 0, 0);
if (IS_ERR(*aead)) {
@@ -2589,9 +2628,14 @@ int tls_set_sw_offload(struct sock *sk, int tx,
*aead = NULL;
goto free_priv;
}
+
+ rc = crypto_aead_setauthsize(*aead, prot->tag_size);
+ if (rc)
+ goto free_aead;
}
- ctx->push_pending_record = tls_sw_push_pending_record;
+ if (tx)
+ ctx->push_pending_record = tls_sw_push_pending_record;
/* setkey is the last operation that could fail during a
* rekey. if it succeeds, we can start modifying the
@@ -2605,12 +2649,6 @@ int tls_set_sw_offload(struct sock *sk, int tx,
goto free_aead;
}
- if (!new_crypto_info) {
- rc = crypto_aead_setauthsize(*aead, prot->tag_size);
- if (rc)
- goto free_aead;
- }
-
if (!tx && !new_crypto_info) {
tfm = crypto_aead_tfm(sw_ctx_rx->aead_recv);
@@ -2624,19 +2662,6 @@ int tls_set_sw_offload(struct sock *sk, int tx,
goto free_aead;
}
- memcpy(cctx->iv, salt, cipher_desc->salt);
- memcpy(cctx->iv + cipher_desc->salt, iv, cipher_desc->iv);
- memcpy(cctx->rec_seq, rec_seq, cipher_desc->rec_seq);
-
- if (new_crypto_info) {
- unsafe_memcpy(crypto_info, new_crypto_info,
- cipher_desc->crypto_info,
- /* size was checked in do_tls_setsockopt_conf */);
- memzero_explicit(new_crypto_info, cipher_desc->crypto_info);
- if (!tx)
- tls_finish_key_update(sk, ctx);
- }
-
goto out;
free_aead:
@@ -2655,3 +2680,57 @@ int tls_set_sw_offload(struct sock *sk, int tx,
out:
return rc;
}
+
+void tls_sw_ctx_finalize(struct sock *sk, int tx,
+ struct tls_crypto_info *new_crypto_info)
+{
+ struct tls_crypto_info *crypto_info, *src_crypto_info;
+ const struct tls_cipher_desc *cipher_desc;
+ struct tls_context *ctx = tls_get_ctx(sk);
+ struct cipher_context *cctx;
+ char *iv, *salt, *rec_seq;
+
+ if (tx) {
+ crypto_info = &ctx->crypto_send.info;
+ cctx = &ctx->tx;
+ } else {
+ crypto_info = &ctx->crypto_recv.info;
+ cctx = &ctx->rx;
+ }
+
+ src_crypto_info = new_crypto_info ?: crypto_info;
+
+ /* Infallible: tls_sw_ctx_init() already validated cipher_type. */
+ cipher_desc = get_cipher_desc(src_crypto_info->cipher_type);
+
+ iv = crypto_info_iv(src_crypto_info, cipher_desc);
+ salt = crypto_info_salt(src_crypto_info, cipher_desc);
+ rec_seq = crypto_info_rec_seq(src_crypto_info, cipher_desc);
+
+ memcpy(cctx->iv, salt, cipher_desc->salt);
+ memcpy(cctx->iv + cipher_desc->salt, iv, cipher_desc->iv);
+ memcpy(cctx->rec_seq, rec_seq, cipher_desc->rec_seq);
+
+ if (new_crypto_info) {
+ unsafe_memcpy(crypto_info, new_crypto_info,
+ cipher_desc->crypto_info,
+ /* size was checked in do_tls_setsockopt_conf */);
+ memzero_explicit(new_crypto_info, cipher_desc->crypto_info);
+
+ if (!tx)
+ tls_finish_key_update(sk, ctx);
+ }
+}
+
+int tls_set_sw_offload(struct sock *sk, int tx,
+ struct tls_crypto_info *new_crypto_info)
+{
+ int rc;
+
+ rc = tls_sw_ctx_init(sk, tx, new_crypto_info);
+ if (rc)
+ return rc;
+
+ tls_sw_ctx_finalize(sk, tx, new_crypto_info);
+ return 0;
+}
diff --git a/net/tls/trace.h b/net/tls/trace.h
index 2d8ce4ff3265b..5b9c1f86d82df 100644
--- a/net/tls/trace.h
+++ b/net/tls/trace.h
@@ -192,6 +192,124 @@ TRACE_EVENT(tls_device_tx_resync_send,
)
);
+TRACE_EVENT(tls_device_rekey_start,
+
+ TP_PROTO(struct sock *sk, u32 copied_seq, u32 nic_boundary,
+ bool inflight),
+
+ TP_ARGS(sk, copied_seq, nic_boundary, inflight),
+
+ TP_STRUCT__entry(
+ __field( struct sock *, sk )
+ __field( u32, copied_seq )
+ __field( u32, nic_boundary )
+ __field( bool, inflight )
+ ),
+
+ TP_fast_assign(
+ __entry->sk = sk;
+ __entry->copied_seq = copied_seq;
+ __entry->nic_boundary = nic_boundary;
+ __entry->inflight = inflight;
+ ),
+
+ TP_printk(
+ "sk=%p copied_seq=%u nic_boundary=%u inflight=%d",
+ __entry->sk, __entry->copied_seq, __entry->nic_boundary,
+ __entry->inflight
+ )
+);
+
+TRACE_EVENT(tls_device_rekey_reencrypt,
+
+ TP_PROTO(struct sock *sk, u32 tcp_seq, u32 nic_boundary),
+
+ TP_ARGS(sk, tcp_seq, nic_boundary),
+
+ TP_STRUCT__entry(
+ __field( struct sock *, sk )
+ __field( u32, tcp_seq )
+ __field( u32, nic_boundary )
+ ),
+
+ TP_fast_assign(
+ __entry->sk = sk;
+ __entry->tcp_seq = tcp_seq;
+ __entry->nic_boundary = nic_boundary;
+ ),
+
+ TP_printk(
+ "sk=%p tcp_seq=%u nic_boundary=%u",
+ __entry->sk, __entry->tcp_seq, __entry->nic_boundary
+ )
+);
+
+TRACE_EVENT(tls_device_rekey_done,
+
+ TP_PROTO(struct sock *sk, u32 tcp_seq, u32 nic_boundary),
+
+ TP_ARGS(sk, tcp_seq, nic_boundary),
+
+ TP_STRUCT__entry(
+ __field( struct sock *, sk )
+ __field( u32, tcp_seq )
+ __field( u32, nic_boundary )
+ ),
+
+ TP_fast_assign(
+ __entry->sk = sk;
+ __entry->tcp_seq = tcp_seq;
+ __entry->nic_boundary = nic_boundary;
+ ),
+
+ TP_printk(
+ "sk=%p tcp_seq=%u nic_boundary=%u",
+ __entry->sk, __entry->tcp_seq, __entry->nic_boundary
+ )
+);
+
+TRACE_EVENT(tls_device_complete_rekey_fail,
+
+ TP_PROTO(struct sock *sk, int rc),
+
+ TP_ARGS(sk, rc),
+
+ TP_STRUCT__entry(
+ __field( struct sock *, sk )
+ __field( int, rc )
+ ),
+
+ TP_fast_assign(
+ __entry->sk = sk;
+ __entry->rc = rc;
+ ),
+
+ TP_printk(
+ "sk=%p rc=%d",
+ __entry->sk, __entry->rc
+ )
+);
+
+TRACE_EVENT(tls_device_complete_rekey_retry,
+
+ TP_PROTO(struct sock *sk),
+
+ TP_ARGS(sk),
+
+ TP_STRUCT__entry(
+ __field( struct sock *, sk )
+ ),
+
+ TP_fast_assign(
+ __entry->sk = sk;
+ ),
+
+ TP_printk(
+ "sk=%p",
+ __entry->sk
+ )
+);
+
#endif /* _TLS_TRACE_H_ */
#undef TRACE_INCLUDE_PATH
diff --git a/tools/testing/selftests/drivers/net/hw/.gitignore b/tools/testing/selftests/drivers/net/hw/.gitignore
index 46540468a7753..911a9bfeaf41e 100644
--- a/tools/testing/selftests/drivers/net/hw/.gitignore
+++ b/tools/testing/selftests/drivers/net/hw/.gitignore
@@ -1,4 +1,5 @@
# SPDX-License-Identifier: GPL-2.0-only
iou-zcrx
ncdevmem
+tls_hw_offload
toeplitz
diff --git a/tools/testing/selftests/drivers/net/hw/Makefile b/tools/testing/selftests/drivers/net/hw/Makefile
index 8aebdc6feb177..b3831c2d09ea5 100644
--- a/tools/testing/selftests/drivers/net/hw/Makefile
+++ b/tools/testing/selftests/drivers/net/hw/Makefile
@@ -46,6 +46,7 @@ TEST_PROGS = \
rss_drv.py \
rss_flow_label.py \
rss_input_xfrm.py \
+ tls_hw_offload.py \
toeplitz.py \
tso.py \
userns_devmem.py \
@@ -80,6 +81,7 @@ YNL_GEN_FILES := \
# end of YNL_GEN_FILES
TEST_GEN_FILES += $(YNL_GEN_FILES)
TEST_GEN_FILES += $(patsubst %.c,%.o,$(wildcard *.bpf.c))
+TEST_GEN_FILES += tls_hw_offload
include ../../../lib.mk
diff --git a/tools/testing/selftests/drivers/net/hw/config b/tools/testing/selftests/drivers/net/hw/config
index d89a9ba176558..169e608516bd5 100644
--- a/tools/testing/selftests/drivers/net/hw/config
+++ b/tools/testing/selftests/drivers/net/hw/config
@@ -22,6 +22,8 @@ CONFIG_NET_IPIP=y
CONFIG_NETKIT=y
CONFIG_NET_SCH_INGRESS=y
CONFIG_SYNC_FILE=y
+CONFIG_TLS=y
+CONFIG_TLS_DEVICE=y
CONFIG_UDMABUF=y
CONFIG_USER_NS=y
CONFIG_VXLAN=y
diff --git a/tools/testing/selftests/drivers/net/hw/tls_hw_offload.c b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.c
new file mode 100644
index 0000000000000..303c6752ace27
--- /dev/null
+++ b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.c
@@ -0,0 +1,1132 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * TLS Hardware Offload Two-Node Test
+ *
+ * Tests kTLS hardware offload between two physical nodes using
+ * hardcoded keys. Supports TLS 1.2/1.3, AES-GCM-128/256, and rekey.
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+#include <limits.h>
+#include <time.h>
+#include <sys/time.h>
+#include <signal.h>
+#include <sys/types.h>
+#include <sys/socket.h>
+#include <netinet/in.h>
+#include <netinet/tcp.h>
+#include <netdb.h>
+#include <linux/tls.h>
+
+#define TLS_RECORD_TYPE_HANDSHAKE 22
+#define TLS_HANDSHAKE_KEY_UPDATE 0x18
+
+/* Large enough for a TLS 1.3 KeyUpdate handshake record's plaintext. */
+#define MIN_BUF_SIZE 16
+
+/* Initial key material */
+static struct tls12_crypto_info_aes_gcm_128 tls_info_key0_128 = {
+ .info = {
+ .version = TLS_1_3_VERSION,
+ .cipher_type = TLS_CIPHER_AES_GCM_128,
+ },
+ .iv = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08 },
+ .key = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08,
+ 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10 },
+ .salt = { 0x01, 0x02, 0x03, 0x04 },
+ .rec_seq = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
+};
+
+static struct tls12_crypto_info_aes_gcm_256 tls_info_key0_256 = {
+ .info = {
+ .version = TLS_1_3_VERSION,
+ .cipher_type = TLS_CIPHER_AES_GCM_256,
+ },
+ .iv = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08 },
+ .key = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08,
+ 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10,
+ 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18,
+ 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, 0x20 },
+ .salt = { 0x01, 0x02, 0x03, 0x04 },
+ .rec_seq = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
+};
+
+static int num_rekeys;
+static int num_iterations = 100;
+static int cipher_type = TLS_CIPHER_AES_GCM_128;
+static int tls_version = TLS_1_3_VERSION;
+static int server_port = 4433;
+static char *server_ip;
+/* Address family to force: AF_UNSPEC (any), AF_INET (-4), AF_INET6 (-6). */
+static int force_family = AF_UNSPEC;
+
+static int send_size = 16384;
+static int random_size_max;
+/* Burst mode: sender keeps pushing records without reading from the peer;
+ * receiver drains without echoing back. Only the client initiates rekey.
+ */
+static int burst_mode;
+static int zc_rx;
+
+/* XOR each byte with the generation so both endpoints derive the
+ * same per-generation key without a real KDF. Generation 0 leaves
+ * the base key unchanged.
+ */
+static void derive_key_fields(unsigned char *key, int key_size,
+ unsigned char *iv, int iv_size,
+ unsigned char *salt, int salt_size,
+ unsigned char *rec_seq, int rec_seq_size,
+ int generation)
+{
+ int i;
+
+ for (i = 0; i < key_size; i++)
+ key[i] ^= generation;
+ for (i = 0; i < iv_size; i++)
+ iv[i] ^= generation;
+ for (i = 0; i < salt_size; i++)
+ salt[i] ^= generation;
+ memset(rec_seq, 0, rec_seq_size);
+}
+
+static void derive_key_128(struct tls12_crypto_info_aes_gcm_128 *key,
+ int generation)
+{
+ memcpy(key, &tls_info_key0_128, sizeof(*key));
+ key->info.version = tls_version;
+ derive_key_fields(key->key, TLS_CIPHER_AES_GCM_128_KEY_SIZE,
+ key->iv, TLS_CIPHER_AES_GCM_128_IV_SIZE,
+ key->salt, TLS_CIPHER_AES_GCM_128_SALT_SIZE,
+ key->rec_seq, TLS_CIPHER_AES_GCM_128_REC_SEQ_SIZE,
+ generation);
+}
+
+static void derive_key_256(struct tls12_crypto_info_aes_gcm_256 *key,
+ int generation)
+{
+ memcpy(key, &tls_info_key0_256, sizeof(*key));
+ key->info.version = tls_version;
+ derive_key_fields(key->key, TLS_CIPHER_AES_GCM_256_KEY_SIZE,
+ key->iv, TLS_CIPHER_AES_GCM_256_IV_SIZE,
+ key->salt, TLS_CIPHER_AES_GCM_256_SALT_SIZE,
+ key->rec_seq, TLS_CIPHER_AES_GCM_256_REC_SEQ_SIZE,
+ generation);
+}
+
+static const char *cipher_name(int cipher)
+{
+ switch (cipher) {
+ case TLS_CIPHER_AES_GCM_128: return "AES-GCM-128";
+ case TLS_CIPHER_AES_GCM_256: return "AES-GCM-256";
+ default: return "unknown";
+ }
+}
+
+static const char *version_name(int version)
+{
+ switch (version) {
+ case TLS_1_2_VERSION: return "TLS 1.2";
+ case TLS_1_3_VERSION: return "TLS 1.3";
+ default: return "unknown";
+ }
+}
+
+static int setup_tls_ulp(int fd)
+{
+ int ret;
+
+ ret = setsockopt(fd, IPPROTO_TCP, TCP_ULP, "tls", sizeof("tls"));
+ if (ret < 0) {
+ printf("SETUP ERROR: TCP_ULP failed: %s\n", strerror(errno));
+ return -1;
+ }
+ return 0;
+}
+
+/* Echo (non-burst) mode drives both directions from a single thread: the
+ * client pushes a whole payload with one blocking send() and only reads the
+ * echo afterwards, while the server blocks in send() mid-echo. If a payload
+ * exceeds the peer's receive window the two sides deadlock - client stuck in
+ * send(), server stuck echoing, neither draining the other. Size the socket
+ * buffers so a full payload always fits in the peer's window (the forward
+ * send() then completes without needing the peer to read concurrently); the
+ * send/recv timeouts armed by set_io_timeouts() turn any residual stall into a
+ * loud EAGAIN instead of a hang.
+ */
+static void configure_echo_socket(int fd, int payload)
+{
+ int want = payload;
+
+ if (want < MIN_BUF_SIZE)
+ want = MIN_BUF_SIZE;
+
+ /* SO_*BUFFORCE bypasses the rmem_max/wmem_max sysctl caps (needs
+ * CAP_NET_ADMIN); fall back to the best-effort, cap-limited option
+ * when unprivileged - the timeouts below still turn any resulting
+ * stall into a loud failure rather than a hang.
+ */
+ if (setsockopt(fd, SOL_SOCKET, SO_RCVBUFFORCE, &want, sizeof(want)) < 0)
+ setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &want, sizeof(want));
+ if (setsockopt(fd, SOL_SOCKET, SO_SNDBUFFORCE, &want, sizeof(want)) < 0)
+ setsockopt(fd, SOL_SOCKET, SO_SNDBUF, &want, sizeof(want));
+}
+
+/* Arm send/recv timeouts so any unexpected stall fails loudly with EAGAIN
+ * instead of hanging until the harness SIGKILLs us. Wanted in both echo and
+ * burst modes - burst mode has no other stall guard.
+ */
+static void set_io_timeouts(int fd)
+{
+ struct timeval tv = { .tv_sec = 8, .tv_usec = 0 };
+
+ setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
+ setsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, &tv, sizeof(tv));
+}
+
+/* Send the whole buffer, looping over short counts. A blocking SOCK_STREAM
+ * send() may return fewer bytes than requested (e.g. when SO_SNDTIMEO fires
+ * after partial progress) without setting errno, so a short count is not an
+ * error - only a negative return is. Looping also sends each iteration as one
+ * uninterrupted run of bytes, which the peer's userspace reassembly in burst
+ * mode counts on to keep iterations aligned.
+ */
+static int send_all(int fd, const char *buf, ssize_t len)
+{
+ ssize_t sent = 0;
+ ssize_t ret;
+
+ while (sent < len) {
+ ret = send(fd, buf + sent, len - sent, 0);
+ if (ret < 0) {
+ printf("FAIL: send failed: %s\n", strerror(errno));
+ return -1;
+ }
+ sent += ret;
+ }
+ return 0;
+}
+
+static int set_zc_rx(int fd)
+{
+ int val = 1;
+
+ if (setsockopt(fd, SOL_TLS, TLS_RX_EXPECT_NO_PAD, &val,
+ sizeof(val)) < 0) {
+ printf("SETUP ERROR: TLS_RX_EXPECT_NO_PAD failed: %s\n",
+ strerror(errno));
+ return -1;
+ }
+ return 0;
+}
+
+/* Send a TLS 1.3 KeyUpdate handshake record. The kernel only
+ * inspects the HandshakeType byte to detect KeyUpdate, so don't
+ * bother with the 3-byte length or request_update fields.
+ */
+static int send_tls_key_update(int fd)
+{
+ char cmsg_buf[CMSG_SPACE(sizeof(unsigned char))];
+ unsigned char key_update_msg = TLS_HANDSHAKE_KEY_UPDATE;
+ struct msghdr msg = {0};
+ struct cmsghdr *cmsg;
+ struct iovec iov;
+
+ iov.iov_base = &key_update_msg;
+ iov.iov_len = sizeof(key_update_msg);
+
+ msg.msg_iov = &iov;
+ msg.msg_iovlen = 1;
+ msg.msg_control = cmsg_buf;
+ msg.msg_controllen = sizeof(cmsg_buf);
+
+ cmsg = CMSG_FIRSTHDR(&msg);
+ cmsg->cmsg_level = SOL_TLS;
+ cmsg->cmsg_type = TLS_SET_RECORD_TYPE;
+ cmsg->cmsg_len = CMSG_LEN(sizeof(unsigned char));
+ *CMSG_DATA(cmsg) = TLS_RECORD_TYPE_HANDSHAKE;
+ msg.msg_controllen = cmsg->cmsg_len;
+
+ if (sendmsg(fd, &msg, 0) < 0) {
+ printf("sendmsg KeyUpdate failed: %s\n", strerror(errno));
+ return -1;
+ }
+
+ printf("Sent TLS KeyUpdate handshake message\n");
+ return 0;
+}
+
+static int recv_tls_message(int fd, char *buf, size_t buflen, int *record_type,
+ int flags)
+{
+ char cmsg_buf[CMSG_SPACE(sizeof(unsigned char))];
+ struct msghdr msg = {0};
+ struct cmsghdr *cmsg;
+ struct iovec iov;
+ int ret;
+
+ iov.iov_base = buf;
+ iov.iov_len = buflen;
+
+ msg.msg_iov = &iov;
+ msg.msg_iovlen = 1;
+ msg.msg_control = cmsg_buf;
+ msg.msg_controllen = sizeof(cmsg_buf);
+
+ ret = recvmsg(fd, &msg, flags);
+ if (ret <= 0)
+ return ret;
+
+ cmsg = CMSG_FIRSTHDR(&msg);
+ if (cmsg && cmsg->cmsg_level == SOL_TLS &&
+ cmsg->cmsg_type == TLS_GET_RECORD_TYPE)
+ *record_type = *((unsigned char *)CMSG_DATA(cmsg));
+
+ return ret;
+}
+
+/* Confirm a handshake record starting with HandshakeType KeyUpdate. */
+static int check_keyupdate(const char *buf, int len, int record_type)
+{
+ if (record_type != TLS_RECORD_TYPE_HANDSHAKE) {
+ printf("Expected handshake record (0x%02x), got 0x%02x\n",
+ TLS_RECORD_TYPE_HANDSHAKE, record_type);
+ return -1;
+ }
+ if (len < 1 || (unsigned char)buf[0] != TLS_HANDSHAKE_KEY_UPDATE) {
+ printf("Expected KeyUpdate (0x%02x), got 0x%02x\n",
+ TLS_HANDSHAKE_KEY_UPDATE,
+ len ? (unsigned char)buf[0] : 0);
+ return -1;
+ }
+ printf("Received TLS KeyUpdate\n");
+ return 0;
+}
+
+static int recv_tls_keyupdate(int fd)
+{
+ char buf[MIN_BUF_SIZE];
+ int record_type = 0;
+ int ret;
+
+ ret = recv_tls_message(fd, buf, sizeof(buf), &record_type, 0);
+ if (ret < 0) {
+ printf("recv_tls_message failed: %s\n", strerror(errno));
+ return -1;
+ }
+
+ return check_keyupdate(buf, ret, record_type);
+}
+
+static int check_ekeyexpired(int fd)
+{
+ char buf[MIN_BUF_SIZE];
+ int ret;
+
+ ret = recv(fd, buf, sizeof(buf), MSG_DONTWAIT);
+ if (ret == -1 && errno == EKEYEXPIRED) {
+ printf("recv() returned EKEYEXPIRED as expected\n");
+ return 0;
+ }
+ if (ret > 0) {
+ printf("FAIL: recv() returned %d bytes, expected EKEYEXPIRED\n",
+ ret);
+ return -1;
+ }
+ if (ret == 0) {
+ printf("FAIL: connection closed during rekey\n");
+ return -1;
+ }
+ printf("FAIL: recv() returned unexpected error: %s\n",
+ strerror(errno));
+ return -1;
+}
+
+static int do_tls_rekey(int fd, int direction, int generation, int cipher)
+{
+ const char *dir = direction == TLS_TX ? "TX" : "RX";
+ int ret;
+
+ printf("%s TLS_%s %s gen %d...\n",
+ generation ? "Rekeying" : "Installing",
+ dir, cipher_name(cipher), generation);
+
+ if (cipher == TLS_CIPHER_AES_GCM_256) {
+ struct tls12_crypto_info_aes_gcm_256 key;
+
+ derive_key_256(&key, generation);
+ ret = setsockopt(fd, SOL_TLS, direction, &key, sizeof(key));
+ } else {
+ struct tls12_crypto_info_aes_gcm_128 key;
+
+ derive_key_128(&key, generation);
+ ret = setsockopt(fd, SOL_TLS, direction, &key, sizeof(key));
+ }
+
+ if (ret < 0) {
+ printf("%sTLS_%s %s gen %d failed: %s\n",
+ generation ? "" : "SETUP ERROR: ", dir,
+ cipher_name(cipher), generation, strerror(errno));
+ return -1;
+ }
+ printf("TLS_%s %s gen %d installed\n",
+ dir, cipher_name(cipher), generation);
+ return 0;
+}
+
+/* Open a TCP connection to server_ip:server_port, switch to the TLS
+ * ULP, and install initial generation-0 TX/RX keys. Works over IPv4 or
+ * IPv6: getaddrinfo() resolves server_ip (honouring any -4/-6 forced
+ * family and %zone scope IDs in link-local addresses). Returns the fd on
+ * success, -1 on error (with the fd already closed).
+ */
+static int client_connect_tls(void)
+{
+ struct addrinfo hints = {0}, *res, *rp;
+ char port_str[16];
+ int csk = -1;
+ int ret;
+
+ hints.ai_family = force_family;
+ hints.ai_socktype = SOCK_STREAM;
+ hints.ai_protocol = IPPROTO_TCP;
+ snprintf(port_str, sizeof(port_str), "%d", server_port);
+
+ ret = getaddrinfo(server_ip, port_str, &hints, &res);
+ if (ret) {
+ printf("SETUP ERROR: getaddrinfo(%s): %s\n", server_ip,
+ gai_strerror(ret));
+ return -1;
+ }
+
+ printf("Connecting to %s:%d...\n", server_ip, server_port);
+ for (rp = res; rp; rp = rp->ai_next) {
+ csk = socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol);
+ if (csk < 0)
+ continue;
+ if (connect(csk, rp->ai_addr, rp->ai_addrlen) == 0)
+ break;
+ close(csk);
+ csk = -1;
+ }
+ freeaddrinfo(res);
+
+ if (csk < 0) {
+ printf("SETUP ERROR: connect to %s:%d failed: %s\n",
+ server_ip, server_port, strerror(errno));
+ return -1;
+ }
+ printf("Connected!\n");
+
+ if (setup_tls_ulp(csk) < 0)
+ goto err;
+
+ if (do_tls_rekey(csk, TLS_TX, 0, cipher_type) < 0 ||
+ do_tls_rekey(csk, TLS_RX, 0, cipher_type) < 0)
+ goto err;
+
+ set_io_timeouts(csk);
+ if (!burst_mode)
+ configure_echo_socket(csk, random_size_max > 0 ?
+ random_size_max : send_size);
+
+ return csk;
+err:
+ close(csk);
+ return -1;
+}
+
+/* Drain `len` echoed bytes from the server and verify they match the
+ * payload we just sent.
+ */
+static int client_recv_echo(int fd, const char *sent, char *echo_buf,
+ ssize_t len)
+{
+ ssize_t total = 0;
+ ssize_t n;
+
+ while (total < len) {
+ n = recv(fd, echo_buf + total, len - total, 0);
+ if (n < 0) {
+ printf("FAIL: Echo recv failed: %s\n", strerror(errno));
+ return -1;
+ }
+ if (n == 0) {
+ printf("FAIL: Connection closed during echo\n");
+ return -1;
+ }
+ total += n;
+ }
+
+ if (memcmp(sent, echo_buf, len) != 0) {
+ printf("FAIL: Echo data mismatch!\n");
+ return -1;
+ }
+ printf("Received echo %zd bytes (ok)\n", total);
+ return 0;
+}
+
+/* Client side of a rekey: send KeyUpdate and rotate TX. In echo mode
+ * also wait for the peer's KeyUpdate and rotate RX.
+ */
+static int client_rekey(int fd, int generation)
+{
+ if (send_tls_key_update(fd) < 0) {
+ printf("FAIL: send KeyUpdate\n");
+ return -1;
+ }
+
+ if (do_tls_rekey(fd, TLS_TX, generation, cipher_type) < 0)
+ return -1;
+
+ if (burst_mode)
+ return 0;
+
+ if (recv_tls_keyupdate(fd) < 0) {
+ printf("FAIL: recv KeyUpdate from server\n");
+ return -1;
+ }
+
+ if (check_ekeyexpired(fd) < 0)
+ return -1;
+
+ return do_tls_rekey(fd, TLS_RX, generation, cipher_type);
+}
+
+static int do_client(void)
+{
+ char *buf = NULL, *echo_buf = NULL;
+ int max_size, rekey_interval;
+ int csk = -1, i;
+ int test_result = -1;
+ int current_gen = 0;
+ int next_rekey_at;
+ ssize_t n;
+
+ max_size = random_size_max > 0 ? random_size_max : send_size;
+ if (max_size < MIN_BUF_SIZE)
+ max_size = MIN_BUF_SIZE;
+ buf = malloc(max_size);
+ if (!burst_mode)
+ echo_buf = malloc(max_size);
+ if (!buf || (!burst_mode && !echo_buf)) {
+ printf("SETUP ERROR: failed to allocate buffers\n");
+ goto out;
+ }
+
+ csk = client_connect_tls();
+ if (csk < 0)
+ goto out;
+
+ if (num_rekeys)
+ printf("TLS %s setup complete. Will perform %d rekey(s).\n",
+ cipher_name(cipher_type), num_rekeys);
+ else
+ printf("TLS setup complete.\n");
+
+ if (random_size_max > 0)
+ printf("Sending %d messages of random size (1..%d bytes)...\n",
+ num_iterations, random_size_max);
+ else
+ printf("Sending %d messages of %d bytes...\n",
+ num_iterations, send_size);
+
+ rekey_interval = num_iterations / (num_rekeys + 1);
+ next_rekey_at = rekey_interval;
+
+ for (i = 1; i <= num_iterations; i++) {
+ int this_size;
+
+ if (random_size_max > 0)
+ this_size = (rand() % random_size_max) + 1;
+ else
+ this_size = send_size;
+
+ /* In burst mode, use a per-iteration fill pattern so the
+ * receiver can detect any plaintext corruption without a
+ * round-trip echo.
+ */
+ if (burst_mode) {
+ memset(buf, i & 0xFF, this_size);
+ } else {
+ int j;
+
+ for (j = 0; j < this_size; j++)
+ buf[j] = rand() & 0xFF;
+ }
+
+ if (send_all(csk, buf, this_size) < 0)
+ goto out;
+ n = this_size;
+
+ if (!burst_mode) {
+ printf("Sent %zd bytes (iteration %d)\n", n, i);
+ if (client_recv_echo(csk, buf, echo_buf, n) < 0)
+ goto out;
+ }
+
+ /* Rekey at intervals. In echo mode this is a full bidirectional
+ * exchange; in burst mode the client only rotates its TX key
+ * and sends KeyUpdate - the peer is expected to follow.
+ */
+ if (num_rekeys && current_gen < num_rekeys &&
+ i == next_rekey_at) {
+ current_gen++;
+ printf("\n=== Client Rekey gen %d ===\n", current_gen);
+
+ if (client_rekey(csk, current_gen) < 0)
+ goto out;
+
+ next_rekey_at += rekey_interval;
+ printf("=== Client Rekey gen %d Complete ===\n\n",
+ current_gen);
+ }
+ }
+
+ test_result = 0;
+out:
+ if (num_rekeys)
+ printf("Rekeys completed: %d/%d\n", current_gen, num_rekeys);
+ if (csk >= 0)
+ close(csk);
+ free(buf);
+ free(echo_buf);
+ return test_result;
+}
+
+/* Bind/listen on server_port, accept one client, switch to the TLS ULP
+ * and install initial generation-0 keys (plus zc_rx if requested).
+ * Returns the connected fd on success and writes the listener fd to
+ * *lsk_out so the caller can close it. Returns -1 on error, with all
+ * intermediate fds already closed and *lsk_out left at -1.
+ */
+static int server_accept_tls(int *lsk_out)
+{
+ struct addrinfo hints = {0}, *res, *rp;
+ int lsk = -1, csk, one = 1;
+ char port_str[16];
+ int ret;
+
+ *lsk_out = -1;
+
+ /* AI_PASSIVE gives a wildcard bind address for the chosen family
+ * (0.0.0.0 / ::). The family is forced by -4/-6; when unspecified,
+ * bind the first entry that works.
+ */
+ hints.ai_family = force_family;
+ hints.ai_socktype = SOCK_STREAM;
+ hints.ai_protocol = IPPROTO_TCP;
+ hints.ai_flags = AI_PASSIVE;
+ snprintf(port_str, sizeof(port_str), "%d", server_port);
+
+ ret = getaddrinfo(NULL, port_str, &hints, &res);
+ if (ret) {
+ printf("SETUP ERROR: getaddrinfo(port %d): %s\n", server_port,
+ gai_strerror(ret));
+ return -1;
+ }
+
+ for (rp = res; rp; rp = rp->ai_next) {
+ lsk = socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol);
+ if (lsk < 0)
+ continue;
+ setsockopt(lsk, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one));
+ if (bind(lsk, rp->ai_addr, rp->ai_addrlen) == 0)
+ break;
+ close(lsk);
+ lsk = -1;
+ }
+ freeaddrinfo(res);
+
+ if (lsk < 0) {
+ printf("SETUP ERROR: failed to bind port %d: %s\n",
+ server_port, strerror(errno));
+ return -1;
+ }
+
+ if (listen(lsk, 1) < 0) {
+ printf("SETUP ERROR: listen failed: %s\n", strerror(errno));
+ close(lsk);
+ return -1;
+ }
+
+ printf("Server listening on port %d\n", server_port);
+ printf("Waiting for client connection...\n");
+
+ /* Bound accept() so a client that never connects (a deploy or connect
+ * failure on the peer) does not block the server forever and leak the
+ * process past the harness timeout. accept() honours SO_RCVTIMEO on the
+ * listening socket; the client connects right after wait_port_listen(),
+ * so 30s is generous.
+ */
+ {
+ struct timeval tv = { .tv_sec = 30, .tv_usec = 0 };
+
+ setsockopt(lsk, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
+ }
+
+ csk = accept(lsk, (struct sockaddr *)NULL, (socklen_t *)NULL);
+ if (csk < 0) {
+ if (errno == EAGAIN || errno == EWOULDBLOCK)
+ printf("SETUP ERROR: accept timed out; client never connected\n");
+ else
+ printf("SETUP ERROR: accept failed: %s\n", strerror(errno));
+ close(lsk);
+ return -1;
+ }
+ printf("Client connected!\n");
+
+ if (setup_tls_ulp(csk) < 0)
+ goto err;
+
+ if (do_tls_rekey(csk, TLS_TX, 0, cipher_type) < 0 ||
+ do_tls_rekey(csk, TLS_RX, 0, cipher_type) < 0)
+ goto err;
+
+ if (zc_rx && set_zc_rx(csk) < 0)
+ goto err;
+
+ set_io_timeouts(csk);
+ if (!burst_mode)
+ configure_echo_socket(csk, random_size_max > 0 ?
+ random_size_max : send_size);
+
+ *lsk_out = lsk;
+ return csk;
+err:
+ close(csk);
+ close(lsk);
+ return -1;
+}
+
+/* Server side of a rekey: confirm recv() reports EKEYEXPIRED, then rotate RX.
+ * In echo mode also send a KeyUpdate back and rotate TX.
+ */
+static int server_rekey(int fd, int generation)
+{
+ if (check_ekeyexpired(fd) < 0)
+ return -1;
+
+ if (do_tls_rekey(fd, TLS_RX, generation, cipher_type) < 0)
+ return -1;
+
+ if (burst_mode)
+ return 0;
+
+ if (send_tls_key_update(fd) < 0) {
+ printf("FAIL: send KeyUpdate\n");
+ return -1;
+ }
+
+ return do_tls_rekey(fd, TLS_TX, generation, cipher_type);
+}
+
+/* Burst mode: verify one reassembled iteration of send_size plaintext bytes,
+ * each filled with (send_iter & 0xff). Catches decrypt-succeeded-but-
+ * plaintext-corrupt bugs that AEAD counters alone would miss.
+ */
+static int server_verify_burst(const char *buf, int send_iter)
+{
+ unsigned char expect = send_iter & 0xFF;
+ int j;
+
+ for (j = 0; j < send_size; j++) {
+ if ((unsigned char)buf[j] != expect) {
+ printf("FAIL: data mismatch iter %d off %d: exp 0x%02x got 0x%02x\n",
+ send_iter, j, expect, (unsigned char)buf[j]);
+ return -1;
+ }
+ }
+ return 0;
+}
+
+static int do_server(void)
+{
+ int lsk = -1, csk = -1;
+ ssize_t n, total = 0;
+ int test_result = -1;
+ int current_gen = 0;
+ int recv_count = 0;
+ int send_iter = 1;
+ char *buf = NULL;
+ int record_type = 0;
+ int filled = 0;
+ int buf_size;
+
+ buf_size = send_size;
+ if (buf_size < MIN_BUF_SIZE)
+ buf_size = MIN_BUF_SIZE;
+ buf = malloc(buf_size);
+ if (!buf) {
+ printf("SETUP ERROR: failed to allocate buffer\n");
+ goto out;
+ }
+
+ csk = server_accept_tls(&lsk);
+ if (csk < 0)
+ goto out;
+
+ printf("TLS %s setup complete. Receiving...\n",
+ cipher_name(cipher_type));
+
+ /* Burst mode: reassemble one iteration (send_size bytes) in userspace
+ * from however much each recv returns, rather than demanding a full
+ * send_size batch in a single MSG_WAITALL call. A blocking MSG_WAITALL
+ * of send_size deadlocks when the client's last record of an iteration
+ * is still partly in flight as its socket buffer fills: the server
+ * waits for bytes the client cannot send until the server reads, and
+ * the server will not read until it has the whole batch. Draining
+ * whatever is available keeps the receive window open and breaks that
+ * cycle. kTLS never splits a record and returns data and control
+ * (KeyUpdate) records separately, and each iteration is a whole number
+ * of records, so capping each recv at the iteration boundary keeps the
+ * reassembly aligned and delivers a KeyUpdate on its own.
+ */
+
+ /* Main receive loop */
+ while (1) {
+ char *dst = burst_mode ? buf + filled : buf;
+ size_t want = burst_mode ? (size_t)(send_size - filled)
+ : (size_t)buf_size;
+
+ n = recv_tls_message(csk, dst, want, &record_type, 0);
+ if (n == 0) {
+ /* A clean close on an iteration boundary is success;
+ * one with a partial iteration still buffered means the
+ * peer dropped the tail - the truncated-data case this
+ * test exists to catch, so fail loudly.
+ */
+ if (burst_mode && filled) {
+ printf("FAIL: closed mid-iteration (%d/%d bytes buffered)\n",
+ filled, send_size);
+ goto out;
+ }
+ printf("Connection closed by client\n");
+ break;
+ }
+ if (n < 0) {
+ printf("FAIL: recv failed: %s\n", strerror(errno));
+ goto out;
+ }
+
+ /* Handle KeyUpdate. In echo mode the server mirrors the
+ * rekey back to the peer; in burst mode it only rotates its
+ * RX key and keeps draining. A KeyUpdate always lands on a
+ * send_size boundary, so no partial iteration must be buffered
+ * when one arrives.
+ */
+ if (record_type == TLS_RECORD_TYPE_HANDSHAKE) {
+ /* Check for a partial iteration before validating the
+ * KeyUpdate, so a mid-iteration arrival fails with this
+ * message rather than a misleading KeyUpdate-OK line.
+ */
+ if (burst_mode && filled) {
+ printf("FAIL: KeyUpdate mid-iteration (%d/%d bytes buffered)\n",
+ filled, send_size);
+ goto out;
+ }
+ if (check_keyupdate(dst, n, record_type) < 0)
+ goto out;
+ current_gen++;
+ printf("\n=== Server Rekey gen %d ===\n", current_gen);
+
+ if (server_rekey(csk, current_gen) < 0)
+ goto out;
+
+ printf("=== Server Rekey gen %d Complete ===\n\n",
+ current_gen);
+ continue;
+ }
+
+ total += n;
+
+ if (burst_mode) {
+ filled += n;
+ if (filled < send_size)
+ continue;
+ if (server_verify_burst(buf, send_iter) < 0)
+ goto out;
+ recv_count++;
+ send_iter++;
+ filled = 0;
+ continue;
+ }
+
+ recv_count++;
+ printf("Received %zd bytes (total: %zd, count: %d)\n",
+ n, total, recv_count);
+
+ if (send_all(csk, buf, n) < 0)
+ goto out;
+ printf("Echoed %zd bytes back to client\n", n);
+ }
+
+ test_result = 0;
+out:
+ printf("Connection closed. Total received: %zd bytes\n", total);
+ if (num_rekeys)
+ printf("Rekeys completed: %d\n", current_gen);
+
+ if (csk >= 0)
+ close(csk);
+ if (lsk >= 0)
+ close(lsk);
+ free(buf);
+ return test_result;
+}
+
+static int parse_int_arg(const char *arg, int min, int max,
+ const char *name, int *out)
+{
+ char *endp;
+ long val;
+
+ errno = 0;
+ val = strtol(arg, &endp, 10);
+ if (errno || endp == arg || *endp != '\0' || val < min || val > max) {
+ if (max == INT_MAX)
+ printf("ERROR: Invalid %s '%s'. Must be >= %d.\n",
+ name, arg, min);
+ else
+ printf("ERROR: Invalid %s '%s'. Must be %d..%d.\n",
+ name, arg, min, max);
+ return -1;
+ }
+ *out = (int)val;
+ return 0;
+}
+
+static int parse_cipher_option(const char *arg)
+{
+ if (strcmp(arg, "128") == 0) {
+ cipher_type = TLS_CIPHER_AES_GCM_128;
+ return 0;
+ } else if (strcmp(arg, "256") == 0) {
+ cipher_type = TLS_CIPHER_AES_GCM_256;
+ return 0;
+ }
+ printf("ERROR: Invalid cipher '%s'. Must be 128 or 256.\n", arg);
+ return -1;
+}
+
+static int parse_version_option(const char *arg)
+{
+ if (strcmp(arg, "1.2") == 0) {
+ tls_version = TLS_1_2_VERSION;
+ return 0;
+ } else if (strcmp(arg, "1.3") == 0) {
+ tls_version = TLS_1_3_VERSION;
+ return 0;
+ }
+ printf("ERROR: Invalid TLS version '%s'. Must be 1.2 or 1.3.\n", arg);
+ return -1;
+}
+
+static void print_usage(const char *prog)
+{
+ printf("TLS Hardware Offload Two-Node Test\n\n");
+ printf("Usage:\n");
+ printf(" %s server [OPTIONS]\n", prog);
+ printf(" %s client -s <ip> [OPTIONS]\n", prog);
+ printf("\nOptions:\n");
+ printf(" -s <ip> Server IP address, v4 or v6 (client, required)\n");
+ printf(" -p <port> Server port (default: 4433)\n");
+ printf(" -4 Force IPv4 (default: auto/either)\n");
+ printf(" -6 Force IPv6 (default: auto/either)\n");
+ printf(" -b <size> Send buffer size in bytes (default: 16384)\n");
+ printf(" -r <max> Use random send buffer sizes (1..<max>)\n");
+ printf(" -v <version> TLS version: 1.2 or 1.3 (default: 1.3)\n");
+ printf(" -c <cipher> Cipher: 128 or 256 (default: 128)\n");
+ printf(" -n <N> Number of send/echo iterations (default: 100)\n");
+ printf(" -k <N> Perform N rekeys (client only, TLS 1.3; N < iterations)\n");
+ printf(" -B Burst mode: client sends continuously without echo;\n");
+ printf(" server drains and handles KeyUpdate without responding.\n");
+ printf(" -Z Set TLS_RX_EXPECT_NO_PAD on the server: TLS 1.3\n");
+ printf(" opt-in to the zero-copy RX fast path. Not needed\n");
+ printf(" for TLS 1.2 (always eligible). Server only.\n");
+ printf(" -h Show this help message\n");
+ printf("\nExample:\n");
+ printf(" Node A: %s server\n", prog);
+ printf(" Node B: %s client -s 192.168.20.2\n", prog);
+ printf("\nRekey Example (3 rekeys, TLS 1.3 only):\n");
+ printf(" Node A: %s server\n", prog);
+ printf(" Node B: %s client -s 192.168.20.2 -k 3\n", prog);
+ printf("\nBurst Mode Example (client stresses TX rekey under load):\n");
+ printf(" Node A: %s server -B\n", prog);
+ printf(" Node B: %s client -s 192.168.20.2 -B -k 3\n", prog);
+ printf("\nIPv6 Example:\n");
+ printf(" Node A: %s server -6\n", prog);
+ printf(" Node B: %s client -6 -s fd00::2\n", prog);
+}
+
+int main(int argc, char *argv[])
+{
+ int send_size_set = 0;
+ int is_server;
+ int opt;
+
+ /* When the peer aborts a TLS connection (e.g. tls_err_abort() on a
+ * failed decrypt), a send() here would raise SIGPIPE and kill us by
+ * signal, so the harness sees only a bare non-zero exit with no
+ * "FAIL:" line. Ignore it and let send()/sendmsg() return EPIPE, which
+ * send_all()/send_tls_key_update() report.
+ */
+ signal(SIGPIPE, SIG_IGN);
+
+ if (argc < 2 ||
+ (strcmp(argv[1], "server") && strcmp(argv[1], "client"))) {
+ print_usage(argv[0]);
+ return 1;
+ }
+ is_server = !strcmp(argv[1], "server");
+
+ optind = 2; /* skip subcommand */
+ while ((opt = getopt(argc, argv, "s:p:b:r:c:v:k:n:BZ46h")) != -1) {
+ switch (opt) {
+ case 's':
+ server_ip = optarg;
+ break;
+ case '4':
+ if (force_family == AF_INET6) {
+ printf("ERROR: -4 and -6 are mutually exclusive\n");
+ return 1;
+ }
+ force_family = AF_INET;
+ break;
+ case '6':
+ if (force_family == AF_INET) {
+ printf("ERROR: -4 and -6 are mutually exclusive\n");
+ return 1;
+ }
+ force_family = AF_INET6;
+ break;
+ case 'B':
+ burst_mode = 1;
+ break;
+ case 'Z':
+ zc_rx = 1;
+ break;
+ case 'p':
+ if (parse_int_arg(optarg, 1, 65535, "port",
+ &server_port) < 0)
+ return 1;
+ break;
+ case 'b':
+ if (parse_int_arg(optarg, 1, INT_MAX, "buffer size",
+ &send_size) < 0)
+ return 1;
+ send_size_set = 1;
+ break;
+ case 'r':
+ if (parse_int_arg(optarg, 1, INT_MAX, "random size",
+ &random_size_max) < 0)
+ return 1;
+ break;
+ case 'c':
+ if (parse_cipher_option(optarg) < 0)
+ return 1;
+ break;
+ case 'v':
+ if (parse_version_option(optarg) < 0)
+ return 1;
+ break;
+ case 'k':
+ if (parse_int_arg(optarg, 1, 255, "rekey count",
+ &num_rekeys) < 0)
+ return 1;
+ break;
+ case 'n':
+ if (parse_int_arg(optarg, 1, INT_MAX, "iteration count",
+ &num_iterations) < 0)
+ return 1;
+ break;
+ case 'h':
+ print_usage(argv[0]);
+ return 0;
+ default:
+ print_usage(argv[0]);
+ return 1;
+ }
+ }
+
+ if (send_size_set && random_size_max > 0) {
+ printf("ERROR: -b and -r are mutually exclusive\n");
+ return 1;
+ }
+
+ if (zc_rx && tls_version != TLS_1_3_VERSION) {
+ printf("ERROR: -Z (TLS_RX_EXPECT_NO_PAD) requires TLS 1.3\n");
+ return 1;
+ }
+
+ if (burst_mode && random_size_max > 0) {
+ printf("ERROR: -B and -r are mutually exclusive\n");
+ return 1;
+ }
+
+ if (burst_mode && send_size < MIN_BUF_SIZE) {
+ printf("ERROR: -b must be >= %d in burst mode (-B)\n",
+ MIN_BUF_SIZE);
+ return 1;
+ }
+
+ if (is_server) {
+ if (server_ip) {
+ printf("warning: -s is ignored in server mode\n");
+ server_ip = NULL;
+ }
+ if (random_size_max > 0) {
+ printf("warning: -r is ignored in server mode\n");
+ random_size_max = 0;
+ }
+ if (num_rekeys) {
+ printf("warning: -k is ignored in server mode\n");
+ num_rekeys = 0;
+ }
+ } else {
+ if (!server_ip) {
+ printf("ERROR: Client requires -s <ip> option\n");
+ return 1;
+ }
+ if (tls_version == TLS_1_2_VERSION && num_rekeys) {
+ printf("ERROR: TLS 1.2 does not support rekey\n");
+ return 1;
+ }
+ if (num_rekeys >= num_iterations) {
+ printf("ERROR: num_rekeys (%d) must be < num_iterations (%d)\n",
+ num_rekeys, num_iterations);
+ return 1;
+ }
+ if (zc_rx) {
+ printf("ERROR: -Z applies to the server (receiver) only\n");
+ return 1;
+ }
+ }
+
+ printf("TLS Version: %s\n", version_name(tls_version));
+ printf("Cipher: %s\n", cipher_name(cipher_type));
+ printf("Address family: %s\n",
+ force_family == AF_INET ? "IPv4" :
+ force_family == AF_INET6 ? "IPv6" : "auto");
+ if (random_size_max > 0)
+ printf("Buffer size: random (1..%d)\n", random_size_max);
+ else
+ printf("Buffer size: %d\n", send_size);
+
+ if (num_rekeys)
+ printf("Rekey testing ENABLED: %d rekey(s)\n", num_rekeys);
+ if (burst_mode)
+ printf("Burst mode ENABLED\n");
+ if (zc_rx)
+ printf("TLS_RX_EXPECT_NO_PAD ENABLED\n");
+
+ srand(time(NULL));
+
+ if (is_server)
+ return do_server() ? 1 : 0;
+
+ return do_client() ? 1 : 0;
+}
diff --git a/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py
new file mode 100755
index 0000000000000..99ae5b3b8996a
--- /dev/null
+++ b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py
@@ -0,0 +1,446 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""Test kTLS hardware offload using a C helper binary."""
+
+from collections import defaultdict
+
+from lib.py import ksft_run, ksft_exit, ksft_pr, KsftSkipEx
+from lib.py import ksft_ge, ksft_eq
+from lib.py import ksft_variants, KsftNamedVariant
+from lib.py import NetDrvEpEnv
+from lib.py import cmd, bkg, wait_port_listen, rand_port
+from lib.py import CmdExitFailure
+
+# Burst variants push hundreds of MB and perform many rekeys, so they
+# need far longer than the default cmd() timeout.
+BURST_TIMEOUT_S = 180
+REKEY_TIMEOUT_S = 90
+
+# Reading /proc/net/tls_stat is trivial locally, but on the remote it runs
+# over ssh, where connection setup can occasionally spike past the short
+# default cmd() timeout. Give these tiny reads plenty of headroom so a slow
+# ssh round-trip doesn't fail an otherwise-good variant.
+STATS_TIMEOUT_S = 30
+
+# Per-packet HW crypto counters exposed via `ethtool -S` on the DUT NIC,
+# keyed by the `ethtool -i` driver name. TlsTxDevice/TlsRxDevice in
+# /proc/net/tls_stat only prove tls_dev_add() accepted the offload; these
+# increment once per packet the NIC actually encrypted/decrypted (mlx5 counts
+# gso_segs, not records), so they prove the HW crypto path was exercised.
+# Names are driver-specific, so the
+# check only runs on drivers listed here and is skipped (not failed) on
+# others, keeping the test portable across NICs.
+HW_CRYPTO_COUNTERS = {
+ 'mlx5_core': {'Tx': 'tx_tls_encrypted_packets',
+ 'Rx': 'rx_tls_decrypted_packets'},
+}
+
+
+def check_tls_support(cfg):
+ """Skip the suite unless both hosts have kTLS and the DUT HW offload."""
+ # The tls module is autoloaded lazily on the first TCP_ULP="tls"
+ # setsockopt, so /proc/net/tls_stat (created from the module's pernet
+ # init) may not exist yet on a freshly booted host. Load the module
+ # explicitly before probing for it.
+ try:
+ cmd("modprobe tls")
+ cmd("modprobe tls", host=cfg.remote)
+ cmd("test -f /proc/net/tls_stat")
+ cmd("test -f /proc/net/tls_stat", host=cfg.remote)
+ except CmdExitFailure as e:
+ raise KsftSkipEx(f"kTLS not supported: {e}") from e
+
+ try:
+ features = cmd(f"ethtool -k {cfg.ifname}").stdout
+ if 'tls-hw-tx-offload: on' not in features:
+ raise KsftSkipEx("Device does not support TLS HW TX offload")
+ if 'tls-hw-rx-offload: on' not in features:
+ raise KsftSkipEx("Device does not support TLS HW RX offload")
+ except CmdExitFailure as e:
+ raise KsftSkipEx(f"Cannot determine TLS HW offload support: {e}") from e
+
+
+def read_tls_stats(host=None):
+ """Snapshot the per-netns TLS MIB from /proc/net/tls_stat as a dict."""
+ # /proc/net/tls_stat exposes the per-netns TLS MIB (TLS_INC_STATS on
+ # sock_net(sk)). The test runs on a real NIC in the host namespace, so
+ # these counters are shared with anything else doing kTLS there. The
+ # strict before/after delta checks (exact rekey-outcome sums, zero error
+ # counters) assume no other kTLS activity in this namespace during a
+ # variant's window; concurrent kTLS users would perturb the deltas and
+ # cause spurious failures. Don't run other kTLS workloads alongside this
+ # test.
+ stats = defaultdict(int)
+ output = cmd("cat /proc/net/tls_stat", host=host, timeout=STATS_TIMEOUT_S)
+ for line in output.stdout.strip().split('\n'):
+ parts = line.split()
+ if len(parts) == 2:
+ stats[parts[0]] = int(parts[1])
+ return stats
+
+
+def nic_driver(cfg):
+ """DUT NIC driver name from `ethtool -i`, or None if undetermined."""
+ try:
+ output = cmd(f"ethtool -i {cfg.ifname}").stdout
+ except CmdExitFailure:
+ return None
+ for line in output.splitlines():
+ if line.startswith('driver:'):
+ return line.split(':', 1)[1].strip()
+ return None
+
+
+def read_nic_stats(cfg):
+ """Snapshot the DUT NIC's `ethtool -S` counters as a dict."""
+ # Driver per-record TLS counters from `ethtool -S` on the DUT NIC. Same
+ # before/after-delta caveat as read_tls_stats(): these are device-wide,
+ # so concurrent kTLS traffic on this NIC would perturb the deltas.
+ stats = defaultdict(int)
+ output = cmd(f"ethtool -S {cfg.ifname}").stdout
+ for line in output.strip().split('\n'):
+ key, sep, val = line.partition(':')
+ if sep and val.strip().isdigit():
+ stats[key.strip()] = int(val.strip())
+ return stats
+
+
+def stat_diff(before, after, key):
+ """Return the delta of counter `key` between two stat snapshots."""
+ return after[key] - before[key]
+
+
+def check_hw_crypto(cfg, before, after, with_tx, with_rx):
+ """DUT-side ethtool -S check: the NIC actually crypto'd records in HW.
+
+ Complements the TlsTxDevice/TlsRxDevice MIBs, which only confirm the
+ offload was installed, not that any record was processed in hardware.
+ Driver-specific; skipped (without failing) on drivers not in
+ HW_CRYPTO_COUNTERS so the test stays portable.
+ """
+ counters = HW_CRYPTO_COUNTERS.get(cfg.nic_driver)
+ if not counters:
+ ksft_pr(f"NOTE: DUT driver '{cfg.nic_driver}' has no known per-record "
+ f"HW crypto counters, skipping ethtool -S check")
+ return
+
+ for direction, active in (('Tx', with_tx), ('Rx', with_rx)):
+ if not active:
+ continue
+ key = counters[direction]
+ if key not in after:
+ ksft_pr(f"NOTE: DUT {direction}: counter '{key}' not exposed by "
+ f"{cfg.nic_driver}, skipping")
+ continue
+ got = stat_diff(before, after, key)
+ ksft_ge(got, 1,
+ comment=f"DUT {direction}: NIC reported no HW crypto "
+ f"({key}={got})")
+
+
+def check_path(before, after, direction, role, require_hw):
+ """On the DUT, require HW offload; on the remote, HW or SW is fine."""
+ dev = stat_diff(before, after, f'Tls{direction}Device')
+ sw = stat_diff(before, after, f'Tls{direction}Sw')
+ if require_hw:
+ ksft_ge(dev, 1,
+ comment=f"{role} {direction}: HW offload not engaged "
+ f"(Device={dev}, Sw={sw})")
+ else:
+ ksft_ge(dev + sw, 1,
+ comment=f"{role} {direction}: no TLS activity "
+ f"(Device={dev}, Sw={sw})")
+
+
+def verify_tls_counters(stats_before, stats_after, expected_rekeys,
+ tls_role, is_dut, burst=False, allow_fallback=False):
+ """Verify TLS counters on one side of the connection.
+
+ tls_role: 'client' or 'server' (TLS role this side played).
+ is_dut: True for the local DUT; requires HW offload counters.
+ burst: burst mode - only the TLS client rotates its TX key; the TLS
+ server only follows with an RX rotation on KeyUpdate receipt.
+ allow_fallback: tolerate rekeys completing in SW (TlsRx/TxRekeyFallback).
+ Default False: a rekey on an up, offload-capable device must stay
+ in HW, so any fallback is a regression. Set True only where SW
+ fallback is expected (e.g. a mid-connection link-flap variant, or
+ the peer, whose offload state is not under test).
+ """
+ role = 'DUT' if is_dut else 'Peer'
+
+ def diff(key):
+ return stat_diff(stats_before, stats_after, key)
+
+ # In burst mode the TLS client only TXs and the TLS server only RXs.
+ # In echo mode both sides drive both directions.
+ with_tx = not burst or tls_role == 'client'
+ with_rx = not burst or tls_role != 'client'
+
+ if with_tx:
+ check_path(stats_before, stats_after, 'Tx', role, require_hw=is_dut)
+ if with_rx:
+ check_path(stats_before, stats_after, 'Rx', role, require_hw=is_dut)
+
+ if expected_rekeys > 0:
+ if with_tx:
+ # Each KeyUpdate yields exactly one terminal outcome, so
+ # TlsTxRekeyOk + TlsTxRekeyAborted + TlsTxRekeyFallback == N.
+ # At most one rekey can be PENDING at socket close (single
+ # TLS_TX_REKEY_PENDING bit), so at most one lands in
+ # TlsTxRekeyAborted. TlsTxRekeyFallback is a legitimate, graceful
+ # degradation: the device did not (re)install the HW context for
+ # that rekey (device gone, dev_add rejected, or a transient
+ # crypto/alloc error) so it completed in SW while the kernel
+ # returned success. It is recoverable - the next KeyUpdate
+ # re-attempts HW offload (tls_device_start_rekey() clears
+ # TLS_TX_REKEY_FAILED). It is folded into the outcome sum below; on
+ # the DUT it must be 0 (allow_fallback=False), on the peer it is
+ # only NOTEd. A genuine rekey bug still surfaces as TlsTxRekeyError.
+ ksft_ge(1, diff('TlsTxRekeyAborted'),
+ comment=f"{role} Tx: TlsTxRekeyAborted expected <= 1")
+ ksft_eq(diff('TlsTxRekeyOk') + diff('TlsTxRekeyAborted') +
+ diff('TlsTxRekeyFallback'), expected_rekeys,
+ comment=f"{role} Tx: rekey outcomes must sum to "
+ f"{expected_rekeys}")
+ fallback = diff('TlsTxRekeyFallback')
+ if allow_fallback:
+ if fallback:
+ ksft_pr(f"NOTE: {role} Tx: {fallback} rekey(s) completed "
+ f"in SW (TlsTxRekeyFallback); HW not re-installed")
+ else:
+ ksft_eq(fallback, 0,
+ comment=f"{role} Tx: TlsTxRekeyFallback expected 0 "
+ f"(rekey must stay in HW offload)")
+ ksft_eq(diff('TlsTxRekeyError'), 0,
+ comment=f"{role} Tx: TlsTxRekeyError expected 0")
+ ksft_eq(diff('TlsCurrTxRekey'), 0,
+ comment=f"{role} Tx: TlsCurrTxRekey expected 0")
+ if with_rx:
+ # As on TX, each received KeyUpdate yields one terminal outcome:
+ # TlsRxRekeyOk + TlsRxRekeyAborted + TlsRxRekeyFallback == N.
+ # At most one rekey can be deferred (single dev_add_pending) at
+ # socket close, landing in TlsRxRekeyAborted. TlsRxRekeyFallback
+ # is a recoverable, graceful degradation (dev_add failed or the
+ # device was gone, so RX temporarily dropped to SW; the next
+ # KeyUpdate re-adds the HW context and clears TLS_RX_DEV_DEGRADED).
+ # It is folded into the outcome sum below; on the DUT it must be 0
+ # (allow_fallback=False), on the peer it is only NOTEd. A genuine
+ # rekey bug still surfaces as TlsRxRekeyError.
+ ksft_ge(1, diff('TlsRxRekeyAborted'),
+ comment=f"{role} Rx: TlsRxRekeyAborted expected <= 1")
+ ksft_eq(diff('TlsRxRekeyOk') + diff('TlsRxRekeyAborted') +
+ diff('TlsRxRekeyFallback'), expected_rekeys,
+ comment=f"{role} Rx: rekey outcomes must sum to "
+ f"{expected_rekeys}")
+ ksft_eq(diff('TlsRxRekeyReceived'), expected_rekeys,
+ comment=f"{role} Rx: TlsRxRekeyReceived expected "
+ f"{expected_rekeys}")
+ fallback = diff('TlsRxRekeyFallback')
+ if allow_fallback:
+ if fallback:
+ ksft_pr(f"NOTE: {role} Rx: {fallback} rekey(s) completed "
+ f"in SW (TlsRxRekeyFallback); HW not re-installed")
+ else:
+ ksft_eq(fallback, 0,
+ comment=f"{role} Rx: TlsRxRekeyFallback expected 0 "
+ f"(rekey must stay in HW offload)")
+ ksft_eq(diff('TlsRxRekeyError'), 0,
+ comment=f"{role} Rx: TlsRxRekeyError expected 0")
+ ksft_eq(diff('TlsCurrRxRekey'), 0,
+ comment=f"{role} Rx: TlsCurrRxRekey expected 0")
+
+ ksft_eq(diff('TlsDecryptError'), 0,
+ comment=f"{role}: TlsDecryptError expected 0")
+
+
+def run_tls_test(cfg, cipher="128", tls_version="1.3", rekey=0,
+ buffer_size=None, random_max=None, burst=False, zc=False,
+ dut_role="client", num_iterations=None, ipver="4"):
+ """Run the TLS offload test.
+
+ dut_role: 'client' (default) - DUT runs the TLS client, remote the server.
+ 'server' - swap: DUT listens, remote connects. Used for burst_rx
+ so the DUT's RX path is the one under rekey pressure.
+
+ ipver: '4' or '6' - IP version to run over. The C helper is forced to the
+ matching family with -4/-6 and connects to the peer's v4/v6 address.
+ Variants requesting '6' skip cleanly when the environment lacks IPv6
+ connectivity (require_ipver()).
+
+ The DUT (local) is the kernel under test; the remote is just a traffic
+ source/sink and may run any kernel without HW offload. Both sides run
+ kTLS because TLS is pairwise, but verify_tls_counters() requires HW
+ offload only on the DUT (is_dut=True); the peer may use SW kTLS.
+
+ Rekey/burst variants additionally require the peer to support TLS 1.3
+ KeyUpdate (as the RX or TX side of the rotation). SW KeyUpdate and its
+ MIB counters landed together in v6.14; an older peer cannot follow the
+ rotation, so those variants are skipped rather than failed when the peer
+ lacks the rekey counters (see the probe below).
+ """
+ cfg.require_ipver(ipver)
+
+ port = rand_port()
+ send_size = random_max or buffer_size
+
+ if dut_role == "client":
+ server_bin, server_host = cfg.bin_remote, cfg.remote
+ client_bin, client_host = cfg.bin_local, None
+ client_target = cfg.remote_addr_v[ipver]
+ else:
+ server_bin, server_host = cfg.bin_local, None
+ client_bin, client_host = cfg.bin_remote, cfg.remote
+ client_target = cfg.addr_v[ipver]
+
+ server_parts = [f"{server_bin} server -p {port} -c {cipher}",
+ f"-v {tls_version}", f"-{ipver}"]
+ if burst:
+ server_parts.append("-B")
+ if zc:
+ server_parts.append("-Z")
+ if send_size:
+ server_parts.append(f"-b {send_size}")
+ server_cmd = " ".join(server_parts)
+
+ client_parts = [f"{client_bin} client -s {client_target}",
+ f"-p {port} -c {cipher} -v {tls_version} -{ipver}"]
+ if rekey:
+ client_parts.append(f"-k {rekey}")
+ if burst:
+ client_parts.append("-B")
+ if num_iterations:
+ client_parts.append(f"-n {num_iterations}")
+ if random_max:
+ client_parts.append(f"-r {random_max}")
+ elif buffer_size:
+ client_parts.append(f"-b {buffer_size}")
+ client_cmd = " ".join(client_parts)
+
+ if burst:
+ cmd_timeout = BURST_TIMEOUT_S
+ elif rekey:
+ cmd_timeout = REKEY_TIMEOUT_S
+ else:
+ cmd_timeout = 20
+
+ stats_before_local = read_tls_stats()
+ stats_before_remote = read_tls_stats(host=cfg.remote)
+ nic_before = read_nic_stats(cfg)
+
+ # /proc/net/tls_stat lists every MIB the running kernel knows (0 or not),
+ # so a missing name means the peer predates that counter. The base rekey
+ # counters (TlsRxRekeyReceived, Tls{Rx,Tx}RekeyOk, Tls{Rx,Tx}RekeyError)
+ # shipped with SW KeyUpdate in v6.14; a peer without them cannot follow a
+ # KeyUpdate, so the rekey/burst variants can't run against it. Skip cleanly
+ # here rather than letting the peer-side rekey-sum / RxRekeyReceived checks
+ # report a confusing "expected N, got 0" later. TlsRxRekeyReceived is a
+ # reliable probe: the peer must bump it to have processed the rotation at all.
+ #
+ # Only a base v6.14 counter is probed. The newer HW-path MIBs (Aborted,
+ # Fallback, CurrRekey) are structurally 0 on a SW-only peer and defaultdict
+ # returns 0 for absent names, so the peer-side checks hold either way.
+ if rekey and 'TlsRxRekeyReceived' not in stats_before_remote:
+ raise KsftSkipEx("Peer kernel lacks TLS 1.3 KeyUpdate support "
+ "(no rekey MIB counters); required for rekey tests")
+
+ with bkg(server_cmd, host=server_host, exit_wait=True):
+ wait_port_listen(port, host=server_host)
+ # Start the client in the background so we keep a handle to it. A
+ # foreground cmd() raises TimeoutExpired from inside its constructor
+ # if the client hangs, and since the child is not killed on timeout
+ # it would be left running with no handle to reap it. A leaked
+ # client keeps bumping the per-netns TLS counters (TlsTxRekeyAborted,
+ # TlsDecryptError, ...) and would corrupt the before/after
+ # measurement window of a later variant. The finally clause reaps it
+ # within this variant's window instead.
+ client = cmd(client_cmd, host=client_host, background=True)
+ try:
+ client.process(terminate=False, fail=True, timeout=cmd_timeout)
+ finally:
+ if client.proc.poll() is None:
+ client.process(terminate=True, fail=False, timeout=5)
+
+ stats_after_local = read_tls_stats()
+ stats_after_remote = read_tls_stats(host=cfg.remote)
+ nic_after = read_nic_stats(cfg)
+
+ peer_tls_role = 'server' if dut_role == 'client' else 'client'
+
+ # Which directions the DUT drives (mirrors verify_tls_counters()): in
+ # burst mode the TLS client only TXs and the server only RXs; echo mode
+ # drives both.
+ dut_with_tx = not burst or dut_role == 'client'
+ dut_with_rx = not burst or dut_role != 'client'
+
+ verify_tls_counters(stats_before_local, stats_after_local,
+ rekey, dut_role, is_dut=True, burst=burst)
+ check_hw_crypto(cfg, nic_before, nic_after, dut_with_tx, dut_with_rx)
+ verify_tls_counters(stats_before_remote, stats_after_remote,
+ rekey, peer_tls_role, is_dut=False, burst=burst,
+ allow_fallback=True)
+
+
+# The cipher/version matrix runs over IPv4; the socket setup is the only
+# IP-version-specific code path, so a single representative variant over
+# IPv6 is enough to cover it (it skips cleanly without v6 connectivity).
+# The rekey and burst suites below likewise stay on IPv4 to bound runtime.
+@ksft_variants([
+ KsftNamedVariant("tls13_aes128", "128", "1.3", "4"),
+ KsftNamedVariant("tls13_aes256", "256", "1.3", "4"),
+ KsftNamedVariant("tls12_aes128", "128", "1.2", "4"),
+ KsftNamedVariant("tls12_aes256", "256", "1.2", "4"),
+ KsftNamedVariant("tls13_aes128_ip6", "128", "1.3", "6"),
+])
+def test_tls_offload(cfg, cipher, tls_version, ipver):
+ """Cipher/version matrix over the HW offload data path, no rekey."""
+ run_tls_test(cfg, cipher=cipher, tls_version=tls_version, ipver=ipver)
+
+
+@ksft_variants([
+ KsftNamedVariant("single", 1),
+ KsftNamedVariant("multiple", 99),
+ KsftNamedVariant("small_buf", 30, 512),
+ KsftNamedVariant("large_buf", 10, 2097152),
+ KsftNamedVariant("random_buf", 20, None, 8192),
+])
+def test_tls_offload_rekey(cfg, rekey, buffer_size=None, random_max=None):
+ """Echo-mode TLS 1.3 KeyUpdate rekeys across a range of buffer sizes."""
+ run_tls_test(cfg, cipher="128", tls_version="1.3", rekey=rekey,
+ buffer_size=buffer_size, random_max=random_max)
+
+
+# Columns: dut_role zc interval rekeys buffer_size
+@ksft_variants([
+ KsftNamedVariant("burst_tx_rekey_every_1", "client", False, 1, 50, 65536),
+ KsftNamedVariant("burst_tx_rekey_every_1000", "client", False, 1000, 3, 65536),
+ KsftNamedVariant("burst_rx_rekey_every_10", "server", False, 10, 20, 65536),
+ KsftNamedVariant("burst_rx_rekey_every_10000", "server", False, 10000, 1, 32768),
+ KsftNamedVariant("burst_rx_zc_rekey_every_100", "server", True, 100, 10, 65536),
+ KsftNamedVariant("burst_rx_zc_rekey_every_20000", "server", True, 20000, 1, 16384),
+])
+def test_tls_offload_burst(cfg, dut_role, zc, interval, rekeys, buffer_size):
+ """High-volume one-directional traffic with frequent rekeys."""
+ run_tls_test(cfg, cipher="128", tls_version="1.3", rekey=rekeys,
+ buffer_size=buffer_size, burst=True, zc=zc, dut_role=dut_role,
+ num_iterations=interval * (rekeys + 1))
+
+
+def main() -> None:
+ """Set up the DUT/peer environment and run the offload test suites."""
+ with NetDrvEpEnv(__file__, nsim_test=False) as cfg:
+ cfg.bin_local = cfg.test_dir / "tls_hw_offload"
+ if not cfg.bin_local.exists():
+ raise KsftSkipEx(f"tls_hw_offload binary not found at {cfg.bin_local}")
+ cfg.bin_remote = cfg.remote.deploy(cfg.bin_local)
+ cfg.require_ipver("4")
+ check_tls_support(cfg)
+ cfg.nic_driver = nic_driver(cfg)
+
+ ksft_run([test_tls_offload, test_tls_offload_rekey,
+ test_tls_offload_burst], args=(cfg, ))
+ ksft_exit()
+
+
+if __name__ == "__main__":
+ main()
|
| 3/2 | 2026/09/20 01:00 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 72958 output: 19 thoughts: 755 Thoughts: **My Assessment of the Git Commit for Syzkaller Fuzzing** Okay, here's the deal. I need to figure out if this recent git commit warrants a fuzzing run with syzkaller. The prompt is clear; it's already in HEAD, so we're evaluating what's been *done*. First things first, I need to remind myself of the core triage philosophy. **Reachability is key.** If the modified code is structurally unreachable within a standard virtualized environment (GCE or QEMU using software emulations), then forget about it. No blind trust in claims of "no functional change" or "refactoring" either; assertions in reachable code? *Definitely* fuzzable. Now, let me review the negative criteria to quickly disqualify any obviously inappropriate candidates: documentation, selftests, anything that's non-kernel, and anything that needs special hardware that's not emulated. We are looking at standard QEMU / GCE availability. Also, unreachable execution paths are out. Okay, positive criteria are core subsystems, reachable drivers and protocols. So far, so good. Now, I need to understand what this patch actually modified. Let's see... *reads the file list* ...Hmm, looks like a mix. Documentation, MAINTAINERS, some includes, some TLS-related code (both kernel and user-space), and a few drivers (Chelsio, Mellanox, Netronome). I see `net/tls/tls_device.c` in there. That's interesting. TLS, especially device offload, is something I need to carefully consider. This is where my expertise comes in. The crucial question: *Is TLS device offload reachable in QEMU?* If it requires some special hardware configuration that isn't emulated in QEMU (or GCE), then it's a dead end. I need to figure this out. I have to think about whether the offload can be exercised without physical hardware. Okay, let's explore. I wonder if `netdevsim` supports TLS device offload. I need to check that. The answer to *that* question is what's going to decide whether we fire up syzkaller on this thing. I need to investigate `netdevsim`... |
| 4/2 | 2026/09/20 01:00 | tool | grepper |
0mArgs:{
"Expression": "tls_dev_add"
}Results: {
"Output": "Documentation/networking/tls-offload.rst=79=as if the offload was never tried.\nDocumentation/networking/tls-offload.rst-80-\nDocumentation/networking/tls-offload.rst:81:Offload request is performed via the :c:member:`tls_dev_add` callback of\nDocumentation/networking/tls-offload.rst-82-:c:type:`struct tlsdev_ops \u003ctlsdev_ops\u003e`:\n--\nDocumentation/networking/tls-offload.rst-85-\nDocumentation/networking/tls-offload.rst:86:\tint (*tls_dev_add)(struct net_device *netdev, struct sock *sk,\nDocumentation/networking/tls-offload.rst-87-\t\t\t enum tls_offload_ctx_dir direction,\n--\nDocumentation/networking/tls-offload.rst=115=between directions or the socket as a whole. After a rekey ``tls_dev_del``,\nDocumentation/networking/tls-offload.rst:116:``tls_dev_add`` may be called again for the same socket and direction to\nDocumentation/networking/tls-offload.rst:117:install the new key. ``tls_dev_del`` is mandatory whenever ``tls_dev_add`` is\nDocumentation/networking/tls-offload.rst-118-provided.\n--\nDocumentation/networking/tls-offload.rst=452=as ready; the device is not touched. The switch itself,\nDocumentation/networking/tls-offload.rst:453::c:func:`tls_dev_del` of the old key followed by :c:func:`tls_dev_add`\nDocumentation/networking/tls-offload.rst-454-of the new one, runs at the start of the next ``sendmsg()`` on the\n--\nDocumentation/networking/tls-offload.rst=483=and the next ``sendmsg()`` retries; the ``tls_device_complete_rekey_retry``\nDocumentation/networking/tls-offload.rst:484:tracepoint fires. A hard failure (:c:func:`tls_dev_add` rejected, or the\nDocumentation/networking/tls-offload.rst-485-netdev gone) is terminal for this KeyUpdate: the connection is pinned to\n--\nDocumentation/networking/tls-offload.rst=520=If old-key records are still queued, installation of the new key through\nDocumentation/networking/tls-offload.rst:521::c:func:`tls_dev_add` is deferred until those records have been consumed;\nDocumentation/networking/tls-offload.rst-522-otherwise it occurs immediately. When the NIC cannot authenticate a record\n--\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c=6382=static int chcr_offload_state(struct adapter *adap,\n--\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c-6424-\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c:6425:static int cxgb4_ktls_dev_add(struct net_device *netdev, struct sock *sk,\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c-6426-\t\t\t enum tls_offload_ctx_dir direction,\n--\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c-6441-\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c:6442:\tret = adap-\u003euld[CXGB4_ULD_KTLS].tlsdev_ops-\u003etls_dev_add(netdev, sk,\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c-6443-\t\t\t\t\t\t\t\tdirection,\n--\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c=6473=static const struct tlsdev_ops cxgb4_ktls_ops = {\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c:6474:\t.tls_dev_add = cxgb4_ktls_dev_add,\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c-6475-\t.tls_dev_del = cxgb4_ktls_dev_del,\n--\ndrivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c=361=static void chcr_ktls_dev_del(struct net_device *netdev,\n--\ndrivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c-403-/*\ndrivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c:404: * chcr_ktls_dev_add: call back for tls_dev_add.\ndrivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c-405- * Create a tcb entry for TP. Also add l2t entry for the connection. And\n--\ndrivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c-411- */\ndrivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c:412:static int chcr_ktls_dev_add(struct net_device *netdev, struct sock *sk,\ndrivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c-413-\t\t\t enum tls_offload_ctx_dir direction,\n--\ndrivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c=2135=static const struct tlsdev_ops chcr_ktls_ops = {\ndrivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c:2136:\t.tls_dev_add = chcr_ktls_dev_add,\ndrivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c-2137-\t.tls_dev_del = chcr_ktls_dev_del,\n--\ndrivers/net/ethernet/fungible/funeth/funeth_ktls.c=126=static const struct tlsdev_ops fun_ktls_ops = {\ndrivers/net/ethernet/fungible/funeth/funeth_ktls.c:127:\t.tls_dev_add = fun_ktls_add,\ndrivers/net/ethernet/fungible/funeth/funeth_ktls.c-128-\t.tls_dev_del = fun_ktls_del,\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c=89=static const struct tlsdev_ops mlx5e_ktls_ops = {\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c:90:\t.tls_dev_add = mlx5e_ktls_add,\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c-91-\t.tls_dev_del = mlx5e_ktls_del,\n--\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c=472=static const struct tlsdev_ops nfp_net_tls_ops = {\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c:473:\t.tls_dev_add = nfp_net_tls_add,\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c-474-\t.tls_dev_del = nfp_net_tls_del,\n--\ninclude/net/tls.h=203=enum tls_context_flags {\n--\ninclude/net/tls.h-217-\t * context). Set in that case, and during a rekey before re-add, and\ninclude/net/tls.h:218:\t * cleared when tls_dev_add re-establishes the context. Readers use it to\ninclude/net/tls.h-219-\t * avoid a second tls_dev_del and to suppress resync while the NIC has no\n--\ninclude/net/tls.h-225-\t * after a failed re-add, or by tls_device_down()); prevents a second\ninclude/net/tls.h:226:\t * tls_dev_del. Cleared when tls_dev_add re-establishes the context.\ninclude/net/tls.h-227-\t */\n--\ninclude/net/tls.h-245-\tTLS_TX_REKEY_FLOOR = 7,\ninclude/net/tls.h:246:\t/* The RX side fell back to SW decryption during a rekey (tls_dev_add()\ninclude/net/tls.h-247-\t * failed, or the netdev is gone) and the socket has been moved from the\n--\ninclude/net/tls.h=335=struct tlsdev_ops {\ninclude/net/tls.h:336:\tint (*tls_dev_add)(struct net_device *netdev, struct sock *sk,\ninclude/net/tls.h-337-\t\t\t enum tls_offload_ctx_dir direction,\n--\ninclude/net/tls.h=366=struct tls_offload_context_rx {\n--\ninclude/net/tls.h-373-\tu8 resync_nh_do_now:1;\ninclude/net/tls.h:374:\t/* tls_dev_add deferred until old key is freed */\ninclude/net/tls.h-375-\tu8 dev_add_pending:1;\n--\nnet/tls/tls_device.c-44-\nnet/tls/tls_device.c:45:/* device_offload_lock is used to synchronize tls_dev_add\nnet/tls/tls_device.c-46- * against NETDEV_DOWN notifications.\n--\nnet/tls/tls_device.c=160=static int tls_device_dev_add_tx(struct sock *sk, struct net_device *netdev,\n--\nnet/tls/tls_device.c-170-\nnet/tls/tls_device.c:171:\trc = netdev-\u003etlsdev_ops-\u003etls_dev_add(netdev, sk, TLS_OFFLOAD_CTX_DIR_TX,\nnet/tls/tls_device.c-172-\t\t\t\t\t crypto_info, write_seq);\n--\nnet/tls/tls_device.c=221=static int tls_device_dev_add_rx(struct sock *sk, struct tls_context *tls_ctx,\n--\nnet/tls/tls_device.c-232-\nnet/tls/tls_device.c:233:\trc = netdev-\u003etlsdev_ops-\u003etls_dev_add(netdev, sk,\nnet/tls/tls_device.c-234-\t\t\t\t\t TLS_OFFLOAD_CTX_DIR_RX,\n--\nnet/tls/tls_device.c=2096=int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,\n--\nnet/tls/tls_device.c-2134-\t\t * the deleted NIC context's old-key XOR) and account the new key\nnet/tls/tls_device.c:2135:\t\t * as a SW fallback in place of the tls_dev_del()/tls_dev_add()\nnet/tls/tls_device.c-2136-\t\t * steps, mirroring the TX side (tls_device_complete_rekey()).\n--\nnet/tls/tls_device.c=2497=static int tls_dev_event(struct notifier_block *this, unsigned long event,\n--\nnet/tls/tls_device.c-2511-\t\tif (!dev-\u003etlsdev_ops ||\nnet/tls/tls_device.c:2512:\t\t !dev-\u003etlsdev_ops-\u003etls_dev_add ||\nnet/tls/tls_device.c-2513-\t\t !dev-\u003etlsdev_ops-\u003etls_dev_del)\n--\ntools/testing/selftests/drivers/net/hw/tls_hw_offload.py=24=STATS_TIMEOUT_S = 30\n--\ntools/testing/selftests/drivers/net/hw/tls_hw_offload.py-27-# keyed by the `ethtool -i` driver name. TlsTxDevice/TlsRxDevice in\ntools/testing/selftests/drivers/net/hw/tls_hw_offload.py:28:# /proc/net/tls_stat only prove tls_dev_add() accepted the offload; these\ntools/testing/selftests/drivers/net/hw/tls_hw_offload.py-29-# increment once per packet the NIC actually encrypted/decrypted (mlx5 counts\n"
} |
| 5/2 | 2026/09/20 01:00 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 77474 output: 28 thoughts: 53 Thoughts: **Examining Network Device Offload Capabilities** Okay, let's dive in. I need to get a handle on the `tlsdev_ops` implementation across a few network drivers. I know `cxgb4`, `funeth`, `mlx5e`, and `nfp` all touch this area. But wait, I have a nagging feeling… doesn’t `netdevsim` also handle this in some way? It's a key piece of our testing infrastructure, so it would make sense. I better quickly verify that. Time to check the source. Let's start by looking in the `drivers/net/netdevsim/` directory, or maybe even the test suites, just to be sure. I need to make sure I am not missing anything when it comes to TLS offload support. |
| 6/2 | 2026/09/20 01:00 | tool | grepper |
0mArgs:{
"Expression": "tlsdev_ops",
"PathPrefix": "drivers/net/"
}Results: {
"Output": "drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c=6382=static int chcr_offload_state(struct adapter *adap,\n--\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c-6391-\t\t}\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c:6392:\t\tif (!adap-\u003euld[CXGB4_ULD_KTLS].tlsdev_ops) {\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c-6393-\t\t\tdev_dbg(adap-\u003epdev_dev,\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c:6394:\t\t\t\t\"ch_ktls driver has no registered tlsdev_ops\\n\");\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c-6395-\t\t\treturn -EOPNOTSUPP;\n--\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c=6425=static int cxgb4_ktls_dev_add(struct net_device *netdev, struct sock *sk,\n--\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c-6441-\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c:6442:\tret = adap-\u003euld[CXGB4_ULD_KTLS].tlsdev_ops-\u003etls_dev_add(netdev, sk,\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c-6443-\t\t\t\t\t\t\t\tdirection,\n--\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c=6455=static void cxgb4_ktls_dev_del(struct net_device *netdev,\n--\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c-6464-\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c:6465:\tadap-\u003euld[CXGB4_ULD_KTLS].tlsdev_ops-\u003etls_dev_del(netdev, tls_ctx,\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c-6466-\t\t\t\t\t\t\t direction);\n--\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c-6472-\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c:6473:static const struct tlsdev_ops cxgb4_ktls_ops = {\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c-6474-\t.tls_dev_add = cxgb4_ktls_dev_add,\n--\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c=6571=static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent)\n--\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c-6808-\t\t\tnetdev-\u003ehw_features |= NETIF_F_HW_TLS_TX;\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c:6809:\t\t\tnetdev-\u003etlsdev_ops = \u0026cxgb4_ktls_ops;\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c-6810-\t\t\t/* initialize the refcount */\n--\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h=473=struct cxgb4_uld_info {\n--\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h-492-#if IS_ENABLED(CONFIG_CHELSIO_TLS_DEVICE)\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h:493:\tconst struct tlsdev_ops *tlsdev_ops;\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.h-494-#endif\n--\ndrivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c=2117=static void *chcr_ktls_uld_add(const struct cxgb4_lld_info *lldi)\n--\ndrivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c-2134-\ndrivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c:2135:static const struct tlsdev_ops chcr_ktls_ops = {\ndrivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c-2136-\t.tls_dev_add = chcr_ktls_dev_add,\n--\ndrivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c=2232=static struct cxgb4_uld_info chcr_ktls_uld_info = {\n--\ndrivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c-2239-\t.state_change = chcr_ktls_uld_state_change,\ndrivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c:2240:\t.tlsdev_ops = \u0026chcr_ktls_ops,\ndrivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c-2241-};\n--\ndrivers/net/ethernet/fungible/funeth/funeth_ktls.c=95=static int fun_ktls_resync(struct net_device *netdev, struct sock *sk, u32 seq,\n--\ndrivers/net/ethernet/fungible/funeth/funeth_ktls.c-125-\ndrivers/net/ethernet/fungible/funeth/funeth_ktls.c:126:static const struct tlsdev_ops fun_ktls_ops = {\ndrivers/net/ethernet/fungible/funeth/funeth_ktls.c-127-\t.tls_dev_add = fun_ktls_add,\n--\ndrivers/net/ethernet/fungible/funeth/funeth_ktls.c=132=int fun_ktls_init(struct net_device *netdev)\n--\ndrivers/net/ethernet/fungible/funeth/funeth_ktls.c-141-\tfp-\u003ektls_id = netdev-\u003edev_port;\ndrivers/net/ethernet/fungible/funeth/funeth_ktls.c:142:\tnetdev-\u003etlsdev_ops = \u0026fun_ktls_ops;\ndrivers/net/ethernet/fungible/funeth/funeth_ktls.c-143-\tnetdev-\u003ehw_features |= NETIF_F_HW_TLS_TX;\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c=78=static int mlx5e_ktls_resync(struct net_device *netdev,\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c-88-\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c:89:static const struct tlsdev_ops mlx5e_ktls_ops = {\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c-90-\t.tls_dev_add = mlx5e_ktls_add,\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c=113=void mlx5e_ktls_build_netdev(struct mlx5e_priv *priv)\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c-128-\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c:129:\tnetdev-\u003etlsdev_ops = \u0026mlx5e_ktls_ops;\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c-130-}\n--\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c=430=nfp_net_tls_resync(struct net_device *netdev, struct sock *sk, u32 seq,\n--\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c-471-\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c:472:static const struct tlsdev_ops nfp_net_tls_ops = {\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c-473-\t.tls_dev_add = nfp_net_tls_add,\n--\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c=567=int nfp_net_tls_init(struct nfp_net *nn)\n--\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c-604-\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c:605:\tnetdev-\u003etlsdev_ops = \u0026nfp_net_tls_ops;\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c-606-\n"
} |
| 7/2 | 2026/09/20 01:00 | llm | patch-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 80205 output: 29 thoughts: 74 Thoughts: Okay, here's my take on those thoughts, framed as an expert in the field: **Digging into TLS Offload Testing** Hold on a second! I need to investigate the testing for TLS offload capabilities. Let's see... what about `tools/testing/selftests/net/tls.c`? That seems like the most likely place to start. I'm curious: does it actually encompass the testing of TLS offload? If so, *how* is that accomplished within the testing framework? What specific aspects are being validated? I need to get a clear picture of its scope. Alright, a more focused approach is in order. Instead of blindly reviewing, I'll leverage my knowledge. To find the specific testing elements I will search directly in `tools/testing/selftests/` for references to `tlsdev_ops` or `netdevsim`. That'll help me quickly identify code pertaining to the device operations or simulation, which would be central to offload testing. This targeted search should quickly surface the relevant sections, whether it's through the device drivers, interface, or simulation within the testing suite. |
| 8/2 | 2026/09/20 01:01 | tool | grepper |
0mError:no matches Args: {
"Expression": "tls",
"PathPrefix": "drivers/net/netdevsim/"
}Results: {
"error": "no matches"
} |
| 9/2 | 2026/09/20 01:01 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 80563 output: 27 thoughts: 2668 Thoughts: **Analysis of KTLS Changes and Implications** Okay, let's break this down. My initial observation is focused on several recent commits impacting kernel TLS (kTLS), specifically changes across `net/sched/sch_generic.c`, `net/tls/*`. I'm seeing a pattern of modifications related to encryption, record handling, and socket configuration, particularly within the software (SW) offload path. First, I'm noting significant changes within `net/tls/tls_sw.c`. The introduction of `tls_sw_ctx_init`, `tls_sw_ctx_finalize`, and the modification of `tls_set_sw_offload` (which calls the init/finalize pair) are key. This suggests a more structured approach to initializing and finalizing the software crypto context. It's likely that this refactoring is preparing for a shift to the software offload path as a primary option or ensuring that the SW and HW offload paths are more consistent. Looking deeper in `tls_do_encryption` and `tls_push_record`, I'm seeing that the code now calls `tls_tx_cipher_ctx(tls_ctx)`. This is likely getting the current cipher context. The code now accesses `iv`, and `rec_seq` from the `tls_tx_cipher_ctx` which is likely using the result of `tls_sw_ctx_init`. These functions are critical in the SW encryption process, handling things like IV generation, sequence number management, and AAD construction. Any issues here would immediately impact the correctness and security of encrypted traffic. Next, I'm seeing a substantial change in `net/tls/tls_main.c` related to `do_tls_setsockopt`. The introduction of a mutex (`tls_get_ctx(sk)->tx_lock`) and its interaction with `lock_sock(sk)` is immediately interesting. This means that a mutex is taken prior to a socket lock, so the locking order has changed. This is a crucial area that requires further examination, because this might be introducing a subtle locking order issue. The added `tx_lock` guards the set of `TLS_TX` calls and ensures that updates to the crypto context and data transfer don't happen concurrently. This also avoids possible data corruption due to concurrent HW-offload rekeying. Further changes in `net/sched/sch_generic.c` are also noteworthy. The added check for `skb_is_decrypted(skb)` within `dequeue_skb` hints at a more robust validation mechanism for crypto-offloaded sockets. This is a good sign, to make sure nothing is emitted as plaintext. The intent seems to be to re-validate skbs that might have had their offload state change while they're queued, to avoid sending unencrypted data. Finally, in `include/net/tcp.h`, the addition of `tcp_write_collapse_fence` and its interaction with `skb->decrypted` are very intriguing. It seems like we now make a stronger distinction between the different states in which an skb can reside on the TCP write queue, to prevent merging of different skbs from different states across a fence, specifically ensuring that decryption occurs before the skb can be written. The main questions that arise are: 1. How does the initialization/finalization impact SW performance? 2. Does the new locking introduce any potential contention or performance bottlenecks? 3. Are there any subtle race conditions related to the interaction between hardware and software offload with these added locks? 4. Given that we are guarding setsockopt, is it safe to assume that a `tls_set_sw_offload(sk, ...)` is called from `do_tls_setsockopt`? I need to trace the execution path of the new locking mechanism and related functions (`tls_device_sendmsg`, `tls_sw_sendmsg`) and understand all of the potential code paths here and ensure that this refactoring is a performance improvement. I need to make sure this code is safe. |
| 10/2 | 2026/09/20 01:01 | tool | grepper |
0mArgs:{
"Expression": "tx_lock",
"PathPrefix": "net/tls/"
}Results: {
"Output": "net/tls/tls_device.c=846=int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)\n--\nnet/tls/tls_device.c-864-\nnet/tls/tls_device.c:865:\tmutex_lock(\u0026tls_ctx-\u003etx_lock);\nnet/tls/tls_device.c-866-\tlock_sock(sk);\n--\nnet/tls/tls_device.c-895-\trelease_sock(sk);\nnet/tls/tls_device.c:896:\tmutex_unlock(\u0026tls_ctx-\u003etx_lock);\nnet/tls/tls_device.c-897-\treturn rc;\n--\nnet/tls/tls_device.c=900=void tls_device_splice_eof(struct socket *sock)\n--\nnet/tls/tls_device.c-909-\nnet/tls/tls_device.c:910:\tmutex_lock(\u0026tls_ctx-\u003etx_lock);\nnet/tls/tls_device.c-911-\tlock_sock(sk);\n--\nnet/tls/tls_device.c-921-\trelease_sock(sk);\nnet/tls/tls_device.c:922:\tmutex_unlock(\u0026tls_ctx-\u003etx_lock);\nnet/tls/tls_device.c-923-}\n--\nnet/tls/tls_main.c=323=void tls_ctx_free(struct sock *sk, struct tls_context *ctx)\n--\nnet/tls/tls_main.c-329-\tmemzero_explicit(\u0026ctx-\u003ecrypto_recv, sizeof(ctx-\u003ecrypto_recv));\nnet/tls/tls_main.c:330:\tmutex_destroy(\u0026ctx-\u003etx_lock);\nnet/tls/tls_main.c-331-\n--\nnet/tls/tls_main.c=904=static int do_tls_setsockopt(struct sock *sk, int optname, sockptr_t optval,\n--\nnet/tls/tls_main.c-911-\tcase TLS_RX: {\nnet/tls/tls_main.c:912:\t\t/* tls_device_sendmsg() holds tx_lock across the lock_sock drop\nnet/tls/tls_main.c-913-\t\t * in sk_stream_wait_memory() with a half-built open_record\n--\nnet/tls/tls_main.c-916-\t\t * corrupting record framing. Serialize TX setsockopt against\nnet/tls/tls_main.c:917:\t\t * the data path with tx_lock, unconditionally for TLS_TX,\nnet/tls/tls_main.c-918-\t\t * since during initial setup there is no sender contending it.\n--\nnet/tls/tls_main.c-922-\t\tif (tx) {\nnet/tls/tls_main.c:923:\t\t\trc = mutex_lock_interruptible(\u0026tls_get_ctx(sk)-\u003etx_lock);\nnet/tls/tls_main.c-924-\t\t\tif (rc)\n--\nnet/tls/tls_main.c-930-\t\tif (tx)\nnet/tls/tls_main.c:931:\t\t\tmutex_unlock(\u0026tls_get_ctx(sk)-\u003etx_lock);\nnet/tls/tls_main.c-932-\t\tbreak;\n--\nnet/tls/tls_main.c=971=struct tls_context *tls_ctx_create(struct sock *sk)\n--\nnet/tls/tls_main.c-979-\nnet/tls/tls_main.c:980:\tmutex_init(\u0026ctx-\u003etx_lock);\nnet/tls/tls_main.c-981-\tctx-\u003esk_proto = READ_ONCE(sk-\u003esk_prot);\n--\nnet/tls/tls_sw.c=1007=int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)\n--\nnet/tls/tls_sw.c-1016-\nnet/tls/tls_sw.c:1017:\tret = mutex_lock_interruptible(\u0026tls_ctx-\u003etx_lock);\nnet/tls/tls_sw.c-1018-\tif (ret)\n--\nnet/tls/tls_sw.c-1022-\trelease_sock(sk);\nnet/tls/tls_sw.c:1023:\tmutex_unlock(\u0026tls_ctx-\u003etx_lock);\nnet/tls/tls_sw.c-1024-\treturn ret;\n--\nnet/tls/tls_sw.c-1030- * Inner logic of tls_sw_splice_eof(), factored out so the device\nnet/tls/tls_sw.c:1031: * TX path can reuse it with tls_ctx-\u003etx_lock and the socket lock\nnet/tls/tls_sw.c-1032- * already held. Callers not already holding both locks must use the\n--\nnet/tls/tls_sw.c=1083=void tls_sw_splice_eof(struct socket *sock)\n--\nnet/tls/tls_sw.c-1091-\nnet/tls/tls_sw.c:1092:\tmutex_lock(\u0026tls_ctx-\u003etx_lock);\nnet/tls/tls_sw.c-1093-\tlock_sock(sk);\n--\nnet/tls/tls_sw.c-1095-\trelease_sock(sk);\nnet/tls/tls_sw.c:1096:\tmutex_unlock(\u0026tls_ctx-\u003etx_lock);\nnet/tls/tls_sw.c-1097-}\n--\nnet/tls/tls_sw.c=2382=static void tx_work_handler(struct work_struct *work)\n--\nnet/tls/tls_sw.c-2400-\nnet/tls/tls_sw.c:2401:\tif (mutex_trylock(\u0026tls_ctx-\u003etx_lock)) {\nnet/tls/tls_sw.c-2402-\t\tlock_sock(sk);\n--\nnet/tls/tls_sw.c-2404-\t\trelease_sock(sk);\nnet/tls/tls_sw.c:2405:\t\tmutex_unlock(\u0026tls_ctx-\u003etx_lock);\nnet/tls/tls_sw.c-2406-\t} else if (!test_and_set_bit(BIT_TX_SCHEDULED, \u0026ctx-\u003etx_bitmask)) {\nnet/tls/tls_sw.c:2407:\t\t/* Someone is holding the tx_lock, they will likely run Tx\nnet/tls/tls_sw.c-2408-\t\t * and cancel the work on their way out of the lock section.\n"
} |
| 11/2 | 2026/09/20 01:01 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 85809 output: 32 thoughts: 994 Thoughts: **Deep Dive into TLS and Networking Concurrency** Okay, so I'm seeing a significant change here in `do_tls_setsockopt` concerning the `TLS_TX` path. It's now acquiring the `tx_lock` before `lock_sock(sk)`, which wasn't the case previously. This is a crucial area to scrutinize as it affects concurrency and potential race conditions in TLS socket operations. This alteration strongly suggests a concern about data integrity or synchronization, likely related to concurrent access to the TLS context during transmission. The original code's behavior suggests the original design had no concern for such lock. Now, this piques my interest, forcing me to ask: what else could be affected by this change? My mind immediately jumps to networking qdisc code, specifically `net/sched/sch_generic.c`, which contains the `dequeue_skb` function. There's logic here related to crypto-offloaded sockets and revalidation of SKBs, especially those that might still be in cleartext, due to state changes like TLS key updates or offload teardown. This clearly connects the change to how we handle offloaded and decrypted SKBs in the networking stack. My examination continues to move to the `tcp_write_collapse_fence` function in `include/net/tcp.h`. The comments there highlight the importance of correctly marking boundaries between TCP states, and the possibility of merging SKBs across those boundaries if not properly handled, especially when dealing with decrypted data. The use of `skb->decrypted` is a key indicator of this concern. The fence marks the end of a transmission state, and this is another hint that the change may be intended to correctly align skb boundaries to the newly acquired locks. I'm now zeroing in on `net/tls/tls_sw.c`. This file is crucial because it contains functions like `tls_sw_ctx_init`, `tls_sw_ctx_finalize`, `tls_set_sw_offload`, `tls_sw_drain_tx`, `tls_sw_splice_eof_locked`, `tls_sw_sendmsg_locked`, `tls_encrypt_async_wait`, and `tls_sw_ctx_tx_init`. These are core TLS software implementations. The change likely interacts with the interactions of the locks with software. It seems the lock is more important in this software-defined context. I then focus on the `tls_do_encryption` function, where I'm seeing direct operations on IV (Initialization Vector) data. The `memcpy`, `tls_xor_iv_with_seq`, and `tls_advance_record_sn` calls are fundamental to TLS encryption, especially in managing record sequence numbers. Any concurrency issues here could lead to devastating security vulnerabilities. I need to be more mindful of these operations. Finally, the function `tls_check_pending_rekey` is called within the software-defined implementations. It may be called after data is released from the software components. The change in locking for `TLS_TX` hints that `tls_check_pending_rekey` might now be operating under different locking conditions. I need to understand how the new locking influences the calls made here. I now need to retrace the calls to `tls_check_pending_rekey` in `net/tls/tls_sw.c` to understand the full impact of this initial socket change, and how they relate to the lock acquisition. |
| 12/2 | 2026/09/20 01:01 | tool | grepper |
0mArgs:{
"Expression": "tls_check_pending_rekey",
"PathPrefix": "net/tls/"
}Results: {
"Output": "net/tls/tls_device.c=257=static void tls_device_deferred_dev_add_rx(struct sock *sk,\n--\nnet/tls/tls_device.c-297-/* Retire the NIC's RX key when a KeyUpdate record is decoded (from\nnet/tls/tls_device.c:298: * tls_check_pending_rekey(), lock_sock held). The NIC must lose the old key\nnet/tls/tls_device.c-299- * now, before it transforms further post-KeyUpdate records that are new-key on\n--\nnet/tls/tls_sw.c=1492=tls_decrypt_device(struct sock *sk, struct msghdr *msg,\n--\nnet/tls/tls_sw.c-1540-\nnet/tls/tls_sw.c:1541:static int tls_check_pending_rekey(struct sock *sk, struct tls_context *ctx,\nnet/tls/tls_sw.c-1542-\t\t\t\t struct sk_buff *skb)\n--\nnet/tls/tls_sw.c=1575=static int tls_rx_one_record(struct sock *sk, struct msghdr *msg,\n--\nnet/tls/tls_sw.c-1595-\nnet/tls/tls_sw.c:1596:\treturn tls_check_pending_rekey(sk, tls_ctx, darg-\u003eskb);\nnet/tls/tls_sw.c-1597-}\n"
} |
| 13/2 | 2026/09/20 01:01 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 87946 output: 24 thoughts: 962 Thoughts: **Deep Dive into Kernel TLS Offload** Okay, so I'm tracing the TLS receive path, and it looks like `tls_rx_one_record` calls `tls_check_pending_rekey`, which in turn calls `tls_device_rx_del_key`. This is all part of the standard RX path, which is interesting. My immediate question is, does TLS device offload require actual hardware, or can it be emulated somehow? Could I reach this functionality in a setup like QEMU or syzkaller? That gets me thinking, can `tls_set_device_offload` even be called from userspace? Time to check. Ah, right, `setsockopt(fd, SOL_TLS, TLS_TX, ...)` should do the trick. Looking into `do_tls_setsockopt_conf`, here's what happens: If it's a TX configuration, the kernel first tries `tls_set_device_offload(sk, ...)` and sets the configuration to `TLS_HW`. Diving into `tls_set_device_offload`, I see this: It gets the `crypto_info`, grabs the `cipher_desc`, and then tries to get the network device associated with the socket. The crucial part is this check: `if (!(netdev->features & NETIF_F_HW_TLS_TX))`. If the network device *doesn't* have `NETIF_F_HW_TLS_TX`, it returns `-EOPNOTSUPP`. That makes sense. So back in `do_tls_setsockopt_conf`, if `tls_set_device_offload` fails (i.e., the netdev lacks the HW feature), the code falls back to `tls_set_sw_offload(sk, ...)` which is essentially the software TLS implementation. Ah, and that `tls_set_sw_offload` was refactored with `tls_sw_ctx_init` and `tls_sw_ctx_finalize` in a recent patch. Got it. But hold on... what if a device *does* have `NETIF_F_HW_TLS_TX`? That's where things get interesting. Does any virtual device, in fact, report that it has `NETIF_F_HW_TLS_TX` or `NETIF_F_HW_TLS_RX`? Time to run a full search of the kernel source for those feature flags! |
| 14/2 | 2026/09/20 01:01 | tool | grepper |
0mArgs:{
"Expression": "NETIF_F_HW_TLS_TX"
}Results: {
"Output": "Documentation/networking/tls-offload.rst=68=During driver initialization device sets the ``NETIF_F_HW_TLS_RX`` and\nDocumentation/networking/tls-offload.rst:69:``NETIF_F_HW_TLS_TX`` features and installs its\nDocumentation/networking/tls-offload.rst-70-:c:type:`struct tlsdev_ops \u003ctlsdev_ops\u003e`\n--\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c=6571=static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent)\n--\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c-6807-\t\tif (pi-\u003eadapter-\u003eparams.crypto \u0026 FW_CAPS_CONFIG_TLS_HW) {\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c:6808:\t\t\tnetdev-\u003ehw_features |= NETIF_F_HW_TLS_TX;\ndrivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c-6809-\t\t\tnetdev-\u003etlsdev_ops = \u0026cxgb4_ktls_ops;\n--\ndrivers/net/ethernet/fungible/funeth/funeth_ktls.c=132=int fun_ktls_init(struct net_device *netdev)\n--\ndrivers/net/ethernet/fungible/funeth/funeth_ktls.c-142-\tnetdev-\u003etlsdev_ops = \u0026fun_ktls_ops;\ndrivers/net/ethernet/fungible/funeth/funeth_ktls.c:143:\tnetdev-\u003ehw_features |= NETIF_F_HW_TLS_TX;\ndrivers/net/ethernet/fungible/funeth/funeth_ktls.c:144:\tnetdev-\u003efeatures |= NETIF_F_HW_TLS_TX;\ndrivers/net/ethernet/fungible/funeth/funeth_ktls.c-145-\treturn 0;\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c=113=void mlx5e_ktls_build_netdev(struct mlx5e_priv *priv)\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c-121-\tif (mlx5e_is_ktls_tx(mdev)) {\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c:122:\t\tnetdev-\u003ehw_features |= NETIF_F_HW_TLS_TX;\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c:123:\t\tnetdev-\u003efeatures |= NETIF_F_HW_TLS_TX;\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c-124-\t}\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_main.c=4524=static netdev_features_t mlx5e_fix_uplink_rep_features(struct net_device *netdev,\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_main.c-4530-\ndrivers/net/ethernet/mellanox/mlx5/core/en_main.c:4531:\tfeatures \u0026= ~NETIF_F_HW_TLS_TX;\ndrivers/net/ethernet/mellanox/mlx5/core/en_main.c:4532:\tif (netdev-\u003efeatures \u0026 NETIF_F_HW_TLS_TX)\ndrivers/net/ethernet/mellanox/mlx5/core/en_main.c-4533-\t\tnetdev_warn(netdev, \"Disabling hw_tls_tx, not supported in switchdev mode\\n\");\n--\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c=567=int nfp_net_tls_init(struct nfp_net *nn)\n--\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c-600-\tif (nn-\u003etlv_caps.crypto_ops \u0026 NFP_NET_TLS_OPCODE_MASK_TX) {\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c:601:\t\tnetdev-\u003ehw_features |= NETIF_F_HW_TLS_TX;\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c:602:\t\tnetdev-\u003efeatures |= NETIF_F_HW_TLS_TX;\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c-603-\t}\n--\ninclude/linux/netdev_features.h=14=enum {\n--\ninclude/linux/netdev_features.h-77-\tNETIF_F_RX_UDP_TUNNEL_PORT_BIT, /* Offload of RX port for UDP tunnels */\ninclude/linux/netdev_features.h:78:\tNETIF_F_HW_TLS_TX_BIT,\t\t/* Hardware TLS TX offload */\ninclude/linux/netdev_features.h-79-\tNETIF_F_HW_TLS_RX_BIT,\t\t/* Hardware TLS RX offload */\n--\ninclude/linux/netdev_features.h-156-#define NETIF_F_GSO_UDP_L4\t__NETIF_F(GSO_UDP_L4)\ninclude/linux/netdev_features.h:157:#define NETIF_F_HW_TLS_TX\t__NETIF_F(HW_TLS_TX)\ninclude/linux/netdev_features.h-158-#define NETIF_F_HW_TLS_RX\t__NETIF_F(HW_TLS_RX)\n--\nnet/core/dev.c=10960=static netdev_features_t netdev_fix_features(struct net_device *dev,\n--\nnet/core/dev.c-11040-\nnet/core/dev.c:11041:\tif ((features \u0026 NETIF_F_HW_TLS_TX) \u0026\u0026 !netdev_has_ip_or_hw_csum(features)) {\nnet/core/dev.c-11042-\t\tnetdev_dbg(dev, \"Dropping TLS TX HW offload feature since no CSUM feature.\\n\");\nnet/core/dev.c:11043:\t\tfeatures \u0026= ~NETIF_F_HW_TLS_TX;\nnet/core/dev.c-11044-\t}\n--\nnet/ethtool/common.c=16=const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN] = {\n--\nnet/ethtool/common.c-69-\t[NETIF_F_RX_UDP_TUNNEL_PORT_BIT] =\t \"rx-udp_tunnel-port-offload\",\nnet/ethtool/common.c:70:\t[NETIF_F_HW_TLS_TX_BIT] =\t \"tls-hw-tx-offload\",\nnet/ethtool/common.c-71-\t[NETIF_F_HW_TLS_RX_BIT] =\t \"tls-hw-rx-offload\",\n--\nnet/tls/tls_device.c=2041=int tls_set_device_offload(struct sock *sk,\n--\nnet/tls/tls_device.c-2082-\nnet/tls/tls_device.c:2083:\tif (!(netdev-\u003efeatures \u0026 NETIF_F_HW_TLS_TX)) {\nnet/tls/tls_device.c-2084-\t\trc = -EOPNOTSUPP;\n--\nnet/tls/tls_device.c=2497=static int tls_dev_event(struct notifier_block *this, unsigned long event,\n--\nnet/tls/tls_device.c-2502-\tif (!dev-\u003etlsdev_ops \u0026\u0026\nnet/tls/tls_device.c:2503:\t !(dev-\u003efeatures \u0026 (NETIF_F_HW_TLS_RX | NETIF_F_HW_TLS_TX)))\nnet/tls/tls_device.c-2504-\t\treturn NOTIFY_DONE;\n"
} |
| 15/2 | 2026/09/20 01:01 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 91548 output: 21 thoughts: 490 Thoughts: **My Analysis of NETIF_F_HW_TLS_TX and its Implications** Okay, so `NETIF_F_HW_TLS_TX` is exclusively implemented on cxgb4, funeth, mlx5, and nfp. All of these, as I recall, are physical PCIe NICs. That's the baseline. But the immediate question is, can a user *force* this feature on through something like netlink, ethtool, or even through a tun/tap interface? That's what I need to dig into. My immediate instinct is to check if `ethtool` can be coerced into enabling `NETIF_F_HW_TLS_TX`, or if it's fundamentally tied to the hardware feature set (`hw_features`). Alright, let's look at `net/core/dev.c` and figure out what can be toggled via `hw_features`. Even if a user *could* somehow set that flag on a device that doesn't provide `netdev->tlsdev_ops`, what actually happens? I need to understand the control path for this. What gets invoked in `tls_dev_event`? What is the role of `tls_device_init()`? I remember there's an `alloc_page` there to allocate a dummy page, but what are the broader ramifications? And then, I need to consider the data path. What happens in `dequeue_skb` within `net/sched/sch_generic.c`? That's going to be called on *every* qdisc packet dequeue, so this is important! Hmm, it looks like `*validate = false` initially. Then if `xfrm_offload(skb)` is true, the validate flag is flipped to `true`. And in the comments... ah, it looks like if the SKB is still cleartext *and* crypto-offloaded then we need to revalidate. Otherwise, the validation can change while the SKB is parked there. Makes sense! Key updates, offload teardown, etc. So the code re-validates to prevent emitting now-unencrypted plaintext. What about `skb_is_decrypted(skb)`? When does *that* return true? I should go check `skb_is_decrypted` itself. |
| 16/2 | 2026/09/20 01:01 | tool | grepper |
0mArgs:{
"Expression": "skb_is_decrypted"
}Results: {
"Output": "include/linux/skbuff.h=1700=static inline int skb_cmp_decrypted(const struct sk_buff *skb1,\n--\ninclude/linux/skbuff.h-1709-\ninclude/linux/skbuff.h:1710:static inline bool skb_is_decrypted(const struct sk_buff *skb)\ninclude/linux/skbuff.h-1711-{\n--\ninclude/net/tls.h=427=static inline bool tls_is_skb_tx_device_offloaded(const struct sk_buff *skb)\n--\ninclude/net/tls.h-453-\t */\ninclude/net/tls.h:454:\treturn validate == \u0026tls_validate_xmit_skb_rekey \u0026\u0026 skb_is_decrypted(skb);\ninclude/net/tls.h-455-#else\n--\nnet/core/dev.c=4011=static struct sk_buff *sk_validate_xmit_skb(struct sk_buff *skb,\n--\nnet/core/dev.c-4028-\t\tskb = sk_validate(sk, dev, skb);\nnet/core/dev.c:4029:\t} else if (unlikely(skb_is_decrypted(skb))) {\nnet/core/dev.c-4030-\t\tpr_warn_ratelimited(\"unencrypted skb with no associated socket - dropping\\n\");\n--\nnet/core/sock.c=2765=static bool can_skb_orphan_partial(const struct sk_buff *skb)\n--\nnet/core/sock.c-2769-\t */\nnet/core/sock.c:2770:\tif (skb_is_decrypted(skb))\nnet/core/sock.c-2771-\t\treturn false;\n--\nnet/sched/sch_generic.c=258=static struct sk_buff *dequeue_skb(struct Qdisc *q, bool *validate,\n--\nnet/sched/sch_generic.c-294-\t\t */\nnet/sched/sch_generic.c:295:\t\tif (skb_is_decrypted(skb))\nnet/sched/sch_generic.c-296-\t\t\t*validate = true;\n--\nnet/tls/tls_device.c=1750=static int tls_device_complete_rekey(struct sock *sk, struct tls_context *ctx,\n--\nnet/tls/tls_device.c-1866-\t * its WARN on the new start marker. The cleartext leak on that path is closed\nnet/tls/tls_device.c:1867:\t * separately by the skb_is_decrypted() gate in tls_sw_fallback(); this is\nnet/tls/tls_device.c-1868-\t * only WARN avoidance. Set once; stays set for the socket's life.\n--\nnet/tls/tls_device_fallback.c=377=static struct sk_buff *tls_sw_fallback(struct sock *sk, struct sk_buff *skb)\n--\nnet/tls/tls_device_fallback.c-416-\t\tif (sync_size \u003c 0 \u0026\u0026 payload_len \u003c= -sync_size \u0026\u0026\nnet/tls/tls_device_fallback.c:417:\t\t !skb_is_decrypted(skb))\nnet/tls/tls_device_fallback.c-418-\t\t\tnskb = skb_get(skb);\n--\nnet/tls/tls_device_fallback.c-439- * start marker. The cleartext leak on that path is closed unconditionally by\nnet/tls/tls_device_fallback.c:440: * the skb_is_decrypted() gate in tls_sw_fallback(); this floor additionally\nnet/tls/tls_device_fallback.c-441- * drops the clone before it reaches the NIC, avoiding the driver's WARN\n--\nnet/tls/tls_device_fallback.c-446- * decrypted straddler whose record is still present (no SKIP_NO_DATA), and a\nnet/tls/tls_device_fallback.c:447: * stale clone is dropped by the skb_is_decrypted() gate in tls_sw_fallback().\nnet/tls/tls_device_fallback.c-448- * Such a clone is exactly a payload skb whose end_seq \u003c= snd_una: the peer has\n--\nnet/tls/tls_device_fallback.c=453=static bool tls_tx_drop_acked_clone(struct sock *sk, struct sk_buff *skb)\n--\nnet/tls/tls_device_fallback.c-467-\t * normal paths (a live record is re-encrypted; a marker/freed-record hit is\nnet/tls/tls_device_fallback.c:468:\t * dropped there too). Both the leak (skb_is_decrypted() gate) and the mlx5\nnet/tls/tls_device_fallback.c-469-\t * WARN only concern the fully-ACKed case handled here.\n--\nnet/tls/tls_device_fallback.c=501=struct sk_buff *tls_validate_xmit_skb_rekey(struct sock *sk,\n--\nnet/tls/tls_device_fallback.c-526-\t\t */\nnet/tls/tls_device_fallback.c:527:\t\tif (payload_len \u003e 0 \u0026\u0026 !skb_is_decrypted(skb) \u0026\u0026\nnet/tls/tls_device_fallback.c-528-\t\t after(tcp_seq + payload_len, snd_una))\n"
} |
| 17/2 | 2026/09/20 01:01 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 93945 output: 25 thoughts: 2803 Thoughts: **Deep Dive into Kernel TLS Locking and Configuration** Okay, so I've been digging into the recent changes in `net/tls/tls_main.c`, specifically focusing on how they relate to the software kTLS implementation, which, as we know, is used in various scenarios, even without dedicated hardware. The first thing that catches my eye is the updated logic within `tls_sk_proto_cleanup`. It appears they've added checks for rekey failures when cleaning up both TX and RX resources, now decrementing SW stats if a HW rekey fails. I'll need to keep this in mind as a potential optimization avenue later, but first, the *locking*. The real interesting bit starts with `do_tls_getsockopt_conf` and `do_tls_setsockopt_conf`. The key update seems to be around `tls_set_sw_offload`. It's used when configuring `TLS_TX` and `TLS_RX`. It calls `tls_sw_ctx_init`, which then gets its changes applied in `tls_sw_ctx_finalize`! The significant code changes in `tls_sw_ctx_init` are related to the allocation and initialization of the AEAD cipher, and it looks like a bug in previous logic has been fixed! This also includes the move of key-related memcpy operations and crypto info handling, which were moved to `tls_sw_ctx_finalize`. The key takeaway here is that they've changed the order of operations, and the conditions under which these initializations are called. The most critical change, however, is the addition of the mutex `tx_lock` inside the `do_tls_setsockopt` for both `TLS_TX` and `TLS_RX`! Crucially, this `tx_lock` is acquired *before* `lock_sock(sk)`. This is a big deal, and my mind immediately starts racing with potential race conditions and locking issues. Now, I need to understand why this was introduced. Based on the comment, it looks to be an effort to serialize TX setsockopt against the data path, particularly to protect against a concurrent HW-offload rekeying corrupting record framing. Even more significant, this is *always* taken on setsockopt TLS_TX, even for pure software TLS! So, the next obvious step is to analyze all the places where `tx_lock` is taken in the kernel to ensure there are no lock inversion possibilities or potential deadlocks. This means scrutinizing functions like `tls_sw_sendmsg`, `tls_device_sendmsg`, and other related code paths. Given my existing knowledge, I expect to see `mutex_lock_interruptible` and `lock_sock` called in various places. I need to be certain that these calls are properly ordered and that no lock inversion situations have been introduced by this change. This is an area of significant concern and will require careful analysis! I'll start by searching for all uses of `tx_lock` and carefully trace the code execution paths. |
| 18/2 | 2026/09/20 01:02 | tool | grepper |
0mArgs:{
"Expression": "tx_lock",
"PathPrefix": "net/"
}Results: {
"Output": "net/can/bcm.c=108=struct bcm_op {\n--\nnet/can/bcm.c-132-\tnetdevice_tracker rx_reg_dev_tracker;\nnet/can/bcm.c:133:\tspinlock_t bcm_tx_lock; /* protect tx data and timer updates */\nnet/can/bcm.c-134-\tspinlock_t bcm_rx_update_lock; /* protect filter/timer data updates */\n--\nnet/can/bcm.c=321=static void bcm_can_tx(struct bcm_op *op, struct canfd_frame *cf)\n--\nnet/can/bcm.c-336-\t\t/* read currframe under lock protection */\nnet/can/bcm.c:337:\t\tspin_lock_bh(\u0026op-\u003ebcm_tx_lock);\nnet/can/bcm.c-338-\t\tidx = op-\u003ecurrframe;\n--\nnet/can/bcm.c-340-\t\tcf = \u0026cframe;\nnet/can/bcm.c:341:\t\tspin_unlock_bh(\u0026op-\u003ebcm_tx_lock);\nnet/can/bcm.c-342-\t}\n--\nnet/can/bcm.c-369-\t/* update currframe and count under lock protection */\nnet/can/bcm.c:370:\tspin_lock_bh(\u0026op-\u003ebcm_tx_lock);\nnet/can/bcm.c-371-\n--\nnet/can/bcm.c-389-\nnet/can/bcm.c:390:\tspin_unlock_bh(\u0026op-\u003ebcm_tx_lock);\nnet/can/bcm.c-391-out:\n--\nnet/can/bcm.c=473=static bool bcm_tx_set_expiry(struct bcm_op *op, struct hrtimer *hrt)\n--\nnet/can/bcm.c-476-\nnet/can/bcm.c:477:\tspin_lock_bh(\u0026op-\u003ebcm_tx_lock);\nnet/can/bcm.c-478-\n--\nnet/can/bcm.c-483-\t} else {\nnet/can/bcm.c:484:\t\tspin_unlock_bh(\u0026op-\u003ebcm_tx_lock);\nnet/can/bcm.c-485-\t\treturn false;\n--\nnet/can/bcm.c-487-\nnet/can/bcm.c:488:\tspin_unlock_bh(\u0026op-\u003ebcm_tx_lock);\nnet/can/bcm.c-489-\n--\nnet/can/bcm.c=501=static enum hrtimer_restart bcm_tx_timeout_handler(struct hrtimer *hrtimer)\n--\nnet/can/bcm.c-509-\t */\nnet/can/bcm.c:510:\tspin_lock_bh(\u0026op-\u003ebcm_tx_lock);\nnet/can/bcm.c-511-\ttx_ival1 = op-\u003ekt_ival1 \u0026\u0026 (op-\u003ecount \u003e 0);\nnet/can/bcm.c-512-\ttx_ival2 = !!op-\u003ekt_ival2;\nnet/can/bcm.c:513:\tspin_unlock_bh(\u0026op-\u003ebcm_tx_lock);\nnet/can/bcm.c-514-\n--\nnet/can/bcm.c-523-\t\t */\nnet/can/bcm.c:524:\t\tspin_lock_bh(\u0026op-\u003ebcm_tx_lock);\nnet/can/bcm.c-525-\t\tflags = op-\u003eflags;\n--\nnet/can/bcm.c-528-\t\tival2 = op-\u003eival2;\nnet/can/bcm.c:529:\t\tspin_unlock_bh(\u0026op-\u003ebcm_tx_lock);\nnet/can/bcm.c-530-\n--\nnet/can/bcm.c=1071=static int bcm_tx_setup(struct bcm_msg_head *msg_head, struct msghdr *msg,\n--\nnet/can/bcm.c-1144-\nnet/can/bcm.c:1145:\t\tspin_lock_bh(\u0026op-\u003ebcm_tx_lock);\nnet/can/bcm.c-1146-\n--\nnet/can/bcm.c-1162-\nnet/can/bcm.c:1163:\t\tspin_unlock_bh(\u0026op-\u003ebcm_tx_lock);\nnet/can/bcm.c-1164-\n--\nnet/can/bcm.c-1173-\nnet/can/bcm.c:1174:\t\tspin_lock_init(\u0026op-\u003ebcm_tx_lock);\nnet/can/bcm.c-1175-\t\top-\u003ecan_id = msg_head-\u003ecan_id;\n--\nnet/can/bcm.c-1240-\t\t/* set timer values */\nnet/can/bcm.c:1241:\t\tspin_lock_bh(\u0026op-\u003ebcm_tx_lock);\nnet/can/bcm.c-1242-\t\top-\u003eival1 = msg_head-\u003eival1;\n--\nnet/can/bcm.c-1245-\t\top-\u003ekt_ival2 = bcm_timeval_to_ktime(msg_head-\u003eival2);\nnet/can/bcm.c:1246:\t\tspin_unlock_bh(\u0026op-\u003ebcm_tx_lock);\nnet/can/bcm.c-1247-\n--\nnet/can/bcm.c=1314=static int bcm_rx_setup(struct bcm_msg_head *msg_head, struct msghdr *msg,\n--\nnet/can/bcm.c-1421-\nnet/can/bcm.c:1422:\t\tspin_lock_init(\u0026op-\u003ebcm_tx_lock);\nnet/can/bcm.c-1423-\t\tspin_lock_init(\u0026op-\u003ebcm_rx_update_lock);\n--\nnet/core/dev.c=4819=int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)\n--\nnet/core/dev.c-4890-\nnet/core/dev.c:4891:\t * Really, it is unlikely that netif_tx_lock protection is necessary\nnet/core/dev.c-4892-\t * here. (f.e. loopback and IP tunnels are clean ignoring statistics\n--\nnet/mac802154/ieee802154_i.h=199=int ieee802154_mlme_tx(struct ieee802154_local *local,\n--\nnet/mac802154/ieee802154_i.h-201-\t\t struct sk_buff *skb);\nnet/mac802154/ieee802154_i.h:202:int ieee802154_mlme_tx_locked(struct ieee802154_local *local,\nnet/mac802154/ieee802154_i.h-203-\t\t\t struct ieee802154_sub_if_data *sdata,\n--\nnet/mac802154/scan.c=741=mac802154_send_association_resp_locked(struct ieee802154_sub_if_data *sdata,\n--\nnet/mac802154/scan.c-781-\nnet/mac802154/scan.c:782:\tret = ieee802154_mlme_tx_locked(local, sdata, skb);\nnet/mac802154/scan.c-783-\tif (ret) {\n--\nnet/mac802154/tx.c=135=int ieee802154_mlme_op_pre(struct ieee802154_local *local)\n--\nnet/mac802154/tx.c-139-\nnet/mac802154/tx.c:140:int ieee802154_mlme_tx_locked(struct ieee802154_local *local,\nnet/mac802154/tx.c-141-\t\t\t struct ieee802154_sub_if_data *sdata,\n--\nnet/mac802154/tx.c=163=int ieee802154_mlme_tx(struct ieee802154_local *local,\n--\nnet/mac802154/tx.c-169-\trtnl_lock();\nnet/mac802154/tx.c:170:\tret = ieee802154_mlme_tx_locked(local, sdata, skb);\nnet/mac802154/tx.c-171-\trtnl_unlock();\n--\nnet/mac802154/tx.c=181=int ieee802154_mlme_tx_one_locked(struct ieee802154_local *local,\n--\nnet/mac802154/tx.c-187-\tieee802154_mlme_op_pre(local);\nnet/mac802154/tx.c:188:\tret = ieee802154_mlme_tx_locked(local, sdata, skb);\nnet/mac802154/tx.c-189-\tieee802154_mlme_op_post(local);\n--\nnet/netfilter/nft_inner.c-25-\nnet/netfilter/nft_inner.c:26:struct nft_inner_tun_ctx_locked {\nnet/netfilter/nft_inner.c-27-\tstruct nft_inner_tun_ctx ctx;\n--\nnet/netfilter/nft_inner.c-30-\nnet/netfilter/nft_inner.c:31:static DEFINE_PER_CPU(struct nft_inner_tun_ctx_locked, nft_pcpu_tun_ctx) = {\nnet/netfilter/nft_inner.c-32-\t.bh_lock = INIT_LOCAL_LOCK(bh_lock),\n--\nnet/qrtr/af_qrtr.c=114=static DEFINE_XARRAY_ALLOC(qrtr_ports);\n--\nnet/qrtr/af_qrtr.c-122- * @qrtr_tx_flow: xarray of qrtr_tx_flow, keyed by node \u003c\u003c 32 | port\nnet/qrtr/af_qrtr.c:123: * @qrtr_tx_lock: lock for qrtr_tx_flow inserts\nnet/qrtr/af_qrtr.c-124- * @hello_sent: hello packet send successful\n--\nnet/qrtr/af_qrtr.c=129=struct qrtr_node {\n--\nnet/qrtr/af_qrtr.c-135-\tstruct xarray qrtr_tx_flow;\nnet/qrtr/af_qrtr.c:136:\tstruct mutex qrtr_tx_lock; /* for qrtr_tx_flow */\nnet/qrtr/af_qrtr.c-137-\n--\nnet/qrtr/af_qrtr.c=264=static int qrtr_tx_wait(struct qrtr_node *node, int dest_node, int dest_port,\n--\nnet/qrtr/af_qrtr.c-275-\nnet/qrtr/af_qrtr.c:276:\tmutex_lock(\u0026node-\u003eqrtr_tx_lock);\nnet/qrtr/af_qrtr.c-277-\tflow = xa_load(\u0026node-\u003eqrtr_tx_flow, key);\n--\nnet/qrtr/af_qrtr.c-288-\t}\nnet/qrtr/af_qrtr.c:289:\tmutex_unlock(\u0026node-\u003eqrtr_tx_lock);\nnet/qrtr/af_qrtr.c-290-\n--\nnet/qrtr/af_qrtr.c=634=int qrtr_endpoint_register(struct qrtr_endpoint *ep, unsigned int nid)\n--\nnet/qrtr/af_qrtr.c-654-\txa_init(\u0026node-\u003eqrtr_tx_flow);\nnet/qrtr/af_qrtr.c:655:\tmutex_init(\u0026node-\u003eqrtr_tx_lock);\nnet/qrtr/af_qrtr.c-656-\n--\nnet/qrtr/af_qrtr.c=675=void qrtr_endpoint_unregister(struct qrtr_endpoint *ep)\n--\nnet/qrtr/af_qrtr.c-706-\t/* Wake up any transmitters waiting for resume-tx from the node */\nnet/qrtr/af_qrtr.c:707:\tmutex_lock(\u0026node-\u003eqrtr_tx_lock);\nnet/qrtr/af_qrtr.c-708-\txa_for_each(\u0026node-\u003eqrtr_tx_flow, index, flow)\nnet/qrtr/af_qrtr.c-709-\t\twake_up_interruptible_all(\u0026flow-\u003eresume_tx);\nnet/qrtr/af_qrtr.c:710:\tmutex_unlock(\u0026node-\u003eqrtr_tx_lock);\nnet/qrtr/af_qrtr.c-711-\n--\nnet/sched/sch_generic.c=353=bool sch_direct_xmit(struct sk_buff *skb, struct Qdisc *q,\n--\nnet/sched/sch_generic.c-414- *\nnet/sched/sch_generic.c:415: * netif_tx_lock serializes accesses to device driver.\nnet/sched/sch_generic.c-416- *\nnet/sched/sch_generic.c:417: * qdisc_lock(q) and netif_tx_lock are mutually exclusive,\nnet/sched/sch_generic.c-418- * if one is grabbed, another must be free.\n--\nnet/sched/sch_generic.c=483=static void netif_freeze_queues(struct net_device *dev)\n--\nnet/sched/sch_generic.c-497-\t\t */\nnet/sched/sch_generic.c:498:\t\t__netif_tx_lock(txq, cpu);\nnet/sched/sch_generic.c-499-\t\tset_bit(__QUEUE_STATE_FROZEN, \u0026txq-\u003estate);\n--\nnet/sched/sch_generic.c-503-\nnet/sched/sch_generic.c:504:void netif_tx_lock(struct net_device *dev)\nnet/sched/sch_generic.c-505-{\n--\nnet/sched/sch_generic.c-508-}\nnet/sched/sch_generic.c:509:EXPORT_SYMBOL(netif_tx_lock);\nnet/sched/sch_generic.c-510-\n--\nnet/sched/sch_generic.c=620=static void netdev_watchdog_down(struct net_device *dev)\nnet/sched/sch_generic.c-621-{\nnet/sched/sch_generic.c:622:\tnetif_tx_lock_bh(dev);\nnet/sched/sch_generic.c-623-\n--\nnet/sunrpc/xprtrdma/svc_rdma_transport.c=180=static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv,\n--\nnet/sunrpc/xprtrdma/svc_rdma_transport.c-182-{\nnet/sunrpc/xprtrdma/svc_rdma_transport.c:183:\tstatic struct lock_class_key svcrdma_rwctx_lock;\nnet/sunrpc/xprtrdma/svc_rdma_transport.c:184:\tstatic struct lock_class_key svcrdma_sctx_lock;\nnet/sunrpc/xprtrdma/svc_rdma_transport.c-185-\tstatic struct lock_class_key svcrdma_dto_lock;\n--\nnet/sunrpc/xprtrdma/svc_rdma_transport.c-206-\tspin_lock_init(\u0026cma_xprt-\u003esc_send_lock);\nnet/sunrpc/xprtrdma/svc_rdma_transport.c:207:\tlockdep_set_class(\u0026cma_xprt-\u003esc_send_lock, \u0026svcrdma_sctx_lock);\nnet/sunrpc/xprtrdma/svc_rdma_transport.c-208-\tspin_lock_init(\u0026cma_xprt-\u003esc_rw_ctxt_lock);\nnet/sunrpc/xprtrdma/svc_rdma_transport.c:209:\tlockdep_set_class(\u0026cma_xprt-\u003esc_rw_ctxt_lock, \u0026svcrdma_rwctx_lock);\nnet/sunrpc/xprtrdma/svc_rdma_transport.c-210-\n--\nnet/tls/tls_device.c=846=int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)\n--\nnet/tls/tls_device.c-864-\nnet/tls/tls_device.c:865:\tmutex_lock(\u0026tls_ctx-\u003etx_lock);\nnet/tls/tls_device.c-866-\tlock_sock(sk);\n--\nnet/tls/tls_device.c-895-\trelease_sock(sk);\nnet/tls/tls_device.c:896:\tmutex_unlock(\u0026tls_ctx-\u003etx_lock);\nnet/tls/tls_device.c-897-\treturn rc;\n--\nnet/tls/tls_device.c=900=void tls_device_splice_eof(struct socket *sock)\n--\nnet/tls/tls_device.c-909-\nnet/tls/tls_device.c:910:\tmutex_lock(\u0026tls_ctx-\u003etx_lock);\nnet/tls/tls_device.c-911-\tlock_sock(sk);\n--\nnet/tls/tls_device.c-921-\trelease_sock(sk);\nnet/tls/tls_device.c:922:\tmutex_unlock(\u0026tls_ctx-\u003etx_lock);\nnet/tls/tls_device.c-923-}\n--\nnet/tls/tls_main.c=323=void tls_ctx_free(struct sock *sk, struct tls_context *ctx)\n--\nnet/tls/tls_main.c-329-\tmemzero_explicit(\u0026ctx-\u003ecrypto_recv, sizeof(ctx-\u003ecrypto_recv));\nnet/tls/tls_main.c:330:\tmutex_destroy(\u0026ctx-\u003etx_lock);\nnet/tls/tls_main.c-331-\n--\nnet/tls/tls_main.c=904=static int do_tls_setsockopt(struct sock *sk, int optname, sockptr_t optval,\n--\nnet/tls/tls_main.c-911-\tcase TLS_RX: {\nnet/tls/tls_main.c:912:\t\t/* tls_device_sendmsg() holds tx_lock across the lock_sock drop\nnet/tls/tls_main.c-913-\t\t * in sk_stream_wait_memory() with a half-built open_record\n--\nnet/tls/tls_main.c-916-\t\t * corrupting record framing. Serialize TX setsockopt against\nnet/tls/tls_main.c:917:\t\t * the data path with tx_lock, unconditionally for TLS_TX,\nnet/tls/tls_main.c-918-\t\t * since during initial setup there is no sender contending it.\n--\nnet/tls/tls_main.c-922-\t\tif (tx) {\nnet/tls/tls_main.c:923:\t\t\trc = mutex_lock_interruptible(\u0026tls_get_ctx(sk)-\u003etx_lock);\nnet/tls/tls_main.c-924-\t\t\tif (rc)\n--\nnet/tls/tls_main.c-930-\t\tif (tx)\nnet/tls/tls_main.c:931:\t\t\tmutex_unlock(\u0026tls_get_ctx(sk)-\u003etx_lock);\nnet/tls/tls_main.c-932-\t\tbreak;\n--\nnet/tls/tls_main.c=971=struct tls_context *tls_ctx_create(struct sock *sk)\n--\nnet/tls/tls_main.c-979-\nnet/tls/tls_main.c:980:\tmutex_init(\u0026ctx-\u003etx_lock);\nnet/tls/tls_main.c-981-\tctx-\u003esk_proto = READ_ONCE(sk-\u003esk_prot);\n--\nnet/tls/tls_sw.c=1007=int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)\n--\nnet/tls/tls_sw.c-1016-\nnet/tls/tls_sw.c:1017:\tret = mutex_lock_interruptible(\u0026tls_ctx-\u003etx_lock);\nnet/tls/tls_sw.c-1018-\tif (ret)\n--\nnet/tls/tls_sw.c-1022-\trelease_sock(sk);\nnet/tls/tls_sw.c:1023:\tmutex_unlock(\u0026tls_ctx-\u003etx_lock);\nnet/tls/tls_sw.c-1024-\treturn ret;\n--\nnet/tls/tls_sw.c-1030- * Inner logic of tls_sw_splice_eof(), factored out so the device\nnet/tls/tls_sw.c:1031: * TX path can reuse it with tls_ctx-\u003etx_lock and the socket lock\nnet/tls/tls_sw.c-1032- * already held. Callers not already holding both locks must use the\n--\nnet/tls/tls_sw.c=1083=void tls_sw_splice_eof(struct socket *sock)\n--\nnet/tls/tls_sw.c-1091-\nnet/tls/tls_sw.c:1092:\tmutex_lock(\u0026tls_ctx-\u003etx_lock);\nnet/tls/tls_sw.c-1093-\tlock_sock(sk);\n--\nnet/tls/tls_sw.c-1095-\trelease_sock(sk);\nnet/tls/tls_sw.c:1096:\tmutex_unlock(\u0026tls_ctx-\u003etx_lock);\nnet/tls/tls_sw.c-1097-}\n--\nnet/tls/tls_sw.c=2382=static void tx_work_handler(struct work_struct *work)\n--\nnet/tls/tls_sw.c-2400-\nnet/tls/tls_sw.c:2401:\tif (mutex_trylock(\u0026tls_ctx-\u003etx_lock)) {\nnet/tls/tls_sw.c-2402-\t\tlock_sock(sk);\n--\nnet/tls/tls_sw.c-2404-\t\trelease_sock(sk);\nnet/tls/tls_sw.c:2405:\t\tmutex_unlock(\u0026tls_ctx-\u003etx_lock);\nnet/tls/tls_sw.c-2406-\t} else if (!test_and_set_bit(BIT_TX_SCHEDULED, \u0026ctx-\u003etx_bitmask)) {\nnet/tls/tls_sw.c:2407:\t\t/* Someone is holding the tx_lock, they will likely run Tx\nnet/tls/tls_sw.c-2408-\t\t * and cancel the work on their way out of the lock section.\n--\nnet/vmw_vsock/virtio_transport.c=30=struct virtio_vsock {\n--\nnet/vmw_vsock/virtio_transport.c-38-\nnet/vmw_vsock/virtio_transport.c:39:\t/* The following fields are protected by tx_lock. vqs[VSOCK_VQ_TX]\nnet/vmw_vsock/virtio_transport.c:40:\t * must be accessed with tx_lock held.\nnet/vmw_vsock/virtio_transport.c-41-\t */\nnet/vmw_vsock/virtio_transport.c:42:\tstruct mutex tx_lock;\nnet/vmw_vsock/virtio_transport.c-43-\tbool tx_run;\n--\nnet/vmw_vsock/virtio_transport.c-66-\t * 'virtio_vsock_probe()'. Both fields are protected\nnet/vmw_vsock/virtio_transport.c:67:\t * by 'tx_lock'. +1 is needed for packet header.\nnet/vmw_vsock/virtio_transport.c-68-\t */\n--\nnet/vmw_vsock/virtio_transport.c=82=static u32 virtio_transport_get_local_cid(void)\n--\nnet/vmw_vsock/virtio_transport.c-99-\nnet/vmw_vsock/virtio_transport.c:100:/* Caller need to hold vsock-\u003etx_lock on vq */\nnet/vmw_vsock/virtio_transport.c-101-static int virtio_transport_send_skb(struct sk_buff *skb, struct virtqueue *vq,\n--\nnet/vmw_vsock/virtio_transport.c=158=virtio_transport_send_pkt_work(struct work_struct *work)\n--\nnet/vmw_vsock/virtio_transport.c-165-\nnet/vmw_vsock/virtio_transport.c:166:\tmutex_lock(\u0026vsock-\u003etx_lock);\nnet/vmw_vsock/virtio_transport.c-167-\n--\nnet/vmw_vsock/virtio_transport.c-207-out:\nnet/vmw_vsock/virtio_transport.c:208:\tmutex_unlock(\u0026vsock-\u003etx_lock);\nnet/vmw_vsock/virtio_transport.c-209-\n--\nnet/vmw_vsock/virtio_transport.c=217=static int virtio_transport_send_skb_fast_path(struct virtio_vsock *vsock, struct sk_buff *skb)\n--\nnet/vmw_vsock/virtio_transport.c-222-\t/* Inside RCU, can't sleep! */\nnet/vmw_vsock/virtio_transport.c:223:\tret = mutex_trylock(\u0026vsock-\u003etx_lock);\nnet/vmw_vsock/virtio_transport.c-224-\tif (unlikely(ret == 0))\n--\nnet/vmw_vsock/virtio_transport.c-230-\nnet/vmw_vsock/virtio_transport.c:231:\tmutex_unlock(\u0026vsock-\u003etx_lock);\nnet/vmw_vsock/virtio_transport.c-232-\n--\nnet/vmw_vsock/virtio_transport.c=342=static void virtio_transport_tx_work(struct work_struct *work)\n--\nnet/vmw_vsock/virtio_transport.c-348-\nnet/vmw_vsock/virtio_transport.c:349:\tmutex_lock(\u0026vsock-\u003etx_lock);\nnet/vmw_vsock/virtio_transport.c-350-\n--\nnet/vmw_vsock/virtio_transport.c-367-out:\nnet/vmw_vsock/virtio_transport.c:368:\tmutex_unlock(\u0026vsock-\u003etx_lock);\nnet/vmw_vsock/virtio_transport.c-369-\n--\nnet/vmw_vsock/virtio_transport.c=727=static void virtio_vsock_vqs_start(struct virtio_vsock *vsock)\nnet/vmw_vsock/virtio_transport.c-728-{\nnet/vmw_vsock/virtio_transport.c:729:\tmutex_lock(\u0026vsock-\u003etx_lock);\nnet/vmw_vsock/virtio_transport.c-730-\tvsock-\u003etx_run = true;\nnet/vmw_vsock/virtio_transport.c:731:\tmutex_unlock(\u0026vsock-\u003etx_lock);\nnet/vmw_vsock/virtio_transport.c-732-\n--\nnet/vmw_vsock/virtio_transport.c=755=static void virtio_vsock_vqs_del(struct virtio_vsock *vsock)\n--\nnet/vmw_vsock/virtio_transport.c-770-\nnet/vmw_vsock/virtio_transport.c:771:\tmutex_lock(\u0026vsock-\u003etx_lock);\nnet/vmw_vsock/virtio_transport.c-772-\tvsock-\u003etx_run = false;\nnet/vmw_vsock/virtio_transport.c:773:\tmutex_unlock(\u0026vsock-\u003etx_lock);\nnet/vmw_vsock/virtio_transport.c-774-\n--\nnet/vmw_vsock/virtio_transport.c-788-\nnet/vmw_vsock/virtio_transport.c:789:\tmutex_lock(\u0026vsock-\u003etx_lock);\nnet/vmw_vsock/virtio_transport.c-790-\twhile ((skb = virtqueue_detach_unused_buf(vsock-\u003evqs[VSOCK_VQ_TX])))\nnet/vmw_vsock/virtio_transport.c-791-\t\tkfree_skb(skb);\nnet/vmw_vsock/virtio_transport.c:792:\tmutex_unlock(\u0026vsock-\u003etx_lock);\nnet/vmw_vsock/virtio_transport.c-793-\n--\nnet/vmw_vsock/virtio_transport.c=800=static int virtio_vsock_probe(struct virtio_device *vdev)\n--\nnet/vmw_vsock/virtio_transport.c-825-\nnet/vmw_vsock/virtio_transport.c:826:\tmutex_init(\u0026vsock-\u003etx_lock);\nnet/vmw_vsock/virtio_transport.c-827-\tmutex_init(\u0026vsock-\u003erx_lock);\n--\nnet/vmw_vsock/virtio_transport_common.c=619=void virtio_transport_consume_skb_sent(struct sk_buff *skb, bool consume)\n--\nnet/vmw_vsock/virtio_transport_common.c-628-\nnet/vmw_vsock/virtio_transport_common.c:629:\t\tspin_lock_bh(\u0026vvs-\u003etx_lock);\nnet/vmw_vsock/virtio_transport_common.c-630-\t\tvvs-\u003ebytes_unsent -= skb-\u003elen;\nnet/vmw_vsock/virtio_transport_common.c:631:\t\tspin_unlock_bh(\u0026vvs-\u003etx_lock);\nnet/vmw_vsock/virtio_transport_common.c-632-\t}\n--\nnet/vmw_vsock/virtio_transport_common.c=639=u32 virtio_transport_get_credit(struct virtio_vsock_sock *vvs, u32 credit)\n--\nnet/vmw_vsock/virtio_transport_common.c-645-\nnet/vmw_vsock/virtio_transport_common.c:646:\tspin_lock_bh(\u0026vvs-\u003etx_lock);\nnet/vmw_vsock/virtio_transport_common.c-647-\tret = min_t(u32, credit, virtio_transport_has_space(vvs));\n--\nnet/vmw_vsock/virtio_transport_common.c-649-\tvvs-\u003ebytes_unsent += ret;\nnet/vmw_vsock/virtio_transport_common.c:650:\tspin_unlock_bh(\u0026vvs-\u003etx_lock);\nnet/vmw_vsock/virtio_transport_common.c-651-\n--\nnet/vmw_vsock/virtio_transport_common.c=656=void virtio_transport_put_credit(struct virtio_vsock_sock *vvs, u32 credit)\n--\nnet/vmw_vsock/virtio_transport_common.c-660-\nnet/vmw_vsock/virtio_transport_common.c:661:\tspin_lock_bh(\u0026vvs-\u003etx_lock);\nnet/vmw_vsock/virtio_transport_common.c-662-\tvvs-\u003etx_cnt -= credit;\nnet/vmw_vsock/virtio_transport_common.c-663-\tvvs-\u003ebytes_unsent -= credit;\nnet/vmw_vsock/virtio_transport_common.c:664:\tspin_unlock_bh(\u0026vvs-\u003etx_lock);\nnet/vmw_vsock/virtio_transport_common.c-665-}\n--\nnet/vmw_vsock/virtio_transport_common.c=978=virtio_transport_seqpacket_enqueue(struct vsock_sock *vsk,\n--\nnet/vmw_vsock/virtio_transport_common.c-983-\nnet/vmw_vsock/virtio_transport_common.c:984:\tspin_lock_bh(\u0026vvs-\u003etx_lock);\nnet/vmw_vsock/virtio_transport_common.c-985-\nnet/vmw_vsock/virtio_transport_common.c-986-\tif (len \u003e virtio_transport_tx_buf_size(vvs)) {\nnet/vmw_vsock/virtio_transport_common.c:987:\t\tspin_unlock_bh(\u0026vvs-\u003etx_lock);\nnet/vmw_vsock/virtio_transport_common.c-988-\t\treturn -EMSGSIZE;\n--\nnet/vmw_vsock/virtio_transport_common.c-990-\nnet/vmw_vsock/virtio_transport_common.c:991:\tspin_unlock_bh(\u0026vvs-\u003etx_lock);\nnet/vmw_vsock/virtio_transport_common.c-992-\n--\nnet/vmw_vsock/virtio_transport_common.c=1048=s64 virtio_transport_stream_has_space(struct vsock_sock *vsk)\n--\nnet/vmw_vsock/virtio_transport_common.c-1052-\nnet/vmw_vsock/virtio_transport_common.c:1053:\tspin_lock_bh(\u0026vvs-\u003etx_lock);\nnet/vmw_vsock/virtio_transport_common.c-1054-\tbytes = virtio_transport_has_space(vvs);\nnet/vmw_vsock/virtio_transport_common.c:1055:\tspin_unlock_bh(\u0026vvs-\u003etx_lock);\nnet/vmw_vsock/virtio_transport_common.c-1056-\n--\nnet/vmw_vsock/virtio_transport_common.c=1061=int virtio_transport_do_socket_init(struct vsock_sock *vsk,\n--\nnet/vmw_vsock/virtio_transport_common.c-1083-\tspin_lock_init(\u0026vvs-\u003erx_lock);\nnet/vmw_vsock/virtio_transport_common.c:1084:\tspin_lock_init(\u0026vvs-\u003etx_lock);\nnet/vmw_vsock/virtio_transport_common.c-1085-\tskb_queue_head_init(\u0026vvs-\u003erx_queue);\n--\nnet/vmw_vsock/virtio_transport_common.c=1281=ssize_t virtio_transport_unsent_bytes(struct vsock_sock *vsk)\n--\nnet/vmw_vsock/virtio_transport_common.c-1285-\nnet/vmw_vsock/virtio_transport_common.c:1286:\tspin_lock_bh(\u0026vvs-\u003etx_lock);\nnet/vmw_vsock/virtio_transport_common.c-1287-\tret = vvs-\u003ebytes_unsent;\nnet/vmw_vsock/virtio_transport_common.c:1288:\tspin_unlock_bh(\u0026vvs-\u003etx_lock);\nnet/vmw_vsock/virtio_transport_common.c-1289-\n--\nnet/vmw_vsock/virtio_transport_common.c=1677=static bool virtio_transport_space_update(struct sock *sk,\n--\nnet/vmw_vsock/virtio_transport_common.c-1693-\t/* buf_alloc and fwd_cnt is always included in the hdr */\nnet/vmw_vsock/virtio_transport_common.c:1694:\tspin_lock_bh(\u0026vvs-\u003etx_lock);\nnet/vmw_vsock/virtio_transport_common.c-1695-\tvvs-\u003epeer_buf_alloc = le32_to_cpu(hdr-\u003ebuf_alloc);\n--\nnet/vmw_vsock/virtio_transport_common.c-1697-\tspace_available = virtio_transport_has_space(vvs);\nnet/vmw_vsock/virtio_transport_common.c:1698:\tspin_unlock_bh(\u0026vvs-\u003etx_lock);\nnet/vmw_vsock/virtio_transport_common.c-1699-\treturn space_available;\n"
} |
| 19/2 | 2026/09/20 01:02 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 107063 output: 37 thoughts: 477 Thoughts: **Analyzing the TLS TX Lock in the Linux Kernel**
Okay, let's break this down. I'm looking at `net/tls/tls_main.c` and immediately notice the `mutex_init(&ctx->tx_lock)` call within `tls_ctx_create()`. This tells me that every TLS socket initialized through `setsockopt(fd, IPPROTO_TCP, TCP_ULP, "tls", sizeof("tls"))` creates a context (`ctx`) and, crucially, initializes `ctx->tx_lock`.
Now, digging into `do_tls_setsockopt()`, specifically handling `TLS_TX` and `TLS_RX` options, I see a clear locking pattern. If `optname` is `TLS_TX` (which corresponds to transmission), a mutex lock (`mutex_lock_interruptible(&tls_get_ctx(sk)->tx_lock)`) is acquired *before* calling `do_tls_setsockopt_conf()`. The socket is then locked, `do_tls_setsockopt_conf` is called, and *then* the mutex is unlocked. Standard pattern, nothing overtly surprising yet.
However, the connection becomes more intriguing when I look at `do_tls_setsockopt_conf()` itself. This function is where the real work happens, and importantly, it's called with `tx == 1` when the `tx_lock` is held. So far, so good: the mutex protects concurrent access during transmission configuration.
My primary concern now is what happens during an *update*. If this is an existing connection, userspace is calling `setsockopt(fd, SOL_TLS, TLS_TX, &crypto_info, sizeof(crypto_info))` on an existing socket! That's a rekeying operation. This would be `update` being true in `do_tls_setsockopt_conf`.
And the critical question is, does this apply to software-based TLS implementations? Can userspace trigger a `TLS_TX` operation to update keys on a software TLS connection?
The answer is yes: With TLS 1.3, KeyUpdate is supported!
Let's check `net/tls/tls_main.c` around `update`.
|
| 20/2 | 2026/09/20 01:02 | tool | read-file |
0mArgs:{
"File": "net/tls/tls_main.c",
"FirstLine": 690,
"LineCount": 80
}Results: {
"Contents": " 690:\t\t\t\treturn -EBUSY;\n 691:\t\t\t}\n 692:\t\n 693:\t\t\tupdate = true;\n 694:\t\t\told_crypto_info = crypto_info;\n 695:\t\t\tcrypto_info = \u0026tmp.info;\n 696:\t\t\tcrypto_ctx = \u0026tmp;\n 697:\t\t}\n 698:\t\n 699:\t\trc = copy_from_sockptr(crypto_info, optval, sizeof(*crypto_info));\n 700:\t\tif (rc) {\n 701:\t\t\trc = -EFAULT;\n 702:\t\t\tgoto err_crypto_info;\n 703:\t\t}\n 704:\t\n 705:\t\tif (update) {\n 706:\t\t\t/* Ensure that TLS version and ciphers are not modified */\n 707:\t\t\tif (crypto_info-\u003eversion != old_crypto_info-\u003eversion ||\n 708:\t\t\t crypto_info-\u003ecipher_type != old_crypto_info-\u003ecipher_type)\n 709:\t\t\t\trc = -EINVAL;\n 710:\t\t} else {\n 711:\t\t\trc = validate_crypto_info(crypto_info, alt_crypto_info);\n 712:\t\t}\n 713:\t\tif (rc)\n 714:\t\t\tgoto err_crypto_info;\n 715:\t\n 716:\t\tcipher_desc = get_cipher_desc(crypto_info-\u003ecipher_type);\n 717:\t\tif (!cipher_desc) {\n 718:\t\t\trc = -EINVAL;\n 719:\t\t\tgoto err_crypto_info;\n 720:\t\t}\n 721:\t\n 722:\t\tif (optlen != cipher_desc-\u003ecrypto_info) {\n 723:\t\t\trc = -EINVAL;\n 724:\t\t\tgoto err_crypto_info;\n 725:\t\t}\n 726:\t\n 727:\t\trc = copy_from_sockptr_offset(crypto_info + 1, optval,\n 728:\t\t\t\t\t sizeof(*crypto_info),\n 729:\t\t\t\t\t optlen - sizeof(*crypto_info));\n 730:\t\tif (rc) {\n 731:\t\t\trc = -EFAULT;\n 732:\t\t\tgoto err_crypto_info;\n 733:\t\t}\n 734:\t\n 735:\t\tif (tx) {\n 736:\t\t\trc = tls_set_device_offload(sk, update ? crypto_info : NULL);\n 737:\t\t\tconf = TLS_HW;\n 738:\t\t\tif (!rc) {\n 739:\t\t\t\tif (!update) {\n 740:\t\t\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXDEVICE);\n 741:\t\t\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);\n 742:\t\t\t\t}\n 743:\t\t\t} else if (update \u0026\u0026 ctx-\u003etx_conf == TLS_HW) {\n 744:\t\t\t\t/* HW rekey failed - return the actual error.\n 745:\t\t\t\t * Cannot fall back to SW for an existing HW connection.\n 746:\t\t\t\t */\n 747:\t\t\t\tgoto err_crypto_info;\n 748:\t\t\t} else {\n 749:\t\t\t\trc = tls_set_sw_offload(sk, 1,\n 750:\t\t\t\t\t\t\tupdate ? crypto_info : NULL);\n 751:\t\t\t\tif (rc)\n 752:\t\t\t\t\tgoto err_crypto_info;\n 753:\t\n 754:\t\t\t\tif (update) {\n 755:\t\t\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYOK);\n 756:\t\t\t\t} else {\n 757:\t\t\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXSW);\n 758:\t\t\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);\n 759:\t\t\t\t}\n 760:\t\t\t\tconf = TLS_SW;\n 761:\t\t\t}\n 762:\t\t} else {\n 763:\t\t\trc = tls_set_device_offload_rx(sk, ctx,\n 764:\t\t\t\t\t\t update ? crypto_info : NULL);\n 765:\t\t\tconf = TLS_HW;\n 766:\t\t\tif (!rc) {\n 767:\t\t\t\tif (!update) {\n 768:\t\t\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXDEVICE);\n 769:\t\t\t\t\tTLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);\n"
} |
| 21/2 | 2026/09/20 01:02 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 109280 output: 26 thoughts: 319 Thoughts: **Analyzing the TLS Offload Patch**
Okay, let's break this down. Line 736... right, `rc = tls_set_device_offload(sk, update ? crypto_info : NULL);`. I'm focusing on the Software TLS connection scenario, so `ctx->tx_conf` is `TLS_SW`. I need to understand what's happening within `tls_set_device_offload` when `new_crypto_info` is present. Let's see...
```c
int tls_set_device_offload(struct sock *sk,
struct tls_crypto_info *new_crypto_info)
{
...
ctx = tls_get_ctx(sk);
/* A rekey of a SW-offloaded socket belongs to tls_set_sw_offload(). */
if (new_crypto_info && ctx->tx_conf != TLS_HW)
return -EINVAL;
```
Ah, it returns `-EINVAL`. Makes sense. Then back in `do_tls_setsockopt_conf`, because `update && ctx->tx_conf == TLS_HW` is false, it proceeds to: `rc = tls_set_sw_offload(sk, 1, update ? crypto_info : NULL);` So `tls_set_sw_offload` is actually what gets executed. Good, that clarifies the flow.
Now, I really need to get a broader view. I need to understand the *purpose* of this patch series. Let me check the commit history. The title and subject should give me the full context. That's the key to understanding the full implications of these changes.
|
| 22/2 | 2026/09/20 01:02 | tool | grepper |
0mArgs:{
"Expression": "tls.*rekey",
"PathPrefix": "Documentation/"
}Results: {
"Output": "Documentation/networking/tls-offload.rst=482=socket) the rekey stays pending, the ``sendmsg()`` proceeds in software,\nDocumentation/networking/tls-offload.rst:483:and the next ``sendmsg()`` retries; the ``tls_device_complete_rekey_retry``\nDocumentation/networking/tls-offload.rst-484-tracepoint fires. A hard failure (:c:func:`tls_dev_add` rejected, or the\n--\nDocumentation/networking/tls-offload.rst=487=and moved from ``TlsCurrTxDevice`` to ``TlsCurrTxSw``; the\nDocumentation/networking/tls-offload.rst:488:``tls_device_complete_rekey_fail`` tracepoint fires. The next ``TLS_TX``\nDocumentation/networking/tls-offload.rst-489-setsockopt re-arms the transition and retries.\n"
} |
| 23/2 | 2026/09/20 01:02 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 110223 output: 24 thoughts: 1444 Thoughts: **Analyzing TLS Offload Implementation** Okay, let's break this down. First, the commit message provides a starting point, even without direct access to `git log`. The key is this TLS 1.3 KeyUpdate message and how it impacts offloaded connections. The kernel needs to handle key switching mid-connection without tearing down the offload, which presents a challenge since old and new keys could be in flight. The software bridge mentioned seems crucial here. Now, let's see which files were actually modified. Looking at the diff, we've got a lot of files to sift through, including documentation, MAINTAINERS, and drivers. The critical part jumps out: these drivers all relate to hardware NICs—Chelsio, Mellanox, and Netronome—specifically, their inline crypto capabilities. This points to TLS offload being directly implemented in the hardware, not simply a software emulation. However, we also have modifications in the `net/` directory, specifically `tls_device.c` and `tls_device_fallback.c`. This is intriguing because these are *not* drivers, but core networking components. My expert intuition raises the question: can these core TLS device components be reached when running a standard QEMU or GCE environment or is this exclusively meant for bare metal hardware? To answer this question, we must understand how a socket's context comes to use `TLS_HW`, indicating hardware-based TLS. That is done with `do_tls_setsockopt_conf` and specifically `tls_set_device_offload`. This function's success depends on the network device having `NETIF_F_HW_TLS_TX` set. Looking at the files, that feature is only set by the NIC drivers we saw earlier. So, the question is: can a virtual device (veth, virtio_net, tun, etc.) also set this flag? Because only those physical NICs sets the flag, and `tls_device.c` can be reached. I need to meticulously check the kernel source to confirm that no virtual device or software device claims to support hardware TLS offload. If not, then this code path is exclusively for bare-metal with dedicated hardware. Let's do a comprehensive grep across the entire kernel source code to verify. The key is to see if any other device besides cxgb4, funeth, mlx5, nfp set `NETIF_F_HW_TLS_TX` and `NETIF_F_HW_TLS_RX`. This will clarify whether `tls_device.c` has any role to play in virtualized environments. |
| 24/2 | 2026/09/20 01:02 | tool | grepper |
0mArgs:{
"Expression": "NETIF_F_HW_TLS_RX"
}Results: {
"Output": "Documentation/networking/tls-offload.rst=65=Device configuration\n--\nDocumentation/networking/tls-offload.rst-67-\nDocumentation/networking/tls-offload.rst:68:During driver initialization device sets the ``NETIF_F_HW_TLS_RX`` and\nDocumentation/networking/tls-offload.rst-69-``NETIF_F_HW_TLS_TX`` features and installs its\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c=113=void mlx5e_ktls_build_netdev(struct mlx5e_priv *priv)\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c-126-\tif (mlx5e_is_ktls_rx(mdev))\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c:127:\t\tnetdev-\u003ehw_features |= NETIF_F_HW_TLS_RX;\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c-128-\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c=152=int mlx5e_ktls_init_rx(struct mlx5e_priv *priv)\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c-162-\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c:163:\tif (priv-\u003enetdev-\u003efeatures \u0026 NETIF_F_HW_TLS_RX) {\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c-164-\t\terr = mlx5e_accel_fs_tcp_create(priv-\u003efs);\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c=175=void mlx5e_ktls_cleanup_rx(struct mlx5e_priv *priv)\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c-179-\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c:180:\tif (priv-\u003enetdev-\u003efeatures \u0026 NETIF_F_HW_TLS_RX)\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c-181-\t\tmlx5e_accel_fs_tcp_destroy(priv-\u003efs);\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_main.c=4489=int mlx5e_set_features(struct net_device *netdev, netdev_features_t features)\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_main.c-4513-#endif\ndrivers/net/ethernet/mellanox/mlx5/core/en_main.c:4514:\terr |= MLX5E_HANDLE_FEATURE(NETIF_F_HW_TLS_RX, mlx5e_ktls_set_feature_rx);\ndrivers/net/ethernet/mellanox/mlx5/core/en_main.c-4515-\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_main.c=4524=static netdev_features_t mlx5e_fix_uplink_rep_features(struct net_device *netdev,\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_main.c-4526-{\ndrivers/net/ethernet/mellanox/mlx5/core/en_main.c:4527:\tfeatures \u0026= ~NETIF_F_HW_TLS_RX;\ndrivers/net/ethernet/mellanox/mlx5/core/en_main.c:4528:\tif (netdev-\u003efeatures \u0026 NETIF_F_HW_TLS_RX)\ndrivers/net/ethernet/mellanox/mlx5/core/en_main.c-4529-\t\tnetdev_warn(netdev, \"Disabling hw_tls_rx, not supported in switchdev mode\\n\");\n--\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c=567=int nfp_net_tls_init(struct nfp_net *nn)\n--\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c-596-\tif (nn-\u003etlv_caps.crypto_ops \u0026 NFP_NET_TLS_OPCODE_MASK_RX) {\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c:597:\t\tnetdev-\u003ehw_features |= NETIF_F_HW_TLS_RX;\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c:598:\t\tnetdev-\u003efeatures |= NETIF_F_HW_TLS_RX;\ndrivers/net/ethernet/netronome/nfp/crypto/tls.c-599-\t}\n--\ninclude/linux/netdev_features.h=14=enum {\n--\ninclude/linux/netdev_features.h-78-\tNETIF_F_HW_TLS_TX_BIT,\t\t/* Hardware TLS TX offload */\ninclude/linux/netdev_features.h:79:\tNETIF_F_HW_TLS_RX_BIT,\t\t/* Hardware TLS RX offload */\ninclude/linux/netdev_features.h-80-\n--\ninclude/linux/netdev_features.h-157-#define NETIF_F_HW_TLS_TX\t__NETIF_F(HW_TLS_TX)\ninclude/linux/netdev_features.h:158:#define NETIF_F_HW_TLS_RX\t__NETIF_F(HW_TLS_RX)\ninclude/linux/netdev_features.h-159-#define NETIF_F_GRO_FRAGLIST\t__NETIF_F(GRO_FRAGLIST)\n--\nnet/core/dev.c=10960=static netdev_features_t netdev_fix_features(struct net_device *dev,\n--\nnet/core/dev.c-11045-\nnet/core/dev.c:11046:\tif ((features \u0026 NETIF_F_HW_TLS_RX) \u0026\u0026 !(features \u0026 NETIF_F_RXCSUM)) {\nnet/core/dev.c-11047-\t\tnetdev_dbg(dev, \"Dropping TLS RX HW offload feature since no RXCSUM feature.\\n\");\nnet/core/dev.c:11048:\t\tfeatures \u0026= ~NETIF_F_HW_TLS_RX;\nnet/core/dev.c-11049-\t}\n--\nnet/ethtool/common.c=16=const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN] = {\n--\nnet/ethtool/common.c-70-\t[NETIF_F_HW_TLS_TX_BIT] =\t \"tls-hw-tx-offload\",\nnet/ethtool/common.c:71:\t[NETIF_F_HW_TLS_RX_BIT] =\t \"tls-hw-rx-offload\",\nnet/ethtool/common.c-72-\t[NETIF_F_GRO_FRAGLIST_BIT] =\t \"rx-gro-list\",\n--\nnet/tls/tls_device.c=2096=int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,\n--\nnet/tls/tls_device.c-2149-\nnet/tls/tls_device.c:2150:\t\tif (!(netdev-\u003efeatures \u0026 NETIF_F_HW_TLS_RX)) {\nnet/tls/tls_device.c-2151-\t\t\trc = -EOPNOTSUPP;\n--\nnet/tls/tls_device.c=2497=static int tls_dev_event(struct notifier_block *this, unsigned long event,\n--\nnet/tls/tls_device.c-2502-\tif (!dev-\u003etlsdev_ops \u0026\u0026\nnet/tls/tls_device.c:2503:\t !(dev-\u003efeatures \u0026 (NETIF_F_HW_TLS_RX | NETIF_F_HW_TLS_TX)))\nnet/tls/tls_device.c-2504-\t\treturn NOTIFY_DONE;\n--\nnet/tls/tls_device.c-2514-\t\t\treturn NOTIFY_BAD;\nnet/tls/tls_device.c:2515:\t\tif ((dev-\u003efeatures \u0026 NETIF_F_HW_TLS_RX) \u0026\u0026\nnet/tls/tls_device.c-2516-\t\t !dev-\u003etlsdev_ops-\u003etls_dev_resync)\n"
} |
| 25/2 | 2026/09/20 01:02 | llm | patch-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 114482 output: 21 thoughts: 3765 Thoughts: **Analysis of Patch Series Impact on Syzkaller/QEMU Reached Code** Alright, let's break this down. My initial focus was hardware TLS, but quickly realized the potential for this series to touch *any* code path reachable by syzkaller or within a QEMU environment. So, let's see where these changes *actually* land within our existing infrastructure. The key thing I am seeing is that the functions `tls_set_device_offload` and `tls_set_device_offload_rx` are called regardless of whether we're configuring hardware or software TLS. Critically, within `tls_set_device_offload`, the crucial lines of code that check `NETIF_F_HW_TLS_TX` and the initialization of the `cipher_desc` are executed on *every* socket. We can force these function calls with the `TLS_TX` and `TLS_RX` setsockopt options. Furthermore, the `mutex_lock_interruptible(&tls_get_ctx(sk)->tx_lock)` is executed on every `TLS_TX` call! This means a lot of the code is executed even when we are not running hardware TLS. `tls_set_sw_offload` is called after `tls_set_device_offload` is called, and is run for every `setsockopt` call. Within `tls_set_sw_offload` the core initialization and finalization are called. This seems important to me, because `tls_sw_ctx_init` and `tls_sw_ctx_finalize` are functions that are brand new to this patch! Diving into the software kTLS data path, I immediately see `tls_tx_cipher_ctx` being called during *every* packet encryption and push. This is a critical code path, so changes here can have widespread effects. `tls_tx_cipher_ctx` is simple, but is called very often. `tls_sw_ctx_tx` is called frequently. I'm also seeing that `skb_is_decrypted` is checked in `net/sched/sch_generic.c` in `dequeue_skb`, a critical function in the qdisc dequeue path. Whenever ANY skb is dequeued, we're hitting this. Finally, I notice that `tcp_write_collapse_fence` is being called - in the path of the TCP stack. This is more of a concern because it is in the TCP core. Based on this analysis, the changes within this patch series *do* modify code that is actively executed when not using hardware TLS. Specifically, functions like `tls_set_device_offload`, `tls_set_device_offload_rx`, `tls_set_sw_offload`, `tls_sw_ctx_init`, `tls_sw_ctx_finalize`, and calls to `tls_tx_cipher_ctx` and `tls_sw_ctx_tx` are executed on every TLS operation. This makes these changes very important to test. |
| 26/2 | 2026/09/20 01:03 | tool | grepper |
0mArgs:{
"Expression": "tcp_write_collapse_fence"
}Results: {
"Output": "include/net/tcp.h=1235=static inline bool tcp_skb_can_collapse(const struct sk_buff *to,\n--\ninclude/net/tcp.h-1237-{\ninclude/net/tcp.h:1238:\t/* skb_cmp_decrypted() not needed, use tcp_write_collapse_fence() */\ninclude/net/tcp.h-1239-\treturn likely(tcp_skb_can_collapse_to(to) \u0026\u0026\n--\ninclude/net/tcp.h=2332=static inline void tcp_rtx_queue_unlink_and_free(struct sk_buff *skb, struct sock *sk)\n--\ninclude/net/tcp.h-2338-\ninclude/net/tcp.h:2339:static inline void tcp_write_collapse_fence(struct sock *sk)\ninclude/net/tcp.h-2340-{\n--\nnet/psp/psp_sock.c=214=psp_sock_set_tx_key(struct sock *sk, struct psp_dev *psd, struct psp_assoc *pas,\n--\nnet/psp/psp_sock.c-231-\tWRITE_ONCE(sk-\u003esk_validate_xmit_skb, psp_validate_xmit);\nnet/psp/psp_sock.c:232:\ttcp_write_collapse_fence(sk);\nnet/psp/psp_sock.c-233-\tpas-\u003eupgrade_seq = tcp_sk(sk)-\u003ercv_nxt;\n--\nnet/tls/tls_device.c=192=static void tls_device_commit_start_marker(struct sock *sk,\n--\nnet/tls/tls_device.c-201-\t */\nnet/tls/tls_device.c:202:\ttcp_write_collapse_fence(sk);\nnet/tls/tls_device.c-203-}\n--\nnet/tls/tls_device.c=349=static void tls_device_commit_rekey_marker(struct sock *sk,\n--\nnet/tls/tls_device.c-378-\nnet/tls/tls_device.c:379:\ttcp_write_collapse_fence(sk);\nnet/tls/tls_device.c-380-}\n--\nnet/tls/tls_device.c=491=static void tls_device_resync_tx(struct sock *sk, struct tls_context *tls_ctx,\n--\nnet/tls/tls_device.c-497-\nnet/tls/tls_device.c:498:\ttcp_write_collapse_fence(sk);\nnet/tls/tls_device.c-499-\trcd_sn = tls_ctx-\u003etx.rec_seq;\n--\nnet/tls/tls_device.c=1596=static int tls_device_start_rekey(struct sock *sk,\n--\nnet/tls/tls_device.c-1724-\t\t/* Prevent a partial record straddling the SW/HW boundary. */\nnet/tls/tls_device.c:1725:\t\ttcp_write_collapse_fence(sk);\nnet/tls/tls_device.c-1726-\n"
} |
| 27/2 | 2026/09/20 01:03 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 119721 output: 33 thoughts: 129 Thoughts: **Investigating TLS Offload with `netdevsim`** Okay, so I'm looking into how TLS offload works, and I just noticed that `tcp_write_collapse_fence(sk)` is also called in `net/psp/psp_sock.c`. That's interesting, maybe there's a connection there I need to understand. But, hang on, the real question is: can I use `netdevsim` to test or even potentially offload TLS processing? That's what I really need to figure out. I *thought* I had already checked the `drivers/net/netdevsim/` directory earlier, grepping for "tls," and came up empty, but let me double-check my work. Right, the next logical step is to dive into `tools/testing/selftests/net/tls.c`. This *should* give me some insight into the current TLS testing landscape within the kernel. I need to understand what this self-test does and if it offers any clues about `netdevsim`'s role in this area. Let's see what's in there. |
| 28/2 | 2026/09/20 01:03 | tool | grepper |
0mError:no matches Args: {
"Expression": "TLS_HW",
"PathPrefix": "tools/testing/selftests/net/"
}Results: {
"error": "no matches"
} |
| 29/2 | 2026/09/20 01:03 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 120140 output: 38 thoughts: 1799 Thoughts: **Analysis of Potential Fuzzing Targets in KTLs Changes** Okay, so I'm looking at this patch and trying to determine if it introduces changes worthy of fuzzing. The initial file, `tools/testing/selftests/net/tls.c`, doesn't seem to touch the hardware TLS (`TLS_HW`) path. I see the other file, `tools/testing/selftests/drivers/net/hw/tls_hw_offload.c`, and its associated Python script (`tls_hw_offload.py`), which are explicitly about testing hardware offload between physical nodes using hardcoded keys. The Python script does a check using `ethtool` to see if the device has the offload support; if not, it skips the test with a `KsftSkipEx`. But wait. The question is whether this change is *solely* about unreachable hardware offload. I have to read the relevant sections from the document carefully: Section 2 (returning `WorthFuzzing=false`) and Section 3 (returning `WorthFuzzing=true`). Section 2 is key: it states that `WorthFuzzing=false` ONLY if *all* modified code falls into certain categories: non-kernel/non-executable changes, structurally unreachable hardware (like vendor-specific PCIe switches, SmartNICs, or GPU drivers not emulated in QEMU), or unreachable execution paths like driver teardown callbacks or code exclusive to different architectures. The phrasing is important: "ONLY IF ALL". The important question is, does *all* of the modified code fall *strictly* into that unreachable hardware category? No! I can see that there are more changes in `net/sched/sch_generic.c`, which touches the core packet scheduler, `include/net/tcp.h` (the function `tcp_write_collapse_fence`), `include/linux/skbuff.h` which includes new sk_buff fields, and the `net/tls/` directory. The changes in `net/tls/` are extensive: there are changes in `tls_main.c` including the addition of `tx_lock`, changes in `do_tls_setsockopt`, and `tls_set_device_offload`. There are modifications to `tls_sw.c`: refactoring into `tls_sw_ctx_init` and `tls_sw_ctx_finalize` and updating references to `tls_tx_cipher_ctx`, new functions and modifications to existing ones: `tls_sw_sendmsg_locked`, `tls_sw_splice_eof_locked`, `tls_sw_drain_tx`, `tls_sw_ctx_tx_init`, and a function called `tls_check_pending_rekey` which also calls `tls_device_rx_del_key`. Crucially, `net/tls/tls_main.c` and `net/tls/tls_sw.c` are part of the *core* networking stack, software TLS (kTLS). This is reachable in any standard environment supporting TCP sockets. Any user program running within a QEMU instance, or in a GCE environment, can create TCP sockets, enable `TCP_ULP="tls"`, and configure keys via `setsockopt(..., SOL_TLS, TLS_TX, ...)` and `setsockopt(..., SOL_TLS, TLS_RX, ...)` calls. Syzkaller even has comprehensive descriptions for these operations (`socket$inet_tcp`, `setsockopt$inet_tcp_TCP_ULP`, `setsockopt$inet_tls_TLS_TX`, `setsockopt$inet_tls_TLS_RX`, etc.). Now, let's analyze what happens with a `setsockopt(..., SOL_TLS, TLS_TX, ...)` call. `do_tls_setsockopt` is called, and this acquires `mutex_lock_interruptible(&tls_get_ctx(sk)->tx_lock)`. This is a critical point! If this lock acquisition gets interrupted by a signal, and there's a concurrent `sendmsg` on that socket which *also* uses `tx_lock`, there's potential for interesting interactions. `do_tls_setsockopt_conf` is then called and `tls_set_device_offload` is called. Because we are not doing a device offload (probably a loopback/veth/dummy device), it returns `-EOPNOTSUPP`, and calls `tls_set_sw_offload`. That then leads to `tls_sw_ctx_init`. This allocates the crypto AEAD tfm, sets `authsize`, checks for `crypto setkey` success. The function `tls_sw_ctx_finalize` then sets up the IV, |
| 30/2 | 2026/09/20 01:03 | tool | read-file |
0mArgs:{
"File": "net/tls/tls_sw.c",
"FirstLine": 2560,
"LineCount": 90
}Results: {
"Contents": "2560:\t\n2561:\t\tWRITE_ONCE(ctx-\u003ekey_update_pending, false);\n2562:\t\t/* wake-up pre-existing poll() */\n2563:\t\tctx-\u003esaved_data_ready(sk);\n2564:\t}\n2565:\t\n2566:\tint tls_sw_ctx_init(struct sock *sk, int tx,\n2567:\t\t\t struct tls_crypto_info *new_crypto_info)\n2568:\t{\n2569:\t\tstruct tls_crypto_info *crypto_info, *src_crypto_info;\n2570:\t\tstruct tls_sw_context_tx *sw_ctx_tx = NULL;\n2571:\t\tstruct tls_sw_context_rx *sw_ctx_rx = NULL;\n2572:\t\tconst struct tls_cipher_desc *cipher_desc;\n2573:\t\tstruct tls_prot_info *prot;\n2574:\t\tstruct crypto_aead **aead;\n2575:\t\tstruct tls_context *ctx;\n2576:\t\tstruct crypto_tfm *tfm;\n2577:\t\tint rc = 0;\n2578:\t\tchar *key;\n2579:\t\n2580:\t\tctx = tls_get_ctx(sk);\n2581:\t\tprot = \u0026ctx-\u003eprot_info;\n2582:\t\n2583:\t\t/* new_crypto_info != NULL means rekey */\n2584:\t\tif (!new_crypto_info) {\n2585:\t\t\tif (tx) {\n2586:\t\t\t\tctx-\u003epriv_ctx_tx = init_ctx_tx(ctx, sk);\n2587:\t\t\t\tif (!ctx-\u003epriv_ctx_tx)\n2588:\t\t\t\t\treturn -ENOMEM;\n2589:\t\t\t} else {\n2590:\t\t\t\tctx-\u003epriv_ctx_rx = init_ctx_rx(ctx);\n2591:\t\t\t\tif (!ctx-\u003epriv_ctx_rx)\n2592:\t\t\t\t\treturn -ENOMEM;\n2593:\t\t\t}\n2594:\t\t}\n2595:\t\n2596:\t\tif (tx) {\n2597:\t\t\tsw_ctx_tx = ctx-\u003epriv_ctx_tx;\n2598:\t\t\tcrypto_info = \u0026ctx-\u003ecrypto_send.info;\n2599:\t\t\taead = \u0026sw_ctx_tx-\u003eaead_send;\n2600:\t\t} else {\n2601:\t\t\tsw_ctx_rx = ctx-\u003epriv_ctx_rx;\n2602:\t\t\tcrypto_info = \u0026ctx-\u003ecrypto_recv.info;\n2603:\t\t\taead = \u0026sw_ctx_rx-\u003eaead_recv;\n2604:\t\t}\n2605:\t\n2606:\t\tsrc_crypto_info = new_crypto_info ?: crypto_info;\n2607:\t\n2608:\t\tcipher_desc = get_cipher_desc(src_crypto_info-\u003ecipher_type);\n2609:\t\tif (!cipher_desc) {\n2610:\t\t\trc = -EINVAL;\n2611:\t\t\tgoto free_priv;\n2612:\t\t}\n2613:\t\n2614:\t\trc = init_prot_info(prot, src_crypto_info, cipher_desc);\n2615:\t\tif (rc)\n2616:\t\t\tgoto free_priv;\n2617:\t\n2618:\t\tkey = crypto_info_key(src_crypto_info, cipher_desc);\n2619:\t\n2620:\t\t/* A rekey normally reuses the existing tfm; the RX HW rekey hands over a\n2621:\t\t * NULL aead (the old one is retained for the drain), so allocate and\n2622:\t\t * configure authsize only when a fresh tfm is created here.\n2623:\t\t */\n2624:\t\tif (!*aead) {\n2625:\t\t\t*aead = crypto_alloc_aead(cipher_desc-\u003ecipher_name, 0, 0);\n2626:\t\t\tif (IS_ERR(*aead)) {\n2627:\t\t\t\trc = PTR_ERR(*aead);\n2628:\t\t\t\t*aead = NULL;\n2629:\t\t\t\tgoto free_priv;\n2630:\t\t\t}\n2631:\t\n2632:\t\t\trc = crypto_aead_setauthsize(*aead, prot-\u003etag_size);\n2633:\t\t\tif (rc)\n2634:\t\t\t\tgoto free_aead;\n2635:\t\t}\n2636:\t\n2637:\t\tif (tx)\n2638:\t\t\tctx-\u003epush_pending_record = tls_sw_push_pending_record;\n2639:\t\n2640:\t\t/* setkey is the last operation that could fail during a\n2641:\t\t * rekey. if it succeeds, we can start modifying the\n2642:\t\t * context.\n2643:\t\t */\n2644:\t\trc = crypto_aead_setkey(*aead, key, cipher_desc-\u003ekey);\n2645:\t\tif (rc) {\n2646:\t\t\tif (new_crypto_info)\n2647:\t\t\t\tgoto out;\n2648:\t\t\telse\n2649:\t\t\t\tgoto free_aead;\n"
} |
| 31/2 | 2026/09/20 01:03 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 124372 output: 38 thoughts: 334 Thoughts: **Analyzing a Potential TLS/SW Regression**
Okay, hold on a second! I need to retrace my steps on this patch. Lines 2637-2638 immediately caught my attention:
```c
if (tx)
ctx->push_pending_record = tls_sw_push_pending_record;
```
That's suspicious, I need to know what that *was* before the patch.
```c
- ctx->push_pending_record = tls_sw_push_pending_record;
```
Ah! Now, let's examine lines 2660-2670 and see what else changed around that part:
```c
if (!tx && !new_crypto_info) {
tfm = crypto_aead_tfm(sw_ctx_rx->aead_recv);
if (crypto_info->version == TLS_1_3_VERSION) {
crypto_info_rec_seq(crypto_info, cipher_desc)[7] = 1;
cctx->rec_seq[7] = 1;
}
rc = crypto_aead_setauthsize(sw_ctx_rx->aead_recv, prot->tag_size);
if (rc)
goto free_aead;
}
```
Okay, the `cctx->rec_seq[7] = 1;` line jumps out at me. I need to figure out where `cctx` is defined. Right, let me check the file `net/tls/tls_sw.c`. I need to track down the initialization. I'll need to go back and look at the `tls_sw_ctx_init` function to fully understand the context.
|
| 32/2 | 2026/09/20 01:03 | tool | read-file |
0mArgs:{
"File": "net/tls/tls_sw.c",
"FirstLine": 2645,
"LineCount": 45
}Results: {
"Contents": "2645:\t\tif (rc) {\n2646:\t\t\tif (new_crypto_info)\n2647:\t\t\t\tgoto out;\n2648:\t\t\telse\n2649:\t\t\t\tgoto free_aead;\n2650:\t\t}\n2651:\t\n2652:\t\tif (!tx \u0026\u0026 !new_crypto_info) {\n2653:\t\t\ttfm = crypto_aead_tfm(sw_ctx_rx-\u003eaead_recv);\n2654:\t\n2655:\t\t\ttls_update_rx_zc_capable(ctx);\n2656:\t\t\tsw_ctx_rx-\u003easync_capable =\n2657:\t\t\t\tsrc_crypto_info-\u003eversion != TLS_1_3_VERSION \u0026\u0026\n2658:\t\t\t\t!!(tfm-\u003e__crt_alg-\u003ecra_flags \u0026 CRYPTO_ALG_ASYNC);\n2659:\t\n2660:\t\t\trc = tls_strp_init(\u0026sw_ctx_rx-\u003estrp, sk);\n2661:\t\t\tif (rc)\n2662:\t\t\t\tgoto free_aead;\n2663:\t\t}\n2664:\t\n2665:\t\tgoto out;\n2666:\t\n2667:\tfree_aead:\n2668:\t\tcrypto_free_aead(*aead);\n2669:\t\t*aead = NULL;\n2670:\tfree_priv:\n2671:\t\tif (!new_crypto_info) {\n2672:\t\t\tif (tx) {\n2673:\t\t\t\tkfree(ctx-\u003epriv_ctx_tx);\n2674:\t\t\t\tctx-\u003epriv_ctx_tx = NULL;\n2675:\t\t\t} else {\n2676:\t\t\t\tkfree(ctx-\u003epriv_ctx_rx);\n2677:\t\t\t\tctx-\u003epriv_ctx_rx = NULL;\n2678:\t\t\t}\n2679:\t\t}\n2680:\tout:\n2681:\t\treturn rc;\n2682:\t}\n2683:\t\n2684:\tvoid tls_sw_ctx_finalize(struct sock *sk, int tx,\n2685:\t\t\t\t struct tls_crypto_info *new_crypto_info)\n2686:\t{\n2687:\t\tstruct tls_crypto_info *crypto_info, *src_crypto_info;\n2688:\t\tconst struct tls_cipher_desc *cipher_desc;\n2689:\t\tstruct tls_context *ctx = tls_get_ctx(sk);\n"
} |
| 33/2 | 2026/09/20 01:03 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 125796 output: 38 thoughts: 28 Thoughts: Okay, let's break this down. In the `tls_sw_ctx_finalize` function... specifically lines 2684 through 2735... I need to understand what's happening within that block. It's likely some crucial clean-up or teardown procedure related to a TLS software context. Given the context, I'd expect it to involve releasing allocated memory, closing resources like file descriptors or sockets, potentially invalidating or zeroing out sensitive data, and maybe even handling error conditions and logging appropriately. I'll need to look closely at memory management calls (malloc/free or their equivalents within the code base), any resource closing functions (close, shutdown), and any data-wiping operations. I will pay attention to the error handling within this block; it may be the reason for an issue, and is crucial for maintaining the integrity and security of the system. I also need to consider any interaction with other parts of the TLS stack during the finalization process, potential race conditions or other concurrency issues that might arise. This is where the core security aspects lie. This part could be really sensitive, which means I should look for the potential for vulnerabilities, such as use-after-free, double-free, memory leaks, or improper cleanup. I'll approach this with the assumption of potential vulnerabilities. |
| 34/2 | 2026/09/20 01:04 | tool | read-file |
0mArgs:{
"File": "net/tls/tls_sw.c",
"FirstLine": 2684,
"LineCount": 55
}Results: {
"Contents": "2684:\tvoid tls_sw_ctx_finalize(struct sock *sk, int tx,\n2685:\t\t\t\t struct tls_crypto_info *new_crypto_info)\n2686:\t{\n2687:\t\tstruct tls_crypto_info *crypto_info, *src_crypto_info;\n2688:\t\tconst struct tls_cipher_desc *cipher_desc;\n2689:\t\tstruct tls_context *ctx = tls_get_ctx(sk);\n2690:\t\tstruct cipher_context *cctx;\n2691:\t\tchar *iv, *salt, *rec_seq;\n2692:\t\n2693:\t\tif (tx) {\n2694:\t\t\tcrypto_info = \u0026ctx-\u003ecrypto_send.info;\n2695:\t\t\tcctx = \u0026ctx-\u003etx;\n2696:\t\t} else {\n2697:\t\t\tcrypto_info = \u0026ctx-\u003ecrypto_recv.info;\n2698:\t\t\tcctx = \u0026ctx-\u003erx;\n2699:\t\t}\n2700:\t\n2701:\t\tsrc_crypto_info = new_crypto_info ?: crypto_info;\n2702:\t\n2703:\t\t/* Infallible: tls_sw_ctx_init() already validated cipher_type. */\n2704:\t\tcipher_desc = get_cipher_desc(src_crypto_info-\u003ecipher_type);\n2705:\t\n2706:\t\tiv = crypto_info_iv(src_crypto_info, cipher_desc);\n2707:\t\tsalt = crypto_info_salt(src_crypto_info, cipher_desc);\n2708:\t\trec_seq = crypto_info_rec_seq(src_crypto_info, cipher_desc);\n2709:\t\n2710:\t\tmemcpy(cctx-\u003eiv, salt, cipher_desc-\u003esalt);\n2711:\t\tmemcpy(cctx-\u003eiv + cipher_desc-\u003esalt, iv, cipher_desc-\u003eiv);\n2712:\t\tmemcpy(cctx-\u003erec_seq, rec_seq, cipher_desc-\u003erec_seq);\n2713:\t\n2714:\t\tif (new_crypto_info) {\n2715:\t\t\tunsafe_memcpy(crypto_info, new_crypto_info,\n2716:\t\t\t\t cipher_desc-\u003ecrypto_info,\n2717:\t\t\t\t /* size was checked in do_tls_setsockopt_conf */);\n2718:\t\t\tmemzero_explicit(new_crypto_info, cipher_desc-\u003ecrypto_info);\n2719:\t\n2720:\t\t\tif (!tx)\n2721:\t\t\t\ttls_finish_key_update(sk, ctx);\n2722:\t\t}\n2723:\t}\n2724:\t\n2725:\tint tls_set_sw_offload(struct sock *sk, int tx,\n2726:\t\t\t struct tls_crypto_info *new_crypto_info)\n2727:\t{\n2728:\t\tint rc;\n2729:\t\n2730:\t\trc = tls_sw_ctx_init(sk, tx, new_crypto_info);\n2731:\t\tif (rc)\n2732:\t\t\treturn rc;\n2733:\t\n2734:\t\ttls_sw_ctx_finalize(sk, tx, new_crypto_info);\n2735:\t\treturn 0;\n2736:\t}\n"
} |
| 35/2 | 2026/09/20 01:04 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 127011 output: 33 thoughts: 1365 Thoughts: **Analysis of Potential Fuzzing Targets** Okay, let's break this down. First, the question: is the code I'm looking at reachable without special hardware in QEMU or GCE? The answer is YES! This centers around core networking and kTLS, which are readily available. Now, I'm going to run through the criteria to determine if these changes warrant fuzzing. The guidelines are pretty clear: trigger `WorthFuzzing=true` if changes touch reachable executable code. This includes modifications in core subsystems (networking, specifically in this case) or refactorings of reachable data structures, and also any new assertions in reachable paths. On the other hand, the code is NOT worth fuzzing if the code falls strictly into one or more of these categories: Non-kernel and non-executable changes, Structurally unreachable hardware and Unreachable execution paths. Let's look at the specific code changes. 1. `net/tls/tls_main.c` has been altered: `do_tls_setsockopt` had its locking modified to acquire `tx_lock` before `lock_sock`. It's a synchronization change for `setsockopt(TLS_TX)` on all TLS sockets, including software kTLS. This clearly falls under the "modifications in core networking" criteria. 2. `net/tls/tls_sw.c`: There's been a refactoring here. Software offload context initialization and finalization have been restructured (`tls_sw_ctx_init`, `tls_sw_ctx_finalize`, `tls_set_sw_offload`) and IV/seq/cipher accessors (`tls_tx_cipher_ctx`) have been updated. That's a refactoring of reachable data structures. 3. `net/sched/sch_generic.c`: The packet scheduler qdisc dequeue logic in `dequeue_skb` was modified. There's a check for `skb_is_decrypted(skb)`. This touches core networking, and it is a modification in the core networking subsystem. 4. `include/net/tcp.h`: `tcp_write_collapse_fence` has been modified. This is another modification in the core networking subsystem. All these paths involve `net/tls/`, `net/sched/` and `net/ipv4/tcp`, and are part of core networking. They're accessible from userspace via standard TCP/kTLS sockets. Given all of this, I'm confident that `WorthFuzzing` should be TRUE! Now, the next step is identifying *FocusSymbols*. The goal is to focus the fuzzer on the relevant functions, and not get diluted by calling ubiquitous lifecycle hot-paths or generic memory management. So, I need to extract kernel functions into FocusSymbols. I must avoid generic functions like memory allocators, deallocators, or synchronization primitives. Instead, I need to target specific feature logic and entrypoints. If changes are static inline, I need to identify non-static callers in `.c` files that exercise them. Let's list the functions added or modified: In `net/tls/tls_main.c`: * `do_tls_setsockopt` (or `tls_setsockopt`) * `do_tls_setsockopt_conf` * `do_tls_getsockopt_conf` * `tls_sk_proto_cleanup` * `tls_sk_proto_close` In `net/tls/tls_sw.c`: * `tls_sw_ctx_init` * `tls_sw_ctx_finalize` * `tls_set_sw_offload` * `tls_do_encryption` * `tls_push_record` * `tls_sw_drain_tx` * `tls_sw_sendmsg_locked` * `tls_sw_splice_eof_locked` * `tls_sw_ctx_tx_init` * `tls_check_pending_rekey` In `net/tls/tls_device.c` (even if device offload requires hardware, let's see what is called): * `tls_set_device_offload` (called by `do_tls_setsockopt_conf`) * `tls_set_device_offload_rx` (called by `do_tls_setsockopt_conf`) * `tls_device_rx_del_key` (called by `tls_check_pending_rekey`) In `net/sched/sch_generic.c`: Here's the problem. `dequeue |
| 36/2 | 2026/09/20 01:04 | tool | grepper |
0mArgs:{
"Expression": "^(int|void|static).*tls_",
"PathPrefix": "net/tls/"
}Results: {
"Output": "\nFull output is too long, showing 500 out of 971 lines.\nUse more precise expression if possible.\n\nnet/tls/tls.h=72=extern const struct tls_cipher_desc tls_cipher_desc[TLS_CIPHER_MAX + 1 - TLS_CIPHER_MIN];\nnet/tls/tls.h-73-\nnet/tls/tls.h:74:static inline const struct tls_cipher_desc *get_cipher_desc(u16 cipher_type)\nnet/tls/tls.h-75-{\n--\nnet/tls/tls.h-81-\nnet/tls/tls.h:82:static inline char *crypto_info_iv(struct tls_crypto_info *crypto_info,\nnet/tls/tls.h-83-\t\t\t\t const struct tls_cipher_desc *cipher_desc)\n--\nnet/tls/tls.h-87-\nnet/tls/tls.h:88:static inline char *crypto_info_key(struct tls_crypto_info *crypto_info,\nnet/tls/tls.h-89-\t\t\t\t const struct tls_cipher_desc *cipher_desc)\n--\nnet/tls/tls.h-93-\nnet/tls/tls.h:94:static inline char *crypto_info_salt(struct tls_crypto_info *crypto_info,\nnet/tls/tls.h-95-\t\t\t\t const struct tls_cipher_desc *cipher_desc)\n--\nnet/tls/tls.h-99-\nnet/tls/tls.h:100:static inline char *crypto_info_rec_seq(struct tls_crypto_info *crypto_info,\nnet/tls/tls.h-101-\t\t\t\t\tconst struct tls_cipher_desc *cipher_desc)\n--\nnet/tls/tls.h=111=struct tls_rec {\n--\nnet/tls/tls.h-135-\nnet/tls/tls.h:136:int __net_init tls_proc_init(struct net *net);\nnet/tls/tls.h:137:void __net_exit tls_proc_fini(struct net *net);\nnet/tls/tls.h-138-\nnet/tls/tls.h=139=struct tls_context *tls_ctx_create(struct sock *sk);\nnet/tls/tls.h:140:void tls_ctx_free(struct sock *sk, struct tls_context *ctx);\nnet/tls/tls.h:141:void update_sk_prot(struct sock *sk, struct tls_context *ctx);\nnet/tls/tls.h-142-\nnet/tls/tls.h=143=int wait_on_pending_writer(struct sock *sk, long *timeo);\nnet/tls/tls.h:144:void tls_err_abort(struct sock *sk, int err);\nnet/tls/tls.h:145:void tls_strp_abort_strp(struct tls_strparser *strp, int err);\nnet/tls/tls.h-146-\nnet/tls/tls.h:147:int init_prot_info(struct tls_prot_info *prot,\nnet/tls/tls.h-148-\t\t const struct tls_crypto_info *crypto_info,\n--\nnet/tls/tls.h-157- */\nnet/tls/tls.h:158:int tls_sw_ctx_init(struct sock *sk, int tx,\nnet/tls/tls.h-159-\t\t struct tls_crypto_info *new_crypto_info);\nnet/tls/tls.h:160:void tls_sw_ctx_finalize(struct sock *sk, int tx,\nnet/tls/tls.h-161-\t\t\t struct tls_crypto_info *new_crypto_info);\nnet/tls/tls.h:162:int tls_set_sw_offload(struct sock *sk, int tx,\nnet/tls/tls.h-163-\t\t struct tls_crypto_info *new_crypto_info);\nnet/tls/tls.h:164:void tls_update_rx_zc_capable(struct tls_context *tls_ctx);\nnet/tls/tls.h:165:void tls_sw_strparser_arm(struct sock *sk, struct tls_context *ctx);\nnet/tls/tls.h:166:void tls_sw_strparser_done(struct tls_context *tls_ctx);\nnet/tls/tls.h:167:int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size);\nnet/tls/tls.h:168:int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size);\nnet/tls/tls.h:169:void tls_sw_ctx_tx_init(struct sock *sk, struct tls_sw_context_tx *sw_ctx);\nnet/tls/tls.h:170:int tls_sw_drain_tx(struct sock *sk, struct tls_context *ctx, int flags);\nnet/tls/tls.h:171:int tls_encrypt_async_wait(struct tls_sw_context_tx *ctx);\nnet/tls/tls.h:172:int tls_sw_push_pending_record(struct sock *sk, int flags);\nnet/tls/tls.h:173:void tls_sw_splice_eof(struct socket *sock);\nnet/tls/tls.h:174:void tls_sw_splice_eof_locked(struct socket *sock);\nnet/tls/tls.h:175:void tls_sw_cancel_work_tx(struct tls_context *tls_ctx);\nnet/tls/tls.h:176:void tls_sw_release_resources_tx(struct sock *sk);\nnet/tls/tls.h:177:void tls_sw_free_ctx_tx(struct tls_context *tls_ctx);\nnet/tls/tls.h:178:void tls_sw_free_resources_rx(struct sock *sk);\nnet/tls/tls.h:179:void tls_sw_release_resources_rx(struct sock *sk);\nnet/tls/tls.h:180:void tls_sw_free_ctx_rx(struct tls_context *tls_ctx);\nnet/tls/tls.h:181:int tls_sw_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,\nnet/tls/tls.h-182-\t\t int flags);\n--\nnet/tls/tls.h=184=ssize_t tls_sw_splice_read(struct socket *sock, loff_t *ppos,\n--\nnet/tls/tls.h-186-\t\t\t size_t len, unsigned int flags);\nnet/tls/tls.h:187:int tls_sw_read_sock(struct sock *sk, read_descriptor_t *desc,\nnet/tls/tls.h-188-\t\t sk_read_actor_t read_actor);\nnet/tls/tls.h-189-\nnet/tls/tls.h:190:int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size);\nnet/tls/tls.h:191:void tls_device_splice_eof(struct socket *sock);\nnet/tls/tls.h:192:int tls_tx_records(struct sock *sk, int flags);\nnet/tls/tls.h-193-\nnet/tls/tls.h:194:void tls_sw_write_space(struct sock *sk, struct tls_context *ctx);\nnet/tls/tls.h:195:void tls_device_write_space(struct sock *sk, struct tls_context *ctx);\nnet/tls/tls.h-196-\nnet/tls/tls.h:197:int tls_process_cmsg(struct sock *sk, struct msghdr *msg,\nnet/tls/tls.h-198-\t\t unsigned char *record_type);\nnet/tls/tls.h=199=int decrypt_skb(struct sock *sk, struct scatterlist *sgout);\nnet/tls/tls.h-200-\nnet/tls/tls.h:201:int tls_sw_fallback_init(struct sock *sk,\nnet/tls/tls.h-202-\t\t\t struct tls_offload_context_tx *offload_ctx,\n--\nnet/tls/tls.h-204-\nnet/tls/tls.h:205:int tls_strp_dev_init(void);\nnet/tls/tls.h:206:void tls_strp_dev_exit(void);\nnet/tls/tls.h-207-\nnet/tls/tls.h:208:void tls_strp_done(struct tls_strparser *strp);\nnet/tls/tls.h:209:void __tls_strp_done(struct tls_strparser *strp);\nnet/tls/tls.h:210:void tls_strp_stop(struct tls_strparser *strp);\nnet/tls/tls.h:211:int tls_strp_init(struct tls_strparser *strp, struct sock *sk);\nnet/tls/tls.h:212:void tls_strp_data_ready(struct tls_strparser *strp);\nnet/tls/tls.h-213-\nnet/tls/tls.h:214:void tls_strp_check_rcv(struct tls_strparser *strp, bool announce);\nnet/tls/tls.h:215:void tls_strp_msg_consume(struct tls_strparser *strp);\nnet/tls/tls.h-216-\nnet/tls/tls.h:217:int tls_rx_msg_size(struct tls_strparser *strp, struct sk_buff *skb);\nnet/tls/tls.h:218:void tls_rx_msg_maybe_announce(struct tls_strparser *strp);\nnet/tls/tls.h-219-\nnet/tls/tls.h=220=bool tls_strp_msg_load(struct tls_strparser *strp, bool force_refresh);\nnet/tls/tls.h:221:int tls_strp_msg_cow(struct tls_sw_context_rx *ctx);\nnet/tls/tls.h-222-struct sk_buff *tls_strp_msg_detach(struct tls_sw_context_rx *ctx);\nnet/tls/tls.h:223:int tls_strp_msg_hold(struct tls_strparser *strp, struct sk_buff_head *dst);\nnet/tls/tls.h-224-\nnet/tls/tls.h:225:static inline struct tls_msg *tls_msg(struct sk_buff *skb)\nnet/tls/tls.h-226-{\n--\nnet/tls/tls.h-231-\nnet/tls/tls.h:232:static inline struct sk_buff *tls_strp_msg(struct tls_sw_context_rx *ctx)\nnet/tls/tls.h-233-{\n--\nnet/tls/tls.h-237-\nnet/tls/tls.h:238:static inline bool tls_strp_msg_ready(struct tls_sw_context_rx *ctx)\nnet/tls/tls.h-239-{\n--\nnet/tls/tls.h-242-\nnet/tls/tls.h:243:static inline bool tls_strp_msg_mixed_decrypted(struct tls_sw_context_rx *ctx)\nnet/tls/tls.h-244-{\n--\nnet/tls/tls.h-248-#ifdef CONFIG_TLS_DEVICE\nnet/tls/tls.h:249:int tls_device_init(void);\nnet/tls/tls.h:250:void tls_device_cleanup(void);\nnet/tls/tls.h:251:int tls_set_device_offload(struct sock *sk,\nnet/tls/tls.h-252-\t\t\t struct tls_crypto_info *crypto_info);\nnet/tls/tls.h:253:void tls_device_free_resources_tx(struct sock *sk);\nnet/tls/tls.h:254:int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,\nnet/tls/tls.h-255-\t\t\t struct tls_crypto_info *crypto_info);\nnet/tls/tls.h:256:void tls_device_offload_cleanup_rx(struct sock *sk);\nnet/tls/tls.h:257:void tls_device_rx_del_key(struct sock *sk, struct tls_context *ctx);\nnet/tls/tls.h:258:void tls_device_rx_resync_new_rec(struct sock *sk, u32 rcd_len, u32 seq);\nnet/tls/tls.h:259:int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx);\nnet/tls/tls.h-260-#else\nnet/tls/tls.h:261:static inline int tls_device_init(void) { return 0; }\nnet/tls/tls.h:262:static inline void tls_device_cleanup(void) {}\nnet/tls/tls.h-263-\n--\nnet/tls/tls.h=265=tls_set_device_offload(struct sock *sk, struct tls_crypto_info *crypto_info)\n--\nnet/tls/tls.h-269-\nnet/tls/tls.h:270:static inline void tls_device_free_resources_tx(struct sock *sk) {}\nnet/tls/tls.h-271-\n--\nnet/tls/tls.h=273=tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,\n--\nnet/tls/tls.h-278-\nnet/tls/tls.h:279:static inline void tls_device_offload_cleanup_rx(struct sock *sk) {}\nnet/tls/tls.h-280-static inline void\n--\nnet/tls/tls.h=286=tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)\n--\nnet/tls/tls.h-291-\nnet/tls/tls.h:292:int tls_push_sg(struct sock *sk, struct tls_context *ctx,\nnet/tls/tls.h-293-\t\tstruct scatterlist *sg, u16 first_offset,\nnet/tls/tls.h-294-\t\tint flags);\nnet/tls/tls.h:295:int tls_push_partial_record(struct sock *sk, struct tls_context *ctx,\nnet/tls/tls.h-296-\t\t\t int flags);\nnet/tls/tls.h:297:void tls_free_partial_record(struct sock *sk, struct tls_context *ctx);\nnet/tls/tls.h-298-\nnet/tls/tls.h:299:static inline bool tls_is_partially_sent_record(struct tls_context *ctx)\nnet/tls/tls.h-300-{\n--\nnet/tls/tls.h-303-\nnet/tls/tls.h:304:static inline bool tls_is_pending_open_record(struct tls_context *tls_ctx)\nnet/tls/tls.h-305-{\n--\nnet/tls/tls.h-308-\nnet/tls/tls.h:309:static inline bool tls_bigint_increment(unsigned char *seq, int len)\nnet/tls/tls.h-310-{\n--\nnet/tls/tls.h-321-\nnet/tls/tls.h:322:static inline void tls_bigint_subtract(unsigned char *seq, int n)\nnet/tls/tls.h-323-{\n--\nnet/tls/tls.h=388=static inline\nnet/tls/tls.h:389:void tls_make_aad(char *buf, size_t size, char *record_sequence,\nnet/tls/tls.h-390-\t\t unsigned char record_type, struct tls_prot_info *prot)\n--\nnet/tls/tls_device.c=50=static struct workqueue_struct *destruct_wq __read_mostly;\nnet/tls/tls_device.c-51-\nnet/tls/tls_device.c:52:static LIST_HEAD(tls_device_list);\nnet/tls/tls_device.c:53:static LIST_HEAD(tls_device_down_list);\nnet/tls/tls_device.c:54:static DEFINE_SPINLOCK(tls_device_lock);\nnet/tls/tls_device.c-55-\nnet/tls/tls_device.c=56=static struct page *dummy_page;\nnet/tls/tls_device.c-57-\nnet/tls/tls_device.c:58:static void tls_device_free_ctx(struct tls_context *ctx)\nnet/tls/tls_device.c-59-{\n--\nnet/tls/tls_device.c-85-\nnet/tls/tls_device.c:86:static void tls_device_tx_del_task(struct work_struct *work)\nnet/tls/tls_device.c-87-{\n--\nnet/tls/tls_device.c-106-\nnet/tls/tls_device.c:107:static void tls_device_queue_ctx_destruction(struct tls_context *ctx)\nnet/tls/tls_device.c-108-{\n--\nnet/tls/tls_device.c=143=static struct net_device *get_netdev_for_sock(struct sock *sk)\n--\nnet/tls/tls_device.c-159-\nnet/tls/tls_device.c:160:static int tls_device_dev_add_tx(struct sock *sk, struct net_device *netdev,\nnet/tls/tls_device.c-161-\t\t\t\t struct tls_crypto_info *crypto_info,\n--\nnet/tls/tls_device.c-181- */\nnet/tls/tls_device.c:182:static void tls_device_add_start_marker(struct sock *sk,\nnet/tls/tls_device.c-183-\t\t\t\t\tstruct tls_offload_context_tx *offload_ctx,\n--\nnet/tls/tls_device.c-191-\nnet/tls/tls_device.c:192:static void tls_device_commit_start_marker(struct sock *sk,\nnet/tls/tls_device.c-193-\t\t\t\t\tstruct tls_offload_context_tx *offload_ctx,\n--\nnet/tls/tls_device.c-210- */\nnet/tls/tls_device.c:211:static void tls_device_rx_rekey_fallback(struct sock *sk,\nnet/tls/tls_device.c-212-\t\t\t\t\t struct tls_context *tls_ctx)\n--\nnet/tls/tls_device.c-220-\nnet/tls/tls_device.c:221:static int tls_device_dev_add_rx(struct sock *sk, struct tls_context *tls_ctx,\nnet/tls/tls_device.c-222-\t\t\t\t struct net_device *netdev,\n--\nnet/tls/tls_device.c-256-\nnet/tls/tls_device.c:257:static void tls_device_deferred_dev_add_rx(struct sock *sk,\nnet/tls/tls_device.c-258-\t\t\t\t\t struct tls_context *tls_ctx,\n--\nnet/tls/tls_device.c-303- */\nnet/tls/tls_device.c:304:void tls_device_rx_del_key(struct sock *sk, struct tls_context *ctx)\nnet/tls/tls_device.c-305-{\n--\nnet/tls/tls_device.c-327-\nnet/tls/tls_device.c:328:static void destroy_record(struct tls_record_info *record)\nnet/tls/tls_device.c-329-{\n--\nnet/tls/tls_device.c-336-\nnet/tls/tls_device.c:337:static void delete_all_records(struct tls_offload_context_tx *offload_ctx)\nnet/tls/tls_device.c-338-{\n--\nnet/tls/tls_device.c-348-\nnet/tls/tls_device.c:349:static void tls_device_commit_rekey_marker(struct sock *sk,\nnet/tls/tls_device.c-350-\t\t\t\t\t struct tls_offload_context_tx *offload_ctx,\n--\nnet/tls/tls_device.c-381-\nnet/tls/tls_device.c:382:static bool tls_has_unacked_records(struct tls_offload_context_tx *offload_ctx)\nnet/tls/tls_device.c-383-{\n--\nnet/tls/tls_device.c-399-\nnet/tls/tls_device.c:400:static void tls_tcp_clean_acked(struct sock *sk, u32 acked_seq)\nnet/tls/tls_device.c-401-{\n--\nnet/tls/tls_device.c-447- */\nnet/tls/tls_device.c:448:void tls_device_sk_destruct(struct sock *sk)\nnet/tls/tls_device.c-449-{\n--\nnet/tls/tls_device.c=465=EXPORT_SYMBOL_GPL(tls_device_sk_destruct);\nnet/tls/tls_device.c-466-\nnet/tls/tls_device.c:467:void tls_device_free_resources_tx(struct sock *sk)\nnet/tls/tls_device.c-468-{\n--\nnet/tls/tls_device.c-481-\nnet/tls/tls_device.c:482:void tls_offload_tx_resync_request(struct sock *sk, u32 got_seq, u32 exp_seq)\nnet/tls/tls_device.c-483-{\n--\nnet/tls/tls_device.c=489=EXPORT_SYMBOL_GPL(tls_offload_tx_resync_request);\nnet/tls/tls_device.c-490-\nnet/tls/tls_device.c:491:static void tls_device_resync_tx(struct sock *sk, struct tls_context *tls_ctx,\nnet/tls/tls_device.c-492-\t\t\t\t u32 seq)\n--\nnet/tls/tls_device.c-515-\nnet/tls/tls_device.c:516:static void tls_append_frag(struct tls_record_info *record,\nnet/tls/tls_device.c-517-\t\t\t struct page_frag *pfrag,\n--\nnet/tls/tls_device.c-537-\nnet/tls/tls_device.c:538:static int tls_push_record(struct sock *sk,\nnet/tls/tls_device.c-539-\t\t\t struct tls_context *ctx,\n--\nnet/tls/tls_device.c-571-\nnet/tls/tls_device.c:572:static void tls_device_record_close(struct sock *sk,\nnet/tls/tls_device.c-573-\t\t\t\t struct tls_context *ctx,\n--\nnet/tls/tls_device.c-609-\nnet/tls/tls_device.c:610:static int tls_create_new_record(struct tls_offload_context_tx *offload_ctx,\nnet/tls/tls_device.c-611-\t\t\t\t struct page_frag *pfrag,\n--\nnet/tls/tls_device.c-633-\nnet/tls/tls_device.c:634:static int tls_do_allocation(struct sock *sk,\nnet/tls/tls_device.c-635-\t\t\t struct tls_offload_context_tx *offload_ctx,\n--\nnet/tls/tls_device.c-663-\nnet/tls/tls_device.c:664:static int tls_device_copy_data(void *addr, size_t bytes, struct iov_iter *i)\nnet/tls/tls_device.c-665-{\n--\nnet/tls/tls_device.c-688-\nnet/tls/tls_device.c:689:static int tls_device_complete_rekey(struct sock *sk, struct tls_context *ctx,\nnet/tls/tls_device.c-690-\t\t\t\t bool deferred, int push_flags);\nnet/tls/tls_device.c-691-\nnet/tls/tls_device.c:692:static int tls_push_data(struct sock *sk,\nnet/tls/tls_device.c-693-\t\t\t struct iov_iter *iter,\n--\nnet/tls/tls_device.c-839- */\nnet/tls/tls_device.c:840:static bool tls_device_tx_uses_sw(const struct tls_context *ctx)\nnet/tls/tls_device.c-841-{\n--\nnet/tls/tls_device.c-845-\nnet/tls/tls_device.c:846:int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)\nnet/tls/tls_device.c-847-{\n--\nnet/tls/tls_device.c-899-\nnet/tls/tls_device.c:900:void tls_device_splice_eof(struct socket *sock)\nnet/tls/tls_device.c-901-{\n--\nnet/tls/tls_device.c=983=EXPORT_SYMBOL(tls_get_record);\nnet/tls/tls_device.c-984-\nnet/tls/tls_device.c:985:static int tls_device_push_pending_record(struct sock *sk, int flags)\nnet/tls/tls_device.c-986-{\n--\nnet/tls/tls_device.c-996-\nnet/tls/tls_device.c:997:void tls_device_write_space(struct sock *sk, struct tls_context *ctx)\nnet/tls/tls_device.c-998-{\n--\nnet/tls/tls_device.c-1023-\nnet/tls/tls_device.c:1024:static void tls_device_resync_rx(struct tls_context *tls_ctx,\nnet/tls/tls_device.c-1025-\t\t\t\t struct sock *sk, u32 seq, u8 *rcd_sn)\n--\nnet/tls/tls_device.c=1041=tls_device_rx_resync_async(struct tls_offload_resync_async *resync_async,\n--\nnet/tls/tls_device.c-1097-\nnet/tls/tls_device.c:1098:void tls_device_rx_resync_new_rec(struct sock *sk, u32 rcd_len, u32 seq)\nnet/tls/tls_device.c-1099-{\n--\nnet/tls/tls_device.c-1164-\nnet/tls/tls_device.c:1165:static void tls_device_core_ctrl_rx_resync(struct tls_context *tls_ctx,\nnet/tls/tls_device.c-1166-\t\t\t\t\t struct tls_offload_context_rx *ctx,\n--\nnet/tls/tls_device.c=1213=tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)\n--\nnet/tls/tls_device.c-1315- */\nnet/tls/tls_device.c:1316:static int tls_device_reencrypt_old_key(struct sock *sk,\nnet/tls/tls_device.c-1317-\t\t\t\t\tstruct tls_offload_context_rx *ctx,\n--\nnet/tls/tls_device.c-1363- */\nnet/tls/tls_device.c:1364:static u32 tls_device_rx_rec_start(struct sock *sk,\nnet/tls/tls_device.c-1365-\t\t\t\t struct tls_sw_context_rx *sw_ctx)\n--\nnet/tls/tls_device.c-1374-\nnet/tls/tls_device.c:1375:int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)\nnet/tls/tls_device.c-1376-{\n--\nnet/tls/tls_device.c-1490-\nnet/tls/tls_device.c:1491:static void tls_device_attach(struct tls_context *ctx, struct sock *sk,\nnet/tls/tls_device.c-1492-\t\t\t struct net_device *netdev)\n--\nnet/tls/tls_device.c-1506-\nnet/tls/tls_device.c:1507:static struct tls_offload_context_tx *alloc_offload_ctx_tx(struct tls_context *ctx)\nnet/tls/tls_device.c-1508-{\n--\nnet/tls/tls_device.c-1535- */\nnet/tls/tls_device.c:1536:static struct crypto_aead *tls_device_build_rekey_aead(\nnet/tls/tls_device.c-1537-\t\t\t\tconst struct tls_cipher_desc *cipher_desc,\n--\nnet/tls/tls_device.c-1557-\nnet/tls/tls_device.c:1558:static void tls_device_copy_rekey_iv_seq(\nnet/tls/tls_device.c-1559-\t\t\t\tstruct tls_offload_context_tx *offload_ctx,\n--\nnet/tls/tls_device.c-1568-\nnet/tls/tls_device.c:1569:static int tls_device_init_rekey_sw(struct sock *sk,\nnet/tls/tls_device.c-1570-\t\t\t\t struct tls_context *ctx,\n--\nnet/tls/tls_device.c-1595-\nnet/tls/tls_device.c:1596:static int tls_device_start_rekey(struct sock *sk,\nnet/tls/tls_device.c-1597-\t\t\t\t struct tls_context *ctx,\n--\nnet/tls/tls_device.c-1749-\nnet/tls/tls_device.c:1750:static int tls_device_complete_rekey(struct sock *sk, struct tls_context *ctx,\nnet/tls/tls_device.c-1751-\t\t\t\t bool deferred, int push_flags)\n--\nnet/tls/tls_device.c-1915-\nnet/tls/tls_device.c:1916:static int tls_set_device_offload_rekey(struct sock *sk,\nnet/tls/tls_device.c-1917-\t\t\t\t\tstruct tls_context *ctx,\n--\nnet/tls/tls_device.c-1956-\nnet/tls/tls_device.c:1957:static int tls_set_device_offload_initial(struct sock *sk,\nnet/tls/tls_device.c-1958-\t\t\t\t\t struct tls_context *ctx,\n--\nnet/tls/tls_device.c-2040-\nnet/tls/tls_device.c:2041:int tls_set_device_offload(struct sock *sk,\nnet/tls/tls_device.c-2042-\t\t\t struct tls_crypto_info *new_crypto_info)\n--\nnet/tls/tls_device.c-2095-\nnet/tls/tls_device.c:2096:int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,\nnet/tls/tls_device.c-2097-\t\t\t struct tls_crypto_info *new_crypto_info)\n--\nnet/tls/tls_device.c-2362-\nnet/tls/tls_device.c:2363:void tls_device_offload_cleanup_rx(struct sock *sk)\nnet/tls/tls_device.c-2364-{\n--\nnet/tls/tls_device.c-2402-\nnet/tls/tls_device.c:2403:static int tls_device_down(struct net_device *netdev)\nnet/tls/tls_device.c-2404-{\n--\nnet/tls/tls_device.c-2496-\nnet/tls/tls_device.c:2497:static int tls_dev_event(struct notifier_block *this, unsigned long event,\nnet/tls/tls_device.c-2498-\t\t\t void *ptr)\n--\nnet/tls/tls_device.c-2525-\nnet/tls/tls_device.c:2526:static struct notifier_block tls_dev_notifier = {\nnet/tls/tls_device.c-2527-\t.notifier_call\t= tls_dev_event,\n--\nnet/tls/tls_device.c-2529-\nnet/tls/tls_device.c:2530:int __init tls_device_init(void)\nnet/tls/tls_device.c-2531-{\n--\nnet/tls/tls_device.c-2571-\nnet/tls/tls_device.c:2572:void __exit tls_device_cleanup(void)\nnet/tls/tls_device.c-2573-{\n--\nnet/tls/tls_device_fallback.c-39-\nnet/tls/tls_device_fallback.c:40:static int tls_enc_record(struct tls_context *tls_ctx,\nnet/tls/tls_device_fallback.c-41-\t\t\t struct aead_request *aead_req,\n--\nnet/tls/tls_device_fallback.c-122-\nnet/tls/tls_device_fallback.c:123:static struct aead_request *tls_alloc_aead_request(struct crypto_aead *aead,\nnet/tls/tls_device_fallback.c-124-\t\t\t\t\t\t gfp_t flags)\n--\nnet/tls/tls_device_fallback.c-135-\nnet/tls/tls_device_fallback.c:136:static int tls_enc_records(struct tls_context *tls_ctx,\nnet/tls/tls_device_fallback.c-137-\t\t\t struct aead_request *aead_req,\n--\nnet/tls/tls_device_fallback.c=291=static void fill_sg_out(struct scatterlist sg_out[3], void *buf,\n--\nnet/tls/tls_device_fallback.c-308-\nnet/tls/tls_device_fallback.c:309:static struct sk_buff *tls_enc_skb(struct tls_context *tls_ctx,\nnet/tls/tls_device_fallback.c-310-\t\t\t\t struct scatterlist sg_out[3],\n--\nnet/tls/tls_device_fallback.c-376-\nnet/tls/tls_device_fallback.c:377:static struct sk_buff *tls_sw_fallback(struct sock *sk, struct sk_buff *skb)\nnet/tls/tls_device_fallback.c-378-{\n--\nnet/tls/tls_device_fallback.c-452- */\nnet/tls/tls_device_fallback.c:453:static bool tls_tx_drop_acked_clone(struct sock *sk, struct sk_buff *skb)\nnet/tls/tls_device_fallback.c-454-{\n--\nnet/tls/tls_device_fallback.c=564=EXPORT_SYMBOL_GPL(tls_encrypt_skb);\nnet/tls/tls_device_fallback.c-565-\nnet/tls/tls_device_fallback.c:566:int tls_sw_fallback_init(struct sock *sk,\nnet/tls/tls_device_fallback.c-567-\t\t\t struct tls_offload_context_tx *offload_ctx,\n--\nnet/tls/tls_main.c=123=static DEFINE_MUTEX(tcpv4_prot_mutex);\nnet/tls/tls_main.c:124:static struct proto tls_prots[TLS_NUM_PROTS][TLS_NUM_CONFIG][TLS_NUM_CONFIG];\nnet/tls/tls_main.c:125:static struct proto_ops tls_proto_ops[TLS_NUM_PROTS][TLS_NUM_CONFIG][TLS_NUM_CONFIG];\nnet/tls/tls_main.c-126-static void build_protos(struct proto prot[TLS_NUM_CONFIG][TLS_NUM_CONFIG],\n--\nnet/tls/tls_main.c-128-\nnet/tls/tls_main.c:129:void update_sk_prot(struct sock *sk, struct tls_context *ctx)\nnet/tls/tls_main.c-130-{\n--\nnet/tls/tls_main.c=139=int wait_on_pending_writer(struct sock *sk, long *timeo)\n--\nnet/tls/tls_main.c-167-\nnet/tls/tls_main.c:168:int tls_push_sg(struct sock *sk,\nnet/tls/tls_main.c-169-\t\tstruct tls_context *ctx,\n--\nnet/tls/tls_main.c-225-\nnet/tls/tls_main.c:226:static int tls_handle_open_record(struct sock *sk, int flags)\nnet/tls/tls_main.c-227-{\n--\nnet/tls/tls_main.c-235-\nnet/tls/tls_main.c:236:int tls_process_cmsg(struct sock *sk, struct msghdr *msg,\nnet/tls/tls_main.c-237-\t\t unsigned char *record_type)\n--\nnet/tls/tls_main.c-267-\nnet/tls/tls_main.c:268:int tls_push_partial_record(struct sock *sk, struct tls_context *ctx,\nnet/tls/tls_main.c-269-\t\t\t int flags)\n--\nnet/tls/tls_main.c-280-\nnet/tls/tls_main.c:281:void tls_free_partial_record(struct sock *sk, struct tls_context *ctx)\nnet/tls/tls_main.c-282-{\n--\nnet/tls/tls_main.c-291-\nnet/tls/tls_main.c:292:static void tls_write_space(struct sock *sk)\nnet/tls/tls_main.c-293-{\n--\nnet/tls/tls_main.c-322- */\nnet/tls/tls_main.c:323:void tls_ctx_free(struct sock *sk, struct tls_context *ctx)\nnet/tls/tls_main.c-324-{\n--\nnet/tls/tls_main.c-337-\nnet/tls/tls_main.c:338:static void tls_sk_proto_cleanup(struct sock *sk,\nnet/tls/tls_main.c-339-\t\t\t\t struct tls_context *ctx, long timeo)\n--\nnet/tls/tls_main.c-374-\nnet/tls/tls_main.c:375:static void tls_sk_proto_close(struct sock *sk, long timeout)\nnet/tls/tls_main.c-376-{\n--\nnet/tls/tls_main.c-412-\nnet/tls/tls_main.c:413:static __poll_t tls_sk_poll(struct file *file, struct socket *sock,\nnet/tls/tls_main.c-414-\t\t\t struct poll_table_struct *wait)\n--\nnet/tls/tls_main.c-440-\nnet/tls/tls_main.c:441:static int do_tls_getsockopt_conf(struct sock *sk, sockopt_t *opt, int tx)\nnet/tls/tls_main.c-442-{\n--\n\n"
} |
| 37/2 | 2026/09/20 01:04 | llm | patch-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 139672 output: 26 thoughts: 306 Thoughts: **Analyzing the Recent TLS Changes** Okay, let's dive into these recent modifications. My initial scan of the code changes has flagged several key functions that have either been altered or newly introduced, all within the TLS subsystem. I'm seeing changes in `net/tls/tls_main.c` including modifications to `do_tls_setsockopt`, `do_tls_setsockopt_conf`, and `do_tls_getsockopt_conf`. These look like they're related to socket option configuration, which is interesting. Then, moving to `net/tls/tls_sw.c`, the changes focus on the software context with the initialization and finalization of `tls_sw_ctx_init` and `tls_sw_ctx_finalize` and methods for the TLS software offload. I also note functions related to encryption (`tls_do_encryption`) and record pushing (`tls_push_record`). Finally, over in `net/tls/tls_device.c`, there are function calls for setting device offload parameters and key deletion. This suggests some deeper integration with hardware acceleration. It looks like `tls_set_device_offload` and `tls_set_device_offload_rx` are the functions for that. I also see that there's a new function to delete keys: `tls_device_rx_del_key`. Before I dig into the details of each function and its implications, I need to check the Kconfig dependencies. I'll need to confirm that `CONFIG_TLS` and `CONFIG_TLS_DEVICE` are correctly set and behaving as expected. That's a crucial step to ensure the changes are integrated properly. |
| 38/2 | 2026/09/20 01:05 | tool | grepper |
0mArgs:{
"Expression": "config TLS",
"PathPrefix": "net/tls/"
}Results: {
"Output": "net/tls/Kconfig-4-#\nnet/tls/Kconfig:5:config TLS\nnet/tls/Kconfig-6-\ttristate \"Transport Layer Security support\"\n--\nnet/tls/Kconfig-18-\nnet/tls/Kconfig:19:config TLS_DEVICE\nnet/tls/Kconfig-20-\tbool \"Transport Layer Security HW offload\"\n"
} |
| 39/2 | 2026/09/20 01:05 | llm | patch-evaluator-compressor |
0mModel:gemini-3.8-flash Args: null Results: null Reply: ### 1. Approaches Tried and Results
- **Initial diff inspection**:
- Reviewed commit `209c4bda0a621712323d0608683304a02d1a2596` introducing TLS 1.3 KeyUpdate / rekey support for kTLS hardware offload, SW fallback handling, socket options, TCP collapse fencing, and Qdisc dequeue validation.
- Affected files:
- Documentation: `Documentation/networking/tls-offload.rst`, `Documentation/networking/tls.rst`
- MAINTAINERS entry updates
- Drivers: Chelsio (`cxgb4`, `chcr_ktls.c`), Mellanox (`mlx5`), Netronome (`nfp`)
- Headers: `include/linux/skbuff.h`, `include/net/tcp.h`, `include/net/tls.h`, `include/uapi/linux/snmp.h`
- Core network & scheduling: `net/sched/sch_generic.c`
- Core kTLS: `net/tls/tls.h`, `net/tls/tls_device.c`, `net/tls/tls_device_fallback.c`, `net/tls/tls_main.c`, `net/tls/tls_proc.c`, `net/tls/tls_sw.c`, `net/tls/trace.h`
- Selftests: `tools/testing/selftests/drivers/net/hw/tls_hw_offload.c`, `.py`, `Makefile`, `.gitignore`, `config`
- **Evaluating HW Offload Reachability (Dead-end investigation)**:
- Investigated whether virtual devices (e.g. `netdevsim`, `virtio_net`, `tun`, `veth`, `loopback`) implement `NETIF_F_HW_TLS_TX` / `NETIF_F_HW_TLS_RX` or `struct tlsdev_ops`.
- Grepped `tlsdev_ops` and `NETIF_F_HW_TLS_TX` / `NETIF_F_HW_TLS_RX`: only Chelsio (`cxgb4`), Fungible (`funeth`), Mellanox (`mlx5`), and Netronome (`nfp`) implement them.
- Grepped `netdevsim`: confirmed zero TLS support (`drivers/net/netdevsim/` returned no matches).
- Dead-end conclusion: Pure hardware-offloaded TLS data paths (e.g. NIC ASIC processing, `netdev->tlsdev_ops->tls_dev_add`) cannot be reached in a standard QEMU environment without physical PCI NICs.
- **Evaluating Core Subsystem & Software kTLS Reachability**:
- Investigated entry paths for userspace TLS configuration (`setsockopt` with `TCP_ULP` "tls" and `SOL_TLS` `TLS_TX` / `TLS_RX`).
- Traced `do_tls_setsockopt()` in `net/tls/tls_main.c`:
- On `TLS_TX`, unconditionally takes `mutex_lock_interruptible(&tls_get_ctx(sk)->tx_lock)` *before* acquiring `lock_sock(sk)`. This applies to **all** kTLS sockets, including pure software TLS (`TLS_SW`).
- `do_tls_setsockopt_conf()` calls `tls_set_device_offload()` and `tls_set_device_offload_rx()`, which execute modified code, check features, and fall back to `tls_set_sw_offload()`.
- Traced `tls_set_sw_offload()` in `net/tls/tls_sw.c`:
- Refactored into two new functions: `tls_sw_ctx_init()` and `tls_sw_ctx_finalize()`.
- Key and IV handling logic (`memcpy`, cipher setup) restructured; executed on every SW kTLS socket creation and rekey.
- Traced SW kTLS TX data path:
- `tls_do_encryption()` and `tls_push_record()` now invoke `tls_tx_cipher_ctx(tls_ctx)` instead of accessing `tls_ctx->tx` directly.
- `tls_sw_ctx_tx()` modified to check `tls_ctx->rekey.sw_ctx`.
- Executed on every `sendmsg()` on software TLS sockets.
- Traced core networking changes:
- `net/sched/sch_generic.c`: `dequeue_skb()` modified to add `if (skb_is_decrypted(skb)) *validate = true;` on Qdisc dequeue.
- `include/net/tcp.h`: `tcp_write_collapse_fence()` modified to inspect `tcp_rtx_queue_tail(sk)`.
- `include/linux/skbuff.h`: `struct sk_buff` layout modified with `decrypt_failed:1` under `CONFIG_SKB_DECRYPTED`.
### 2. Current Hypotheses and Lines of Investigation
- **Reachability Assessment**:
- The patch modifies core subsystems (`net/sched/`, `include/net/tcp.h`, `net/tls/`) reachable via standard AF_INET/AF_INET6 TCP sockets configured with `TCP_ULP` "tls" and `SOL_TLS` options.
- `WorthFuzzing=true` is clearly indicated because:
1. Core networking (`net/sched/sch_generic.c`, `include/net/tcp.h`) logic is modified.
2. Software kTLS (`net/tls/tls_sw.c`, `net/tls/tls_main.c`) is fully reachable without specialized hardware in QEMU.
3. Locking restructuring: `tx_lock` mutex is introduced into `do_tls_setsockopt()` before `lock_sock(sk)`, altering lock acquisition hierarchies and creating potential lockdep / deadlock / race surfaces.
4. Data path helpers (`tls_tx_cipher_ctx()`, `tls_sw_ctx_init()`, `tls_sw_ctx_finalize()`) run on software kTLS connections.
- **Active Lines of Investigation**:
- Determine optimal `FocusSymbols`:
- Avoid ubiquitous functions (`dequeue_skb`).
- Focus on functions altered in reachable code:
- `tls_sw_ctx_init`
- `tls_sw_ctx_finalize`
- `tls_set_sw_offload`
- `do_tls_setsockopt` / `do_tls_setsockopt_conf`
- `tls_sw_sendmsg` / `tls_sw_sendmsg_locked`
- `tls_sw_drain_tx`
- `tls_sw_splice_eof_locked`
- `tls_tx_cipher_ctx` (static inline -> caller `tls_do_encryption`, `tls_push_record`)
- `tcp_write_collapse_fence` (static inline -> caller `psp_sock_set_tx_key` or TLS entry points)
- `tls_set_device_offload` / `tls_set_device_offload_rx` (exercised during setsockopt before SW fallback)
- Determine required `EnableConfigs`:
- Need `TLS` (`CONFIG_TLS`)
- Need `TLS_DEVICE` (`CONFIG_TLS_DEVICE`) for the `tls_device.c` entries and `CONFIG_SKB_DECRYPTED`
- Check dependencies in `net/tls/Kconfig`.
### 3. Critical File Paths and Code Snippets
- `net/tls/tls_main.c`:
- Locking in `do_tls_setsockopt()`:
```c
case TLS_TX:
case TLS_RX: {
bool tx = optname == TLS_TX;
if (tx) {
rc = mutex_lock_interruptible(&tls_get_ctx(sk)->tx_lock);
if (rc)
break;
}
lock_sock(sk);
rc = do_tls_setsockopt_conf(sk, optval, optlen, tx);
release_sock(sk);
if (tx)
mutex_unlock(&tls_get_ctx(sk)->tx_lock);
break;
}
```
- Offload configuration & fallback in `do_tls_setsockopt_conf()`:
```c
if (tx) {
rc = tls_set_device_offload(sk, update ? crypto_info : NULL);
conf = TLS_HW;
if (!rc) {
...
} else if (update && ctx->tx_conf == TLS_HW) {
goto err_crypto_info;
} else {
rc = tls_set_sw_offload(sk, 1, update ? crypto_info : NULL);
...
conf = TLS_SW;
}
}
```
- `net/tls/tls_sw.c`:
- `tls_sw_ctx_init(struct sock *sk, int tx, struct tls_crypto_info *new_crypto_info)`
- `tls_sw_ctx_finalize(struct sock *sk, int tx, struct tls_crypto_info *new_crypto_info)`
- `tls_set_sw_offload(struct sock *sk, int tx, struct tls_crypto_info *new_crypto_info)`
- `tls_sw_drain_tx(struct sock *sk, struct tls_context *ctx, int flags)`
- `tls_do_encryption(struct sock *sk, ...)`:
- Uses `tls_tx_cipher_ctx(tls_ctx)` for `iv` and `rec_seq`.
- `include/net/tls.h`:
- `tls_tx_cipher_ctx()`:
```c
static inline struct cipher_context *tls_tx_cipher_ctx(
const struct tls_context *tls_ctx)
{
struct cipher_context *rekey_ctx = READ_ONCE(tls_ctx->rekey.cipher_ctx);
if (unlikely(rekey_ctx))
return rekey_ctx;
return (struct cipher_context *)&tls_ctx->tx;
}
```
- `tls_sw_ctx_tx()`:
```c
static inline struct tls_sw_context_tx *tls_sw_ctx_tx(
const struct tls_context *tls_ctx)
{
struct tls_sw_context_tx *rekey_ctx = READ_ONCE(tls_ctx->rekey.sw_ctx);
if (unlikely(rekey_ctx))
return rekey_ctx;
return (struct tls_sw_context_tx *)tls_ctx->priv_ctx_tx;
}
```
- `net/sched/sch_generic.c`:
- `dequeue_skb()`:
```c
if (skb_is_decrypted(skb))
*validate = true;
```
- `include/net/tcp.h`:
- `tcp_write_collapse_fence()`:
```c
static inline void tcp_write_collapse_fence(struct sock *sk)
{
struct sk_buff *skb = tcp_write_queue_tail(sk);
if (!skb)
skb = tcp_rtx_queue_tail(sk);
if (skb)
TCP_SKB_CB(skb)->eor = 1;
}
```
### 4. Watch Out / Potential Loops
- Avoid repeatedly grepping drivers for hardware offload support: it is already confirmed that no virtual driver supports `NETIF_F_HW_TLS_TX` / `NETIF_F_HW_TLS_RX`.
- Focus directly on finalizing the Kconfig requirements (`TLS`, `TLS_DEVICE`, `INET`, etc.) and extracting targeted, non-diluting `FocusSymbols` from `net/tls/`.Tokens: input: 106709 output: 2771 thoughts: 0 |
| 40/2 | 2026/09/20 01:05 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 92524 output: 32 thoughts: 0 |
| 41/2 | 2026/09/20 01:05 | tool | read-file |
0mArgs:{
"File": "net/tls/Kconfig",
"FirstLine": 1,
"LineCount": 35
}Results: {
"Contents": " 1:\t# SPDX-License-Identifier: GPL-2.0-only\n 2:\t#\n 3:\t# TLS configuration\n 4:\t#\n 5:\tconfig TLS\n 6:\t\ttristate \"Transport Layer Security support\"\n 7:\t\tdepends on INET\n 8:\t\tselect CRYPTO\n 9:\t\tselect CRYPTO_AES\n 10:\t\tselect CRYPTO_GCM\n 11:\t\tselect NET_SOCK_MSG\n 12:\t\tdefault n\n 13:\t\thelp\n 14:\t\tEnable kernel support for TLS protocol. This allows symmetric\n 15:\t\tencryption handling of the TLS protocol to be done in-kernel.\n 16:\t\n 17:\t\tIf unsure, say N.\n 18:\t\n 19:\tconfig TLS_DEVICE\n 20:\t\tbool \"Transport Layer Security HW offload\"\n 21:\t\tdepends on TLS\n 22:\t\tselect SKB_DECRYPTED\n 23:\t\tselect SOCK_VALIDATE_XMIT\n 24:\t\tselect SOCK_RX_QUEUE_MAPPING\n 25:\t\tdefault n\n 26:\t\thelp\n 27:\t\tEnable kernel support for HW offload of the TLS protocol.\n 28:\t\n 29:\t\tIf unsure, say N.\n"
} |
| 42/2 | 2026/09/20 01:05 | llm | patch-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 92881 output: 29 thoughts: 2535 Thoughts: **Analysis of the kTLS Patch: WorthFuzzing Evaluation** Alright, let's break this down. First, the patch introduces TLS 1.3 KeyUpdate/rekey support for kTLS, which immediately flags it as potentially interesting. Although there's a hardware offload aspect, the key here is the significant modification of *reachable* code. The commit message mentions this is under review. The primary point of concern is the mutex lock order change within `do_tls_setsockopt()`. Taking `tx_lock` before `lock_sock(sk)` on ALL kTLS sockets is a big red flag. Given that `tls_device_sendmsg()` and `tls_sw_sendmsg()` *already* lock `tx_lock` then `lock_sock`, this needs immediate attention. The fact that `tls_sw_splice_eof()` does the same is even more concerning. This screams "potential lock inversion" and "deadlock" waiting to happen, ripe for lockdep analysis. This alone is enough to say, "Yes, this is worth fuzzing." Furthermore, the refactoring of `tls_sw_ctx_init()` and `tls_sw_ctx_finalize()` in `net/tls/tls_sw.c` is another area that needs close scrutiny. These are critical initialization and cleanup functions within the software path. The fact they are refactored means that fuzzing this code is important. The use of `tls_tx_cipher_ctx()` within `tls_do_encryption()` and `tls_push_record()` adds another layer of complexity to the software path. This is also worth fuzzing. The modifications to `tcp_write_collapse_fence()` and the inclusion of `dequeue_skb()` in the generic Qdisc dequeue path highlight the impact of this change on the core networking stack. While `dequeue_skb()` is ubiquitous, its modification in this context, coupled with everything else, necessitates fuzzing. Thus, based on my understanding, modifying reachable executable code in core networking, especially with the mutex lock order change and refactoring, it's clear: `WorthFuzzing = true`. Now, let's identify what to feed the fuzzer. First, for `EnableConfigs`: * `TLS` (depends on `INET`) - the foundation. * `TLS_DEVICE` (depends on `TLS`, and selects `SKB_DECRYPTED`, `SOCK_VALIDATE_XMIT`, `SOCK_RX_QUEUE_MAPPING`) - the core kTLS feature. * `INET` - Networking is a given. Now to the `FocusSymbols`: I need to pick specific and relevant functions, avoiding the ubiquitous ones. Here are the ones that jump out: * `tls_set_sw_offload` (Important entry point) * `tls_sw_ctx_init` (Core kTLS initialization path) * `tls_sw_ctx_finalize` (Core kTLS finalization path) * `do_tls_setsockopt` (Entrypoint for setsockopt and config) * `tls_set_device_offload` (Device configuration) * `tls_set_device_offload_rx` (Device RX configuration) * `tls_sw_sendmsg_locked` (The main encryption/tx function) * `tls_sw_drain_tx` (TX drain functionality) * `tls_do_encryption` (Core encryption) * `tls_push_record` (Sending the records) * `tls_validate_xmit_skb_rekey` (validation/security check) By targeting these, we should maximize our fuzzing efficiency and find issues related to the core logic. |
| 43/2 | 2026/09/20 01:05 | tool | grepper |
0mArgs:{
"Expression": "tls_setsockopt",
"PathPrefix": "net/tls/"
}Results: {
"Output": "net/tls/tls_device.c=1596=static int tls_device_start_rekey(struct sock *sk,\n--\nnet/tls/tls_device.c-1743-\t\t cipher_desc-\u003ecrypto_info,\nnet/tls/tls_device.c:1744:\t\t /* checked in do_tls_setsockopt_conf */);\nnet/tls/tls_device.c-1745-\tmemzero_explicit(new_crypto_info, cipher_desc-\u003ecrypto_info);\n--\nnet/tls/tls_main.c=621=static int validate_crypto_info(const struct tls_crypto_info *crypto_info,\n--\nnet/tls/tls_main.c-645-\nnet/tls/tls_main.c:646:static int do_tls_setsockopt_conf(struct sock *sk, sockptr_t optval,\nnet/tls/tls_main.c-647-\t\t\t\t unsigned int optlen, int tx)\n--\nnet/tls/tls_main.c-825-\nnet/tls/tls_main.c:826:static int do_tls_setsockopt_tx_zc(struct sock *sk, sockptr_t optval,\nnet/tls/tls_main.c-827-\t\t\t\t unsigned int optlen)\n--\nnet/tls/tls_main.c-845-\nnet/tls/tls_main.c:846:static int do_tls_setsockopt_no_pad(struct sock *sk, sockptr_t optval,\nnet/tls/tls_main.c-847-\t\t\t\t unsigned int optlen)\n--\nnet/tls/tls_main.c-877-\nnet/tls/tls_main.c:878:static int do_tls_setsockopt_tx_payload_len(struct sock *sk, sockptr_t optval,\nnet/tls/tls_main.c-879-\t\t\t\t\t unsigned int optlen)\n--\nnet/tls/tls_main.c-903-\nnet/tls/tls_main.c:904:static int do_tls_setsockopt(struct sock *sk, int optname, sockptr_t optval,\nnet/tls/tls_main.c-905-\t\t\t unsigned int optlen)\n--\nnet/tls/tls_main.c-927-\t\tlock_sock(sk);\nnet/tls/tls_main.c:928:\t\trc = do_tls_setsockopt_conf(sk, optval, optlen, tx);\nnet/tls/tls_main.c-929-\t\trelease_sock(sk);\n--\nnet/tls/tls_main.c-935-\t\tlock_sock(sk);\nnet/tls/tls_main.c:936:\t\trc = do_tls_setsockopt_tx_zc(sk, optval, optlen);\nnet/tls/tls_main.c-937-\t\trelease_sock(sk);\n--\nnet/tls/tls_main.c-939-\tcase TLS_RX_EXPECT_NO_PAD:\nnet/tls/tls_main.c:940:\t\trc = do_tls_setsockopt_no_pad(sk, optval, optlen);\nnet/tls/tls_main.c-941-\t\tbreak;\n--\nnet/tls/tls_main.c-943-\t\tlock_sock(sk);\nnet/tls/tls_main.c:944:\t\trc = do_tls_setsockopt_tx_payload_len(sk, optval, optlen);\nnet/tls/tls_main.c-945-\t\trelease_sock(sk);\n--\nnet/tls/tls_main.c-953-\nnet/tls/tls_main.c:954:static int tls_setsockopt(struct sock *sk, int level, int optname,\nnet/tls/tls_main.c-955-\t\t\t sockptr_t optval, unsigned int optlen)\n--\nnet/tls/tls_main.c-962-\nnet/tls/tls_main.c:963:\treturn do_tls_setsockopt(sk, optname, optval, optlen);\nnet/tls/tls_main.c-964-}\n--\nnet/tls/tls_main.c=1057=static void build_protos(struct proto prot[TLS_NUM_CONFIG][TLS_NUM_CONFIG],\n--\nnet/tls/tls_main.c-1060-\tprot[TLS_BASE][TLS_BASE] = *base;\nnet/tls/tls_main.c:1061:\tprot[TLS_BASE][TLS_BASE].setsockopt\t= tls_setsockopt;\nnet/tls/tls_main.c-1062-\tprot[TLS_BASE][TLS_BASE].getsockopt\t= tls_getsockopt;\n--\nnet/tls/tls_sw.c=2684=void tls_sw_ctx_finalize(struct sock *sk, int tx,\n--\nnet/tls/tls_sw.c-2716-\t\t\t cipher_desc-\u003ecrypto_info,\nnet/tls/tls_sw.c:2717:\t\t\t /* size was checked in do_tls_setsockopt_conf */);\nnet/tls/tls_sw.c-2718-\t\tmemzero_explicit(new_crypto_info, cipher_desc-\u003ecrypto_info);\n"
} |
| 44/2 | 2026/09/20 01:05 | llm | patch-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 97610 output: 489 thoughts: 986 Thoughts: **kTLS Software and Device Offload Analysis** Okay, let's break this down. My initial assessment identifies several key kTLS functions and their locations: `tls_setsockopt`, `do_tls_setsockopt`, `do_tls_setsockopt_conf`, and the `tls_set_*` family. I see `tls_sw_ctx_init`, `tls_sw_ctx_finalize` in the software side, and the device-specific offload functions `tls_set_device_offload`, and `tls_set_device_offload_rx`. Then there are the data path elements: `tls_sw_drain_tx`, `tls_sw_sendmsg_locked`, `tls_sw_splice_eof_locked`, `tls_do_encryption`, and `tls_push_record`. Finally, `tls_validate_xmit_skb_rekey`. My first thought is to verify that the `FocusSymbols` list accurately reflects the code changes. I've re-checked that the listed symbols are the ones relevant to the kTLS changes that are the focus. Next, I need to ensure that these symbols are specific to kTLS and not generic utilities. I confirm that none of these relate to memory allocation, page faults, or fundamental lock primitives. They're all kTLS-specific. Good. Now for the configurations. `TLS` is the base, of course, enabling kTLS. `TLS_DEVICE` is critical as it enables device offload, which is a major point of the proposed change and triggers the necessary setsockopt paths. `INET` is obviously required. Okay, let's summarize the key changes, keeping in mind the "WorthFuzzing" flag and the configuration requirements: 1. `do_tls_setsockopt` now acquires a mutex `tx_lock` before `lock_sock`, modifying lock ordering on setsockopt calls across all kTLS sockets. This is a significant synchronization change. 2. `do_tls_setsockopt_conf` now routes setsockopt configuration through the device offload setup functions (`tls_set_device_offload` and `tls_set_device_offload_rx`) before any software fallback. 3. `tls_set_sw_offload` has been refactored into `tls_sw_ctx_init` and `tls_sw_ctx_finalize`, which change software cipher and context initialization and finalization. 4. The software data path encryption, including `tls_do_encryption` and `tls_push_record`, now uses new cipher context accessor logic. 5. There are changes in core network code, specifically in `net/sched/sch_generic.c` (`dequeue_skb`) and `include/net/tcp.h` (`tcp_write_collapse_fence`). I can confirm that these changes are reachable via standard TCP sockets configuring kTLS ULP, even in a virtualized environment. While true hardware offload requires physical SmartNICs (e.g., mlx5, nfp), these software paths are still substantially altered. Looks like everything is clear, precise, and meets all criteria. Now I'm ready to call `set-results`. |
| 45/2 | 2026/09/20 01:06 | tool | set-results |
0mArgs:{
"EnableConfigs": [
"TLS",
"TLS_DEVICE",
"INET"
],
"FocusSymbols": [
"tls_setsockopt",
"do_tls_setsockopt",
"do_tls_setsockopt_conf",
"tls_set_sw_offload",
"tls_sw_ctx_init",
"tls_sw_ctx_finalize",
"tls_set_device_offload",
"tls_set_device_offload_rx",
"tls_sw_drain_tx",
"tls_sw_sendmsg_locked",
"tls_validate_xmit_skb_rekey"
],
"Reasoning": "The patch modifies both core networking and the kTLS subsystem to support TLS 1.3 KeyUpdate/rekeying. While dedicated hardware offload execution relies on physical SmartNICs, the patch significantly alters reachable software paths that execute in standard virtualized environments (QEMU):\n1. `do_tls_setsockopt` introduces mutex acquisition of `tx_lock` prior to `lock_sock`, altering the locking hierarchy for all kTLS sockets (software or hardware).\n2. `do_tls_setsockopt_conf` invokes `tls_set_device_offload` and `tls_set_device_offload_rx` for initial and rekey configuration checks before falling back to `tls_set_sw_offload`.\n3. `tls_set_sw_offload` has been refactored into `tls_sw_ctx_init` and `tls_sw_ctx_finalize`, changing cipher allocation, setkey, and sequence initialization in software kTLS.\n4. Data path encryption (`tls_do_encryption`, `tls_push_record`) and transmission paths (`tls_sw_sendmsg_locked`, `tls_sw_drain_tx`) now integrate rekey cipher context accessors.\n5. Core networking changes in `net/sched/sch_generic.c` (`dequeue_skb`) and `include/net/tcp.h` (`tcp_write_collapse_fence`) affect socket packet handling.\nThese changes modify reachable logic and synchronization invariants, warranting fuzzing.",
"WorthFuzzing": true
}Results: {
"EnableConfigs": [
"TLS",
"TLS_DEVICE",
"INET"
],
"FocusSymbols": [
"tls_setsockopt",
"do_tls_setsockopt",
"do_tls_setsockopt_conf",
"tls_set_sw_offload",
"tls_sw_ctx_init",
"tls_sw_ctx_finalize",
"tls_set_device_offload",
"tls_set_device_offload_rx",
"tls_sw_drain_tx",
"tls_sw_sendmsg_locked",
"tls_validate_xmit_skb_rekey"
],
"Reasoning": "The patch modifies both core networking and the kTLS subsystem to support TLS 1.3 KeyUpdate/rekeying. While dedicated hardware offload execution relies on physical SmartNICs, the patch significantly alters reachable software paths that execute in standard virtualized environments (QEMU):\n1. `do_tls_setsockopt` introduces mutex acquisition of `tx_lock` prior to `lock_sock`, altering the locking hierarchy for all kTLS sockets (software or hardware).\n2. `do_tls_setsockopt_conf` invokes `tls_set_device_offload` and `tls_set_device_offload_rx` for initial and rekey configuration checks before falling back to `tls_set_sw_offload`.\n3. `tls_set_sw_offload` has been refactored into `tls_sw_ctx_init` and `tls_sw_ctx_finalize`, changing cipher allocation, setkey, and sequence initialization in software kTLS.\n4. Data path encryption (`tls_do_encryption`, `tls_push_record`) and transmission paths (`tls_sw_sendmsg_locked`, `tls_sw_drain_tx`) now integrate rekey cipher context accessors.\n5. Core networking changes in `net/sched/sch_generic.c` (`dequeue_skb`) and `include/net/tcp.h` (`tcp_write_collapse_fence`) affect socket packet handling.\nThese changes modify reachable logic and synchronization invariants, warranting fuzzing.",
"WorthFuzzing": true
} |
| 46/1 | 2026/09/20 01:06 | action | If |
5mArgs:{
"WorthFuzzing": true
}Results: null |
| 47/2 | 2026/09/20 01:06 | agent | kmsan-evaluator |
5mModel:core Args: null Results: {
"KMSANReasoning": "The patch series implements TLS 1.3 hardware offload and key rotation (KeyUpdate) support for kTLS (in net/tls and driver offloads like mlx5).\n\nAnalysis of memory initialization and safety:\n1. Heap/Page Allocations:\n - All newly added context structures (`tls_offload_context_tx`, `tls_offload_context_rx`) continue to be allocated via kzalloc (`kzalloc_obj`), guaranteeing zero-initialization of newly added fields (e.g. `rekey`).\n - The start marker record (`offload_ctx-\u003erekey.start_marker`) is allocated and its fields (`end_seq`, `len = 0`, `num_frags = 0`) are explicitly initialized prior to being queued on `records_list`; its `frags` are not accessed when `num_frags == 0`.\n - The fallback page `dummy_page` in `tls_device_init()` is explicitly allocated with `__GFP_ZERO` and initialized with an identity map specifically to avoid any uninitialized byte leak.\n2. User-Space Info Leaks (getsockopt):\n - In `do_tls_getsockopt_conf()`, reading the rekey crypto info accesses `offload_ctx-\u003erekey.crypto_send.info`. This union is originally populated from user input into a zero-initialized union (`tmp = {}`) and copied into the kzalloc'd `offload_ctx`. When returned via `copy_to_iter()`, strictly `cipher_desc-\u003ecrypto_info` bytes are copied, leaving no uninitialized padding or fields.\n3. sk_buff Bitfield:\n - The new `decrypt_failed` bitfield in `struct sk_buff` resides in the header area zeroed out by `__alloc_skb()` upon allocation, and is maintained via `skb_copy_decrypted()`.\n4. KASAN vs KMSAN Applicability:\n - The risks introduced by this patch series pertain to asynchronous state machine transitions, concurrent RCU dereferences / reader-writer locking (`tx_lock`, `device_offload_lock`, `lock_sock`), and lifetime management of crypto AEAD transforms during fallback / rekeying.\n - Any bugs arising from these changes (e.g. use-after-free of AEAD transforms, double frees, list corruption, or deadlock) are detected by KASAN and LOCKDEP.\n - There are no risks of uninitialized memory reads or information leaks that would require KMSAN detection.\n\nTherefore, a dedicated KMSAN fuzzing session is not justified.",
"NeedsKMSAN": false
}Instruction: You are an expert Linux kernel security engineer specializing in kernel memory
error detectors (KASAN and KMSAN). Your job is to review the provided patch series and
determine if the code changes justify spawning a dedicated KMSAN (KernelMemorySanitizer)
fuzzing session in addition to standard KASAN fuzzing.
CRITICAL DISTINCTION BETWEEN KASAN AND KMSAN:
- Standard KASAN kernel builds (upstream-apparmor-kasan.config) already enable
a comprehensive suite of debugging tools and sanitizers, including KASAN
(out-of-bounds accesses, use-after-free, double free, invalid free), LOCKDEP
(locking bugs and deadlocks), UB-sanitizers, and memory corruption checks.
- KMSAN (KernelMemorySanitizer) detects reads of UNINITIALIZED memory (stack, heap,
or page allocations) and kernel-to-user memory info-leaks.
Rule: THERE IS NO SENSE IN RUNNING A KMSAN SESSION IF A BUG CAN BE CAUGHT BY KASAN,
LOCKDEP, OR OTHER STANDARD BUG DETECTORS.
A dedicated KMSAN fuzzing session incurs significant resource costs. You must ONLY
set NeedsKMSAN=true if the code changes introduce or expose UNINITIALIZED MEMORY risks
that are detected ONLY by KMSAN.
Look holistically at the patch series and surrounding code. Even if no direct
uninitialized field accesses or new buffer allocations are added in the diff itself,
a patch may alter control flow, bounds checking, or data length calculations in ways
that change how the rest of the code operates on existing buffers (e.g. allowing
uninitialized stack/heap memory to be read, copied to user space, or used in control
flow). Do not hesitate to use your code access tools to inspect the surrounding code,
called functions, and callers.
Set NeedsKMSAN=true ONLY IF the patch introduces or modifies:
1. Kernel structures sent to user space (via copy_to_user, put_user, netlink skb
attributes, ioctl output arguments, socket options, or BPF buffers) where fields
or structure padding might not be fully initialized/zeroed.
2. Conditional logic or branching that depends on potentially uninitialized variables
or struct fields.
3. Allocation or initialization of complex data structures where uninitialized fields
could be read later in reachable code paths.
4. Bounds checks, lengths, or logic in a way that allows surrounding code to access
uninitialized bytes of existing buffers.
Set NeedsKMSAN=false IF:
- The code changes primarily risk out-of-bounds access, array overflows, NULL pointer
dereferences, locking deadlocks, or use-after-free bugs (these are already caught
by KASAN, LOCKDEP, or standard bug detectors).
- All stack/heap structures touched or introduced by the patch are fully zeroed
or initialized (e.g. using = {0}, memset, kzalloc) before being read or copied.
- The patch does not introduce any risk of uninitialized memory usage or info-leaks.
Use your code access tools to inspect the surrounding code if necessary, then provide
detailed KMSANReasoning contrasting KASAN vs KMSAN applicability for this patch.
Prefer calling several tools at the same time to save round-trips.
Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.
Prompt: Target architecture: amd64
For your convenience, here is the diff of the changes:
commit 209c4bda0a621712323d0608683304a02d1a2596
Author: syz-cluster <triage@syzkaller.com>
Date: Sun Sep 20 01:00:09 2026 +0000
syz-cluster: applied patch under review
diff --git a/Documentation/networking/tls-offload.rst b/Documentation/networking/tls-offload.rst
index e5802bcd4d22d..cdf84f4b817a7 100644
--- a/Documentation/networking/tls-offload.rst
+++ b/Documentation/networking/tls-offload.rst
@@ -99,9 +99,8 @@ at the end of kernel structures (see :c:member:`driver_state` members
in ``include/net/tls.h``) to avoid additional allocations and pointer
dereferences.
-When the offloaded connection is destroyed the core calls
-the :c:member:`tls_dev_del` callback so the driver can release per-direction
-state:
+The core calls the :c:member:`tls_dev_del` callback so the driver can release
+per-direction state:
.. code-block:: c
@@ -109,7 +108,14 @@ state:
struct tls_context *ctx,
enum tls_offload_ctx_dir direction);
-``tls_dev_del`` is mandatory whenever ``tls_dev_add`` is provided.
+``tls_dev_del`` is called either when the offloaded connection is destroyed or,
+for a TLS 1.3 connection, when the old key is retired during a rekey (see the
+`Rekey`_ section). It operates on a single ``direction``, so the driver must
+release only the state for that direction and must not free state shared
+between directions or the socket as a whole. After a rekey ``tls_dev_del``,
+``tls_dev_add`` may be called again for the same socket and direction to
+install the new key. ``tls_dev_del`` is mandatory whenever ``tls_dev_add`` is
+provided.
The third TLS device callback is :c:member:`tls_dev_resync`, called by the core
to synchronize the TCP stream with the record boundaries:
@@ -205,7 +211,10 @@ Upon reception of a TLS offloaded packet, the driver sets
the :c:member:`decrypted` mark in :c:type:`struct sk_buff <sk_buff>`
corresponding to the segment. Networking stack makes sure decrypted
and non-decrypted segments do not get coalesced (e.g. by GRO or socket layer)
-and takes care of partial decryption.
+and takes care of partial decryption. A segment the device processed but
+could not authenticate may instead carry the :c:member:`decrypt_failed`
+mark; see the `Error handling`_ section for what the mark implies about
+the payload.
Resync handling
===============
@@ -404,8 +413,121 @@ records, then after 4 records, after 8, after 16... up until every
Rekey
=====
-Offload does not currently support TLS 1.3, therefore key rotation
-is not a concern for offloaded connections at this point.
+TLS 1.3 allows traffic keys to be updated mid-connection using the
+KeyUpdate message. Offloaded TLS 1.3 connections must therefore switch
+keys without tearing down the offload. The device cannot simply be given
+the new key because records encrypted (TX) or transformed (RX) with the
+old key may still be in flight. The stack retains the necessary old-key
+state and bridges the transition in software.
+
+TX
+--
+
+On TX, the new key is installed in a temporary software context, and
+sendmsg is routed through the software path. If no hardware-offloaded
+records remain unacknowledged, the switch completes inline during
+setsockopt. Otherwise the rekey is left pending and is completed later,
+on the sender's next ``sendmsg()`` after all old-key records have been
+ACKed (see `Completing a deferred rekey`_). Completion calls
+:c:func:`tls_dev_del` for the old key and reinstalls hardware offload
+with the new key at the current TCP write sequence. If reinstallation
+fails, the connection keeps encrypting in software with the new key; the
+next KeyUpdate re-arms the transition and retries the hardware
+installation.
+
+Unlike the software path, a ``TLS_TX`` setsockopt on an offloaded
+connection first flushes the open and partially sent hardware records to
+TCP before installing the new key. It therefore behaves like a blocking
+``send()`` of that record: it may wait for send buffer space (bounded by
+``SO_SNDTIMEO``), and on a non-blocking socket it fails with ``-EAGAIN``
+and must be retried once the socket is writable. The new key is not
+installed until the call succeeds; the connection keeps using the old key
+in the meantime.
+
+Completing a deferred rekey
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A deferred rekey is completed by the sender, not by the ACK path. When
+the last old-key record is acknowledged the stack only marks the rekey
+as ready; the device is not touched. The switch itself,
+:c:func:`tls_dev_del` of the old key followed by :c:func:`tls_dev_add`
+of the new one, runs at the start of the next ``sendmsg()`` on the
+socket, and that ``sendmsg()`` is the first to be encrypted by hardware
+again. No other event completes it: ``splice_eof()``, write-space
+wakeups, retransmissions and pure ACKs all leave the connection on the
+software path.
+
+This is intentional. Completion has to flush the software context's
+open record to TCP and may sleep for send buffer space, which rules out
+the ACK and write-space paths. Beyond that, the stack only switches when
+it has new data to hand to the device: the software path is fully
+correct with the new key, so deferring the switch costs host CPU but
+nothing else, and it keeps the device from being programmed for a
+connection that may never send again.
+
+Two consequences follow. A connection that stops sending after a
+KeyUpdate stays in the deferred state until it is closed: it is
+encrypted in software with the new key, it is counted in
+``TlsCurrTxRekey``, and at close it is reported as
+``TlsTxRekeyAborted``. That counter therefore includes senders that
+simply had nothing more to send, not only sockets torn down
+mid-transition, and is not by itself an error indication. And the return
+to hardware is delayed by at least one ACK round trip after the last
+old-key record, plus however long the application waits before its next
+``sendmsg()``. A sender that wants the hardware path back promptly can
+issue a small ``sendmsg()`` once its old data has been acknowledged.
+
+Completion can fail transiently or permanently. If the software flush
+cannot get send buffer space (``-EAGAIN``, or a signal on a blocking
+socket) the rekey stays pending, the ``sendmsg()`` proceeds in software,
+and the next ``sendmsg()`` retries; the ``tls_device_complete_rekey_retry``
+tracepoint fires. A hard failure (:c:func:`tls_dev_add` rejected, or the
+netdev gone) is terminal for this KeyUpdate: the connection is pinned to
+software encryption with the new key, counted in ``TlsTxRekeyFallback``
+and moved from ``TlsCurrTxDevice`` to ``TlsCurrTxSw``; the
+``tls_device_complete_rekey_fail`` tracepoint fires. The next ``TLS_TX``
+setsockopt re-arms the transition and retries.
+
+The decision to defer is taken at the start of the ``TLS_TX``
+setsockopt, before the open hardware record is flushed to TCP. That
+flush may block for send buffer space, and old-key records acknowledged
+while it sleeps do not change the decision: the rekey is still deferred
+and completes on a following ``sendmsg()`` rather than inline. This is
+conservative, not a correctness issue. The boundary is fixed at the
+write sequence after the flush, so the acknowledgment of the flushed
+record itself arms completion; the cost is one more ACK round trip and
+one more ``sendmsg()``. Applications should not expect an inline switch
+whenever the socket has unacknowledged data at the time of the
+setsockopt.
+
+RX
+--
+
+On RX, the NIC may already have transformed in-flight records with the
+old key before the peer's KeyUpdate is parsed. When the KeyUpdate is
+decoded, the stack removes the old key from the NIC but retains the old
+AEAD, IV, and record sequence in the software offload context.
+
+Each record is classified by the TCP sequence of its first byte relative
+to the boundary at which the NIC stopped using the old key. Records
+starting after that boundary carry new-key wire encryption, so the old
+software AEAD state can be released. Records before the boundary that
+remain fully encrypted are passed to the software path. Records that
+were partially transformed by the NIC are re-encrypted with the old key
+to restore the new-key ciphertext, allowing the software AEAD to decrypt
+them with the new key.
+
+If old-key records are still queued, installation of the new key through
+:c:func:`tls_dev_add` is deferred until those records have been consumed;
+otherwise it occurs immediately. When the NIC cannot authenticate a record
+processed during the transition, the affected fragments are delivered with
+``skb->decrypt_failed`` set, following the contract described in the
+`Error handling`_ section. In a mixed record such a fragment was
+transformed (XORed) with the old key, and the re-encrypt path uses this to
+undo the transform on those fragments with the old key while leaving
+untouched fragments intact. A non-mixed record carrying
+``skb->decrypt_failed`` was not transformed; it is still wire ciphertext
+and is decrypted directly by the software AEAD under the new key.
Error handling
==============
@@ -442,8 +564,43 @@ to the host's stack as it was on the wire (recovering original packet in the
driver if device provides precise error is sufficient).
The Linux networking stack does not provide a way of reporting per-packet
-decryption and authentication errors, packets with errors must simply not
-have the :c:member:`decrypted` mark set.
+decryption and authentication errors. A packet with errors must not have
+the :c:member:`decrypted` mark set. In addition, the driver may set the
+:c:member:`decrypt_failed` mark on a segment the device matched to an
+offloaded connection and processed but could not authenticate. The two
+marks are mutually exclusive.
+
+The stack interprets :c:member:`decrypt_failed` per record, relative to the
+:c:member:`decrypted` mark of the other segments making up the same record.
+Coalescing (GRO, socket layer) and record classification are keyed on
+:c:member:`decrypted` alone, so :c:member:`decrypt_failed` segments may be
+merged with unmarked ones. A driver setting the mark must therefore honour
+the following contract:
+
+ * In a record none of whose segments carry :c:member:`decrypted`, every
+ segment, including one with :c:member:`decrypt_failed` set, must hold
+ the payload exactly as it was on the wire. This is the general rule
+ above: if the device did not successfully decrypt any part of a record
+ it must hand the whole record over untouched. The stack passes such a
+ record to software decryption directly and does not consult
+ :c:member:`decrypt_failed`.
+
+ * In a record where some segments carry :c:member:`decrypted` (a mixed
+ record), a segment with :c:member:`decrypt_failed` set must hold payload
+ the device has already transformed (XORed with the cipher keystream) but
+ failed to authenticate, and a segment with neither mark must hold the
+ payload as it was on the wire. The stack re-encrypts the
+ :c:member:`decrypted` and :c:member:`decrypt_failed` segments to restore
+ the ciphertext, leaves the unmarked segments intact, and authenticates
+ the whole record in software.
+
+A transformed segment delivered without :c:member:`decrypt_failed`, or an
+untransformed segment of a mixed record delivered with it, is restored
+incorrectly and the record fails software authentication. A device which
+cannot tell the driver whether a failed segment was transformed must
+recover the original packet before handing it to the stack, as described
+above, and leave both marks clear. During a TLS 1.3 rekey the mark also
+tells the stack which key the device applied; see the `Rekey`_ section.
A packet should also not be handled by the TLS offload if it contains
incorrect checksums.
diff --git a/Documentation/networking/tls.rst b/Documentation/networking/tls.rst
index 980c442d7161a..cf05543260d85 100644
--- a/Documentation/networking/tls.rst
+++ b/Documentation/networking/tls.rst
@@ -314,6 +314,11 @@ TLS implementation exposes the following per-namespace statistics
number of TX and RX sessions currently installed where NIC handles
cryptography
+- ``TlsCurrTxRekey``, ``TlsCurrRxRekey`` -
+ number of TX and RX sessions currently undergoing a deferred rekey,
+ i.e. a rekey which could not be applied immediately and is waiting for
+ in-flight records to drain before the new key is installed in hardware
+
- ``TlsTxSw``, ``TlsRxSw`` -
number of TX and RX sessions opened with host cryptography
@@ -344,3 +349,15 @@ TLS implementation exposes the following per-namespace statistics
- ``TlsRxRekeyReceived`` -
number of received KeyUpdate handshake messages, requiring userspace
to provide a new RX key
+
+- ``TlsTxRekeyFallback``, ``TlsRxRekeyFallback`` -
+ number of rekeys on existing sessions for TX and RX which could not be
+ offloaded to the NIC and fell back to software cryptography
+
+- ``TlsTxRekeyAborted``, ``TlsRxRekeyAborted`` -
+ number of deferred rekeys for TX and RX which were still pending when
+ the socket was destroyed, and so never completed. For TX hardware
+ offload this includes senders that sent nothing further after the
+ KeyUpdate, since the switch back to hardware only happens on
+ ``sendmsg()`` (see the Rekey section of
+ Documentation/networking/tls-offload.rst)
diff --git a/MAINTAINERS b/MAINTAINERS
index 0e04d92d1b098..4f1645bf2ee5e 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -19255,6 +19255,8 @@ F: Documentation/networking/tls*
F: include/net/tls.h
F: include/uapi/linux/tls.h
F: net/tls/
+F: tools/testing/selftests/drivers/net/hw/tls_hw_offload.c
+F: tools/testing/selftests/drivers/net/hw/tls_hw_offload.py
F: tools/testing/selftests/net/tls.c
NETWORKING [SOCKETS]
diff --git a/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c b/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c
index f5acd4be1e69d..29e108ce67645 100644
--- a/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c
+++ b/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c
@@ -431,6 +431,9 @@ static int chcr_ktls_dev_add(struct net_device *netdev, struct sock *sk,
atomic64_inc(&port_stats->ktls_tx_connection_open);
u_ctx = adap->uld[CXGB4_ULD_KTLS].handle;
+ if (crypto_info->version != TLS_1_2_VERSION)
+ goto out;
+
if (direction == TLS_OFFLOAD_CTX_DIR_RX) {
pr_err("not expecting for RX direction\n");
goto out;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
index 07a04a142a2ea..0469ca6a0762e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
@@ -30,7 +30,9 @@ static inline bool mlx5e_is_ktls_device(struct mlx5_core_dev *mdev)
return false;
return (MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_128) ||
- MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_256));
+ MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_256) ||
+ MLX5_CAP_TLS(mdev, tls_1_3_aes_gcm_128) ||
+ MLX5_CAP_TLS(mdev, tls_1_3_aes_gcm_256));
}
static inline bool mlx5e_ktls_type_check(struct mlx5_core_dev *mdev,
@@ -40,10 +42,14 @@ static inline bool mlx5e_ktls_type_check(struct mlx5_core_dev *mdev,
case TLS_CIPHER_AES_GCM_128:
if (crypto_info->version == TLS_1_2_VERSION)
return MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_128);
+ else if (crypto_info->version == TLS_1_3_VERSION)
+ return MLX5_CAP_TLS(mdev, tls_1_3_aes_gcm_128);
break;
case TLS_CIPHER_AES_GCM_256:
if (crypto_info->version == TLS_1_2_VERSION)
return MLX5_CAP_TLS(mdev, tls_1_2_aes_gcm_256);
+ else if (crypto_info->version == TLS_1_3_VERSION)
+ return MLX5_CAP_TLS(mdev, tls_1_3_aes_gcm_256);
break;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
index bca45679e2016..8ec40f5fd5b50 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
@@ -602,7 +602,18 @@ void mlx5e_ktls_handle_rx_skb(struct mlx5e_rq *rq, struct sk_buff *skb,
stats->tls_resync_req_pkt++;
resync_update_sn(rq, skb);
break;
- default: /* CQE_TLS_OFFLOAD_ERROR: */
+ case CQE_TLS_OFFLOAD_ERROR:
+ /* The device could not authenticate the payload. Depending on
+ * where the failure occurred the bytes may have been transformed
+ * (XORed) or left as wire ciphertext. Flag it so that, during a
+ * TLS 1.3 rekey transition, the re-encrypt path undoes the
+ * transform on any XORed frag of a mixed record while software
+ * re-authenticates; a non-mixed record stays wire ciphertext and
+ * is decrypted directly.
+ */
+ skb->decrypt_failed = 1;
+ fallthrough;
+ default: /* CQE_TLS_OFFLOAD_NOT_DECRYPTED: */
stats->tls_err++;
break;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
index 570a912dd6faf..f3f1be1d40343 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c
@@ -6,6 +6,7 @@
enum {
MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2 = 0x2,
+ MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_3 = 0x3,
};
enum {
@@ -15,8 +16,10 @@ enum {
#define EXTRACT_INFO_FIELDS do { \
salt = info->salt; \
rec_seq = info->rec_seq; \
+ iv = info->iv; \
salt_sz = sizeof(info->salt); \
rec_seq_sz = sizeof(info->rec_seq); \
+ iv_sz = sizeof(info->iv); \
} while (0)
static void
@@ -24,9 +27,9 @@ fill_static_params(struct mlx5_wqe_tls_static_params_seg *params,
union mlx5e_crypto_info *crypto_info,
u32 key_id, u32 resync_tcp_sn)
{
+ u16 salt_sz, rec_seq_sz, iv_sz;
+ char *salt, *rec_seq, *iv;
char *initial_rn, *gcm_iv;
- u16 salt_sz, rec_seq_sz;
- char *salt, *rec_seq;
u8 tls_version;
u8 *ctx;
@@ -59,7 +62,12 @@ fill_static_params(struct mlx5_wqe_tls_static_params_seg *params,
memcpy(gcm_iv, salt, salt_sz);
memcpy(initial_rn, rec_seq, rec_seq_sz);
- tls_version = MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2;
+ if (crypto_info->crypto_info.version == TLS_1_3_VERSION) {
+ memcpy(gcm_iv + salt_sz, iv, iv_sz);
+ tls_version = MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_3;
+ } else {
+ tls_version = MLX5E_STATIC_PARAMS_CONTEXT_TLS_1_2;
+ }
MLX5_SET(tls_static_params, ctx, tls_version, tls_version);
MLX5_SET(tls_static_params, ctx, const_1, 1);
diff --git a/drivers/net/ethernet/netronome/nfp/crypto/tls.c b/drivers/net/ethernet/netronome/nfp/crypto/tls.c
index 9983d7aa2b9cd..13864c6a55dce 100644
--- a/drivers/net/ethernet/netronome/nfp/crypto/tls.c
+++ b/drivers/net/ethernet/netronome/nfp/crypto/tls.c
@@ -287,6 +287,9 @@ nfp_net_tls_add(struct net_device *netdev, struct sock *sk,
BUILD_BUG_ON(offsetof(struct nfp_net_tls_offload_ctx, rx_end) >
TLS_DRIVER_STATE_SIZE_RX);
+ if (crypto_info->version != TLS_1_2_VERSION)
+ return -EOPNOTSUPP;
+
if (!nfp_net_cipher_supported(nn, crypto_info->cipher_type, direction))
return -EOPNOTSUPP;
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 421f6fc454511..5da2c1149d982 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -851,6 +851,10 @@ enum skb_tstamp_type {
* unreadable.
* @dst_pending_confirm: need to confirm neighbour
* @decrypted: Decrypted SKB
+ * @decrypt_failed: hardware could not authenticate this skb's TLS payload.
+ * The payload may have been transformed (XORed) or left as wire
+ * ciphertext, so software must re-authenticate the record and undo the
+ * transform on any XORed fragment before it can be decrypted
* @slow_gro: state present at GRO time, slower prepare step required
* @tstamp_type: When set, skb->tstamp has the
* delivery_time clock base of skb->tstamp.
@@ -1025,6 +1029,7 @@ struct sk_buff {
#endif
#ifdef CONFIG_SKB_DECRYPTED
__u8 decrypted:1;
+ __u8 decrypt_failed:1;
#endif
__u8 slow_gro:1;
#if IS_ENABLED(CONFIG_IP_SCTP)
@@ -1716,6 +1721,7 @@ static inline void skb_copy_decrypted(struct sk_buff *to,
{
#ifdef CONFIG_SKB_DECRYPTED
to->decrypted = from->decrypted;
+ to->decrypt_failed = from->decrypt_failed;
#endif
}
diff --git a/include/net/tcp.h b/include/net/tcp.h
index 5e5f5f9b89a38..8c6d90e962c43 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -2340,6 +2340,15 @@ static inline void tcp_write_collapse_fence(struct sock *sk)
{
struct sk_buff *skb = tcp_write_queue_tail(sk);
+ /* When nothing is queued for transmit, the last skb of the current
+ * state is the rtx queue tail (its end_seq == snd_nxt == write_seq).
+ * Fence that instead, otherwise the boundary is left unmarked and a
+ * later tcp_retrans_try_collapse()/tcp_shift_skb_data() can merge it
+ * with the first skb of the next state across the fence (they only test
+ * the tail's EOR, not skb->decrypted).
+ */
+ if (!skb)
+ skb = tcp_rtx_queue_tail(sk);
if (skb)
TCP_SKB_CB(skb)->eor = 1;
}
diff --git a/include/net/tls.h b/include/net/tls.h
index e57bef58851ea..6844a685d6e08 100644
--- a/include/net/tls.h
+++ b/include/net/tls.h
@@ -155,6 +155,22 @@ struct tls_record_info {
skb_frag_t frags[MAX_SKB_FRAGS];
};
+struct cipher_context {
+ char iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE];
+ char rec_seq[TLS_MAX_REC_SEQ_SIZE];
+};
+
+union tls_crypto_context {
+ struct tls_crypto_info info;
+ union {
+ struct tls12_crypto_info_aes_gcm_128 aes_gcm_128;
+ struct tls12_crypto_info_aes_gcm_256 aes_gcm_256;
+ struct tls12_crypto_info_chacha20_poly1305 chacha20_poly1305;
+ struct tls12_crypto_info_sm4_gcm sm4_gcm;
+ struct tls12_crypto_info_sm4_ccm sm4_ccm;
+ };
+};
+
#define TLS_DRIVER_STATE_SIZE_TX 16
struct tls_offload_context_tx {
struct crypto_aead *aead_send;
@@ -169,6 +185,14 @@ struct tls_offload_context_tx {
void (*sk_destruct)(struct sock *sk);
struct work_struct destruct_work;
struct tls_context *ctx;
+
+ struct {
+ struct tls_sw_context_tx sw; /* SW context for new key */
+ struct cipher_context tx; /* IV, rec_seq for new key */
+ union tls_crypto_context crypto_send; /* Crypto for new key */
+ struct tls_record_info *start_marker;
+ } rekey;
+
/* The TLS layer reserves room for driver specific state
* Currently the belief is that there is not enough
* driver specific state to justify another layer of indirection
@@ -187,28 +211,46 @@ enum tls_context_flags {
* to be atomic.
*/
TLS_TX_SYNC_SCHED = 1,
- /* tls_dev_del was called for the RX side, device state was released,
- * but tls_ctx->netdev might still be kept, because TX-side driver
- * resources might not be released yet. Used to prevent the second
- * tls_dev_del call in tls_device_down if it happens simultaneously.
+ /* tls_dev_del was called for the RX side, releasing the NIC's RX
+ * offload context, while tls_ctx->netdev is still kept (TX-side driver
+ * resources may not be released yet, or a rekey is about to re-add the
+ * context). Set in that case, and during a rekey before re-add, and
+ * cleared when tls_dev_add re-establishes the context. Readers use it to
+ * avoid a second tls_dev_del and to suppress resync while the NIC has no
+ * key. tls_device_down() sets it too, so the rekey paths can test the bit
+ * alone.
*/
TLS_RX_DEV_CLOSED = 2,
-};
-
-struct cipher_context {
- char iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE];
- char rec_seq[TLS_MAX_REC_SEQ_SIZE];
-};
-
-union tls_crypto_context {
- struct tls_crypto_info info;
- union {
- struct tls12_crypto_info_aes_gcm_128 aes_gcm_128;
- struct tls12_crypto_info_aes_gcm_256 aes_gcm_256;
- struct tls12_crypto_info_chacha20_poly1305 chacha20_poly1305;
- struct tls12_crypto_info_sm4_gcm sm4_gcm;
- struct tls12_crypto_info_sm4_ccm sm4_ccm;
- };
+ /* TX HW context has been tls_dev_del()'d (mid-rekey before the re-add,
+ * after a failed re-add, or by tls_device_down()); prevents a second
+ * tls_dev_del. Cleared when tls_dev_add re-establishes the context.
+ */
+ TLS_TX_DEV_CLOSED = 3,
+ /* TX rekey is pending, waiting for old-key data to be ACKed.
+ * While set, new data uses SW path with new key, HW keeps old key
+ * for retransmissions.
+ */
+ TLS_TX_REKEY_PENDING = 4,
+ /* All old-key data has been ACKed, ready to install new key in HW. */
+ TLS_TX_REKEY_READY = 5,
+ /* HW rekey failed; TX stays on the SW rekey context until the next
+ * KeyUpdate re-arms the transition (tls_device_start_rekey()). Also
+ * stops tls_tcp_clean_acked() from re-setting TLS_TX_REKEY_READY.
+ */
+ TLS_TX_REKEY_FAILED = 6,
+ /* A rekey has completed on this socket at least once; that arms
+ * tls_tx_drop_acked_clone() (see its header for the rationale). WARN
+ * avoidance only.
+ */
+ TLS_TX_REKEY_FLOOR = 7,
+ /* The RX side fell back to SW decryption during a rekey (tls_dev_add()
+ * failed, or the netdev is gone) and the socket has been moved from the
+ * TlsCurrRxDevice to the TlsCurrRxSw gauge while rx_conf stays TLS_HW.
+ * Accounting only: the functional state is TLS_RX_DEV_{DEGRADED,CLOSED}.
+ * Cleared, moving the socket back, when a later rekey re-adds the NIC
+ * context. Mirrors TLS_TX_REKEY_FAILED for the close-time decrement.
+ */
+ TLS_RX_REKEY_FAILED = 8,
};
struct tls_prot_info {
@@ -257,6 +299,20 @@ struct tls_context {
*/
unsigned long flags;
+ struct {
+ /* TCP sequence number boundary for pending rekey.
+ * Packets with seq < this use old key, >= use new key.
+ */
+ u32 boundary_seq;
+
+ /* SW encryption contexts for the new key, non-NULL only while
+ * TLS_TX_REKEY_{PENDING,FAILED}; consulted by tls_sw_ctx_tx() and
+ * tls_tx_cipher_ctx().
+ */
+ struct tls_sw_context_tx *sw_ctx;
+ struct cipher_context *cipher_ctx;
+ } rekey;
+
/* cache cold stuff */
struct proto *sk_proto;
struct sock *sk;
@@ -315,6 +371,14 @@ struct tls_offload_context_rx {
u8 resync_nh_reset:1;
/* CORE_NEXT_HINT-only member, but use the hole here */
u8 resync_nh_do_now:1;
+ /* tls_dev_add deferred until old key is freed */
+ u8 dev_add_pending:1;
+ struct {
+ struct crypto_aead *old_aead_recv; /* old key AEAD cipher */
+ char old_iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE]; /* old key IV */
+ char old_rec_seq[TLS_MAX_REC_SEQ_SIZE]; /* old key TLS record seq */
+ u32 old_nic_boundary; /* TCP seq below which the NIC may have used the old key */
+ } rekey;
union {
/* TLS_OFFLOAD_SYNC_TYPE_DRIVER_REQ */
struct {
@@ -356,15 +420,38 @@ tls_validate_xmit_skb(struct sock *sk, struct net_device *dev,
struct sk_buff *
tls_validate_xmit_skb_sw(struct sock *sk, struct net_device *dev,
struct sk_buff *skb);
+struct sk_buff *
+tls_validate_xmit_skb_rekey(struct sock *sk, struct net_device *dev,
+ struct sk_buff *skb);
static inline bool tls_is_skb_tx_device_offloaded(const struct sk_buff *skb)
{
#ifdef CONFIG_TLS_DEVICE
struct sock *sk = skb->sk;
+ typeof(sk->sk_validate_xmit_skb) validate;
+
+ if (!sk || !sk_fullsock(sk))
+ return false;
- return sk && sk_fullsock(sk) &&
- (smp_load_acquire(&sk->sk_validate_xmit_skb) ==
- &tls_validate_xmit_skb);
+ /* Pairs with the smp_store_release() that installs or swaps the
+ * validator (tls_set_device_offload() / tls_device_start_rekey()): the
+ * pointer read here is published together with the offload state it
+ * guards, so a non-NULL validator implies that state is visible.
+ */
+ validate = smp_load_acquire(&sk->sk_validate_xmit_skb);
+ if (likely(validate == &tls_validate_xmit_skb))
+ return true;
+
+ /* A TX rekey (tls_device_start_rekey()) can swap in the rekey validator
+ * between this skb's validate_xmit_skb(), where the old validator
+ * passed it through as HW-offload plaintext, and here. A skb->decrypted
+ * skb under the rekey validator is therefore that straddler: old-key
+ * plaintext whose HW context is still installed (tls_dev_del() runs in
+ * tls_device_complete_rekey() only after a synchronize_net() that drains
+ * this in-flight xmit), so the NIC must still encrypt it. Everything else
+ * the rekey validator emits is ciphertext (skb->decrypted == 0).
+ */
+ return validate == &tls_validate_xmit_skb_rekey && skb_is_decrypted(skb);
#else
return false;
#endif
@@ -389,9 +476,25 @@ static inline struct tls_sw_context_rx *tls_sw_ctx_rx(
static inline struct tls_sw_context_tx *tls_sw_ctx_tx(
const struct tls_context *tls_ctx)
{
+ struct tls_sw_context_tx *rekey_ctx = READ_ONCE(tls_ctx->rekey.sw_ctx);
+
+ if (unlikely(rekey_ctx))
+ return rekey_ctx;
+
return (struct tls_sw_context_tx *)tls_ctx->priv_ctx_tx;
}
+static inline struct cipher_context *tls_tx_cipher_ctx(
+ const struct tls_context *tls_ctx)
+{
+ struct cipher_context *rekey_ctx = READ_ONCE(tls_ctx->rekey.cipher_ctx);
+
+ if (unlikely(rekey_ctx))
+ return rekey_ctx;
+
+ return (struct cipher_context *)&tls_ctx->tx;
+}
+
static inline struct tls_offload_context_tx *
tls_offload_ctx_tx(const struct tls_context *tls_ctx)
{
diff --git a/include/uapi/linux/snmp.h b/include/uapi/linux/snmp.h
index 49f5640092a0d..423aec9ae4cac 100644
--- a/include/uapi/linux/snmp.h
+++ b/include/uapi/linux/snmp.h
@@ -369,6 +369,12 @@ enum
LINUX_MIB_TLSTXREKEYOK, /* TlsTxRekeyOk */
LINUX_MIB_TLSTXREKEYERROR, /* TlsTxRekeyError */
LINUX_MIB_TLSRXREKEYRECEIVED, /* TlsRxRekeyReceived */
+ LINUX_MIB_TLSTXREKEYFALLBACK, /* TlsTxRekeyFallback */
+ LINUX_MIB_TLSRXREKEYFALLBACK, /* TlsRxRekeyFallback */
+ LINUX_MIB_TLSCURRTXREKEY, /* TlsCurrTxRekey */
+ LINUX_MIB_TLSCURRRXREKEY, /* TlsCurrRxRekey */
+ LINUX_MIB_TLSTXREKEYABORTED, /* TlsTxRekeyAborted */
+ LINUX_MIB_TLSRXREKEYABORTED, /* TlsRxRekeyAborted */
__LINUX_MIB_TLSMAX
};
diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index 6f6a6f0d5eb0d..fc8ef0d13f5e7 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -285,6 +285,15 @@ static struct sk_buff *dequeue_skb(struct Qdisc *q, bool *validate,
*validate = false;
if (xfrm_offload(skb))
*validate = true;
+ /* A still-cleartext skb of a crypto-offloaded socket was validated
+ * against that socket's offload state at the time. That state
+ * (sk->sk_validate_xmit_skb) can change while the skb is parked here
+ * e.g. a TLS key update or offload teardown, so re-validate it,
+ * letting the current callback decide how it reaches the wire instead
+ * of emitting now-unencrypted plaintext.
+ */
+ if (skb_is_decrypted(skb))
+ *validate = true;
/* check the reason of requeuing without tx lock first */
txq = skb_get_tx_queue(txq->dev, skb);
if (!netif_xmit_frozen_or_stopped(txq)) {
diff --git a/net/tls/tls.h b/net/tls/tls.h
index 60a37bdaaa250..5d8f4d458df8a 100644
--- a/net/tls/tls.h
+++ b/net/tls/tls.h
@@ -147,13 +147,31 @@ void tls_strp_abort_strp(struct tls_strparser *strp, int err);
int init_prot_info(struct tls_prot_info *prot,
const struct tls_crypto_info *crypto_info,
const struct tls_cipher_desc *cipher_desc);
+/* tls_sw_ctx_init() and tls_sw_ctx_finalize() are two halves of installing
+ * a SW crypto context, split so the device path can attach the NIC between
+ * them. finalize() may only be called after an init() that returned 0, and
+ * both must be called with the same tx and new_crypto_info; on a rekey
+ * (new_crypto_info != NULL) the two must also see the same
+ * new_crypto_info->cipher_type. finalize() commits state and cannot fail,
+ * so violating this leaves the context inconsistent without any error.
+ */
+int tls_sw_ctx_init(struct sock *sk, int tx,
+ struct tls_crypto_info *new_crypto_info);
+void tls_sw_ctx_finalize(struct sock *sk, int tx,
+ struct tls_crypto_info *new_crypto_info);
int tls_set_sw_offload(struct sock *sk, int tx,
struct tls_crypto_info *new_crypto_info);
void tls_update_rx_zc_capable(struct tls_context *tls_ctx);
void tls_sw_strparser_arm(struct sock *sk, struct tls_context *ctx);
void tls_sw_strparser_done(struct tls_context *tls_ctx);
int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size);
+int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size);
+void tls_sw_ctx_tx_init(struct sock *sk, struct tls_sw_context_tx *sw_ctx);
+int tls_sw_drain_tx(struct sock *sk, struct tls_context *ctx, int flags);
+int tls_encrypt_async_wait(struct tls_sw_context_tx *ctx);
+int tls_sw_push_pending_record(struct sock *sk, int flags);
void tls_sw_splice_eof(struct socket *sock);
+void tls_sw_splice_eof_locked(struct socket *sock);
void tls_sw_cancel_work_tx(struct tls_context *tls_ctx);
void tls_sw_release_resources_tx(struct sock *sk);
void tls_sw_free_ctx_tx(struct tls_context *tls_ctx);
@@ -230,10 +248,13 @@ static inline bool tls_strp_msg_mixed_decrypted(struct tls_sw_context_rx *ctx)
#ifdef CONFIG_TLS_DEVICE
int tls_device_init(void);
void tls_device_cleanup(void);
-int tls_set_device_offload(struct sock *sk);
+int tls_set_device_offload(struct sock *sk,
+ struct tls_crypto_info *crypto_info);
void tls_device_free_resources_tx(struct sock *sk);
-int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx);
+int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,
+ struct tls_crypto_info *crypto_info);
void tls_device_offload_cleanup_rx(struct sock *sk);
+void tls_device_rx_del_key(struct sock *sk, struct tls_context *ctx);
void tls_device_rx_resync_new_rec(struct sock *sk, u32 rcd_len, u32 seq);
int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx);
#else
@@ -241,7 +262,7 @@ static inline int tls_device_init(void) { return 0; }
static inline void tls_device_cleanup(void) {}
static inline int
-tls_set_device_offload(struct sock *sk)
+tls_set_device_offload(struct sock *sk, struct tls_crypto_info *crypto_info)
{
return -EOPNOTSUPP;
}
@@ -249,13 +270,16 @@ tls_set_device_offload(struct sock *sk)
static inline void tls_device_free_resources_tx(struct sock *sk) {}
static inline int
-tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
+tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,
+ struct tls_crypto_info *crypto_info)
{
return -EOPNOTSUPP;
}
static inline void tls_device_offload_cleanup_rx(struct sock *sk) {}
static inline void
+tls_device_rx_del_key(struct sock *sk, struct tls_context *ctx) {}
+static inline void
tls_device_rx_resync_new_rec(struct sock *sk, u32 rcd_len, u32 seq) {}
static inline int
diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
index f11d0528fc431..5f45c097bad3c 100644
--- a/net/tls/tls_device.c
+++ b/net/tls/tls_device.c
@@ -57,11 +57,28 @@ static struct page *dummy_page;
static void tls_device_free_ctx(struct tls_context *ctx)
{
- if (ctx->tx_conf == TLS_HW)
- kfree(tls_offload_ctx_tx(ctx));
+ if (ctx->tx_conf == TLS_HW) {
+ struct tls_offload_context_tx *offload_ctx =
+ tls_offload_ctx_tx(ctx);
+
+ kfree(offload_ctx->rekey.start_marker);
+ memzero_explicit(&offload_ctx->rekey,
+ sizeof(offload_ctx->rekey));
+ kfree(offload_ctx);
+ }
+
+ if (ctx->rx_conf == TLS_HW) {
+ struct tls_offload_context_rx *offload_ctx =
+ tls_offload_ctx_rx(ctx);
- if (ctx->rx_conf == TLS_HW)
- kfree(tls_offload_ctx_rx(ctx));
+ /* Normally freed and NULLed in tls_device_offload_cleanup_rx();
+ * free defensively here so a future path can't leak the tfm.
+ */
+ crypto_free_aead(offload_ctx->rekey.old_aead_recv);
+ memzero_explicit(&offload_ctx->rekey,
+ sizeof(offload_ctx->rekey));
+ kfree(offload_ctx);
+ }
tls_ctx_free(NULL, ctx);
}
@@ -79,7 +96,9 @@ static void tls_device_tx_del_task(struct work_struct *work)
netdev = rcu_dereference_protected(ctx->netdev,
!refcount_read(&ctx->refcount));
- netdev->tlsdev_ops->tls_dev_del(netdev, ctx, TLS_OFFLOAD_CTX_DIR_TX);
+ if (!test_bit(TLS_TX_DEV_CLOSED, &ctx->flags))
+ netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
+ TLS_OFFLOAD_CTX_DIR_TX);
dev_put(netdev);
ctx->netdev = NULL;
tls_device_free_ctx(ctx);
@@ -138,6 +157,174 @@ static struct net_device *get_netdev_for_sock(struct sock *sk)
return lowest_dev;
}
+static int tls_device_dev_add_tx(struct sock *sk, struct net_device *netdev,
+ struct tls_crypto_info *crypto_info,
+ u32 write_seq)
+{
+ const struct tls_cipher_desc *cipher_desc;
+ char *rec_seq;
+ int rc;
+
+ cipher_desc = get_cipher_desc(crypto_info->cipher_type);
+ DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
+
+ rc = netdev->tlsdev_ops->tls_dev_add(netdev, sk, TLS_OFFLOAD_CTX_DIR_TX,
+ crypto_info, write_seq);
+ rec_seq = crypto_info_rec_seq(crypto_info, cipher_desc);
+ trace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_TX,
+ write_seq, rec_seq, rc);
+ return rc;
+}
+
+/* Caller controls locking: initial-offload path is lock-free (pre-publish);
+ * rekey path holds offload_ctx->lock.
+ */
+static void tls_device_add_start_marker(struct sock *sk,
+ struct tls_offload_context_tx *offload_ctx,
+ struct tls_record_info *start_marker_record)
+{
+ start_marker_record->end_seq = tcp_sk(sk)->write_seq;
+ start_marker_record->len = 0;
+ start_marker_record->num_frags = 0;
+ list_add_tail_rcu(&start_marker_record->list, &offload_ctx->records_list);
+}
+
+static void tls_device_commit_start_marker(struct sock *sk,
+ struct tls_offload_context_tx *offload_ctx,
+ struct tls_record_info *start_marker_record)
+{
+ tls_device_add_start_marker(sk, offload_ctx, start_marker_record);
+
+ /* TLS offload is greatly simplified if we don't send
+ * SKBs where only part of the payload needs to be encrypted.
+ * So mark the last skb in the write queue as end of record.
+ */
+ tcp_write_collapse_fence(sk);
+}
+
+/* Account a rekey that could not (re)install the RX key on the NIC. The event
+ * counter is bumped every time; the gauges move only on the first fallback
+ * since the socket was last offloaded, so the recurring post-NETDEV_DOWN
+ * rekeys and repeated failed adds do not drift them. The matching move back is
+ * in tls_device_dev_add_rx(); the close-time decrement keys off the bit.
+ */
+static void tls_device_rx_rekey_fallback(struct sock *sk,
+ struct tls_context *tls_ctx)
+{
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYFALLBACK);
+ if (!test_and_set_bit(TLS_RX_REKEY_FAILED, &tls_ctx->flags)) {
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXSW);
+ }
+}
+
+static int tls_device_dev_add_rx(struct sock *sk, struct tls_context *tls_ctx,
+ struct net_device *netdev,
+ struct tls_crypto_info *crypto_info,
+ u32 cur_seq, bool is_rekey)
+{
+ const struct tls_cipher_desc *cipher_desc;
+ char *rec_seq;
+ int rc;
+
+ cipher_desc = get_cipher_desc(crypto_info->cipher_type);
+ DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
+
+ rc = netdev->tlsdev_ops->tls_dev_add(netdev, sk,
+ TLS_OFFLOAD_CTX_DIR_RX,
+ crypto_info, cur_seq);
+ rec_seq = crypto_info_rec_seq(crypto_info, cipher_desc);
+ trace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_RX,
+ cur_seq, rec_seq, rc);
+ if (!rc) {
+ clear_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags);
+ clear_bit(TLS_RX_DEV_CLOSED, &tls_ctx->flags);
+ /* Back on the NIC after an earlier SW fallback: undo its move. */
+ if (test_and_clear_bit(TLS_RX_REKEY_FAILED, &tls_ctx->flags)) {
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXSW);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);
+ }
+ if (is_rekey)
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);
+ } else if (is_rekey) {
+ set_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags);
+ set_bit(TLS_RX_DEV_CLOSED, &tls_ctx->flags);
+ tls_device_rx_rekey_fallback(sk, tls_ctx);
+ }
+ return rc;
+}
+
+static void tls_device_deferred_dev_add_rx(struct sock *sk,
+ struct tls_context *tls_ctx,
+ struct tls_offload_context_rx *ctx,
+ u32 rec_start_seq)
+{
+ const struct tls_cipher_desc *cipher_desc;
+ union tls_crypto_context crypto_ctx;
+ struct net_device *netdev;
+
+ ctx->dev_add_pending = 0;
+
+ /* crypto_recv.info.rec_seq is frozen at the value setsockopt() passed
+ * in: the new key's first record number. The records that drained
+ * between setsockopt() and this boundary crossing were SW-decrypted
+ * under the new key and advanced tls_ctx->rx.rec_seq, so the record
+ * starting at rec_start_seq, the one being decrypted right now,
+ * before tls_rx_one_record() calls tls_advance_record_sn(), is
+ * numbered by rx.rec_seq, not by the blob. Hand the NIC the live
+ * (TCP seq, record number) pair, as getsockopt(TLS_RX) already does.
+ */
+ cipher_desc = get_cipher_desc(tls_ctx->crypto_recv.info.cipher_type);
+ DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
+ crypto_ctx = tls_ctx->crypto_recv;
+ memcpy(crypto_info_rec_seq(&crypto_ctx.info, cipher_desc),
+ tls_ctx->rx.rec_seq, cipher_desc->rec_seq);
+
+ down_read(&device_offload_lock);
+ netdev = rcu_dereference_protected(tls_ctx->netdev,
+ lockdep_is_held(&device_offload_lock));
+ if (netdev)
+ tls_device_dev_add_rx(sk, tls_ctx, netdev,
+ &crypto_ctx.info,
+ rec_start_seq, true);
+ else
+ tls_device_rx_rekey_fallback(sk, tls_ctx);
+ up_read(&device_offload_lock);
+ memzero_explicit(&crypto_ctx, sizeof(crypto_ctx));
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXREKEY);
+}
+
+/* Retire the NIC's RX key when a KeyUpdate record is decoded (from
+ * tls_check_pending_rekey(), lock_sock held). The NIC must lose the old key
+ * now, before it transforms further post-KeyUpdate records that are new-key on
+ * the wire. TLS_RX_DEV_CLOSED is re-tested under device_offload_lock because
+ * tls_device_down() can run in between; synchronize_net() drains the RX path
+ * before the driver frees its context.
+ */
+void tls_device_rx_del_key(struct sock *sk, struct tls_context *ctx)
+{
+ struct net_device *netdev;
+
+ if (ctx->rx_conf != TLS_HW)
+ return;
+ if (test_bit(TLS_RX_DEV_CLOSED, &ctx->flags))
+ return;
+
+ down_read(&device_offload_lock);
+ netdev = rcu_dereference_protected(ctx->netdev,
+ lockdep_is_held(&device_offload_lock));
+ if (!netdev || test_bit(TLS_RX_DEV_CLOSED, &ctx->flags)) {
+ up_read(&device_offload_lock);
+ return;
+ }
+
+ set_bit(TLS_RX_DEV_CLOSED, &ctx->flags);
+ synchronize_net();
+ netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
+ TLS_OFFLOAD_CTX_DIR_RX);
+ up_read(&device_offload_lock);
+}
+
static void destroy_record(struct tls_record_info *record)
{
int i;
@@ -159,6 +346,57 @@ static void delete_all_records(struct tls_offload_context_tx *offload_ctx)
offload_ctx->retransmit_hint = NULL;
}
+static void tls_device_commit_rekey_marker(struct sock *sk,
+ struct tls_offload_context_tx *offload_ctx,
+ struct tls_record_info *start_marker_record)
+{
+ struct tls_record_info *info, *temp;
+ unsigned long flags;
+ __be64 rcd_sn;
+
+ spin_lock_irqsave(&offload_ctx->lock, flags);
+
+ /* The deferred path reaches here with an empty list; the inline
+ * path may still hold the old start marker (never a real record,
+ * since tls_has_unacked_records() was false). Only markers are
+ * ever at the head, so stop at the first non-marker.
+ */
+ list_for_each_entry_safe(info, temp, &offload_ctx->records_list, list) {
+ if (!tls_record_is_start_marker(info))
+ break;
+ list_del(&info->list);
+ destroy_record(info);
+ }
+ offload_ctx->retransmit_hint = NULL;
+
+ memcpy(&rcd_sn, offload_ctx->rekey.tx.rec_seq, sizeof(rcd_sn));
+ offload_ctx->unacked_record_sn = be64_to_cpu(rcd_sn) - 1;
+
+ tls_device_add_start_marker(sk, offload_ctx, start_marker_record);
+
+ spin_unlock_irqrestore(&offload_ctx->lock, flags);
+
+ tcp_write_collapse_fence(sk);
+}
+
+static bool tls_has_unacked_records(struct tls_offload_context_tx *offload_ctx)
+{
+ struct tls_record_info *info;
+ bool has_unacked = false;
+ unsigned long flags;
+
+ spin_lock_irqsave(&offload_ctx->lock, flags);
+ list_for_each_entry(info, &offload_ctx->records_list, list) {
+ if (!tls_record_is_start_marker(info)) {
+ has_unacked = true;
+ break;
+ }
+ }
+ spin_unlock_irqrestore(&offload_ctx->lock, flags);
+
+ return has_unacked;
+}
+
static void tls_tcp_clean_acked(struct sock *sk, u32 acked_seq)
{
struct tls_context *tls_ctx = tls_get_ctx(sk);
@@ -187,6 +425,19 @@ static void tls_tcp_clean_acked(struct sock *sk, u32 acked_seq)
}
ctx->unacked_record_sn += deleted_records;
+
+ /* Once all old-key HW records are ACKed, set REKEY_READY to
+ * let sendmsg know it can finish the rekey and switch back
+ * to HW offload.
+ */
+ if (test_bit(TLS_TX_REKEY_PENDING, &tls_ctx->flags) &&
+ !test_bit(TLS_TX_REKEY_FAILED, &tls_ctx->flags)) {
+ u32 boundary_seq = READ_ONCE(tls_ctx->rekey.boundary_seq);
+
+ if (!before(acked_seq, boundary_seq))
+ set_bit(TLS_TX_REKEY_READY, &tls_ctx->flags);
+ }
+
spin_unlock_irqrestore(&ctx->lock, flags);
}
@@ -217,7 +468,15 @@ void tls_device_free_resources_tx(struct sock *sk)
{
struct tls_context *tls_ctx = tls_get_ctx(sk);
- tls_free_partial_record(sk, tls_ctx);
+ if (unlikely(tls_ctx->rekey.sw_ctx))
+ tls_sw_release_resources_tx(sk);
+ else
+ tls_free_partial_record(sk, tls_ctx);
+
+ if (test_bit(TLS_TX_REKEY_PENDING, &tls_ctx->flags)) {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYABORTED);
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXREKEY);
+ }
}
void tls_offload_tx_resync_request(struct sock *sk, u32 got_seq, u32 exp_seq)
@@ -317,25 +576,34 @@ static void tls_device_record_close(struct sock *sk,
unsigned char record_type)
{
struct tls_prot_info *prot = &ctx->prot_info;
- struct page_frag dummy_tag_frag;
-
- /* append tag
- * device will fill in the tag, we just need to append a placeholder
- * use socket memory to improve coalescing (re-using a single buffer
- * increases frag count)
- * if we can't allocate memory now use the dummy page
+ int tail = prot->tag_size + prot->tail_size;
+
+ /* Append tail: tag for TLS 1.2, content_type + tag for TLS 1.3.
+ * Device fills in the tag, we just need to append a placeholder.
+ * Use socket memory to improve coalescing (re-using a single buffer
+ * increases frag count); if allocation fails use dummy_page
+ * (offset = record_type gives correct content_type byte via
+ * identity mapping)
*/
- if (unlikely(pfrag->size - pfrag->offset < prot->tag_size) &&
- !skb_page_frag_refill(prot->tag_size, pfrag, sk->sk_allocation)) {
- dummy_tag_frag.page = dummy_page;
- dummy_tag_frag.offset = 0;
- pfrag = &dummy_tag_frag;
+ if (unlikely(!pfrag->page || pfrag->size - pfrag->offset < tail) &&
+ !skb_page_frag_refill(tail, pfrag, sk->sk_allocation)) {
+ struct page_frag dummy_pfrag = {
+ .page = dummy_page,
+ .offset = record_type,
+ };
+ tls_append_frag(record, &dummy_pfrag, tail);
+ } else {
+ if (prot->tail_size) {
+ char *content_type_addr = page_address(pfrag->page) +
+ pfrag->offset;
+ *content_type_addr = record_type;
+ }
+ tls_append_frag(record, pfrag, tail);
}
- tls_append_frag(record, pfrag, prot->tag_size);
/* fill prepend */
tls_fill_prepend(ctx, skb_frag_address(&record->frags[0]),
- record->len - prot->overhead_size,
+ record->len - prot->overhead_size + prot->tail_size,
record_type);
}
@@ -418,6 +686,9 @@ static int tls_device_copy_data(void *addr, size_t bytes, struct iov_iter *i)
return 0;
}
+static int tls_device_complete_rekey(struct sock *sk, struct tls_context *ctx,
+ bool deferred, int push_flags);
+
static int tls_push_data(struct sock *sk,
struct iov_iter *iter,
size_t size, int flags,
@@ -563,18 +834,54 @@ static int tls_push_data(struct sock *sk,
return rc;
}
+/* True while TX is routed through the temporary SW rekey context: a rekey is in
+ * progress (PENDING) or has failed and the socket stays pinned to SW (FAILED).
+ */
+static bool tls_device_tx_uses_sw(const struct tls_context *ctx)
+{
+ return test_bit(TLS_TX_REKEY_PENDING, &ctx->flags) ||
+ test_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+}
+
int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)
{
unsigned char record_type = TLS_RECORD_TYPE_DATA;
struct tls_context *tls_ctx = tls_get_ctx(sk);
int rc;
+ /* Reject unsupported flags up front. tls_push_data() enforces the same
+ * set, but during a rekey the send is routed to tls_sw_sendmsg_locked(),
+ * which is the _locked variant and does not re-check; without this,
+ * MSG_ZEROCOPY / MSG_OOB etc. would reach tcp_sendmsg_locked() on the
+ * kernel-owned record pages while PENDING/FAILED.
+ */
+ if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL |
+ MSG_SPLICE_PAGES | MSG_EOR))
+ return -EOPNOTSUPP;
+
if (!tls_ctx->zerocopy_sendfile)
msg->msg_flags &= ~MSG_SPLICE_PAGES;
mutex_lock(&tls_ctx->tx_lock);
lock_sock(sk);
+ /* Old-key records all ACKed; switch back to HW. */
+ if (test_bit(TLS_TX_REKEY_READY, &tls_ctx->flags)) {
+ rc = tls_device_complete_rekey(sk, tls_ctx, true, msg->msg_flags);
+ /* Non-zero here is the transient -EAGAIN retry,
+ * the next sendmsg retries. Hard failures return 0 after
+ * falling back to SW and emit tls_device_complete_rekey_fail
+ * from the fallback path.
+ */
+ if (rc)
+ trace_tls_device_complete_rekey_retry(sk);
+ }
+
+ if (tls_device_tx_uses_sw(tls_ctx)) {
+ rc = tls_sw_sendmsg_locked(sk, msg, size);
+ goto out;
+ }
+
if (unlikely(msg->msg_controllen)) {
rc = tls_process_cmsg(sk, msg, &record_type);
if (rc)
@@ -603,8 +910,10 @@ void tls_device_splice_eof(struct socket *sock)
mutex_lock(&tls_ctx->tx_lock);
lock_sock(sk);
- if (tls_is_partially_sent_record(tls_ctx) ||
- tls_is_pending_open_record(tls_ctx)) {
+ if (tls_device_tx_uses_sw(tls_ctx)) {
+ tls_sw_splice_eof_locked(sock);
+ } else if (tls_is_partially_sent_record(tls_ctx) ||
+ tls_is_pending_open_record(tls_ctx)) {
iov_iter_bvec(&iter, ITER_SOURCE, NULL, 0, 0);
tls_push_data(sk, &iter, 0, 0, TLS_RECORD_TYPE_DATA);
}
@@ -675,14 +984,30 @@ EXPORT_SYMBOL(tls_get_record);
static int tls_device_push_pending_record(struct sock *sk, int flags)
{
+ struct tls_context *tls_ctx = tls_get_ctx(sk);
struct iov_iter iter;
+ if (tls_device_tx_uses_sw(tls_ctx))
+ return tls_sw_push_pending_record(sk, flags);
+
iov_iter_kvec(&iter, ITER_SOURCE, NULL, 0, 0);
return tls_push_data(sk, &iter, 0, flags, TLS_RECORD_TYPE_DATA);
}
void tls_device_write_space(struct sock *sk, struct tls_context *ctx)
{
+ if (tls_device_tx_uses_sw(ctx)) {
+ struct tls_offload_context_tx *offload_ctx;
+ unsigned long flags;
+
+ offload_ctx = tls_offload_ctx_tx(ctx);
+ spin_lock_irqsave(&offload_ctx->lock, flags);
+ if (tls_device_tx_uses_sw(ctx))
+ tls_sw_write_space(sk, ctx);
+ spin_unlock_irqrestore(&offload_ctx->lock, flags);
+ return;
+ }
+
if (tls_is_partially_sent_record(ctx)) {
gfp_t sk_allocation = sk->sk_allocation;
@@ -785,6 +1110,8 @@ void tls_device_rx_resync_new_rec(struct sock *sk, u32 rcd_len, u32 seq)
return;
if (unlikely(test_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags)))
return;
+ if (unlikely(test_bit(TLS_RX_DEV_CLOSED, &tls_ctx->flags)))
+ return;
prot = &tls_ctx->prot_info;
rx_ctx = tls_offload_ctx_rx(tls_ctx);
@@ -886,6 +1213,7 @@ static int
tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
{
struct tls_sw_context_rx *sw_ctx = tls_sw_ctx_rx(tls_ctx);
+ struct tls_prot_info *prot = &tls_ctx->prot_info;
const struct tls_cipher_desc *cipher_desc;
int err, offset, copy, data_len, pos;
struct sk_buff *skb, *skb_iter;
@@ -897,7 +1225,7 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
rxm = strp_msg(tls_strp_msg(sw_ctx));
- orig_buf = kmalloc(rxm->full_len + TLS_HEADER_SIZE + cipher_desc->iv,
+ orig_buf = kmalloc(rxm->full_len + prot->prepend_size,
sk->sk_allocation);
if (!orig_buf)
return -ENOMEM;
@@ -912,9 +1240,8 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
offset = rxm->offset;
sg_init_table(sg, 1);
- sg_set_buf(&sg[0], buf,
- rxm->full_len + TLS_HEADER_SIZE + cipher_desc->iv);
- err = skb_copy_bits(skb, offset, buf, TLS_HEADER_SIZE + cipher_desc->iv);
+ sg_set_buf(&sg[0], buf, rxm->full_len + prot->prepend_size);
+ err = skb_copy_bits(skb, offset, buf, prot->prepend_size);
if (err)
goto free_buf;
@@ -930,7 +1257,7 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
if (skb_pagelen(skb) > offset) {
copy = min_t(int, skb_pagelen(skb) - offset, data_len);
- if (skb->decrypted) {
+ if (skb->decrypted || skb->decrypt_failed) {
err = skb_store_bits(skb, offset, buf, copy);
if (err)
goto free_buf;
@@ -957,7 +1284,7 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
copy = min_t(int, skb_iter->len - frag_pos,
data_len + rxm->offset - offset);
- if (skb_iter->decrypted) {
+ if (skb_iter->decrypted || skb_iter->decrypt_failed) {
err = skb_store_bits(skb_iter, frag_pos, buf, copy);
if (err)
goto free_buf;
@@ -974,6 +1301,77 @@ tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)
return err;
}
+/*
+ * Reconstruct a boundary record whose frags the NIC XORed with the old key,
+ * then hand it to the SW AEAD under the current (new) key.
+ *
+ * These are deliberately two different keys: the sender has already done its
+ * TX KeyUpdate, so the record on the wire is AEAD-encrypted with the new key,
+ * but the RX NIC still holds the old key and CTR-XORed some frags with the old
+ * keystream. tls_device_reencrypt() must undo that XOR with the *old* key to
+ * restore the pristine new-key ciphertext, so swap the old key in only for the
+ * reconstruction and restore the current key before returning; the SW AEAD
+ * decrypt that follows then runs under the new key, matching the wire record.
+ */
+static int tls_device_reencrypt_old_key(struct sock *sk,
+ struct tls_offload_context_rx *ctx,
+ struct tls_sw_context_rx *sw_ctx,
+ struct tls_context *tls_ctx)
+{
+ struct crypto_aead *saved_aead = sw_ctx->aead_recv;
+ char saved_iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE];
+ char saved_rec_seq[TLS_MAX_REC_SEQ_SIZE];
+ int ret;
+
+ memcpy(saved_iv, tls_ctx->rx.iv, sizeof(saved_iv));
+ memcpy(saved_rec_seq, tls_ctx->rx.rec_seq, sizeof(saved_rec_seq));
+
+ sw_ctx->aead_recv = ctx->rekey.old_aead_recv;
+ memcpy(tls_ctx->rx.iv, ctx->rekey.old_iv, sizeof(ctx->rekey.old_iv));
+ memcpy(tls_ctx->rx.rec_seq, ctx->rekey.old_rec_seq,
+ sizeof(ctx->rekey.old_rec_seq));
+
+ ret = tls_device_reencrypt(sk, tls_ctx);
+
+ memcpy(ctx->rekey.old_rec_seq, tls_ctx->rx.rec_seq,
+ sizeof(ctx->rekey.old_rec_seq));
+
+ sw_ctx->aead_recv = saved_aead;
+ memcpy(tls_ctx->rx.iv, saved_iv, sizeof(saved_iv));
+ memcpy(tls_ctx->rx.rec_seq, saved_rec_seq, sizeof(saved_rec_seq));
+
+ if (ret)
+ return ret;
+
+ tls_bigint_increment(ctx->rekey.old_rec_seq,
+ tls_ctx->prot_info.rec_seq_size);
+ ctx->resync_nh_reset = 1;
+
+ return 0;
+}
+
+/*
+ * TCP sequence of the first byte of the record the strparser currently holds
+ * or is still collecting. In non-copy mode tcp_sk(sk)->copied_seq is left at
+ * the record start until tls_strp_msg_consume(). In copy mode
+ * tls_strp_read_copy() zeroes stm.offset and anchor->len and then
+ * tls_strp_read_copyin() -> tcp_read_sock() advances copied_seq by every byte
+ * it appends to the anchor, a complete parsed-ahead record, a partial one
+ * under rmem pressure, or only header bytes, so subtract anchor->len to get
+ * back to the record start. Both the recv path and the setsockopt rekey path
+ * must classify records against the same start, so share this helper.
+ */
+static u32 tls_device_rx_rec_start(struct sock *sk,
+ struct tls_sw_context_rx *sw_ctx)
+{
+ u32 copied_seq = tcp_sk(sk)->copied_seq;
+
+ if (sw_ctx->strp.copy_mode)
+ return copied_seq - sw_ctx->strp.anchor->len;
+
+ return copied_seq;
+}
+
int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)
{
struct tls_offload_context_rx *ctx = tls_offload_ctx_rx(tls_ctx);
@@ -981,6 +1379,7 @@ int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)
struct sk_buff *skb = tls_strp_msg(sw_ctx);
struct strp_msg *rxm = strp_msg(skb);
int is_decrypted, is_encrypted;
+ u32 rec_start_seq;
if (!tls_strp_msg_mixed_decrypted(sw_ctx)) {
is_decrypted = skb->decrypted;
@@ -990,10 +1389,77 @@ int tls_device_decrypted(struct sock *sk, struct tls_context *tls_ctx)
is_encrypted = 0;
}
- trace_tls_device_decrypted(sk, tcp_sk(sk)->copied_seq - rxm->full_len,
+ rec_start_seq = tls_device_rx_rec_start(sk, sw_ctx);
+
+ trace_tls_device_decrypted(sk, rec_start_seq,
tls_ctx->rx.rec_seq, rxm->full_len,
is_encrypted, is_decrypted);
+ if (unlikely(ctx->rekey.old_aead_recv)) {
+ bool nic_touched = !is_encrypted || skb->decrypt_failed;
+ bool before_nic_boundary;
+
+ /* old_nic_boundary is the TCP stack's view at setsockopt time
+ * (rcv_nxt plus the out-of-order tail), not the NIC's last
+ * transformed byte. A segment the NIC transformed with the old
+ * key before tls_dev_del returned can still be in the RQ/CQ, in
+ * a GRO list or in the socket backlog when that snapshot is
+ * taken and reach TCP later, above it. While old_aead_recv is
+ * held the NIC has no RX context for this socket at all: the
+ * old one was deleted before old_aead_recv was set and the new
+ * one is only installed once it is freed below. So a NIC mark
+ * seen here can only be the old key's transform, wherever the
+ * record sits relative to the snapshot. Slide the boundary out
+ * over such a record instead of retiring the old key on it; the
+ * old key is retired only on a record the NIC never saw.
+ */
+ if (nic_touched &&
+ !before(rec_start_seq, ctx->rekey.old_nic_boundary))
+ ctx->rekey.old_nic_boundary = rec_start_seq + rxm->full_len;
+
+ before_nic_boundary =
+ before(rec_start_seq, ctx->rekey.old_nic_boundary);
+
+ if (before_nic_boundary) {
+ /* Non-mixed (skb->decrypted clear) is untouched wire
+ * ciphertext even if skb->decrypt_failed is set, so advance
+ * old_rec_seq and let the SW AEAD decrypt it directly.
+ * old_rec_seq tracks the stream's record number, which the
+ * NIC also advances for records it did not transform, so
+ * keeping it in step lets a later NIC-touched record be undone
+ * with the right nonce. A mixed record carries NIC-XORed frags
+ * (skb->decrypt_failed or skb->decrypted) and takes the
+ * old-key reencrypt path below, which undoes the transform per
+ * frag before the SW AEAD decrypts.
+ */
+ if (is_encrypted) {
+ tls_bigint_increment(ctx->rekey.old_rec_seq,
+ tls_ctx->prot_info.rec_seq_size);
+ return 0;
+ }
+
+ trace_tls_device_rekey_reencrypt(sk, rec_start_seq,
+ ctx->rekey.old_nic_boundary);
+
+ return tls_device_reencrypt_old_key(sk, ctx,
+ sw_ctx, tls_ctx);
+ }
+
+ trace_tls_device_rekey_done(sk, rec_start_seq,
+ ctx->rekey.old_nic_boundary);
+ crypto_free_aead(ctx->rekey.old_aead_recv);
+ ctx->rekey.old_aead_recv = NULL;
+
+ /* Anchor the NIC on the start of this first post-boundary
+ * record. rec_start_seq already accounts for copy_mode, where
+ * copied_seq has advanced past the record end; using it keeps
+ * the (TCP seq, record number) pair consistent in both modes.
+ */
+ if (ctx->dev_add_pending)
+ tls_device_deferred_dev_add_rx(sk, tls_ctx, ctx,
+ rec_start_seq);
+ }
+
if (unlikely(test_bit(TLS_RX_DEV_DEGRADED, &tls_ctx->flags))) {
if (likely(is_encrypted || is_decrypted))
return is_decrypted;
@@ -1062,62 +1528,457 @@ static struct tls_offload_context_tx *alloc_offload_ctx_tx(struct tls_context *c
return offload_ctx;
}
-int tls_set_device_offload(struct sock *sk)
+/* Build a fresh AEAD tfm for the rekey with the given key, so it can be
+ * swapped in only on success. Re-keying a live tfm in place is not atomic:
+ * a failed crypto_aead_setkey() leaves it with CRYPTO_TFM_NEED_KEY set,
+ * destroying the previous key. Returns an ERR_PTR() on failure.
+ */
+static struct crypto_aead *tls_device_build_rekey_aead(
+ const struct tls_cipher_desc *cipher_desc,
+ char *key, u32 alg_flags)
{
- struct tls_record_info *start_marker_record;
- struct tls_offload_context_tx *offload_ctx;
+ struct crypto_aead *aead;
+ int rc;
+
+ aead = crypto_alloc_aead(cipher_desc->cipher_name, 0, alg_flags);
+ if (IS_ERR(aead))
+ return aead;
+
+ rc = crypto_aead_setkey(aead, key, cipher_desc->key);
+ if (!rc)
+ rc = crypto_aead_setauthsize(aead, cipher_desc->tag);
+ if (rc) {
+ crypto_free_aead(aead);
+ return ERR_PTR(rc);
+ }
+
+ return aead;
+}
+
+static void tls_device_copy_rekey_iv_seq(
+ struct tls_offload_context_tx *offload_ctx,
+ const struct tls_cipher_desc *cipher_desc,
+ char *salt, char *iv, char *rec_seq)
+{
+ memcpy(offload_ctx->rekey.tx.iv, salt, cipher_desc->salt);
+ memcpy(offload_ctx->rekey.tx.iv + cipher_desc->salt, iv,
+ cipher_desc->iv);
+ memcpy(offload_ctx->rekey.tx.rec_seq, rec_seq, cipher_desc->rec_seq);
+}
+
+static int tls_device_init_rekey_sw(struct sock *sk,
+ struct tls_context *ctx,
+ struct tls_offload_context_tx *offload_ctx,
+ struct tls_crypto_info *new_crypto_info)
+{
+ struct tls_sw_context_tx *sw_ctx = &offload_ctx->rekey.sw;
+ const struct tls_cipher_desc *cipher_desc;
+ char *key;
+ int rc;
+
+ cipher_desc = get_cipher_desc(new_crypto_info->cipher_type);
+ DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
+
+ memset(sw_ctx, 0, sizeof(*sw_ctx));
+ tls_sw_ctx_tx_init(sk, sw_ctx);
+
+ key = crypto_info_key(new_crypto_info, cipher_desc);
+ sw_ctx->aead_send = tls_device_build_rekey_aead(cipher_desc, key, 0);
+ if (IS_ERR(sw_ctx->aead_send)) {
+ rc = PTR_ERR(sw_ctx->aead_send);
+ sw_ctx->aead_send = NULL;
+ return rc;
+ }
+
+ return 0;
+}
+
+static int tls_device_start_rekey(struct sock *sk,
+ struct tls_context *ctx,
+ struct tls_offload_context_tx *offload_ctx,
+ struct tls_crypto_info *new_crypto_info)
+{
+ bool rekey_pending = test_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+ bool rekey_failed = test_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+ const struct tls_cipher_desc *cipher_desc;
+ struct crypto_aead *new_aead, *old_aead;
+ char *key, *iv, *rec_seq, *salt;
+ int push_flags = MSG_NOSIGNAL;
+ unsigned long flags;
+ int rc;
+
+ cipher_desc = get_cipher_desc(new_crypto_info->cipher_type);
+ DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
+
+ key = crypto_info_key(new_crypto_info, cipher_desc);
+ iv = crypto_info_iv(new_crypto_info, cipher_desc);
+ rec_seq = crypto_info_rec_seq(new_crypto_info, cipher_desc);
+ salt = crypto_info_salt(new_crypto_info, cipher_desc);
+
+ /* The record flushes below hand the open/partially sent HW record to
+ * TCP and may have to wait for send buffer space. Honour the socket's
+ * non-blocking mode so an O_NONBLOCK application is not put to sleep
+ * inside setsockopt(): it gets -EAGAIN and retries once the socket is
+ * writable. Kernel sockets (no backing file, e.g. nvme-tcp) keep the
+ * blocking semantics, matching how they call sendmsg().
+ */
+ if (sk->sk_socket && sk->sk_socket->file &&
+ (sk->sk_socket->file->f_flags & O_NONBLOCK))
+ push_flags |= MSG_DONTWAIT;
+
+ if (rekey_pending || rekey_failed) {
+ /* Flush any SW open_record before swapping the key. -EINPROGRESS
+ * means an async AEAD accepted the record for encryption; it is a
+ * success, waited for by tls_encrypt_async_wait() just below (as
+ * tls_process_cmsg()/tls_sw_drain_tx() also treat it).
+ */
+ if (tls_is_pending_open_record(ctx)) {
+ rc = ctx->push_pending_record(sk, push_flags);
+ if (rc < 0 && rc != -EINPROGRESS)
+ return rc;
+ }
+
+ /* Wait for in-flight async encryptions submitted to this tfm
+ * with the previous key before changing it.
+ */
+ rc = tls_encrypt_async_wait(&offload_ctx->rekey.sw);
+ if (rc)
+ return rc;
+
+ /* Build the new key into a fresh tfm and swap it in only on
+ * success; A failed rekey here must leave the SW fallback
+ * path able to encrypt.
+ */
+ new_aead = tls_device_build_rekey_aead(cipher_desc, key, 0);
+ if (IS_ERR(new_aead))
+ return PTR_ERR(new_aead);
+
+ old_aead = offload_ctx->rekey.sw.aead_send;
+ offload_ctx->rekey.sw.aead_send = new_aead;
+ crypto_free_aead(old_aead);
+
+ tls_device_copy_rekey_iv_seq(offload_ctx, cipher_desc,
+ salt, iv, rec_seq);
+
+ if (rekey_failed) {
+ /* Re-arm FAILED -> PENDING under device_offload_lock. The
+ * PENDING set and FAILED clear are two stores to ctx->flags,
+ * and tls_device_down() tests !PENDING && !FAILED as two
+ * separate loads; without the lock those loads could straddle
+ * the flip and see neither bit, letting tls_device_down()
+ * install tls_validate_xmit_skb_sw with PENDING set (dropping
+ * all new-key ciphertext). The lock keeps PENDING || FAILED
+ * observable throughout. Non-blocking, so no NETDEV_DOWN stall.
+ */
+ down_read(&device_offload_lock);
+ spin_lock_irqsave(&offload_ctx->lock, flags);
+ WRITE_ONCE(ctx->rekey.boundary_seq, tcp_sk(sk)->snd_una);
+ set_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+ spin_unlock_irqrestore(&offload_ctx->lock, flags);
+ /* Release pairs with test_bit_acquire() in the validator:
+ * a TX seeing FAILED clear must see the fresh boundary_seq.
+ */
+ clear_bit_unlock(TLS_TX_REKEY_FAILED, &ctx->flags);
+ up_read(&device_offload_lock);
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
+ }
+ } else {
+ /* Drain partially sent record and flush open HW record
+ * before switching to SW.
+ */
+ if (tls_is_partially_sent_record(ctx)) {
+ rc = tls_push_partial_record(sk, ctx,
+ MSG_SENDPAGE_DECRYPTED |
+ push_flags);
+ if (rc < 0)
+ return rc;
+ }
+ if (tls_is_pending_open_record(ctx)) {
+ rc = ctx->push_pending_record(sk, push_flags);
+ if (rc < 0)
+ return rc;
+ }
+
+ rc = tls_device_init_rekey_sw(sk, ctx, offload_ctx,
+ new_crypto_info);
+ if (rc)
+ return rc;
+
+ tls_device_copy_rekey_iv_seq(offload_ctx, cipher_desc,
+ salt, iv, rec_seq);
+
+ /* Publish the rekey under device_offload_lock so that setting
+ * TLS_TX_REKEY_PENDING and installing the rekey validator is
+ * atomic against tls_device_down(), which under down_write() tests
+ * !PENDING and installs tls_validate_xmit_skb_sw. Otherwise the two
+ * validator stores could interleave to leave PENDING set with the
+ * SW validator, and every new-key ciphertext (never on the offload
+ * records_list) would then be dropped by tls_sw_fallback(). The
+ * blocking flush and crypto_alloc above deliberately run WITHOUT
+ * this lock, so a stalled peer cannot hold up NETDEV_DOWN (which
+ * takes down_write() under RTNL) or any other down_read() user.
+ */
+ down_read(&device_offload_lock);
+
+ /* Prevent a partial record straddling the SW/HW boundary. */
+ tcp_write_collapse_fence(sk);
+
+ WRITE_ONCE(ctx->rekey.sw_ctx, &offload_ctx->rekey.sw);
+ WRITE_ONCE(ctx->rekey.cipher_ctx, &offload_ctx->rekey.tx);
+
+ spin_lock_irqsave(&offload_ctx->lock, flags);
+ WRITE_ONCE(ctx->rekey.boundary_seq, tcp_sk(sk)->write_seq);
+ set_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+ spin_unlock_irqrestore(&offload_ctx->lock, flags);
+
+ /* Switch to rekey validator; new sends won't use HW offload */
+ smp_store_release(&sk->sk_validate_xmit_skb,
+ tls_validate_xmit_skb_rekey);
+
+ up_read(&device_offload_lock);
+ }
+
+ unsafe_memcpy(&offload_ctx->rekey.crypto_send.info, new_crypto_info,
+ cipher_desc->crypto_info,
+ /* checked in do_tls_setsockopt_conf */);
+ memzero_explicit(new_crypto_info, cipher_desc->crypto_info);
+
+ return 0;
+}
+
+static int tls_device_complete_rekey(struct sock *sk, struct tls_context *ctx,
+ bool deferred, int push_flags)
+{
+ struct tls_offload_context_tx *offload_ctx = tls_offload_ctx_tx(ctx);
+ struct crypto_aead *new_aead, *old_aead, *old_sw_aead;
const struct tls_cipher_desc *cipher_desc;
- struct tls_crypto_info *crypto_info;
- struct tls_prot_info *prot;
struct net_device *netdev;
- struct tls_context *ctx;
- char *iv, *rec_seq;
+ unsigned long flags;
+ char *key;
int rc;
- ctx = tls_get_ctx(sk);
- prot = &ctx->prot_info;
+ cipher_desc = get_cipher_desc(offload_ctx->rekey.crypto_send.info.cipher_type);
+ DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
- if (ctx->priv_ctx_tx)
- return -EEXIST;
+ DEBUG_NET_WARN_ON_ONCE(!offload_ctx->rekey.start_marker);
- netdev = get_netdev_for_sock(sk);
+ rc = tls_sw_drain_tx(sk, ctx, push_flags);
+ /* -EAGAIN (sndbuf full) and a signal (-EINTR/-ERESTARTSYS from
+ * sk_stream_wait_memory()) are transient: leave the rekey PENDING and
+ * retry on the next sendmsg rather than permanently dropping HW offload.
+ * tls_tx_records() likewise passes these through without aborting.
+ */
+ if (rc == -EAGAIN || rc == -EINTR || rc == -ERESTARTSYS)
+ return rc;
+ if (rc)
+ goto rekey_fallback; /* hard failure: fall back to SW */
+
+ down_read(&device_offload_lock);
+
+ netdev = rcu_dereference_protected(ctx->netdev,
+ lockdep_is_held(&device_offload_lock));
if (!netdev) {
- pr_err_ratelimited("%s: netdev not found\n", __func__);
- return -EINVAL;
+ rc = -ENODEV;
+ goto release_lock;
}
- if (!(netdev->features & NETIF_F_HW_TLS_TX)) {
- rc = -EOPNOTSUPP;
- goto release_netdev;
+ /* Drain in-flight xmit users before tls_dev_del() and before freeing the
+ * old fallback aead_send: (1) under the rekey validator a decrypted
+ * straddler may still be inside the driver on the HW context (same swap ->
+ * synchronize_net -> dev_del order as tls_device_down(), which also keeps a
+ * decrypted skb from reaching a torn-down context); (2) pre-boundary
+ * retransmits routed to tls_sw_fallback() read aead_send locklessly. No new
+ * fallback can start here: every pre-boundary record is ACKed and freed, so
+ * fill_sg_in() bails.
+ */
+ synchronize_net();
+
+ if (!test_bit(TLS_TX_DEV_CLOSED, &ctx->flags)) {
+ netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
+ TLS_OFFLOAD_CTX_DIR_TX);
+ set_bit(TLS_TX_DEV_CLOSED, &ctx->flags);
}
- crypto_info = &ctx->crypto_send.info;
- if (crypto_info->version != TLS_1_2_VERSION) {
- rc = -EOPNOTSUPP;
- goto release_netdev;
+ /* Build the new SW-fallback key into a fresh tfm and swap it in only
+ * on success. Doing this while the HW context is torn down
+ * (TLS_TX_DEV_CLOSED set) means a failure falls into rekey_fallback
+ * with HW off, so the SW fallback is coherent, same as a dev_add
+ * failure.
+ */
+ key = crypto_info_key(&offload_ctx->rekey.crypto_send.info, cipher_desc);
+ new_aead = tls_device_build_rekey_aead(cipher_desc, key, CRYPTO_ALG_ASYNC);
+ if (IS_ERR(new_aead)) {
+ rc = PTR_ERR(new_aead);
+ goto release_lock;
}
- cipher_desc = get_cipher_desc(crypto_info->cipher_type);
- if (!cipher_desc || !cipher_desc->offloadable) {
- rc = -EINVAL;
- goto release_netdev;
+ /* crypto_send.info.rec_seq is frozen at setsockopt time; the SW context
+ * advanced rekey.tx.rec_seq for every record it sent, so hand the NIC the
+ * live record number (mirrors the RX deferred add).
+ */
+ memcpy(crypto_info_rec_seq(&offload_ctx->rekey.crypto_send.info, cipher_desc),
+ offload_ctx->rekey.tx.rec_seq, cipher_desc->rec_seq);
+
+ rc = tls_device_dev_add_tx(sk, netdev, &offload_ctx->rekey.crypto_send.info,
+ tcp_sk(sk)->write_seq);
+ if (rc) {
+ crypto_free_aead(new_aead);
+ goto release_lock;
}
- rc = init_prot_info(prot, crypto_info, cipher_desc);
+ /* Point of no return: HW is live with the new key. Swap in the new
+ * fallback tfm and drop the old one; the remaining steps cannot fail.
+ */
+ old_aead = offload_ctx->aead_send;
+ offload_ctx->aead_send = new_aead;
+ crypto_free_aead(old_aead);
+ clear_bit(TLS_TX_DEV_CLOSED, &ctx->flags);
+
+ memcpy(ctx->tx.iv, offload_ctx->rekey.tx.iv,
+ cipher_desc->salt + cipher_desc->iv);
+ memcpy(ctx->tx.rec_seq, offload_ctx->rekey.tx.rec_seq,
+ cipher_desc->rec_seq);
+ unsafe_memcpy(&ctx->crypto_send.info,
+ &offload_ctx->rekey.crypto_send.info,
+ cipher_desc->crypto_info,
+ /* checked during rekey setup */);
+
+ /* Start marker: the NIC passes through everything before
+ * write_seq untouched (it is already SW-encrypted ciphertext),
+ * same as during initial offload setup. Also drops the stale
+ * marker and rebases unacked_record_sn so the record-sequence
+ * bookkeeping stays consistent on the inline path.
+ */
+ tls_device_commit_rekey_marker(sk, offload_ctx,
+ offload_ctx->rekey.start_marker);
+
+ old_sw_aead = tls_sw_ctx_tx(ctx)->aead_send;
+
+ spin_lock_irqsave(&offload_ctx->lock, flags);
+ clear_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+ clear_bit(TLS_TX_REKEY_READY, &ctx->flags);
+ clear_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+
+ /* Arm the drop floor before restoring the HW validator: from now on
+ * tls_validate_xmit_skb() drops payload retransmits of fully-ACKed data, so
+ * a stale clone whose record was purged here does not reach the NIC and trip
+ * its WARN on the new start marker. The cleartext leak on that path is closed
+ * separately by the skb_is_decrypted() gate in tls_sw_fallback(); this is
+ * only WARN avoidance. Set once; stays set for the socket's life.
+ */
+ set_bit(TLS_TX_REKEY_FLOOR, &ctx->flags);
+
+ /* Switch back to HW offload validator */
+ smp_store_release(&sk->sk_validate_xmit_skb, tls_validate_xmit_skb);
+
+ WRITE_ONCE(ctx->rekey.sw_ctx, NULL);
+ WRITE_ONCE(ctx->rekey.cipher_ctx, NULL);
+ spin_unlock_irqrestore(&offload_ctx->lock, flags);
+
+ memzero_explicit(&offload_ctx->rekey, sizeof(offload_ctx->rekey));
+ crypto_free_aead(old_sw_aead);
+
+ up_read(&device_offload_lock);
+
+ if (deferred)
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXREKEY);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYOK);
+ return 0;
+
+release_lock:
+ up_read(&device_offload_lock);
+
+rekey_fallback:
+ kfree(offload_ctx->rekey.start_marker);
+ offload_ctx->rekey.start_marker = NULL;
+ spin_lock_irqsave(&offload_ctx->lock, flags);
+ set_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+ clear_bit(TLS_TX_REKEY_READY, &ctx->flags);
+ clear_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+ spin_unlock_irqrestore(&offload_ctx->lock, flags);
+ if (deferred)
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXREKEY);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYFALLBACK);
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
+
+ /* Hard failure: HW rekey gave up and the connection is now pinned to
+ * SW encryption. The call site only sees the transient -EAGAIN retry
+ * (rc is not propagated here), so emit the trace from the fallback
+ * path itself; rc still holds the originating error.
+ */
+ trace_tls_device_complete_rekey_fail(sk, rc);
+
+ return 0;
+}
+
+static int tls_set_device_offload_rekey(struct sock *sk,
+ struct tls_context *ctx,
+ struct tls_crypto_info *new_crypto_info)
+{
+ struct tls_offload_context_tx *offload_ctx = tls_offload_ctx_tx(ctx);
+ bool rekey_pending = test_bit(TLS_TX_REKEY_PENDING, &ctx->flags);
+ bool rekey_failed = test_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+ bool defer = true;
+ int rc;
+
+ /* Defer the switch back to HW until any in-flight old-key records are
+ * ACKed. A partially_sent_record needs no separate check: its record is
+ * on records_list before it is sent (tls_push_record()) and stays there
+ * until ACKed, so tls_has_unacked_records() already covers it.
+ */
+ if (!rekey_pending && !rekey_failed)
+ defer = tls_has_unacked_records(offload_ctx) ||
+ tls_is_pending_open_record(ctx);
+
+ if (!offload_ctx->rekey.start_marker) {
+ offload_ctx->rekey.start_marker =
+ kmalloc_obj(*offload_ctx->rekey.start_marker);
+ if (!offload_ctx->rekey.start_marker)
+ return -ENOMEM;
+ }
+
+ rc = tls_device_start_rekey(sk, ctx, offload_ctx, new_crypto_info);
if (rc)
- goto release_netdev;
+ return rc;
+
+ if (defer) {
+ if (!rekey_pending)
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXREKEY);
+ else
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXREKEYOK);
+ return 0;
+ }
+
+ return tls_device_complete_rekey(sk, ctx, false, 0);
+}
+
+static int tls_set_device_offload_initial(struct sock *sk,
+ struct tls_context *ctx,
+ struct net_device *netdev,
+ struct tls_crypto_info *crypto_info,
+ const struct tls_cipher_desc *cipher_desc)
+{
+ struct tls_prot_info *prot = &ctx->prot_info;
+ struct tls_record_info *start_marker_record;
+ struct tls_offload_context_tx *offload_ctx;
+ char *iv, *rec_seq;
+ int rc;
iv = crypto_info_iv(crypto_info, cipher_desc);
rec_seq = crypto_info_rec_seq(crypto_info, cipher_desc);
+ rc = init_prot_info(prot, crypto_info, cipher_desc);
+ if (rc)
+ return rc;
+
memcpy(ctx->tx.iv + cipher_desc->salt, iv, cipher_desc->iv);
memcpy(ctx->tx.rec_seq, rec_seq, cipher_desc->rec_seq);
start_marker_record = kmalloc_obj(*start_marker_record);
- if (!start_marker_record) {
- rc = -ENOMEM;
- goto release_netdev;
- }
+ if (!start_marker_record)
+ return -ENOMEM;
offload_ctx = alloc_offload_ctx_tx(ctx);
if (!offload_ctx) {
@@ -1129,20 +1990,11 @@ int tls_set_device_offload(struct sock *sk)
if (rc)
goto free_offload_ctx;
- start_marker_record->end_seq = tcp_sk(sk)->write_seq;
- start_marker_record->len = 0;
- start_marker_record->num_frags = 0;
- list_add_tail(&start_marker_record->list, &offload_ctx->records_list);
+ tls_device_commit_start_marker(sk, offload_ctx, start_marker_record);
clean_acked_data_enable(tcp_sk(sk), &tls_tcp_clean_acked);
ctx->push_pending_record = tls_device_push_pending_record;
- /* TLS offload is greatly simplified if we don't send
- * SKBs where only part of the payload needs to be encrypted.
- * So mark the last skb in the write queue as end of record.
- */
- tcp_write_collapse_fence(sk);
-
/* Avoid offloading if the device is down
* We don't want to offload new flows after
* the NETDEV_DOWN event
@@ -1158,11 +2010,8 @@ int tls_set_device_offload(struct sock *sk)
}
ctx->priv_ctx_tx = offload_ctx;
- rc = netdev->tlsdev_ops->tls_dev_add(netdev, sk, TLS_OFFLOAD_CTX_DIR_TX,
- &ctx->crypto_send.info,
- tcp_sk(sk)->write_seq);
- trace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_TX,
- tcp_sk(sk)->write_seq, rec_seq, rc);
+ rc = tls_device_dev_add_tx(sk, netdev, crypto_info,
+ tcp_sk(sk)->write_seq);
if (rc)
goto release_lock;
@@ -1174,7 +2023,6 @@ int tls_set_device_offload(struct sock *sk)
* by the netdev's xmit function.
*/
smp_store_release(&sk->sk_validate_xmit_skb, tls_validate_xmit_skb);
- dev_put(netdev);
return 0;
@@ -1187,20 +2035,44 @@ int tls_set_device_offload(struct sock *sk)
ctx->priv_ctx_tx = NULL;
free_marker_record:
kfree(start_marker_record);
-release_netdev:
- dev_put(netdev);
return rc;
}
-int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
+int tls_set_device_offload(struct sock *sk,
+ struct tls_crypto_info *new_crypto_info)
{
- struct tls12_crypto_info_aes_gcm_128 *info;
- struct tls_offload_context_rx *context;
+ struct tls_crypto_info *crypto_info, *src_crypto_info;
+ const struct tls_cipher_desc *cipher_desc;
struct net_device *netdev;
- int rc = 0;
+ struct tls_context *ctx;
+ int rc;
- if (ctx->crypto_recv.info.version != TLS_1_2_VERSION)
- return -EOPNOTSUPP;
+ ctx = tls_get_ctx(sk);
+
+ /* A rekey of a SW-offloaded socket belongs to tls_set_sw_offload(). */
+ if (new_crypto_info && ctx->tx_conf != TLS_HW)
+ return -EINVAL;
+
+ crypto_info = &ctx->crypto_send.info;
+ src_crypto_info = new_crypto_info ?: crypto_info;
+ cipher_desc = get_cipher_desc(src_crypto_info->cipher_type);
+ if (!cipher_desc || !cipher_desc->offloadable)
+ return -EINVAL;
+
+ /* A rekey targets the device already holding the HW TX context
+ * (ctx->netdev), which can differ from the socket's current route after
+ * a route change or bond/team failover; tls_set_device_offload_rekey()
+ * and tls_device_complete_rekey() resolve it from ctx->netdev under
+ * device_offload_lock. Only the initial install needs the route device.
+ */
+ if (new_crypto_info)
+ return tls_set_device_offload_rekey(sk, ctx, src_crypto_info);
+
+ /* Initial install: a HW TX context must not already exist, otherwise
+ * alloc_offload_ctx_tx() below would silently overwrite it.
+ */
+ if (ctx->priv_ctx_tx)
+ return -EEXIST;
netdev = get_netdev_for_sock(sk);
if (!netdev) {
@@ -1208,50 +2080,249 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
return -EINVAL;
}
- if (!(netdev->features & NETIF_F_HW_TLS_RX)) {
+ if (!(netdev->features & NETIF_F_HW_TLS_TX)) {
rc = -EOPNOTSUPP;
goto release_netdev;
}
- /* Avoid offloading if the device is down
- * We don't want to offload new flows after
- * the NETDEV_DOWN event
- *
- * device_offload_lock is taken in tls_devices's NETDEV_DOWN
- * handler thus protecting from the device going down before
- * ctx was added to tls_device_list.
- */
- down_read(&device_offload_lock);
- if (!(netdev->flags & IFF_UP)) {
- rc = -EINVAL;
- goto release_lock;
+ rc = tls_set_device_offload_initial(sk, ctx, netdev, src_crypto_info,
+ cipher_desc);
+
+release_netdev:
+ dev_put(netdev);
+ return rc;
+}
+
+int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx,
+ struct tls_crypto_info *new_crypto_info)
+{
+ struct tls_crypto_info *crypto_info, *src_crypto_info;
+ const struct tls_cipher_desc *cipher_desc;
+ u32 drain_start = tcp_sk(sk)->copied_seq;
+ struct tls_offload_context_rx *context;
+ struct net_device *netdev;
+ bool was_dev_add_pending;
+ bool moved_aead_recv = false;
+ bool retired_pending = false;
+ bool put_netdev = false;
+ int rc = 0;
+
+ /* A rekey of a SW-offloaded socket belongs to tls_set_sw_offload(). */
+ if (new_crypto_info && ctx->rx_conf != TLS_HW)
+ return -EINVAL;
+
+ crypto_info = &ctx->crypto_recv.info;
+ src_crypto_info = new_crypto_info ?: crypto_info;
+ cipher_desc = get_cipher_desc(src_crypto_info->cipher_type);
+ if (!cipher_desc || !cipher_desc->offloadable)
+ return -EINVAL;
+
+ if (new_crypto_info) {
+ /* Rekey targets the device holding the HW RX context, which
+ * can differ from the socket's route after a route change or
+ * bond/team failover. Resolve it from ctx->netdev under
+ * device_offload_lock, like the other del/add-key paths, not
+ * via get_netdev_for_sock(). The context owns the reference,
+ * so don't take an extra one here.
+ *
+ * A NULL netdev means tls_device_down() already ran: the HW RX
+ * context is deleted, TLS_RX_DEV_{DEGRADED,CLOSED} are set and
+ * every record is decrypted in SW, but rx_conf stays TLS_HW.
+ * The rekey is still required, the peer's KeyUpdate was parsed
+ * and recvmsg() returns -EKEYEXPIRED until the new key lands,
+ * so run the same state machine (queued records may still carry
+ * the deleted NIC context's old-key XOR) and account the new key
+ * as a SW fallback in place of the tls_dev_del()/tls_dev_add()
+ * steps, mirroring the TX side (tls_device_complete_rekey()).
+ * Do not fail the setsockopt.
+ */
+ down_read(&device_offload_lock);
+ netdev = rcu_dereference_protected(ctx->netdev,
+ lockdep_is_held(&device_offload_lock));
+ } else {
+ netdev = get_netdev_for_sock(sk);
+ if (!netdev) {
+ pr_err_ratelimited("%s: netdev not found\n", __func__);
+ return -EINVAL;
+ }
+ put_netdev = true;
+
+ if (!(netdev->features & NETIF_F_HW_TLS_RX)) {
+ rc = -EOPNOTSUPP;
+ goto release_netdev;
+ }
+
+ /* Avoid offloading if the device is down
+ * We don't want to offload new flows after
+ * the NETDEV_DOWN event
+ *
+ * device_offload_lock is taken in tls_devices's NETDEV_DOWN
+ * handler thus protecting from the device going down before
+ * ctx was added to tls_device_list.
+ */
+ down_read(&device_offload_lock);
+ if (!(netdev->flags & IFF_UP)) {
+ rc = -EINVAL;
+ goto release_lock;
+ }
}
- context = kzalloc_obj(*context);
- if (!context) {
- rc = -ENOMEM;
- goto release_lock;
+ if (!new_crypto_info) {
+ context = kzalloc_obj(*context);
+ if (!context) {
+ rc = -ENOMEM;
+ goto release_lock;
+ }
+ ctx->priv_ctx_rx = context;
+ } else {
+ context = tls_offload_ctx_rx(ctx);
}
+ was_dev_add_pending = context->dev_add_pending;
context->resync_nh_reset = 1;
- ctx->priv_ctx_rx = context;
- rc = tls_set_sw_offload(sk, 0, NULL);
+ if (new_crypto_info) {
+ struct tls_sw_context_rx *sw_ctx = tls_sw_ctx_rx(ctx);
+
+ /* Classify against the record start, not the raw copied_seq: in
+ * strparser copy mode tcp_read_sock() has already advanced
+ * copied_seq past a parsed-ahead (possibly partial) record the
+ * user has not received, which may still carry the old NIC key's
+ * XOR. tls_device_decrypted() compensates the same way; keeping
+ * both in sync is what lets a drained-vs-still-draining decision
+ * here match the reencrypt-key decision there.
+ */
+ drain_start = tls_device_rx_rec_start(sk, sw_ctx);
+
+ /* netdev is NULL only after tls_device_down(), which already
+ * deleted the HW RX context and set TLS_RX_DEV_CLOSED; the
+ * netdev check just makes that dependency explicit.
+ */
+ if (netdev && !test_bit(TLS_RX_DEV_CLOSED, &ctx->flags)) {
+ set_bit(TLS_RX_DEV_CLOSED, &ctx->flags);
+ synchronize_net();
+ netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
+ TLS_OFFLOAD_CTX_DIR_RX);
+ }
+
+ if (context->rekey.old_aead_recv &&
+ before(drain_start, context->rekey.old_nic_boundary)) {
+ /* Previous rekey still draining. Keep rekey.old_aead_recv,
+ * it is the only key that can undo the NIC-XOR on queued
+ * records. sw_ctx->aead_recv may be re-setkey'd by
+ * tls_sw_ctx_init(); that intermediate key was never on
+ * the NIC and its wire era is drained, so it is needed
+ * for neither undo nor AEAD. Defer dev_add; the new key
+ * is installed once drain_start crosses rekey.old_nic_boundary.
+ */
+ context->dev_add_pending = 1;
+ trace_tls_device_rekey_start(sk, drain_start,
+ context->rekey.old_nic_boundary,
+ true);
+ } else {
+ struct tcp_sock *tp = tcp_sk(sk);
+ u32 nic_end;
+
+ if (context->rekey.old_aead_recv) {
+ /* Prior rekey's era already drained (drain_start is
+ * past old_nic_boundary), so retiring its key here
+ * is a boundary crossing, same as the free in
+ * tls_device_decrypted(); mark it done.
+ */
+ trace_tls_device_rekey_done(sk, drain_start,
+ context->rekey.old_nic_boundary);
+ crypto_free_aead(context->rekey.old_aead_recv);
+ context->rekey.old_aead_recv = NULL;
+ }
+
+ /* Flush the backlog so TCP's view is current, then take the
+ * highest byte TCP holds, including the out-of-order tail:
+ * a NIC-transformed segment behind a host-side drop sits
+ * above rcv_nxt until the retransmit fills the hole and
+ * must still be classified against the old key. This is
+ * still only the stack's view, a transformed segment the
+ * NIC has not delivered yet is caught in-band by
+ * tls_device_decrypted(), which slides the boundary.
+ */
+ __sk_flush_backlog(sk);
+ nic_end = tp->rcv_nxt;
+ if (!RB_EMPTY_ROOT(&tp->out_of_order_queue) &&
+ after(TCP_SKB_CB(tp->ooo_last_skb)->end_seq, nic_end))
+ nic_end = TCP_SKB_CB(tp->ooo_last_skb)->end_seq;
+
+ if (before(drain_start, nic_end)) {
+ context->rekey.old_aead_recv = sw_ctx->aead_recv;
+ /* NULL so tls_sw_ctx_init() allocates a fresh tfm
+ * for the new key instead of re-keying the one we
+ * must keep for the drain.
+ */
+ sw_ctx->aead_recv = NULL;
+ moved_aead_recv = true;
+ memcpy(context->rekey.old_iv, ctx->rx.iv,
+ sizeof(context->rekey.old_iv));
+ memcpy(context->rekey.old_rec_seq, ctx->rx.rec_seq,
+ sizeof(context->rekey.old_rec_seq));
+ context->rekey.old_nic_boundary = nic_end;
+ context->dev_add_pending = 1;
+ } else if (was_dev_add_pending) {
+ /* A prior rekey's deferred dev_add can no longer
+ * run: its trigger (old_aead_recv) was just freed
+ * above and no new drain replaces it. Its era
+ * drained successfully (drain_start is already past
+ * old_nic_boundary), so retire it and let the new
+ * key install immediately below. retired_pending
+ * defers its OK/gauge accounting to the post-init
+ * block, past the error goto, so a failed
+ * tls_sw_ctx_init() needs no counter undo.
+ */
+ context->dev_add_pending = 0;
+ retired_pending = true;
+ }
+ trace_tls_device_rekey_start(sk, drain_start, nic_end,
+ before(drain_start, nic_end));
+ }
+ }
+
+ rc = tls_sw_ctx_init(sk, 0, new_crypto_info);
if (rc)
goto release_ctx;
- rc = netdev->tlsdev_ops->tls_dev_add(netdev, sk, TLS_OFFLOAD_CTX_DIR_RX,
- &ctx->crypto_recv.info,
- tcp_sk(sk)->copied_seq);
- info = (void *)&ctx->crypto_recv.info;
- trace_tls_device_offload_set(sk, TLS_OFFLOAD_CTX_DIR_RX,
- tcp_sk(sk)->copied_seq, info->rec_seq, rc);
- if (rc)
- goto free_sw_resources;
+ if (!context->dev_add_pending) {
+ if (retired_pending) {
+ /* Account the superseded rekey that drained OK, mirroring
+ * the deferred-add path: one RXREKEYOK and release its
+ * in-flight gauge. The new key's own OK/FALLBACK is counted
+ * by tls_device_dev_add_rx() just below.
+ */
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXREKEY);
+ }
+ if (netdev) {
+ rc = tls_device_dev_add_rx(sk, ctx, netdev,
+ src_crypto_info, drain_start,
+ !!new_crypto_info);
+ } else {
+ /* No device after tls_device_down(); the SW path keeps
+ * decrypting.
+ */
+ tls_device_rx_rekey_fallback(sk, ctx);
+ }
+ if (!new_crypto_info) {
+ if (rc)
+ goto free_sw_resources;
+ tls_device_attach(ctx, sk, netdev);
+ }
+ } else if (!was_dev_add_pending) {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXREKEY);
+ } else {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYOK);
+ }
+
+ tls_sw_ctx_finalize(sk, 0, new_crypto_info);
- tls_device_attach(ctx, sk, netdev);
up_read(&device_offload_lock);
- dev_put(netdev);
+ if (put_netdev)
+ dev_put(netdev);
return 0;
@@ -1260,17 +2331,39 @@ int tls_set_device_offload_rx(struct sock *sk, struct tls_context *ctx)
tls_sw_free_resources_rx(sk);
down_read(&device_offload_lock);
release_ctx:
- ctx->priv_ctx_rx = NULL;
+ if (!new_crypto_info) {
+ ctx->priv_ctx_rx = NULL;
+ } else {
+ /* A failed RX rekey is terminal, so there is no HW state to roll
+ * back to. KeyUpdate is directional and the peer's TX has already
+ * switched keys, so once the new RX key fails to install the old
+ * SW key restored below cannot decrypt any further record; the
+ * socket is dead and the app must close it. The half-torn HW
+ * context (tls_dev_del already ran) and any dangling
+ * dev_add_pending / old_aead_recv are reclaimed by
+ * tls_device_offload_cleanup_rx() on close.
+ */
+ context->dev_add_pending = was_dev_add_pending;
+ if (moved_aead_recv) {
+ struct tls_sw_context_rx *sw_ctx = tls_sw_ctx_rx(ctx);
+
+ crypto_free_aead(sw_ctx->aead_recv);
+ sw_ctx->aead_recv = context->rekey.old_aead_recv;
+ context->rekey.old_aead_recv = NULL;
+ }
+ }
release_lock:
up_read(&device_offload_lock);
release_netdev:
- dev_put(netdev);
+ if (put_netdev)
+ dev_put(netdev);
return rc;
}
void tls_device_offload_cleanup_rx(struct sock *sk)
{
struct tls_context *tls_ctx = tls_get_ctx(sk);
+ struct tls_offload_context_rx *rx_ctx;
struct net_device *netdev;
down_read(&device_offload_lock);
@@ -1279,8 +2372,9 @@ void tls_device_offload_cleanup_rx(struct sock *sk)
if (!netdev)
goto out;
- netdev->tlsdev_ops->tls_dev_del(netdev, tls_ctx,
- TLS_OFFLOAD_CTX_DIR_RX);
+ if (!test_bit(TLS_RX_DEV_CLOSED, &tls_ctx->flags))
+ netdev->tlsdev_ops->tls_dev_del(netdev, tls_ctx,
+ TLS_OFFLOAD_CTX_DIR_RX);
if (tls_ctx->tx_conf != TLS_HW) {
dev_put(netdev);
@@ -1290,6 +2384,19 @@ void tls_device_offload_cleanup_rx(struct sock *sk)
}
out:
up_read(&device_offload_lock);
+
+ rx_ctx = tls_offload_ctx_rx(tls_ctx);
+ if (rx_ctx && rx_ctx->rekey.old_aead_recv) {
+ crypto_free_aead(rx_ctx->rekey.old_aead_recv);
+ rx_ctx->rekey.old_aead_recv = NULL;
+ }
+
+ if (rx_ctx && rx_ctx->dev_add_pending) {
+ rx_ctx->dev_add_pending = 0;
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYABORTED);
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXREKEY);
+ }
+
tls_sw_release_resources_rx(sk);
}
@@ -1317,10 +2424,16 @@ static int tls_device_down(struct net_device *netdev)
spin_unlock_irqrestore(&tls_device_lock, flags);
list_for_each_entry_safe(ctx, tmp, &list, list) {
- /* Stop offloaded TX and switch to the fallback.
- * tls_is_skb_tx_device_offloaded will return false.
+ /* Stop offloaded TX and switch to the fallback. For a socket not
+ * mid-rekey, tls_is_skb_tx_device_offloaded() then returns false; a
+ * PENDING/FAILED socket keeps the rekey validator (under which only a
+ * decrypted straddler still offloads), and the synchronize_net()
+ * below drains any such in-flight skb before tls_dev_del().
*/
- WRITE_ONCE(ctx->sk->sk_validate_xmit_skb, tls_validate_xmit_skb_sw);
+ if (!test_bit(TLS_TX_REKEY_PENDING, &ctx->flags) &&
+ !test_bit(TLS_TX_REKEY_FAILED, &ctx->flags))
+ WRITE_ONCE(ctx->sk->sk_validate_xmit_skb,
+ tls_validate_xmit_skb_sw);
/* Stop the RX and TX resync.
* tls_dev_resync must not be called after tls_dev_del.
@@ -1337,13 +2450,18 @@ static int tls_device_down(struct net_device *netdev)
synchronize_net();
/* Release the offload context on the driver side. */
- if (ctx->tx_conf == TLS_HW)
+ if (ctx->tx_conf == TLS_HW &&
+ !test_bit(TLS_TX_DEV_CLOSED, &ctx->flags)) {
netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
TLS_OFFLOAD_CTX_DIR_TX);
+ set_bit(TLS_TX_DEV_CLOSED, &ctx->flags);
+ }
if (ctx->rx_conf == TLS_HW &&
- !test_bit(TLS_RX_DEV_CLOSED, &ctx->flags))
+ !test_bit(TLS_RX_DEV_CLOSED, &ctx->flags)) {
netdev->tlsdev_ops->tls_dev_del(netdev, ctx,
TLS_OFFLOAD_CTX_DIR_RX);
+ set_bit(TLS_RX_DEV_CLOSED, &ctx->flags);
+ }
dev_put(netdev);
@@ -1411,12 +2529,27 @@ static struct notifier_block tls_dev_notifier = {
int __init tls_device_init(void)
{
- int err;
+ unsigned char *page_addr;
+ int err, i;
- dummy_page = alloc_page(GFP_KERNEL);
+ dummy_page = alloc_page(GFP_KERNEL | __GFP_ZERO);
if (!dummy_page)
return -ENOMEM;
+ /* Pre-populate the first 256 bytes with an identity map so that,
+ * when this page is used as the tail-frag fallback (allocation
+ * failure in tls_device_record_close()), dummy_page[record_type]
+ * yields the correct TLS 1.3 content_type byte for any record_type
+ * without runtime validation.
+ *
+ * A high record_type pushes the tag placeholder past the identity
+ * map, so __GFP_ZERO is what keeps tag-placeholder bytes defined
+ * rather than exposing uninitialized page contents.
+ */
+ page_addr = page_address(dummy_page);
+ for (i = 0; i < 256; i++)
+ page_addr[i] = (unsigned char)i;
+
destruct_wq = alloc_workqueue("ktls_device_destruct", WQ_PERCPU, 0);
if (!destruct_wq) {
err = -ENOMEM;
diff --git a/net/tls/tls_device_fallback.c b/net/tls/tls_device_fallback.c
index 3b7d0ab2bcf17..f2a0ae827bb2a 100644
--- a/net/tls/tls_device_fallback.c
+++ b/net/tls/tls_device_fallback.c
@@ -37,14 +37,15 @@
#include "tls.h"
-static int tls_enc_record(struct aead_request *aead_req,
+static int tls_enc_record(struct tls_context *tls_ctx,
+ struct aead_request *aead_req,
struct crypto_aead *aead, char *aad,
char *iv, __be64 rcd_sn,
struct scatter_walk *in,
- struct scatter_walk *out, int *in_len,
- struct tls_prot_info *prot)
+ struct scatter_walk *out, int *in_len)
{
unsigned char buf[TLS_HEADER_SIZE + TLS_MAX_IV_SIZE];
+ struct tls_prot_info *prot = &tls_ctx->prot_info;
const struct tls_cipher_desc *cipher_desc;
struct scatterlist sg_in[3];
struct scatterlist sg_out[3];
@@ -55,7 +56,7 @@ static int tls_enc_record(struct aead_request *aead_req,
cipher_desc = get_cipher_desc(prot->cipher_type);
DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
- buf_size = TLS_HEADER_SIZE + cipher_desc->iv;
+ buf_size = prot->prepend_size;
len = min_t(int, *in_len, buf_size);
memcpy_from_scatterwalk(buf, in, len);
@@ -66,16 +67,27 @@ static int tls_enc_record(struct aead_request *aead_req,
return 0;
len = buf[4] | (buf[3] << 8);
- len -= cipher_desc->iv;
+ if (prot->version != TLS_1_3_VERSION)
+ len -= cipher_desc->iv;
tls_make_aad(aad, len - cipher_desc->tag, (char *)&rcd_sn, buf[0], prot);
- memcpy(iv + cipher_desc->salt, buf + TLS_HEADER_SIZE, cipher_desc->iv);
+ if (prot->version == TLS_1_3_VERSION) {
+ void *iv_src = crypto_info_iv(&tls_ctx->crypto_send.info,
+ cipher_desc);
+
+ memcpy(iv + cipher_desc->salt, iv_src, cipher_desc->iv);
+ } else {
+ memcpy(iv + cipher_desc->salt, buf + TLS_HEADER_SIZE,
+ cipher_desc->iv);
+ }
+
+ tls_xor_iv_with_seq(prot, iv, (char *)&rcd_sn);
sg_init_table(sg_in, ARRAY_SIZE(sg_in));
sg_init_table(sg_out, ARRAY_SIZE(sg_out));
- sg_set_buf(sg_in, aad, TLS_AAD_SPACE_SIZE);
- sg_set_buf(sg_out, aad, TLS_AAD_SPACE_SIZE);
+ sg_set_buf(sg_in, aad, prot->aad_size);
+ sg_set_buf(sg_out, aad, prot->aad_size);
scatterwalk_get_sglist(in, sg_in + 1);
scatterwalk_get_sglist(out, sg_out + 1);
@@ -108,13 +120,6 @@ static int tls_enc_record(struct aead_request *aead_req,
return rc;
}
-static void tls_init_aead_request(struct aead_request *aead_req,
- struct crypto_aead *aead)
-{
- aead_request_set_tfm(aead_req, aead);
- aead_request_set_ad(aead_req, TLS_AAD_SPACE_SIZE);
-}
-
static struct aead_request *tls_alloc_aead_request(struct crypto_aead *aead,
gfp_t flags)
{
@@ -124,14 +129,15 @@ static struct aead_request *tls_alloc_aead_request(struct crypto_aead *aead,
aead_req = kzalloc(req_size, flags);
if (aead_req)
- tls_init_aead_request(aead_req, aead);
+ aead_request_set_tfm(aead_req, aead);
return aead_req;
}
-static int tls_enc_records(struct aead_request *aead_req,
+static int tls_enc_records(struct tls_context *tls_ctx,
+ struct aead_request *aead_req,
struct crypto_aead *aead, struct scatterlist *sg_in,
struct scatterlist *sg_out, char *aad, char *iv,
- u64 rcd_sn, int len, struct tls_prot_info *prot)
+ u64 rcd_sn, int len)
{
struct scatter_walk out, in;
int rc;
@@ -140,8 +146,8 @@ static int tls_enc_records(struct aead_request *aead_req,
scatterwalk_start(&out, sg_out);
do {
- rc = tls_enc_record(aead_req, aead, aad, iv,
- cpu_to_be64(rcd_sn), &in, &out, &len, prot);
+ rc = tls_enc_record(tls_ctx, aead_req, aead, aad, iv,
+ cpu_to_be64(rcd_sn), &in, &out, &len);
rcd_sn++;
} while (rc == 0 && len);
@@ -184,6 +190,14 @@ static void complete_skb(struct sk_buff *nskb, struct sk_buff *skb, int headln)
skb_copy_header(nskb, skb);
+ /* nskb now carries ciphertext, but skb_copy_header() inherited
+ * skb->decrypted from the plaintext original. Clear it so the bit keeps
+ * meaning "still-plaintext, needs an encryptor": otherwise a requeued
+ * nskb would be needlessly re-validated (and re-encrypted) and would trip
+ * the NIC's decrypted-vs-start-marker WARN.
+ */
+ nskb->decrypted = 0;
+
skb_put(nskb, skb->len);
memcpy(nskb->data, skb->data, headln);
@@ -314,7 +328,10 @@ static struct sk_buff *tls_enc_skb(struct tls_context *tls_ctx,
cipher_desc = get_cipher_desc(tls_ctx->crypto_send.info.cipher_type);
DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable);
- buf_len = cipher_desc->salt + cipher_desc->iv + TLS_AAD_SPACE_SIZE +
+ aead_request_set_ad(aead_req, tls_ctx->prot_info.aad_size);
+
+ buf_len = cipher_desc->salt + cipher_desc->iv +
+ tls_ctx->prot_info.aad_size +
sync_size + cipher_desc->tag;
buf = kmalloc(buf_len, GFP_ATOMIC);
if (!buf)
@@ -324,7 +341,7 @@ static struct sk_buff *tls_enc_skb(struct tls_context *tls_ctx,
salt = crypto_info_salt(&tls_ctx->crypto_send.info, cipher_desc);
memcpy(iv, salt, cipher_desc->salt);
aad = buf + cipher_desc->salt + cipher_desc->iv;
- dummy_buf = aad + TLS_AAD_SPACE_SIZE;
+ dummy_buf = aad + tls_ctx->prot_info.aad_size;
nskb = alloc_skb(skb_headroom(skb) + skb->len, GFP_ATOMIC);
if (!nskb)
@@ -335,9 +352,8 @@ static struct sk_buff *tls_enc_skb(struct tls_context *tls_ctx,
fill_sg_out(sg_out, buf, tls_ctx, nskb, tcp_payload_offset,
payload_len, sync_size, dummy_buf);
- if (tls_enc_records(aead_req, ctx->aead_send, sg_in, sg_out, aad, iv,
- rcd_sn, sync_size + payload_len,
- &tls_ctx->prot_info) < 0)
+ if (tls_enc_records(tls_ctx, aead_req, ctx->aead_send, sg_in, sg_out,
+ aad, iv, rcd_sn, sync_size + payload_len) < 0)
goto free_nskb;
complete_skb(nskb, skb, tcp_payload_offset);
@@ -388,8 +404,17 @@ static struct sk_buff *tls_sw_fallback(struct sock *sk, struct sk_buff *skb)
sg_init_table(sg_out, ARRAY_SIZE(sg_out));
if (fill_sg_in(sg_in, skb, ctx, &rcd_sn, &sync_size, &resync_sgs)) {
- /* bypass packets before kernel TLS socket option was set */
- if (sync_size < 0 && payload_len <= -sync_size)
+ /* Below the record range (start marker / already-freed record).
+ * Pass through only cleartext that was never offload-encrypted
+ * (skb->decrypted == 0): genuine pre-TLS bytes sent before the
+ * socket option was set, or SW-encrypted rekey ciphertext. A
+ * decrypted=1 skb here is offload-record plaintext whose record was
+ * purged (e.g. a rekey installed a new start marker above its seq);
+ * it must never reach the wire in the clear, so continue on and
+ * drop it (nskb stays NULL).
+ */
+ if (sync_size < 0 && payload_len <= -sync_size &&
+ !skb_is_decrypted(skb))
nskb = skb_get(skb);
goto put_sg;
}
@@ -408,11 +433,57 @@ static struct sk_buff *tls_sw_fallback(struct sock *sk, struct sk_buff *skb)
return nskb;
}
+/* Post-rekey drop floor. Once a rekey has completed (TLS_TX_REKEY_FLOOR set), a
+ * stale retransmit clone of already-ACKed data may still be dequeued from a
+ * qdisc; if its offload record was purged at completion it now maps to a rekey
+ * start marker. The cleartext leak on that path is closed unconditionally by
+ * the skb_is_decrypted() gate in tls_sw_fallback(); this floor additionally
+ * drops the clone before it reaches the NIC, avoiding the driver's WARN
+ * (mlx5e_ktls_handle_tx_skb() SKIP_NO_DATA) on an otherwise-legitimate race.
+ * Only needed by tls_validate_xmit_skb() (the restored HW-offload validator):
+ * only there can a purged-record clone reach the NIC and hit the new start
+ * marker. Under the rekey/SW validators the only skb the NIC offloads is a
+ * decrypted straddler whose record is still present (no SKIP_NO_DATA), and a
+ * stale clone is dropped by the skb_is_decrypted() gate in tls_sw_fallback().
+ * Such a clone is exactly a payload skb whose end_seq <= snd_una: the peer has
+ * already ACKed that data, so dropping it is always safe. Live/unacked data
+ * (including a legitimate retransmit, or a straddler ending past snd_una) is
+ * never touched; pure ACKs and zero-window probes carry no payload and pass.
+ */
+static bool tls_tx_drop_acked_clone(struct sock *sk, struct sk_buff *skb)
+{
+ int payload_len = skb->len - skb_tcp_all_headers(skb);
+ u32 end_seq;
+
+ if (likely(!test_bit(TLS_TX_REKEY_FLOOR, &tls_get_ctx(sk)->flags)))
+ return false;
+
+ if (payload_len <= 0)
+ return false;
+
+ /* Drop only when the whole payload is already ACKed (end_seq <= snd_una):
+ * such a skb is purely a stale retransmit clone the peer already has. A
+ * clone straddling snd_una still carries unacked bytes, so leave it to the
+ * normal paths (a live record is re-encrypted; a marker/freed-record hit is
+ * dropped there too). Both the leak (skb_is_decrypted() gate) and the mlx5
+ * WARN only concern the fully-ACKed case handled here.
+ */
+ end_seq = ntohl(tcp_hdr(skb)->seq) + payload_len;
+ return !after(end_seq, READ_ONCE(tcp_sk(sk)->snd_una));
+}
+
struct sk_buff *tls_validate_xmit_skb(struct sock *sk,
struct net_device *dev,
struct sk_buff *skb)
{
- if (dev == rcu_dereference_bh(tls_get_ctx(sk)->netdev) ||
+ struct tls_context *tls_ctx = tls_get_ctx(sk);
+
+ if (unlikely(tls_tx_drop_acked_clone(sk, skb))) {
+ kfree_skb(skb);
+ return NULL;
+ }
+
+ if (dev == rcu_dereference_bh(tls_ctx->netdev) ||
netif_is_bond_master(dev))
return skb;
@@ -427,6 +498,65 @@ struct sk_buff *tls_validate_xmit_skb_sw(struct sock *sk,
return tls_sw_fallback(sk, skb);
}
+struct sk_buff *tls_validate_xmit_skb_rekey(struct sock *sk,
+ struct net_device *dev,
+ struct sk_buff *skb)
+{
+ struct tls_context *tls_ctx = tls_get_ctx(sk);
+ u32 tcp_seq = ntohl(tcp_hdr(skb)->seq);
+ u32 pivot_seq;
+
+ /* acquire pairs with clear_bit_unlock() on re-arm; makes the refreshed
+ * boundary_seq visible in the else branch below.
+ */
+ if (test_bit_acquire(TLS_TX_REKEY_FAILED, &tls_ctx->flags)) {
+ int payload_len = skb->len - skb_tcp_all_headers(skb);
+ u32 snd_una = READ_ONCE(tcp_sk(sk)->snd_una);
+
+ /* FAILED: HW context gone and all old-key plaintext ACKed
+ * (snd_una >= boundary_seq). seq < boundary_seq is old-key data
+ * whose records are freed, so tls_sw_fallback() drops it. seq >=
+ * boundary_seq is SW ciphertext with no record. A retransmit is
+ * built at seq == snd_una (tcp_trim_head()), so an ACK landing
+ * before we run can move snd_una past seq while the tail is
+ * unacked; pivoting on snd_una alone would drop that live data
+ * and force an RTO. Pass through any non-decrypted skb ending
+ * past snd_una (mirrors tls_tx_drop_acked_clone()); fully-ACKed
+ * clones fall to the pivot and are dropped.
+ */
+ if (payload_len > 0 && !skb_is_decrypted(skb) &&
+ after(tcp_seq + payload_len, snd_una))
+ return skb;
+
+ pivot_seq = snd_una;
+ } else {
+ /* PENDING: new-key data is SW-encrypted at seq >= boundary_seq;
+ * old-key data below it is still unacked.
+ *
+ * On the first arm, boundary_seq is published by the
+ * smp_store_release() of sk_validate_xmit_skb in
+ * tls_device_start_rekey(); the xmit path loads that pointer with a
+ * plain read (net/core/dev.c), so pair it here with an smp_rmb()
+ * before reading boundary_seq. A stale boundary_seq (0) would pass an
+ * unacked old-key plaintext skb through; tls_is_skb_tx_device_offloaded()
+ * would still HW-encrypt it with the installed old key, so not a leak,
+ * but the barrier keeps the pivot accurate.
+ */
+ smp_rmb();
+ pivot_seq = READ_ONCE(tls_ctx->rekey.boundary_seq);
+ }
+
+ /* At or after the pivot: already correctly encrypted, pass through */
+ if (!before(tcp_seq, pivot_seq))
+ return skb;
+
+ /* Below the pivot: retransmit of old data, SW fallback with old key */
+ return tls_sw_fallback(sk, skb);
+}
+
+/* Address taken by tls_is_skb_tx_device_offloaded() in the offload drivers. */
+EXPORT_SYMBOL_GPL(tls_validate_xmit_skb_rekey);
+
struct sk_buff *tls_encrypt_skb(struct sk_buff *skb)
{
return tls_sw_fallback(skb->sk, skb);
diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c
index fbb274287aa5f..0a9e7d15fa95b 100644
--- a/net/tls/tls_main.c
+++ b/net/tls/tls_main.c
@@ -347,16 +347,28 @@ static void tls_sk_proto_cleanup(struct sock *sk,
tls_sw_release_resources_tx(sk);
TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
} else if (ctx->tx_conf == TLS_HW) {
+ bool rekey_failed = test_bit(TLS_TX_REKEY_FAILED, &ctx->flags);
+
tls_device_free_resources_tx(sk);
- TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
+
+ if (rekey_failed)
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXSW);
+ else
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
}
if (ctx->rx_conf == TLS_SW) {
tls_sw_release_resources_rx(sk);
TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXSW);
} else if (ctx->rx_conf == TLS_HW) {
+ bool rekey_failed = test_bit(TLS_RX_REKEY_FAILED, &ctx->flags);
+
tls_device_offload_cleanup_rx(sk);
- TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);
+
+ if (rekey_failed)
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXSW);
+ else
+ TLS_DEC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);
}
}
@@ -369,6 +381,8 @@ static void tls_sk_proto_close(struct sock *sk, long timeout)
if (ctx->tx_conf == TLS_SW)
tls_sw_cancel_work_tx(ctx);
+ else if (ctx->tx_conf == TLS_HW && ctx->rekey.sw_ctx)
+ tls_sw_cancel_work_tx(ctx);
lock_sock(sk);
free_ctx = ctx->tx_conf != TLS_HW && ctx->rx_conf != TLS_HW;
@@ -445,8 +459,17 @@ static int do_tls_getsockopt_conf(struct sock *sk, sockopt_t *opt, int tx)
/* get user crypto info */
if (tx) {
- crypto_info = &ctx->crypto_send.info;
- cctx = &ctx->tx;
+ /* Select the cipher context via the same accessor the data path
+ * uses, so getsockopt reports the IV/rec_seq that sendmsg encrypts
+ * with (the pending rekey's while one is in flight, else the
+ * active key). crypto_info has no accessor; select it the same way.
+ * lock_sock is held, so rekey.cipher_ctx cannot change under us.
+ */
+ cctx = tls_tx_cipher_ctx(ctx);
+ if (ctx->rekey.cipher_ctx)
+ crypto_info = &tls_offload_ctx_tx(ctx)->rekey.crypto_send.info;
+ else
+ crypto_info = &ctx->crypto_send.info;
} else {
crypto_info = &ctx->crypto_recv.info;
cctx = &ctx->rx;
@@ -710,11 +733,18 @@ static int do_tls_setsockopt_conf(struct sock *sk, sockptr_t optval,
}
if (tx) {
- rc = tls_set_device_offload(sk);
+ rc = tls_set_device_offload(sk, update ? crypto_info : NULL);
conf = TLS_HW;
if (!rc) {
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXDEVICE);
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
+ if (!update) {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSTXDEVICE);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRTXDEVICE);
+ }
+ } else if (update && ctx->tx_conf == TLS_HW) {
+ /* HW rekey failed - return the actual error.
+ * Cannot fall back to SW for an existing HW connection.
+ */
+ goto err_crypto_info;
} else {
rc = tls_set_sw_offload(sk, 1,
update ? crypto_info : NULL);
@@ -730,11 +760,19 @@ static int do_tls_setsockopt_conf(struct sock *sk, sockptr_t optval,
conf = TLS_SW;
}
} else {
- rc = tls_set_device_offload_rx(sk, ctx);
+ rc = tls_set_device_offload_rx(sk, ctx,
+ update ? crypto_info : NULL);
conf = TLS_HW;
if (!rc) {
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXDEVICE);
- TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);
+ if (!update) {
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXDEVICE);
+ TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSCURRRXDEVICE);
+ }
+ } else if (update && ctx->rx_conf == TLS_HW) {
+ /* HW rekey failed - return the actual error.
+ * Cannot fall back to SW for an existing HW connection.
+ */
+ goto err_crypto_info;
} else {
rc = tls_set_sw_offload(sk, 0,
update ? crypto_info : NULL);
@@ -773,7 +811,11 @@ static int do_tls_setsockopt_conf(struct sock *sk, sockptr_t optval,
return 0;
err_crypto_info:
- if (update) {
+ /* -EAGAIN is a transient sndbuf-full condition on a non-blocking rekey,
+ * not a failed KeyUpdate: the old key stays installed and userspace
+ * retries once the socket is writable, so don't count it as an error.
+ */
+ if (update && rc != -EAGAIN) {
TLS_INC_STATS(sock_net(sk), tx ? LINUX_MIB_TLSTXREKEYERROR
: LINUX_MIB_TLSRXREKEYERROR);
}
@@ -866,12 +908,29 @@ static int do_tls_setsockopt(struct sock *sk, int optname, sockptr_t optval,
switch (optname) {
case TLS_TX:
- case TLS_RX:
+ case TLS_RX: {
+ /* tls_device_sendmsg() holds tx_lock across the lock_sock drop
+ * in sk_stream_wait_memory() with a half-built open_record
+ * exposed. A concurrent HW-offload rekey (tls_device_start_rekey())
+ * would flush that record and swap the key under the sender,
+ * corrupting record framing. Serialize TX setsockopt against
+ * the data path with tx_lock, unconditionally for TLS_TX,
+ * since during initial setup there is no sender contending it.
+ */
+ bool tx = optname == TLS_TX;
+
+ if (tx) {
+ rc = mutex_lock_interruptible(&tls_get_ctx(sk)->tx_lock);
+ if (rc)
+ break;
+ }
lock_sock(sk);
- rc = do_tls_setsockopt_conf(sk, optval, optlen,
- optname == TLS_TX);
+ rc = do_tls_setsockopt_conf(sk, optval, optlen, tx);
release_sock(sk);
+ if (tx)
+ mutex_unlock(&tls_get_ctx(sk)->tx_lock);
break;
+ }
case TLS_TX_ZEROCOPY_RO:
lock_sock(sk);
rc = do_tls_setsockopt_tx_zc(sk, optval, optlen);
diff --git a/net/tls/tls_proc.c b/net/tls/tls_proc.c
index 4012c4372d4c0..6255f7b07eb76 100644
--- a/net/tls/tls_proc.c
+++ b/net/tls/tls_proc.c
@@ -27,6 +27,12 @@ static const struct snmp_mib tls_mib_list[] = {
SNMP_MIB_ITEM("TlsTxRekeyOk", LINUX_MIB_TLSTXREKEYOK),
SNMP_MIB_ITEM("TlsTxRekeyError", LINUX_MIB_TLSTXREKEYERROR),
SNMP_MIB_ITEM("TlsRxRekeyReceived", LINUX_MIB_TLSRXREKEYRECEIVED),
+ SNMP_MIB_ITEM("TlsTxRekeyFallback", LINUX_MIB_TLSTXREKEYFALLBACK),
+ SNMP_MIB_ITEM("TlsRxRekeyFallback", LINUX_MIB_TLSRXREKEYFALLBACK),
+ SNMP_MIB_ITEM("TlsCurrTxRekey", LINUX_MIB_TLSCURRTXREKEY),
+ SNMP_MIB_ITEM("TlsCurrRxRekey", LINUX_MIB_TLSCURRRXREKEY),
+ SNMP_MIB_ITEM("TlsTxRekeyAborted", LINUX_MIB_TLSTXREKEYABORTED),
+ SNMP_MIB_ITEM("TlsRxRekeyAborted", LINUX_MIB_TLSRXREKEYABORTED),
};
static int tls_statistics_seq_show(struct seq_file *seq, void *v)
diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index d1ad31986cf2c..d546091dd5240 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -522,7 +522,7 @@ static void tls_encrypt_done(void *data, int err)
complete(&ctx->async_wait.completion);
}
-static int tls_encrypt_async_wait(struct tls_sw_context_tx *ctx)
+int tls_encrypt_async_wait(struct tls_sw_context_tx *ctx)
{
if (!atomic_dec_and_test(&ctx->encrypt_pending))
crypto_wait_req(-EINPROGRESS, &ctx->async_wait);
@@ -555,11 +555,11 @@ static int tls_do_encryption(struct sock *sk,
break;
}
- memcpy(&rec->iv_data[iv_offset], tls_ctx->tx.iv,
+ memcpy(&rec->iv_data[iv_offset], tls_tx_cipher_ctx(tls_ctx)->iv,
prot->iv_size + prot->salt_size);
tls_xor_iv_with_seq(prot, rec->iv_data + iv_offset,
- tls_ctx->tx.rec_seq);
+ tls_tx_cipher_ctx(tls_ctx)->rec_seq);
sge->offset += prot->prepend_size;
sge->length -= prot->prepend_size;
@@ -610,7 +610,7 @@ static int tls_do_encryption(struct sock *sk,
/* Unhook the record from context if encryption is not failure */
ctx->open_rec = NULL;
- tls_advance_record_sn(sk, prot, &tls_ctx->tx);
+ tls_advance_record_sn(sk, prot, tls_tx_cipher_ctx(tls_ctx));
return rc;
}
@@ -676,7 +676,7 @@ static int tls_push_record(struct sock *sk, int flags,
sg_chain(rec->sg_aead_out, 2, &msg_en->sg.data[i]);
tls_make_aad(rec->aad_space, msg_pl->sg.size + prot->tail_size,
- tls_ctx->tx.rec_seq, record_type, prot);
+ tls_tx_cipher_ctx(tls_ctx)->rec_seq, record_type, prot);
tls_fill_prepend(tls_ctx,
page_address(sg_page(&msg_en->sg.data[i])) +
@@ -712,7 +712,7 @@ static int bpf_exec_tx_verdict(struct sk_msg *msg, struct sock *sk,
return err;
}
-static int tls_sw_push_pending_record(struct sock *sk, int flags)
+int tls_sw_push_pending_record(struct sock *sk, int flags)
{
struct tls_context *tls_ctx = tls_get_ctx(sk);
struct tls_sw_context_tx *ctx = tls_sw_ctx_tx(tls_ctx);
@@ -763,8 +763,7 @@ static int tls_sw_sendmsg_splice(struct sock *sk, struct msghdr *msg,
return 0;
}
-static int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg,
- size_t size)
+int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
{
long timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT);
struct tls_context *tls_ctx = tls_get_ctx(sk);
@@ -1027,8 +1026,13 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)
/*
* Handle unexpected EOF during splice without SPLICE_F_MORE set.
+ *
+ * Inner logic of tls_sw_splice_eof(), factored out so the device
+ * TX path can reuse it with tls_ctx->tx_lock and the socket lock
+ * already held. Callers not already holding both locks must use the
+ * tls_sw_splice_eof() wrapper instead.
*/
-void tls_sw_splice_eof(struct socket *sock)
+void tls_sw_splice_eof_locked(struct socket *sock)
{
struct sock *sk = sock->sk;
struct tls_context *tls_ctx = tls_get_ctx(sk);
@@ -1039,21 +1043,15 @@ void tls_sw_splice_eof(struct socket *sock)
bool retrying = false;
int ret = 0;
- if (!ctx->open_rec)
- return;
-
- mutex_lock(&tls_ctx->tx_lock);
- lock_sock(sk);
-
retry:
- /* same checks as in tls_sw_push_pending_record() */
+ /* same open_rec / empty-record checks as tls_sw_push_pending_record() */
rec = ctx->open_rec;
if (!rec)
- goto unlock;
+ return;
msg_pl = &rec->msg_plaintext;
if (msg_pl->sg.size == 0)
- goto unlock;
+ return;
/* Perform transmission. */
ret = bpf_exec_tx_verdict(msg_pl, sk, TLS_RECORD_TYPE_DATA,
@@ -1062,26 +1060,38 @@ void tls_sw_splice_eof(struct socket *sock)
case 0:
case -EAGAIN:
if (retrying)
- goto unlock;
+ return;
retrying = true;
goto retry;
case -EINPROGRESS:
break;
default:
- goto unlock;
+ return;
}
/* Wait for pending encryptions to get completed */
if (tls_encrypt_async_wait(ctx))
- goto unlock;
+ return;
/* Transmit if any encryptions have completed */
if (test_and_clear_bit(BIT_TX_SCHEDULED, &ctx->tx_bitmask)) {
cancel_delayed_work(&ctx->tx_work.work);
tls_tx_records(sk, 0);
}
+}
+
+void tls_sw_splice_eof(struct socket *sock)
+{
+ struct sock *sk = sock->sk;
+ struct tls_context *tls_ctx = tls_get_ctx(sk);
+ struct tls_sw_context_tx *ctx = tls_sw_ctx_tx(tls_ctx);
-unlock:
+ if (!ctx->open_rec)
+ return;
+
+ mutex_lock(&tls_ctx->tx_lock);
+ lock_sock(sk);
+ tls_sw_splice_eof_locked(sock);
release_sock(sk);
mutex_unlock(&tls_ctx->tx_lock);
}
@@ -1551,6 +1561,7 @@ static int tls_check_pending_rekey(struct sock *sk, struct tls_context *ctx,
if (hs_type == TLS_HANDSHAKE_KEYUPDATE) {
struct tls_sw_context_rx *rx_ctx = ctx->priv_ctx_rx;
+ tls_device_rx_del_key(sk, ctx);
WRITE_ONCE(rx_ctx->key_update_pending, true);
TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXREKEYRECEIVED);
}
@@ -2401,6 +2412,40 @@ static void tx_work_handler(struct work_struct *work)
}
}
+void tls_sw_ctx_tx_init(struct sock *sk, struct tls_sw_context_tx *sw_ctx)
+{
+ crypto_init_wait(&sw_ctx->async_wait);
+ atomic_set(&sw_ctx->encrypt_pending, 1);
+ INIT_LIST_HEAD(&sw_ctx->tx_list);
+ INIT_DELAYED_WORK(&sw_ctx->tx_work.work, tx_work_handler);
+ sw_ctx->tx_work.sk = sk;
+}
+
+int tls_sw_drain_tx(struct sock *sk, struct tls_context *ctx, int flags)
+{
+ struct tls_sw_context_tx *sw_ctx = tls_sw_ctx_tx(ctx);
+ int rc;
+
+ flags = (flags & MSG_DONTWAIT) | MSG_NOSIGNAL;
+
+ if (sw_ctx->open_rec)
+ tls_sw_push_pending_record(sk, flags);
+ rc = tls_encrypt_async_wait(sw_ctx);
+ if (rc)
+ return rc;
+ rc = tls_tx_records(sk, flags);
+ if (rc < 0 || tls_is_partially_sent_record(ctx) ||
+ tls_is_pending_open_record(ctx) ||
+ !list_empty(&sw_ctx->tx_list))
+ return rc < 0 ? rc : -EAGAIN;
+
+ tls_free_open_rec(sk);
+
+ cancel_delayed_work_sync(&sw_ctx->tx_work.work);
+ clear_bit(BIT_TX_SCHEDULED, &sw_ctx->tx_bitmask);
+ return 0;
+}
+
static bool tls_is_tx_ready(struct tls_sw_context_tx *ctx)
{
struct tls_rec *rec;
@@ -2452,11 +2497,7 @@ static struct tls_sw_context_tx *init_ctx_tx(struct tls_context *ctx, struct soc
sw_ctx_tx = ctx->priv_ctx_tx;
}
- crypto_init_wait(&sw_ctx_tx->async_wait);
- atomic_set(&sw_ctx_tx->encrypt_pending, 1);
- INIT_LIST_HEAD(&sw_ctx_tx->tx_list);
- INIT_DELAYED_WORK(&sw_ctx_tx->tx_work.work, tx_work_handler);
- sw_ctx_tx->tx_work.sk = sk;
+ tls_sw_ctx_tx_init(sk, sw_ctx_tx);
return sw_ctx_tx;
}
@@ -2522,20 +2563,19 @@ static void tls_finish_key_update(struct sock *sk, struct tls_context *tls_ctx)
ctx->saved_data_ready(sk);
}
-int tls_set_sw_offload(struct sock *sk, int tx,
- struct tls_crypto_info *new_crypto_info)
+int tls_sw_ctx_init(struct sock *sk, int tx,
+ struct tls_crypto_info *new_crypto_info)
{
struct tls_crypto_info *crypto_info, *src_crypto_info;
struct tls_sw_context_tx *sw_ctx_tx = NULL;
struct tls_sw_context_rx *sw_ctx_rx = NULL;
const struct tls_cipher_desc *cipher_desc;
- char *iv, *rec_seq, *key, *salt;
- struct cipher_context *cctx;
struct tls_prot_info *prot;
struct crypto_aead **aead;
struct tls_context *ctx;
struct crypto_tfm *tfm;
int rc = 0;
+ char *key;
ctx = tls_get_ctx(sk);
prot = &ctx->prot_info;
@@ -2556,12 +2596,10 @@ int tls_set_sw_offload(struct sock *sk, int tx,
if (tx) {
sw_ctx_tx = ctx->priv_ctx_tx;
crypto_info = &ctx->crypto_send.info;
- cctx = &ctx->tx;
aead = &sw_ctx_tx->aead_send;
} else {
sw_ctx_rx = ctx->priv_ctx_rx;
crypto_info = &ctx->crypto_recv.info;
- cctx = &ctx->rx;
aead = &sw_ctx_rx->aead_recv;
}
@@ -2577,11 +2615,12 @@ int tls_set_sw_offload(struct sock *sk, int tx,
if (rc)
goto free_priv;
- iv = crypto_info_iv(src_crypto_info, cipher_desc);
key = crypto_info_key(src_crypto_info, cipher_desc);
- salt = crypto_info_salt(src_crypto_info, cipher_desc);
- rec_seq = crypto_info_rec_seq(src_crypto_info, cipher_desc);
+ /* A rekey normally reuses the existing tfm; the RX HW rekey hands over a
+ * NULL aead (the old one is retained for the drain), so allocate and
+ * configure authsize only when a fresh tfm is created here.
+ */
if (!*aead) {
*aead = crypto_alloc_aead(cipher_desc->cipher_name, 0, 0);
if (IS_ERR(*aead)) {
@@ -2589,9 +2628,14 @@ int tls_set_sw_offload(struct sock *sk, int tx,
*aead = NULL;
goto free_priv;
}
+
+ rc = crypto_aead_setauthsize(*aead, prot->tag_size);
+ if (rc)
+ goto free_aead;
}
- ctx->push_pending_record = tls_sw_push_pending_record;
+ if (tx)
+ ctx->push_pending_record = tls_sw_push_pending_record;
/* setkey is the last operation that could fail during a
* rekey. if it succeeds, we can start modifying the
@@ -2605,12 +2649,6 @@ int tls_set_sw_offload(struct sock *sk, int tx,
goto free_aead;
}
- if (!new_crypto_info) {
- rc = crypto_aead_setauthsize(*aead, prot->tag_size);
- if (rc)
- goto free_aead;
- }
-
if (!tx && !new_crypto_info) {
tfm = crypto_aead_tfm(sw_ctx_rx->aead_recv);
@@ -2624,19 +2662,6 @@ int tls_set_sw_offload(struct sock *sk, int tx,
goto free_aead;
}
- memcpy(cctx->iv, salt, cipher_desc->salt);
- memcpy(cctx->iv + cipher_desc->salt, iv, cipher_desc->iv);
- memcpy(cctx->rec_seq, rec_seq, cipher_desc->rec_seq);
-
- if (new_crypto_info) {
- unsafe_memcpy(crypto_info, new_crypto_info,
- cipher_desc->crypto_info,
- /* size was checked in do_tls_setsockopt_conf */);
- memzero_explicit(new_crypto_info, cipher_desc->crypto_info);
- if (!tx)
- tls_finish_key_update(sk, ctx);
- }
-
goto out;
free_aead:
@@ -2655,3 +2680,57 @@ int tls_set_sw_offload(struct sock *sk, int tx,
out:
return rc;
}
+
+void tls_sw_ctx_finalize(struct sock *sk, int tx,
+ struct tls_crypto_info *new_crypto_info)
+{
+ struct tls_crypto_info *crypto_info, *src_crypto_info;
+ const struct tls_cipher_desc *cipher_desc;
+ struct tls_context *ctx = tls_get_ctx(sk);
+ struct cipher_context *cctx;
+ char *iv, *salt, *rec_seq;
+
+ if (tx) {
+ crypto_info = &ctx->crypto_send.info;
+ cctx = &ctx->tx;
+ } else {
+ crypto_info = &ctx->crypto_recv.info;
+ cctx = &ctx->rx;
+ }
+
+ src_crypto_info = new_crypto_info ?: crypto_info;
+
+ /* Infallible: tls_sw_ctx_init() already validated cipher_type. */
+ cipher_desc = get_cipher_desc(src_crypto_info->cipher_type);
+
+ iv = crypto_info_iv(src_crypto_info, cipher_desc);
+ salt = crypto_info_salt(src_crypto_info, cipher_desc);
+ rec_seq = crypto_info_rec_seq(src_crypto_info, cipher_desc);
+
+ memcpy(cctx->iv, salt, cipher_desc->salt);
+ memcpy(cctx->iv + cipher_desc->salt, iv, cipher_desc->iv);
+ memcpy(cctx->rec_seq, rec_seq, cipher_desc->rec_seq);
+
+ if (new_crypto_info) {
+ unsafe_memcpy(crypto_info, new_crypto_info,
+ cipher_desc->crypto_info,
+ /* size was checked in do_tls_setsockopt_conf */);
+ memzero_explicit(new_crypto_info, cipher_desc->crypto_info);
+
+ if (!tx)
+ tls_finish_key_update(sk, ctx);
+ }
+}
+
+int tls_set_sw_offload(struct sock *sk, int tx,
+ struct tls_crypto_info *new_crypto_info)
+{
+ int rc;
+
+ rc = tls_sw_ctx_init(sk, tx, new_crypto_info);
+ if (rc)
+ return rc;
+
+ tls_sw_ctx_finalize(sk, tx, new_crypto_info);
+ return 0;
+}
diff --git a/net/tls/trace.h b/net/tls/trace.h
index 2d8ce4ff3265b..5b9c1f86d82df 100644
--- a/net/tls/trace.h
+++ b/net/tls/trace.h
@@ -192,6 +192,124 @@ TRACE_EVENT(tls_device_tx_resync_send,
)
);
+TRACE_EVENT(tls_device_rekey_start,
+
+ TP_PROTO(struct sock *sk, u32 copied_seq, u32 nic_boundary,
+ bool inflight),
+
+ TP_ARGS(sk, copied_seq, nic_boundary, inflight),
+
+ TP_STRUCT__entry(
+ __field( struct sock *, sk )
+ __field( u32, copied_seq )
+ __field( u32, nic_boundary )
+ __field( bool, inflight )
+ ),
+
+ TP_fast_assign(
+ __entry->sk = sk;
+ __entry->copied_seq = copied_seq;
+ __entry->nic_boundary = nic_boundary;
+ __entry->inflight = inflight;
+ ),
+
+ TP_printk(
+ "sk=%p copied_seq=%u nic_boundary=%u inflight=%d",
+ __entry->sk, __entry->copied_seq, __entry->nic_boundary,
+ __entry->inflight
+ )
+);
+
+TRACE_EVENT(tls_device_rekey_reencrypt,
+
+ TP_PROTO(struct sock *sk, u32 tcp_seq, u32 nic_boundary),
+
+ TP_ARGS(sk, tcp_seq, nic_boundary),
+
+ TP_STRUCT__entry(
+ __field( struct sock *, sk )
+ __field( u32, tcp_seq )
+ __field( u32, nic_boundary )
+ ),
+
+ TP_fast_assign(
+ __entry->sk = sk;
+ __entry->tcp_seq = tcp_seq;
+ __entry->nic_boundary = nic_boundary;
+ ),
+
+ TP_printk(
+ "sk=%p tcp_seq=%u nic_boundary=%u",
+ __entry->sk, __entry->tcp_seq, __entry->nic_boundary
+ )
+);
+
+TRACE_EVENT(tls_device_rekey_done,
+
+ TP_PROTO(struct sock *sk, u32 tcp_seq, u32 nic_boundary),
+
+ TP_ARGS(sk, tcp_seq, nic_boundary),
+
+ TP_STRUCT__entry(
+ __field( struct sock *, sk )
+ __field( u32, tcp_seq )
+ __field( u32, nic_boundary )
+ ),
+
+ TP_fast_assign(
+ __entry->sk = sk;
+ __entry->tcp_seq = tcp_seq;
+ __entry->nic_boundary = nic_boundary;
+ ),
+
+ TP_printk(
+ "sk=%p tcp_seq=%u nic_boundary=%u",
+ __entry->sk, __entry->tcp_seq, __entry->nic_boundary
+ )
+);
+
+TRACE_EVENT(tls_device_complete_rekey_fail,
+
+ TP_PROTO(struct sock *sk, int rc),
+
+ TP_ARGS(sk, rc),
+
+ TP_STRUCT__entry(
+ __field( struct sock *, sk )
+ __field( int, rc )
+ ),
+
+ TP_fast_assign(
+ __entry->sk = sk;
+ __entry->rc = rc;
+ ),
+
+ TP_printk(
+ "sk=%p rc=%d",
+ __entry->sk, __entry->rc
+ )
+);
+
+TRACE_EVENT(tls_device_complete_rekey_retry,
+
+ TP_PROTO(struct sock *sk),
+
+ TP_ARGS(sk),
+
+ TP_STRUCT__entry(
+ __field( struct sock *, sk )
+ ),
+
+ TP_fast_assign(
+ __entry->sk = sk;
+ ),
+
+ TP_printk(
+ "sk=%p",
+ __entry->sk
+ )
+);
+
#endif /* _TLS_TRACE_H_ */
#undef TRACE_INCLUDE_PATH
diff --git a/tools/testing/selftests/drivers/net/hw/.gitignore b/tools/testing/selftests/drivers/net/hw/.gitignore
index 46540468a7753..911a9bfeaf41e 100644
--- a/tools/testing/selftests/drivers/net/hw/.gitignore
+++ b/tools/testing/selftests/drivers/net/hw/.gitignore
@@ -1,4 +1,5 @@
# SPDX-License-Identifier: GPL-2.0-only
iou-zcrx
ncdevmem
+tls_hw_offload
toeplitz
diff --git a/tools/testing/selftests/drivers/net/hw/Makefile b/tools/testing/selftests/drivers/net/hw/Makefile
index 8aebdc6feb177..b3831c2d09ea5 100644
--- a/tools/testing/selftests/drivers/net/hw/Makefile
+++ b/tools/testing/selftests/drivers/net/hw/Makefile
@@ -46,6 +46,7 @@ TEST_PROGS = \
rss_drv.py \
rss_flow_label.py \
rss_input_xfrm.py \
+ tls_hw_offload.py \
toeplitz.py \
tso.py \
userns_devmem.py \
@@ -80,6 +81,7 @@ YNL_GEN_FILES := \
# end of YNL_GEN_FILES
TEST_GEN_FILES += $(YNL_GEN_FILES)
TEST_GEN_FILES += $(patsubst %.c,%.o,$(wildcard *.bpf.c))
+TEST_GEN_FILES += tls_hw_offload
include ../../../lib.mk
diff --git a/tools/testing/selftests/drivers/net/hw/config b/tools/testing/selftests/drivers/net/hw/config
index d89a9ba176558..169e608516bd5 100644
--- a/tools/testing/selftests/drivers/net/hw/config
+++ b/tools/testing/selftests/drivers/net/hw/config
@@ -22,6 +22,8 @@ CONFIG_NET_IPIP=y
CONFIG_NETKIT=y
CONFIG_NET_SCH_INGRESS=y
CONFIG_SYNC_FILE=y
+CONFIG_TLS=y
+CONFIG_TLS_DEVICE=y
CONFIG_UDMABUF=y
CONFIG_USER_NS=y
CONFIG_VXLAN=y
diff --git a/tools/testing/selftests/drivers/net/hw/tls_hw_offload.c b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.c
new file mode 100644
index 0000000000000..303c6752ace27
--- /dev/null
+++ b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.c
@@ -0,0 +1,1132 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * TLS Hardware Offload Two-Node Test
+ *
+ * Tests kTLS hardware offload between two physical nodes using
+ * hardcoded keys. Supports TLS 1.2/1.3, AES-GCM-128/256, and rekey.
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+#include <limits.h>
+#include <time.h>
+#include <sys/time.h>
+#include <signal.h>
+#include <sys/types.h>
+#include <sys/socket.h>
+#include <netinet/in.h>
+#include <netinet/tcp.h>
+#include <netdb.h>
+#include <linux/tls.h>
+
+#define TLS_RECORD_TYPE_HANDSHAKE 22
+#define TLS_HANDSHAKE_KEY_UPDATE 0x18
+
+/* Large enough for a TLS 1.3 KeyUpdate handshake record's plaintext. */
+#define MIN_BUF_SIZE 16
+
+/* Initial key material */
+static struct tls12_crypto_info_aes_gcm_128 tls_info_key0_128 = {
+ .info = {
+ .version = TLS_1_3_VERSION,
+ .cipher_type = TLS_CIPHER_AES_GCM_128,
+ },
+ .iv = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08 },
+ .key = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08,
+ 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10 },
+ .salt = { 0x01, 0x02, 0x03, 0x04 },
+ .rec_seq = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
+};
+
+static struct tls12_crypto_info_aes_gcm_256 tls_info_key0_256 = {
+ .info = {
+ .version = TLS_1_3_VERSION,
+ .cipher_type = TLS_CIPHER_AES_GCM_256,
+ },
+ .iv = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08 },
+ .key = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08,
+ 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10,
+ 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18,
+ 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, 0x20 },
+ .salt = { 0x01, 0x02, 0x03, 0x04 },
+ .rec_seq = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 },
+};
+
+static int num_rekeys;
+static int num_iterations = 100;
+static int cipher_type = TLS_CIPHER_AES_GCM_128;
+static int tls_version = TLS_1_3_VERSION;
+static int server_port = 4433;
+static char *server_ip;
+/* Address family to force: AF_UNSPEC (any), AF_INET (-4), AF_INET6 (-6). */
+static int force_family = AF_UNSPEC;
+
+static int send_size = 16384;
+static int random_size_max;
+/* Burst mode: sender keeps pushing records without reading from the peer;
+ * receiver drains without echoing back. Only the client initiates rekey.
+ */
+static int burst_mode;
+static int zc_rx;
+
+/* XOR each byte with the generation so both endpoints derive the
+ * same per-generation key without a real KDF. Generation 0 leaves
+ * the base key unchanged.
+ */
+static void derive_key_fields(unsigned char *key, int key_size,
+ unsigned char *iv, int iv_size,
+ unsigned char *salt, int salt_size,
+ unsigned char *rec_seq, int rec_seq_size,
+ int generation)
+{
+ int i;
+
+ for (i = 0; i < key_size; i++)
+ key[i] ^= generation;
+ for (i = 0; i < iv_size; i++)
+ iv[i] ^= generation;
+ for (i = 0; i < salt_size; i++)
+ salt[i] ^= generation;
+ memset(rec_seq, 0, rec_seq_size);
+}
+
+static void derive_key_128(struct tls12_crypto_info_aes_gcm_128 *key,
+ int generation)
+{
+ memcpy(key, &tls_info_key0_128, sizeof(*key));
+ key->info.version = tls_version;
+ derive_key_fields(key->key, TLS_CIPHER_AES_GCM_128_KEY_SIZE,
+ key->iv, TLS_CIPHER_AES_GCM_128_IV_SIZE,
+ key->salt, TLS_CIPHER_AES_GCM_128_SALT_SIZE,
+ key->rec_seq, TLS_CIPHER_AES_GCM_128_REC_SEQ_SIZE,
+ generation);
+}
+
+static void derive_key_256(struct tls12_crypto_info_aes_gcm_256 *key,
+ int generation)
+{
+ memcpy(key, &tls_info_key0_256, sizeof(*key));
+ key->info.version = tls_version;
+ derive_key_fields(key->key, TLS_CIPHER_AES_GCM_256_KEY_SIZE,
+ key->iv, TLS_CIPHER_AES_GCM_256_IV_SIZE,
+ key->salt, TLS_CIPHER_AES_GCM_256_SALT_SIZE,
+ key->rec_seq, TLS_CIPHER_AES_GCM_256_REC_SEQ_SIZE,
+ generation);
+}
+
+static const char *cipher_name(int cipher)
+{
+ switch (cipher) {
+ case TLS_CIPHER_AES_GCM_128: return "AES-GCM-128";
+ case TLS_CIPHER_AES_GCM_256: return "AES-GCM-256";
+ default: return "unknown";
+ }
+}
+
+static const char *version_name(int version)
+{
+ switch (version) {
+ case TLS_1_2_VERSION: return "TLS 1.2";
+ case TLS_1_3_VERSION: return "TLS 1.3";
+ default: return "unknown";
+ }
+}
+
+static int setup_tls_ulp(int fd)
+{
+ int ret;
+
+ ret = setsockopt(fd, IPPROTO_TCP, TCP_ULP, "tls", sizeof("tls"));
+ if (ret < 0) {
+ printf("SETUP ERROR: TCP_ULP failed: %s\n", strerror(errno));
+ return -1;
+ }
+ return 0;
+}
+
+/* Echo (non-burst) mode drives both directions from a single thread: the
+ * client pushes a whole payload with one blocking send() and only reads the
+ * echo afterwards, while the server blocks in send() mid-echo. If a payload
+ * exceeds the peer's receive window the two sides deadlock - client stuck in
+ * send(), server stuck echoing, neither draining the other. Size the socket
+ * buffers so a full payload always fits in the peer's window (the forward
+ * send() then completes without needing the peer to read concurrently); the
+ * send/recv timeouts armed by set_io_timeouts() turn any residual stall into a
+ * loud EAGAIN instead of a hang.
+ */
+static void configure_echo_socket(int fd, int payload)
+{
+ int want = payload;
+
+ if (want < MIN_BUF_SIZE)
+ want = MIN_BUF_SIZE;
+
+ /* SO_*BUFFORCE bypasses the rmem_max/wmem_max sysctl caps (needs
+ * CAP_NET_ADMIN); fall back to the best-effort, cap-limited option
+ * when unprivileged - the timeouts below still turn any resulting
+ * stall into a loud failure rather than a hang.
+ */
+ if (setsockopt(fd, SOL_SOCKET, SO_RCVBUFFORCE, &want, sizeof(want)) < 0)
+ setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &want, sizeof(want));
+ if (setsockopt(fd, SOL_SOCKET, SO_SNDBUFFORCE, &want, sizeof(want)) < 0)
+ setsockopt(fd, SOL_SOCKET, SO_SNDBUF, &want, sizeof(want));
+}
+
+/* Arm send/recv timeouts so any unexpected stall fails loudly with EAGAIN
+ * instead of hanging until the harness SIGKILLs us. Wanted in both echo and
+ * burst modes - burst mode has no other stall guard.
+ */
+static void set_io_timeouts(int fd)
+{
+ struct timeval tv = { .tv_sec = 8, .tv_usec = 0 };
+
+ setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
+ setsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, &tv, sizeof(tv));
+}
+
+/* Send the whole buffer, looping over short counts. A blocking SOCK_STREAM
+ * send() may return fewer bytes than requested (e.g. when SO_SNDTIMEO fires
+ * after partial progress) without setting errno, so a short count is not an
+ * error - only a negative return is. Looping also sends each iteration as one
+ * uninterrupted run of bytes, which the peer's userspace reassembly in burst
+ * mode counts on to keep iterations aligned.
+ */
+static int send_all(int fd, const char *buf, ssize_t len)
+{
+ ssize_t sent = 0;
+ ssize_t ret;
+
+ while (sent < len) {
+ ret = send(fd, buf + sent, len - sent, 0);
+ if (ret < 0) {
+ printf("FAIL: send failed: %s\n", strerror(errno));
+ return -1;
+ }
+ sent += ret;
+ }
+ return 0;
+}
+
+static int set_zc_rx(int fd)
+{
+ int val = 1;
+
+ if (setsockopt(fd, SOL_TLS, TLS_RX_EXPECT_NO_PAD, &val,
+ sizeof(val)) < 0) {
+ printf("SETUP ERROR: TLS_RX_EXPECT_NO_PAD failed: %s\n",
+ strerror(errno));
+ return -1;
+ }
+ return 0;
+}
+
+/* Send a TLS 1.3 KeyUpdate handshake record. The kernel only
+ * inspects the HandshakeType byte to detect KeyUpdate, so don't
+ * bother with the 3-byte length or request_update fields.
+ */
+static int send_tls_key_update(int fd)
+{
+ char cmsg_buf[CMSG_SPACE(sizeof(unsigned char))];
+ unsigned char key_update_msg = TLS_HANDSHAKE_KEY_UPDATE;
+ struct msghdr msg = {0};
+ struct cmsghdr *cmsg;
+ struct iovec iov;
+
+ iov.iov_base = &key_update_msg;
+ iov.iov_len = sizeof(key_update_msg);
+
+ msg.msg_iov = &iov;
+ msg.msg_iovlen = 1;
+ msg.msg_control = cmsg_buf;
+ msg.msg_controllen = sizeof(cmsg_buf);
+
+ cmsg = CMSG_FIRSTHDR(&msg);
+ cmsg->cmsg_level = SOL_TLS;
+ cmsg->cmsg_type = TLS_SET_RECORD_TYPE;
+ cmsg->cmsg_len = CMSG_LEN(sizeof(unsigned char));
+ *CMSG_DATA(cmsg) = TLS_RECORD_TYPE_HANDSHAKE;
+ msg.msg_controllen = cmsg->cmsg_len;
+
+ if (sendmsg(fd, &msg, 0) < 0) {
+ printf("sendmsg KeyUpdate failed: %s\n", strerror(errno));
+ return -1;
+ }
+
+ printf("Sent TLS KeyUpdate handshake message\n");
+ return 0;
+}
+
+static int recv_tls_message(int fd, char *buf, size_t buflen, int *record_type,
+ int flags)
+{
+ char cmsg_buf[CMSG_SPACE(sizeof(unsigned char))];
+ struct msghdr msg = {0};
+ struct cmsghdr *cmsg;
+ struct iovec iov;
+ int ret;
+
+ iov.iov_base = buf;
+ iov.iov_len = buflen;
+
+ msg.msg_iov = &iov;
+ msg.msg_iovlen = 1;
+ msg.msg_control = cmsg_buf;
+ msg.msg_controllen = sizeof(cmsg_buf);
+
+ ret = recvmsg(fd, &msg, flags);
+ if (ret <= 0)
+ return ret;
+
+ cmsg = CMSG_FIRSTHDR(&msg);
+ if (cmsg && cmsg->cmsg_level == SOL_TLS &&
+ cmsg->cmsg_type == TLS_GET_RECORD_TYPE)
+ *record_type = *((unsigned char *)CMSG_DATA(cmsg));
+
+ return ret;
+}
+
+/* Confirm a handshake record starting with HandshakeType KeyUpdate. */
+static int check_keyupdate(const char *buf, int len, int record_type)
+{
+ if (record_type != TLS_RECORD_TYPE_HANDSHAKE) {
+ printf("Expected handshake record (0x%02x), got 0x%02x\n",
+ TLS_RECORD_TYPE_HANDSHAKE, record_type);
+ return -1;
+ }
+ if (len < 1 || (unsigned char)buf[0] != TLS_HANDSHAKE_KEY_UPDATE) {
+ printf("Expected KeyUpdate (0x%02x), got 0x%02x\n",
+ TLS_HANDSHAKE_KEY_UPDATE,
+ len ? (unsigned char)buf[0] : 0);
+ return -1;
+ }
+ printf("Received TLS KeyUpdate\n");
+ return 0;
+}
+
+static int recv_tls_keyupdate(int fd)
+{
+ char buf[MIN_BUF_SIZE];
+ int record_type = 0;
+ int ret;
+
+ ret = recv_tls_message(fd, buf, sizeof(buf), &record_type, 0);
+ if (ret < 0) {
+ printf("recv_tls_message failed: %s\n", strerror(errno));
+ return -1;
+ }
+
+ return check_keyupdate(buf, ret, record_type);
+}
+
+static int check_ekeyexpired(int fd)
+{
+ char buf[MIN_BUF_SIZE];
+ int ret;
+
+ ret = recv(fd, buf, sizeof(buf), MSG_DONTWAIT);
+ if (ret == -1 && errno == EKEYEXPIRED) {
+ printf("recv() returned EKEYEXPIRED as expected\n");
+ return 0;
+ }
+ if (ret > 0) {
+ printf("FAIL: recv() returned %d bytes, expected EKEYEXPIRED\n",
+ ret);
+ return -1;
+ }
+ if (ret == 0) {
+ printf("FAIL: connection closed during rekey\n");
+ return -1;
+ }
+ printf("FAIL: recv() returned unexpected error: %s\n",
+ strerror(errno));
+ return -1;
+}
+
+static int do_tls_rekey(int fd, int direction, int generation, int cipher)
+{
+ const char *dir = direction == TLS_TX ? "TX" : "RX";
+ int ret;
+
+ printf("%s TLS_%s %s gen %d...\n",
+ generation ? "Rekeying" : "Installing",
+ dir, cipher_name(cipher), generation);
+
+ if (cipher == TLS_CIPHER_AES_GCM_256) {
+ struct tls12_crypto_info_aes_gcm_256 key;
+
+ derive_key_256(&key, generation);
+ ret = setsockopt(fd, SOL_TLS, direction, &key, sizeof(key));
+ } else {
+ struct tls12_crypto_info_aes_gcm_128 key;
+
+ derive_key_128(&key, generation);
+ ret = setsockopt(fd, SOL_TLS, direction, &key, sizeof(key));
+ }
+
+ if (ret < 0) {
+ printf("%sTLS_%s %s gen %d failed: %s\n",
+ generation ? "" : "SETUP ERROR: ", dir,
+ cipher_name(cipher), generation, strerror(errno));
+ return -1;
+ }
+ printf("TLS_%s %s gen %d installed\n",
+ dir, cipher_name(cipher), generation);
+ return 0;
+}
+
+/* Open a TCP connection to server_ip:server_port, switch to the TLS
+ * ULP, and install initial generation-0 TX/RX keys. Works over IPv4 or
+ * IPv6: getaddrinfo() resolves server_ip (honouring any -4/-6 forced
+ * family and %zone scope IDs in link-local addresses). Returns the fd on
+ * success, -1 on error (with the fd already closed).
+ */
+static int client_connect_tls(void)
+{
+ struct addrinfo hints = {0}, *res, *rp;
+ char port_str[16];
+ int csk = -1;
+ int ret;
+
+ hints.ai_family = force_family;
+ hints.ai_socktype = SOCK_STREAM;
+ hints.ai_protocol = IPPROTO_TCP;
+ snprintf(port_str, sizeof(port_str), "%d", server_port);
+
+ ret = getaddrinfo(server_ip, port_str, &hints, &res);
+ if (ret) {
+ printf("SETUP ERROR: getaddrinfo(%s): %s\n", server_ip,
+ gai_strerror(ret));
+ return -1;
+ }
+
+ printf("Connecting to %s:%d...\n", server_ip, server_port);
+ for (rp = res; rp; rp = rp->ai_next) {
+ csk = socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol);
+ if (csk < 0)
+ continue;
+ if (connect(csk, rp->ai_addr, rp->ai_addrlen) == 0)
+ break;
+ close(csk);
+ csk = -1;
+ }
+ freeaddrinfo(res);
+
+ if (csk < 0) {
+ printf("SETUP ERROR: connect to %s:%d failed: %s\n",
+ server_ip, server_port, strerror(errno));
+ return -1;
+ }
+ printf("Connected!\n");
+
+ if (setup_tls_ulp(csk) < 0)
+ goto err;
+
+ if (do_tls_rekey(csk, TLS_TX, 0, cipher_type) < 0 ||
+ do_tls_rekey(csk, TLS_RX, 0, cipher_type) < 0)
+ goto err;
+
+ set_io_timeouts(csk);
+ if (!burst_mode)
+ configure_echo_socket(csk, random_size_max > 0 ?
+ random_size_max : send_size);
+
+ return csk;
+err:
+ close(csk);
+ return -1;
+}
+
+/* Drain `len` echoed bytes from the server and verify they match the
+ * payload we just sent.
+ */
+static int client_recv_echo(int fd, const char *sent, char *echo_buf,
+ ssize_t len)
+{
+ ssize_t total = 0;
+ ssize_t n;
+
+ while (total < len) {
+ n = recv(fd, echo_buf + total, len - total, 0);
+ if (n < 0) {
+ printf("FAIL: Echo recv failed: %s\n", strerror(errno));
+ return -1;
+ }
+ if (n == 0) {
+ printf("FAIL: Connection closed during echo\n");
+ return -1;
+ }
+ total += n;
+ }
+
+ if (memcmp(sent, echo_buf, len) != 0) {
+ printf("FAIL: Echo data mismatch!\n");
+ return -1;
+ }
+ printf("Received echo %zd bytes (ok)\n", total);
+ return 0;
+}
+
+/* Client side of a rekey: send KeyUpdate and rotate TX. In echo mode
+ * also wait for the peer's KeyUpdate and rotate RX.
+ */
+static int client_rekey(int fd, int generation)
+{
+ if (send_tls_key_update(fd) < 0) {
+ printf("FAIL: send KeyUpdate\n");
+ return -1;
+ }
+
+ if (do_tls_rekey(fd, TLS_TX, generation, cipher_type) < 0)
+ return -1;
+
+ if (burst_mode)
+ return 0;
+
+ if (recv_tls_keyupdate(fd) < 0) {
+ printf("FAIL: recv KeyUpdate from server\n");
+ return -1;
+ }
+
+ if (check_ekeyexpired(fd) < 0)
+ return -1;
+
+ return do_tls_rekey(fd, TLS_RX, generation, cipher_type);
+}
+
+static int do_client(void)
+{
+ char *buf = NULL, *echo_buf = NULL;
+ int max_size, rekey_interval;
+ int csk = -1, i;
+ int test_result = -1;
+ int current_gen = 0;
+ int next_rekey_at;
+ ssize_t n;
+
+ max_size = random_size_max > 0 ? random_size_max : send_size;
+ if (max_size < MIN_BUF_SIZE)
+ max_size = MIN_BUF_SIZE;
+ buf = malloc(max_size);
+ if (!burst_mode)
+ echo_buf = malloc(max_size);
+ if (!buf || (!burst_mode && !echo_buf)) {
+ printf("SETUP ERROR: failed to allocate buffers\n");
+ goto out;
+ }
+
+ csk = client_connect_tls();
+ if (csk < 0)
+ goto out;
+
+ if (num_rekeys)
+ printf("TLS %s setup complete. Will perform %d rekey(s).\n",
+ cipher_name(cipher_type), num_rekeys);
+ else
+ printf("TLS setup complete.\n");
+
+ if (random_size_max > 0)
+ printf("Sending %d messages of random size (1..%d bytes)...\n",
+ num_iterations, random_size_max);
+ else
+ printf("Sending %d messages of %d bytes...\n",
+ num_iterations, send_size);
+
+ rekey_interval = num_iterations / (num_rekeys + 1);
+ next_rekey_at = rekey_interval;
+
+ for (i = 1; i <= num_iterations; i++) {
+ int this_size;
+
+ if (random_size_max > 0)
+ this_size = (rand() % random_size_max) + 1;
+ else
+ this_size = send_size;
+
+ /* In burst mode, use a per-iteration fill pattern so the
+ * receiver can detect any plaintext corruption without a
+ * round-trip echo.
+ */
+ if (burst_mode) {
+ memset(buf, i & 0xFF, this_size);
+ } else {
+ int j;
+
+ for (j = 0; j < this_size; j++)
+ buf[j] = rand() & 0xFF;
+ }
+
+ if (send_all(csk, buf, this_size) < 0)
+ goto out;
+ n = this_size;
+
+ if (!burst_mode) {
+ printf("Sent %zd bytes (iteration %d)\n", n, i);
+ if (client_recv_echo(csk, buf, echo_buf, n) < 0)
+ goto out;
+ }
+
+ /* Rekey at intervals. In echo mode this is a full bidirectional
+ * exchange; in burst mode the client only rotates its TX key
+ * and sends KeyUpdate - the peer is expected to follow.
+ */
+ if (num_rekeys && current_gen < num_rekeys &&
+ i == next_rekey_at) {
+ current_gen++;
+ printf("\n=== Client Rekey gen %d ===\n", current_gen);
+
+ if (client_rekey(csk, current_gen) < 0)
+ goto out;
+
+ next_rekey_at += rekey_interval;
+ printf("=== Client Rekey gen %d Complete ===\n\n",
+ current_gen);
+ }
+ }
+
+ test_result = 0;
+out:
+ if (num_rekeys)
+ printf("Rekeys completed: %d/%d\n", current_gen, num_rekeys);
+ if (csk >= 0)
+ close(csk);
+ free(buf);
+ free(echo_buf);
+ return test_result;
+}
+
+/* Bind/listen on server_port, accept one client, switch to the TLS ULP
+ * and install initial generation-0 keys (plus zc_rx if requested).
+ * Returns the connected fd on success and writes the listener fd to
+ * *lsk_out so the caller can close it. Returns -1 on error, with all
+ * intermediate fds already closed and *lsk_out left at -1.
+ */
+static int server_accept_tls(int *lsk_out)
+{
+ struct addrinfo hints = {0}, *res, *rp;
+ int lsk = -1, csk, one = 1;
+ char port_str[16];
+ int ret;
+
+ *lsk_out = -1;
+
+ /* AI_PASSIVE gives a wildcard bind address for the chosen family
+ * (0.0.0.0 / ::). The family is forced by -4/-6; when unspecified,
+ * bind the first entry that works.
+ */
+ hints.ai_family = force_family;
+ hints.ai_socktype = SOCK_STREAM;
+ hints.ai_protocol = IPPROTO_TCP;
+ hints.ai_flags = AI_PASSIVE;
+ snprintf(port_str, sizeof(port_str), "%d", server_port);
+
+ ret = getaddrinfo(NULL, port_str, &hints, &res);
+ if (ret) {
+ printf("SETUP ERROR: getaddrinfo(port %d): %s\n", server_port,
+ gai_strerror(ret));
+ return -1;
+ }
+
+ for (rp = res; rp; rp = rp->ai_next) {
+ lsk = socket(rp->ai_family, rp->ai_socktype, rp->ai_protocol);
+ if (lsk < 0)
+ continue;
+ setsockopt(lsk, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one));
+ if (bind(lsk, rp->ai_addr, rp->ai_addrlen) == 0)
+ break;
+ close(lsk);
+ lsk = -1;
+ }
+ freeaddrinfo(res);
+
+ if (lsk < 0) {
+ printf("SETUP ERROR: failed to bind port %d: %s\n",
+ server_port, strerror(errno));
+ return -1;
+ }
+
+ if (listen(lsk, 1) < 0) {
+ printf("SETUP ERROR: listen failed: %s\n", strerror(errno));
+ close(lsk);
+ return -1;
+ }
+
+ printf("Server listening on port %d\n", server_port);
+ printf("Waiting for client connection...\n");
+
+ /* Bound accept() so a client that never connects (a deploy or connect
+ * failure on the peer) does not block the server forever and leak the
+ * process past the harness timeout. accept() honours SO_RCVTIMEO on the
+ * listening socket; the client connects right after wait_port_listen(),
+ * so 30s is generous.
+ */
+ {
+ struct timeval tv = { .tv_sec = 30, .tv_usec = 0 };
+
+ setsockopt(lsk, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
+ }
+
+ csk = accept(lsk, (struct sockaddr *)NULL, (socklen_t *)NULL);
+ if (csk < 0) {
+ if (errno == EAGAIN || errno == EWOULDBLOCK)
+ printf("SETUP ERROR: accept timed out; client never connected\n");
+ else
+ printf("SETUP ERROR: accept failed: %s\n", strerror(errno));
+ close(lsk);
+ return -1;
+ }
+ printf("Client connected!\n");
+
+ if (setup_tls_ulp(csk) < 0)
+ goto err;
+
+ if (do_tls_rekey(csk, TLS_TX, 0, cipher_type) < 0 ||
+ do_tls_rekey(csk, TLS_RX, 0, cipher_type) < 0)
+ goto err;
+
+ if (zc_rx && set_zc_rx(csk) < 0)
+ goto err;
+
+ set_io_timeouts(csk);
+ if (!burst_mode)
+ configure_echo_socket(csk, random_size_max > 0 ?
+ random_size_max : send_size);
+
+ *lsk_out = lsk;
+ return csk;
+err:
+ close(csk);
+ close(lsk);
+ return -1;
+}
+
+/* Server side of a rekey: confirm recv() reports EKEYEXPIRED, then rotate RX.
+ * In echo mode also send a KeyUpdate back and rotate TX.
+ */
+static int server_rekey(int fd, int generation)
+{
+ if (check_ekeyexpired(fd) < 0)
+ return -1;
+
+ if (do_tls_rekey(fd, TLS_RX, generation, cipher_type) < 0)
+ return -1;
+
+ if (burst_mode)
+ return 0;
+
+ if (send_tls_key_update(fd) < 0) {
+ printf("FAIL: send KeyUpdate\n");
+ return -1;
+ }
+
+ return do_tls_rekey(fd, TLS_TX, generation, cipher_type);
+}
+
+/* Burst mode: verify one reassembled iteration of send_size plaintext bytes,
+ * each filled with (send_iter & 0xff). Catches decrypt-succeeded-but-
+ * plaintext-corrupt bugs that AEAD counters alone would miss.
+ */
+static int server_verify_burst(const char *buf, int send_iter)
+{
+ unsigned char expect = send_iter & 0xFF;
+ int j;
+
+ for (j = 0; j < send_size; j++) {
+ if ((unsigned char)buf[j] != expect) {
+ printf("FAIL: data mismatch iter %d off %d: exp 0x%02x got 0x%02x\n",
+ send_iter, j, expect, (unsigned char)buf[j]);
+ return -1;
+ }
+ }
+ return 0;
+}
+
+static int do_server(void)
+{
+ int lsk = -1, csk = -1;
+ ssize_t n, total = 0;
+ int test_result = -1;
+ int current_gen = 0;
+ int recv_count = 0;
+ int send_iter = 1;
+ char *buf = NULL;
+ int record_type = 0;
+ int filled = 0;
+ int buf_size;
+
+ buf_size = send_size;
+ if (buf_size < MIN_BUF_SIZE)
+ buf_size = MIN_BUF_SIZE;
+ buf = malloc(buf_size);
+ if (!buf) {
+ printf("SETUP ERROR: failed to allocate buffer\n");
+ goto out;
+ }
+
+ csk = server_accept_tls(&lsk);
+ if (csk < 0)
+ goto out;
+
+ printf("TLS %s setup complete. Receiving...\n",
+ cipher_name(cipher_type));
+
+ /* Burst mode: reassemble one iteration (send_size bytes) in userspace
+ * from however much each recv returns, rather than demanding a full
+ * send_size batch in a single MSG_WAITALL call. A blocking MSG_WAITALL
+ * of send_size deadlocks when the client's last record of an iteration
+ * is still partly in flight as its socket buffer fills: the server
+ * waits for bytes the client cannot send until the server reads, and
+ * the server will not read until it has the whole batch. Draining
+ * whatever is available keeps the receive window open and breaks that
+ * cycle. kTLS never splits a record and returns data and control
+ * (KeyUpdate) records separately, and each iteration is a whole number
+ * of records, so capping each recv at the iteration boundary keeps the
+ * reassembly aligned and delivers a KeyUpdate on its own.
+ */
+
+ /* Main receive loop */
+ while (1) {
+ char *dst = burst_mode ? buf + filled : buf;
+ size_t want = burst_mode ? (size_t)(send_size - filled)
+ : (size_t)buf_size;
+
+ n = recv_tls_message(csk, dst, want, &record_type, 0);
+ if (n == 0) {
+ /* A clean close on an iteration boundary is success;
+ * one with a partial iteration still buffered means the
+ * peer dropped the tail - the truncated-data case this
+ * test exists to catch, so fail loudly.
+ */
+ if (burst_mode && filled) {
+ printf("FAIL: closed mid-iteration (%d/%d bytes buffered)\n",
+ filled, send_size);
+ goto out;
+ }
+ printf("Connection closed by client\n");
+ break;
+ }
+ if (n < 0) {
+ printf("FAIL: recv failed: %s\n", strerror(errno));
+ goto out;
+ }
+
+ /* Handle KeyUpdate. In echo mode the server mirrors the
+ * rekey back to the peer; in burst mode it only rotates its
+ * RX key and keeps draining. A KeyUpdate always lands on a
+ * send_size boundary, so no partial iteration must be buffered
+ * when one arrives.
+ */
+ if (record_type == TLS_RECORD_TYPE_HANDSHAKE) {
+ /* Check for a partial iteration before validating the
+ * KeyUpdate, so a mid-iteration arrival fails with this
+ * message rather than a misleading KeyUpdate-OK line.
+ */
+ if (burst_mode && filled) {
+ printf("FAIL: KeyUpdate mid-iteration (%d/%d bytes buffered)\n",
+ filled, send_size);
+ goto out;
+ }
+ if (check_keyupdate(dst, n, record_type) < 0)
+ goto out;
+ current_gen++;
+ printf("\n=== Server Rekey gen %d ===\n", current_gen);
+
+ if (server_rekey(csk, current_gen) < 0)
+ goto out;
+
+ printf("=== Server Rekey gen %d Complete ===\n\n",
+ current_gen);
+ continue;
+ }
+
+ total += n;
+
+ if (burst_mode) {
+ filled += n;
+ if (filled < send_size)
+ continue;
+ if (server_verify_burst(buf, send_iter) < 0)
+ goto out;
+ recv_count++;
+ send_iter++;
+ filled = 0;
+ continue;
+ }
+
+ recv_count++;
+ printf("Received %zd bytes (total: %zd, count: %d)\n",
+ n, total, recv_count);
+
+ if (send_all(csk, buf, n) < 0)
+ goto out;
+ printf("Echoed %zd bytes back to client\n", n);
+ }
+
+ test_result = 0;
+out:
+ printf("Connection closed. Total received: %zd bytes\n", total);
+ if (num_rekeys)
+ printf("Rekeys completed: %d\n", current_gen);
+
+ if (csk >= 0)
+ close(csk);
+ if (lsk >= 0)
+ close(lsk);
+ free(buf);
+ return test_result;
+}
+
+static int parse_int_arg(const char *arg, int min, int max,
+ const char *name, int *out)
+{
+ char *endp;
+ long val;
+
+ errno = 0;
+ val = strtol(arg, &endp, 10);
+ if (errno || endp == arg || *endp != '\0' || val < min || val > max) {
+ if (max == INT_MAX)
+ printf("ERROR: Invalid %s '%s'. Must be >= %d.\n",
+ name, arg, min);
+ else
+ printf("ERROR: Invalid %s '%s'. Must be %d..%d.\n",
+ name, arg, min, max);
+ return -1;
+ }
+ *out = (int)val;
+ return 0;
+}
+
+static int parse_cipher_option(const char *arg)
+{
+ if (strcmp(arg, "128") == 0) {
+ cipher_type = TLS_CIPHER_AES_GCM_128;
+ return 0;
+ } else if (strcmp(arg, "256") == 0) {
+ cipher_type = TLS_CIPHER_AES_GCM_256;
+ return 0;
+ }
+ printf("ERROR: Invalid cipher '%s'. Must be 128 or 256.\n", arg);
+ return -1;
+}
+
+static int parse_version_option(const char *arg)
+{
+ if (strcmp(arg, "1.2") == 0) {
+ tls_version = TLS_1_2_VERSION;
+ return 0;
+ } else if (strcmp(arg, "1.3") == 0) {
+ tls_version = TLS_1_3_VERSION;
+ return 0;
+ }
+ printf("ERROR: Invalid TLS version '%s'. Must be 1.2 or 1.3.\n", arg);
+ return -1;
+}
+
+static void print_usage(const char *prog)
+{
+ printf("TLS Hardware Offload Two-Node Test\n\n");
+ printf("Usage:\n");
+ printf(" %s server [OPTIONS]\n", prog);
+ printf(" %s client -s <ip> [OPTIONS]\n", prog);
+ printf("\nOptions:\n");
+ printf(" -s <ip> Server IP address, v4 or v6 (client, required)\n");
+ printf(" -p <port> Server port (default: 4433)\n");
+ printf(" -4 Force IPv4 (default: auto/either)\n");
+ printf(" -6 Force IPv6 (default: auto/either)\n");
+ printf(" -b <size> Send buffer size in bytes (default: 16384)\n");
+ printf(" -r <max> Use random send buffer sizes (1..<max>)\n");
+ printf(" -v <version> TLS version: 1.2 or 1.3 (default: 1.3)\n");
+ printf(" -c <cipher> Cipher: 128 or 256 (default: 128)\n");
+ printf(" -n <N> Number of send/echo iterations (default: 100)\n");
+ printf(" -k <N> Perform N rekeys (client only, TLS 1.3; N < iterations)\n");
+ printf(" -B Burst mode: client sends continuously without echo;\n");
+ printf(" server drains and handles KeyUpdate without responding.\n");
+ printf(" -Z Set TLS_RX_EXPECT_NO_PAD on the server: TLS 1.3\n");
+ printf(" opt-in to the zero-copy RX fast path. Not needed\n");
+ printf(" for TLS 1.2 (always eligible). Server only.\n");
+ printf(" -h Show this help message\n");
+ printf("\nExample:\n");
+ printf(" Node A: %s server\n", prog);
+ printf(" Node B: %s client -s 192.168.20.2\n", prog);
+ printf("\nRekey Example (3 rekeys, TLS 1.3 only):\n");
+ printf(" Node A: %s server\n", prog);
+ printf(" Node B: %s client -s 192.168.20.2 -k 3\n", prog);
+ printf("\nBurst Mode Example (client stresses TX rekey under load):\n");
+ printf(" Node A: %s server -B\n", prog);
+ printf(" Node B: %s client -s 192.168.20.2 -B -k 3\n", prog);
+ printf("\nIPv6 Example:\n");
+ printf(" Node A: %s server -6\n", prog);
+ printf(" Node B: %s client -6 -s fd00::2\n", prog);
+}
+
+int main(int argc, char *argv[])
+{
+ int send_size_set = 0;
+ int is_server;
+ int opt;
+
+ /* When the peer aborts a TLS connection (e.g. tls_err_abort() on a
+ * failed decrypt), a send() here would raise SIGPIPE and kill us by
+ * signal, so the harness sees only a bare non-zero exit with no
+ * "FAIL:" line. Ignore it and let send()/sendmsg() return EPIPE, which
+ * send_all()/send_tls_key_update() report.
+ */
+ signal(SIGPIPE, SIG_IGN);
+
+ if (argc < 2 ||
+ (strcmp(argv[1], "server") && strcmp(argv[1], "client"))) {
+ print_usage(argv[0]);
+ return 1;
+ }
+ is_server = !strcmp(argv[1], "server");
+
+ optind = 2; /* skip subcommand */
+ while ((opt = getopt(argc, argv, "s:p:b:r:c:v:k:n:BZ46h")) != -1) {
+ switch (opt) {
+ case 's':
+ server_ip = optarg;
+ break;
+ case '4':
+ if (force_family == AF_INET6) {
+ printf("ERROR: -4 and -6 are mutually exclusive\n");
+ return 1;
+ }
+ force_family = AF_INET;
+ break;
+ case '6':
+ if (force_family == AF_INET) {
+ printf("ERROR: -4 and -6 are mutually exclusive\n");
+ return 1;
+ }
+ force_family = AF_INET6;
+ break;
+ case 'B':
+ burst_mode = 1;
+ break;
+ case 'Z':
+ zc_rx = 1;
+ break;
+ case 'p':
+ if (parse_int_arg(optarg, 1, 65535, "port",
+ &server_port) < 0)
+ return 1;
+ break;
+ case 'b':
+ if (parse_int_arg(optarg, 1, INT_MAX, "buffer size",
+ &send_size) < 0)
+ return 1;
+ send_size_set = 1;
+ break;
+ case 'r':
+ if (parse_int_arg(optarg, 1, INT_MAX, "random size",
+ &random_size_max) < 0)
+ return 1;
+ break;
+ case 'c':
+ if (parse_cipher_option(optarg) < 0)
+ return 1;
+ break;
+ case 'v':
+ if (parse_version_option(optarg) < 0)
+ return 1;
+ break;
+ case 'k':
+ if (parse_int_arg(optarg, 1, 255, "rekey count",
+ &num_rekeys) < 0)
+ return 1;
+ break;
+ case 'n':
+ if (parse_int_arg(optarg, 1, INT_MAX, "iteration count",
+ &num_iterations) < 0)
+ return 1;
+ break;
+ case 'h':
+ print_usage(argv[0]);
+ return 0;
+ default:
+ print_usage(argv[0]);
+ return 1;
+ }
+ }
+
+ if (send_size_set && random_size_max > 0) {
+ printf("ERROR: -b and -r are mutually exclusive\n");
+ return 1;
+ }
+
+ if (zc_rx && tls_version != TLS_1_3_VERSION) {
+ printf("ERROR: -Z (TLS_RX_EXPECT_NO_PAD) requires TLS 1.3\n");
+ return 1;
+ }
+
+ if (burst_mode && random_size_max > 0) {
+ printf("ERROR: -B and -r are mutually exclusive\n");
+ return 1;
+ }
+
+ if (burst_mode && send_size < MIN_BUF_SIZE) {
+ printf("ERROR: -b must be >= %d in burst mode (-B)\n",
+ MIN_BUF_SIZE);
+ return 1;
+ }
+
+ if (is_server) {
+ if (server_ip) {
+ printf("warning: -s is ignored in server mode\n");
+ server_ip = NULL;
+ }
+ if (random_size_max > 0) {
+ printf("warning: -r is ignored in server mode\n");
+ random_size_max = 0;
+ }
+ if (num_rekeys) {
+ printf("warning: -k is ignored in server mode\n");
+ num_rekeys = 0;
+ }
+ } else {
+ if (!server_ip) {
+ printf("ERROR: Client requires -s <ip> option\n");
+ return 1;
+ }
+ if (tls_version == TLS_1_2_VERSION && num_rekeys) {
+ printf("ERROR: TLS 1.2 does not support rekey\n");
+ return 1;
+ }
+ if (num_rekeys >= num_iterations) {
+ printf("ERROR: num_rekeys (%d) must be < num_iterations (%d)\n",
+ num_rekeys, num_iterations);
+ return 1;
+ }
+ if (zc_rx) {
+ printf("ERROR: -Z applies to the server (receiver) only\n");
+ return 1;
+ }
+ }
+
+ printf("TLS Version: %s\n", version_name(tls_version));
+ printf("Cipher: %s\n", cipher_name(cipher_type));
+ printf("Address family: %s\n",
+ force_family == AF_INET ? "IPv4" :
+ force_family == AF_INET6 ? "IPv6" : "auto");
+ if (random_size_max > 0)
+ printf("Buffer size: random (1..%d)\n", random_size_max);
+ else
+ printf("Buffer size: %d\n", send_size);
+
+ if (num_rekeys)
+ printf("Rekey testing ENABLED: %d rekey(s)\n", num_rekeys);
+ if (burst_mode)
+ printf("Burst mode ENABLED\n");
+ if (zc_rx)
+ printf("TLS_RX_EXPECT_NO_PAD ENABLED\n");
+
+ srand(time(NULL));
+
+ if (is_server)
+ return do_server() ? 1 : 0;
+
+ return do_client() ? 1 : 0;
+}
diff --git a/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py
new file mode 100755
index 0000000000000..99ae5b3b8996a
--- /dev/null
+++ b/tools/testing/selftests/drivers/net/hw/tls_hw_offload.py
@@ -0,0 +1,446 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+"""Test kTLS hardware offload using a C helper binary."""
+
+from collections import defaultdict
+
+from lib.py import ksft_run, ksft_exit, ksft_pr, KsftSkipEx
+from lib.py import ksft_ge, ksft_eq
+from lib.py import ksft_variants, KsftNamedVariant
+from lib.py import NetDrvEpEnv
+from lib.py import cmd, bkg, wait_port_listen, rand_port
+from lib.py import CmdExitFailure
+
+# Burst variants push hundreds of MB and perform many rekeys, so they
+# need far longer than the default cmd() timeout.
+BURST_TIMEOUT_S = 180
+REKEY_TIMEOUT_S = 90
+
+# Reading /proc/net/tls_stat is trivial locally, but on the remote it runs
+# over ssh, where connection setup can occasionally spike past the short
+# default cmd() timeout. Give these tiny reads plenty of headroom so a slow
+# ssh round-trip doesn't fail an otherwise-good variant.
+STATS_TIMEOUT_S = 30
+
+# Per-packet HW crypto counters exposed via `ethtool -S` on the DUT NIC,
+# keyed by the `ethtool -i` driver name. TlsTxDevice/TlsRxDevice in
+# /proc/net/tls_stat only prove tls_dev_add() accepted the offload; these
+# increment once per packet the NIC actually encrypted/decrypted (mlx5 counts
+# gso_segs, not records), so they prove the HW crypto path was exercised.
+# Names are driver-specific, so the
+# check only runs on drivers listed here and is skipped (not failed) on
+# others, keeping the test portable across NICs.
+HW_CRYPTO_COUNTERS = {
+ 'mlx5_core': {'Tx': 'tx_tls_encrypted_packets',
+ 'Rx': 'rx_tls_decrypted_packets'},
+}
+
+
+def check_tls_support(cfg):
+ """Skip the suite unless both hosts have kTLS and the DUT HW offload."""
+ # The tls module is autoloaded lazily on the first TCP_ULP="tls"
+ # setsockopt, so /proc/net/tls_stat (created from the module's pernet
+ # init) may not exist yet on a freshly booted host. Load the module
+ # explicitly before probing for it.
+ try:
+ cmd("modprobe tls")
+ cmd("modprobe tls", host=cfg.remote)
+ cmd("test -f /proc/net/tls_stat")
+ cmd("test -f /proc/net/tls_stat", host=cfg.remote)
+ except CmdExitFailure as e:
+ raise KsftSkipEx(f"kTLS not supported: {e}") from e
+
+ try:
+ features = cmd(f"ethtool -k {cfg.ifname}").stdout
+ if 'tls-hw-tx-offload: on' not in features:
+ raise KsftSkipEx("Device does not support TLS HW TX offload")
+ if 'tls-hw-rx-offload: on' not in features:
+ raise KsftSkipEx("Device does not support TLS HW RX offload")
+ except CmdExitFailure as e:
+ raise KsftSkipEx(f"Cannot determine TLS HW offload support: {e}") from e
+
+
+def read_tls_stats(host=None):
+ """Snapshot the per-netns TLS MIB from /proc/net/tls_stat as a dict."""
+ # /proc/net/tls_stat exposes the per-netns TLS MIB (TLS_INC_STATS on
+ # sock_net(sk)). The test runs on a real NIC in the host namespace, so
+ # these counters are shared with anything else doing kTLS there. The
+ # strict before/after delta checks (exact rekey-outcome sums, zero error
+ # counters) assume no other kTLS activity in this namespace during a
+ # variant's window; concurrent kTLS users would perturb the deltas and
+ # cause spurious failures. Don't run other kTLS workloads alongside this
+ # test.
+ stats = defaultdict(int)
+ output = cmd("cat /proc/net/tls_stat", host=host, timeout=STATS_TIMEOUT_S)
+ for line in output.stdout.strip().split('\n'):
+ parts = line.split()
+ if len(parts) == 2:
+ stats[parts[0]] = int(parts[1])
+ return stats
+
+
+def nic_driver(cfg):
+ """DUT NIC driver name from `ethtool -i`, or None if undetermined."""
+ try:
+ output = cmd(f"ethtool -i {cfg.ifname}").stdout
+ except CmdExitFailure:
+ return None
+ for line in output.splitlines():
+ if line.startswith('driver:'):
+ return line.split(':', 1)[1].strip()
+ return None
+
+
+def read_nic_stats(cfg):
+ """Snapshot the DUT NIC's `ethtool -S` counters as a dict."""
+ # Driver per-record TLS counters from `ethtool -S` on the DUT NIC. Same
+ # before/after-delta caveat as read_tls_stats(): these are device-wide,
+ # so concurrent kTLS traffic on this NIC would perturb the deltas.
+ stats = defaultdict(int)
+ output = cmd(f"ethtool -S {cfg.ifname}").stdout
+ for line in output.strip().split('\n'):
+ key, sep, val = line.partition(':')
+ if sep and val.strip().isdigit():
+ stats[key.strip()] = int(val.strip())
+ return stats
+
+
+def stat_diff(before, after, key):
+ """Return the delta of counter `key` between two stat snapshots."""
+ return after[key] - before[key]
+
+
+def check_hw_crypto(cfg, before, after, with_tx, with_rx):
+ """DUT-side ethtool -S check: the NIC actually crypto'd records in HW.
+
+ Complements the TlsTxDevice/TlsRxDevice MIBs, which only confirm the
+ offload was installed, not that any record was processed in hardware.
+ Driver-specific; skipped (without failing) on drivers not in
+ HW_CRYPTO_COUNTERS so the test stays portable.
+ """
+ counters = HW_CRYPTO_COUNTERS.get(cfg.nic_driver)
+ if not counters:
+ ksft_pr(f"NOTE: DUT driver '{cfg.nic_driver}' has no known per-record "
+ f"HW crypto counters, skipping ethtool -S check")
+ return
+
+ for direction, active in (('Tx', with_tx), ('Rx', with_rx)):
+ if not active:
+ continue
+ key = counters[direction]
+ if key not in after:
+ ksft_pr(f"NOTE: DUT {direction}: counter '{key}' not exposed by "
+ f"{cfg.nic_driver}, skipping")
+ continue
+ got = stat_diff(before, after, key)
+ ksft_ge(got, 1,
+ comment=f"DUT {direction}: NIC reported no HW crypto "
+ f"({key}={got})")
+
+
+def check_path(before, after, direction, role, require_hw):
+ """On the DUT, require HW offload; on the remote, HW or SW is fine."""
+ dev = stat_diff(before, after, f'Tls{direction}Device')
+ sw = stat_diff(before, after, f'Tls{direction}Sw')
+ if require_hw:
+ ksft_ge(dev, 1,
+ comment=f"{role} {direction}: HW offload not engaged "
+ f"(Device={dev}, Sw={sw})")
+ else:
+ ksft_ge(dev + sw, 1,
+ comment=f"{role} {direction}: no TLS activity "
+ f"(Device={dev}, Sw={sw})")
+
+
+def verify_tls_counters(stats_before, stats_after, expected_rekeys,
+ tls_role, is_dut, burst=False, allow_fallback=False):
+ """Verify TLS counters on one side of the connection.
+
+ tls_role: 'client' or 'server' (TLS role this side played).
+ is_dut: True for the local DUT; requires HW offload counters.
+ burst: burst mode - only the TLS client rotates its TX key; the TLS
+ server only follows with an RX rotation on KeyUpdate receipt.
+ allow_fallback: tolerate rekeys completing in SW (TlsRx/TxRekeyFallback).
+ Default False: a rekey on an up, offload-capable device must stay
+ in HW, so any fallback is a regression. Set True only where SW
+ fallback is expected (e.g. a mid-connection link-flap variant, or
+ the peer, whose offload state is not under test).
+ """
+ role = 'DUT' if is_dut else 'Peer'
+
+ def diff(key):
+ return stat_diff(stats_before, stats_after, key)
+
+ # In burst mode the TLS client only TXs and the TLS server only RXs.
+ # In echo mode both sides drive both directions.
+ with_tx = not burst or tls_role == 'client'
+ with_rx = not burst or tls_role != 'client'
+
+ if with_tx:
+ check_path(stats_before, stats_after, 'Tx', role, require_hw=is_dut)
+ if with_rx:
+ check_path(stats_before, stats_after, 'Rx', role, require_hw=is_dut)
+
+ if expected_rekeys > 0:
+ if with_tx:
+ # Each KeyUpdate yields exactly one terminal outcome, so
+ # TlsTxRekeyOk + TlsTxRekeyAborted + TlsTxRekeyFallback == N.
+ # At most one rekey can be PENDING at socket close (single
+ # TLS_TX_REKEY_PENDING bit), so at most one lands in
+ # TlsTxRekeyAborted. TlsTxRekeyFallback is a legitimate, graceful
+ # degradation: the device did not (re)install the HW context for
+ # that rekey (device gone, dev_add rejected, or a transient
+ # crypto/alloc error) so it completed in SW while the kernel
+ # returned success. It is recoverable - the next KeyUpdate
+ # re-attempts HW offload (tls_device_start_rekey() clears
+ # TLS_TX_REKEY_FAILED). It is folded into the outcome sum below; on
+ # the DUT it must be 0 (allow_fallback=False), on the peer it is
+ # only NOTEd. A genuine rekey bug still surfaces as TlsTxRekeyError.
+ ksft_ge(1, diff('TlsTxRekeyAborted'),
+ comment=f"{role} Tx: TlsTxRekeyAborted expected <= 1")
+ ksft_eq(diff('TlsTxRekeyOk') + diff('TlsTxRekeyAborted') +
+ diff('TlsTxRekeyFallback'), expected_rekeys,
+ comment=f"{role} Tx: rekey outcomes must sum to "
+ f"{expected_rekeys}")
+ fallback = diff('TlsTxRekeyFallback')
+ if allow_fallback:
+ if fallback:
+ ksft_pr(f"NOTE: {role} Tx: {fallback} rekey(s) completed "
+ f"in SW (TlsTxRekeyFallback); HW not re-installed")
+ else:
+ ksft_eq(fallback, 0,
+ comment=f"{role} Tx: TlsTxRekeyFallback expected 0 "
+ f"(rekey must stay in HW offload)")
+ ksft_eq(diff('TlsTxRekeyError'), 0,
+ comment=f"{role} Tx: TlsTxRekeyError expected 0")
+ ksft_eq(diff('TlsCurrTxRekey'), 0,
+ comment=f"{role} Tx: TlsCurrTxRekey expected 0")
+ if with_rx:
+ # As on TX, each received KeyUpdate yields one terminal outcome:
+ # TlsRxRekeyOk + TlsRxRekeyAborted + TlsRxRekeyFallback == N.
+ # At most one rekey can be deferred (single dev_add_pending) at
+ # socket close, landing in TlsRxRekeyAborted. TlsRxRekeyFallback
+ # is a recoverable, graceful degradation (dev_add failed or the
+ # device was gone, so RX temporarily dropped to SW; the next
+ # KeyUpdate re-adds the HW context and clears TLS_RX_DEV_DEGRADED).
+ # It is folded into the outcome sum below; on the DUT it must be 0
+ # (allow_fallback=False), on the peer it is only NOTEd. A genuine
+ # rekey bug still surfaces as TlsRxRekeyError.
+ ksft_ge(1, diff('TlsRxRekeyAborted'),
+ comment=f"{role} Rx: TlsRxRekeyAborted expected <= 1")
+ ksft_eq(diff('TlsRxRekeyOk') + diff('TlsRxRekeyAborted') +
+ diff('TlsRxRekeyFallback'), expected_rekeys,
+ comment=f"{role} Rx: rekey outcomes must sum to "
+ f"{expected_rekeys}")
+ ksft_eq(diff('TlsRxRekeyReceived'), expected_rekeys,
+ comment=f"{role} Rx: TlsRxRekeyReceived expected "
+ f"{expected_rekeys}")
+ fallback = diff('TlsRxRekeyFallback')
+ if allow_fallback:
+ if fallback:
+ ksft_pr(f"NOTE: {role} Rx: {fallback} rekey(s) completed "
+ f"in SW (TlsRxRekeyFallback); HW not re-installed")
+ else:
+ ksft_eq(fallback, 0,
+ comment=f"{role} Rx: TlsRxRekeyFallback expected 0 "
+ f"(rekey must stay in HW offload)")
+ ksft_eq(diff('TlsRxRekeyError'), 0,
+ comment=f"{role} Rx: TlsRxRekeyError expected 0")
+ ksft_eq(diff('TlsCurrRxRekey'), 0,
+ comment=f"{role} Rx: TlsCurrRxRekey expected 0")
+
+ ksft_eq(diff('TlsDecryptError'), 0,
+ comment=f"{role}: TlsDecryptError expected 0")
+
+
+def run_tls_test(cfg, cipher="128", tls_version="1.3", rekey=0,
+ buffer_size=None, random_max=None, burst=False, zc=False,
+ dut_role="client", num_iterations=None, ipver="4"):
+ """Run the TLS offload test.
+
+ dut_role: 'client' (default) - DUT runs the TLS client, remote the server.
+ 'server' - swap: DUT listens, remote connects. Used for burst_rx
+ so the DUT's RX path is the one under rekey pressure.
+
+ ipver: '4' or '6' - IP version to run over. The C helper is forced to the
+ matching family with -4/-6 and connects to the peer's v4/v6 address.
+ Variants requesting '6' skip cleanly when the environment lacks IPv6
+ connectivity (require_ipver()).
+
+ The DUT (local) is the kernel under test; the remote is just a traffic
+ source/sink and may run any kernel without HW offload. Both sides run
+ kTLS because TLS is pairwise, but verify_tls_counters() requires HW
+ offload only on the DUT (is_dut=True); the peer may use SW kTLS.
+
+ Rekey/burst variants additionally require the peer to support TLS 1.3
+ KeyUpdate (as the RX or TX side of the rotation). SW KeyUpdate and its
+ MIB counters landed together in v6.14; an older peer cannot follow the
+ rotation, so those variants are skipped rather than failed when the peer
+ lacks the rekey counters (see the probe below).
+ """
+ cfg.require_ipver(ipver)
+
+ port = rand_port()
+ send_size = random_max or buffer_size
+
+ if dut_role == "client":
+ server_bin, server_host = cfg.bin_remote, cfg.remote
+ client_bin, client_host = cfg.bin_local, None
+ client_target = cfg.remote_addr_v[ipver]
+ else:
+ server_bin, server_host = cfg.bin_local, None
+ client_bin, client_host = cfg.bin_remote, cfg.remote
+ client_target = cfg.addr_v[ipver]
+
+ server_parts = [f"{server_bin} server -p {port} -c {cipher}",
+ f"-v {tls_version}", f"-{ipver}"]
+ if burst:
+ server_parts.append("-B")
+ if zc:
+ server_parts.append("-Z")
+ if send_size:
+ server_parts.append(f"-b {send_size}")
+ server_cmd = " ".join(server_parts)
+
+ client_parts = [f"{client_bin} client -s {client_target}",
+ f"-p {port} -c {cipher} -v {tls_version} -{ipver}"]
+ if rekey:
+ client_parts.append(f"-k {rekey}")
+ if burst:
+ client_parts.append("-B")
+ if num_iterations:
+ client_parts.append(f"-n {num_iterations}")
+ if random_max:
+ client_parts.append(f"-r {random_max}")
+ elif buffer_size:
+ client_parts.append(f"-b {buffer_size}")
+ client_cmd = " ".join(client_parts)
+
+ if burst:
+ cmd_timeout = BURST_TIMEOUT_S
+ elif rekey:
+ cmd_timeout = REKEY_TIMEOUT_S
+ else:
+ cmd_timeout = 20
+
+ stats_before_local = read_tls_stats()
+ stats_before_remote = read_tls_stats(host=cfg.remote)
+ nic_before = read_nic_stats(cfg)
+
+ # /proc/net/tls_stat lists every MIB the running kernel knows (0 or not),
+ # so a missing name means the peer predates that counter. The base rekey
+ # counters (TlsRxRekeyReceived, Tls{Rx,Tx}RekeyOk, Tls{Rx,Tx}RekeyError)
+ # shipped with SW KeyUpdate in v6.14; a peer without them cannot follow a
+ # KeyUpdate, so the rekey/burst variants can't run against it. Skip cleanly
+ # here rather than letting the peer-side rekey-sum / RxRekeyReceived checks
+ # report a confusing "expected N, got 0" later. TlsRxRekeyReceived is a
+ # reliable probe: the peer must bump it to have processed the rotation at all.
+ #
+ # Only a base v6.14 counter is probed. The newer HW-path MIBs (Aborted,
+ # Fallback, CurrRekey) are structurally 0 on a SW-only peer and defaultdict
+ # returns 0 for absent names, so the peer-side checks hold either way.
+ if rekey and 'TlsRxRekeyReceived' not in stats_before_remote:
+ raise KsftSkipEx("Peer kernel lacks TLS 1.3 KeyUpdate support "
+ "(no rekey MIB counters); required for rekey tests")
+
+ with bkg(server_cmd, host=server_host, exit_wait=True):
+ wait_port_listen(port, host=server_host)
+ # Start the client in the background so we keep a handle to it. A
+ # foreground cmd() raises TimeoutExpired from inside its constructor
+ # if the client hangs, and since the child is not killed on timeout
+ # it would be left running with no handle to reap it. A leaked
+ # client keeps bumping the per-netns TLS counters (TlsTxRekeyAborted,
+ # TlsDecryptError, ...) and would corrupt the before/after
+ # measurement window of a later variant. The finally clause reaps it
+ # within this variant's window instead.
+ client = cmd(client_cmd, host=client_host, background=True)
+ try:
+ client.process(terminate=False, fail=True, timeout=cmd_timeout)
+ finally:
+ if client.proc.poll() is None:
+ client.process(terminate=True, fail=False, timeout=5)
+
+ stats_after_local = read_tls_stats()
+ stats_after_remote = read_tls_stats(host=cfg.remote)
+ nic_after = read_nic_stats(cfg)
+
+ peer_tls_role = 'server' if dut_role == 'client' else 'client'
+
+ # Which directions the DUT drives (mirrors verify_tls_counters()): in
+ # burst mode the TLS client only TXs and the server only RXs; echo mode
+ # drives both.
+ dut_with_tx = not burst or dut_role == 'client'
+ dut_with_rx = not burst or dut_role != 'client'
+
+ verify_tls_counters(stats_before_local, stats_after_local,
+ rekey, dut_role, is_dut=True, burst=burst)
+ check_hw_crypto(cfg, nic_before, nic_after, dut_with_tx, dut_with_rx)
+ verify_tls_counters(stats_before_remote, stats_after_remote,
+ rekey, peer_tls_role, is_dut=False, burst=burst,
+ allow_fallback=True)
+
+
+# The cipher/version matrix runs over IPv4; the socket setup is the only
+# IP-version-specific code path, so a single representative variant over
+# IPv6 is enough to cover it (it skips cleanly without v6 connectivity).
+# The rekey and burst suites below likewise stay on IPv4 to bound runtime.
+@ksft_variants([
+ KsftNamedVariant("tls13_aes128", "128", "1.3", "4"),
+ KsftNamedVariant("tls13_aes256", "256", "1.3", "4"),
+ KsftNamedVariant("tls12_aes128", "128", "1.2", "4"),
+ KsftNamedVariant("tls12_aes256", "256", "1.2", "4"),
+ KsftNamedVariant("tls13_aes128_ip6", "128", "1.3", "6"),
+])
+def test_tls_offload(cfg, cipher, tls_version, ipver):
+ """Cipher/version matrix over the HW offload data path, no rekey."""
+ run_tls_test(cfg, cipher=cipher, tls_version=tls_version, ipver=ipver)
+
+
+@ksft_variants([
+ KsftNamedVariant("single", 1),
+ KsftNamedVariant("multiple", 99),
+ KsftNamedVariant("small_buf", 30, 512),
+ KsftNamedVariant("large_buf", 10, 2097152),
+ KsftNamedVariant("random_buf", 20, None, 8192),
+])
+def test_tls_offload_rekey(cfg, rekey, buffer_size=None, random_max=None):
+ """Echo-mode TLS 1.3 KeyUpdate rekeys across a range of buffer sizes."""
+ run_tls_test(cfg, cipher="128", tls_version="1.3", rekey=rekey,
+ buffer_size=buffer_size, random_max=random_max)
+
+
+# Columns: dut_role zc interval rekeys buffer_size
+@ksft_variants([
+ KsftNamedVariant("burst_tx_rekey_every_1", "client", False, 1, 50, 65536),
+ KsftNamedVariant("burst_tx_rekey_every_1000", "client", False, 1000, 3, 65536),
+ KsftNamedVariant("burst_rx_rekey_every_10", "server", False, 10, 20, 65536),
+ KsftNamedVariant("burst_rx_rekey_every_10000", "server", False, 10000, 1, 32768),
+ KsftNamedVariant("burst_rx_zc_rekey_every_100", "server", True, 100, 10, 65536),
+ KsftNamedVariant("burst_rx_zc_rekey_every_20000", "server", True, 20000, 1, 16384),
+])
+def test_tls_offload_burst(cfg, dut_role, zc, interval, rekeys, buffer_size):
+ """High-volume one-directional traffic with frequent rekeys."""
+ run_tls_test(cfg, cipher="128", tls_version="1.3", rekey=rekeys,
+ buffer_size=buffer_size, burst=True, zc=zc, dut_role=dut_role,
+ num_iterations=interval * (rekeys + 1))
+
+
+def main() -> None:
+ """Set up the DUT/peer environment and run the offload test suites."""
+ with NetDrvEpEnv(__file__, nsim_test=False) as cfg:
+ cfg.bin_local = cfg.test_dir / "tls_hw_offload"
+ if not cfg.bin_local.exists():
+ raise KsftSkipEx(f"tls_hw_offload binary not found at {cfg.bin_local}")
+ cfg.bin_remote = cfg.remote.deploy(cfg.bin_local)
+ cfg.require_ipver("4")
+ check_tls_support(cfg)
+ cfg.nic_driver = nic_driver(cfg)
+
+ ksft_run([test_tls_offload, test_tls_offload_rekey,
+ test_tls_offload_burst], args=(cfg, ))
+ ksft_exit()
+
+
+if __name__ == "__main__":
+ main()
|
| 48/3 | 2026/09/20 01:06 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 72170 output: 21 thoughts: 1094 Thoughts: **KMSAN Evaluation of kTLS TLS 1.3 Offload Patch**
Okay, let's break this down. The task is to determine if this kTLS patch series, which adds TLS 1.3 hardware offload and rekeying support, necessitates a dedicated KMSAN fuzzing session. My expertise tells me we need to be very precise here; KMSAN is resource-intensive, so we only use it when *absolutely* necessary. KASAN and the standard detectors are already pretty comprehensive.
First, I need to remind myself of the critical distinction: KMSAN is all about detecting uninitialized memory reads and information leaks. KASAN already handles out-of-bounds, use-after-free, double-free, and general memory corruption. Therefore, the goal is to see if this patch introduces risks *specific* to uninitialized memory. If KASAN could flag it, then there is no need for KMSAN.
The guidelines are clear. We set `NeedsKMSAN=true` only if the patch:
1. Exposes kernel structures to user space with potential uninitialized fields.
2. Involves conditional logic based on potentially uninitialized variables.
3. Deals with complex data structures where uninitialized fields could be read later.
4. Modifies bounds checks in a way that allows access to uninitialized data.
Conversely, we set `NeedsKMSAN=false` if the changes primarily risk out-of-bounds, overflows, NULL dereferences, or use-after-free bugs. It's also false if all touched structures are fully initialized or zeroed before reading or copying, and if it introduces no risk of uninitialized memory.
So, let's analyze the patch. It's focused on adding TLS 1.3 hardware offload, primarily to `net/tls` and Mellanox's `mlx5` driver.
Let's review the diffs:
1. **Documentation/Selftests**: Documentation updates and user-space selftests don't directly introduce KMSAN risks.
2. **Drivers**:
* `chcr_ktls.c` and `nfp/crypto/tls.c`: Simply reject TLS 1.3 for Chelsio and Netronome. No uninitialized memory concerns here.
* `mlx5`: Now *this* is interesting. The `mlx5` part is the key:
* `mlx5e_ktls_type_check` explicitly *allows* TLS 1.3.
* `mlx5e_ktls_handle_rx_skb`: Adds a `decrypt_failed` flag to the `sk_buff` structure. This `decrypt_failed` flag is a single bit field within a byte in `struct sk_buff`. The code seems to copy this flag around. The crucial question is: is the `sk_buff` itself initialized and zeroed before use? Specifically, are the bitfields around `decrypted` initialized? This *could* be a KMSAN candidate, if `decrypt_failed` is read before initialization.
* `fill_static_params`: This adds `iv` and `iv_sz` to the `EXTRACT_INFO_FIELDS`. Importantly, the code distinguishes between TLS 1.2 and TLS 1.3 and uses `memcpy` to copy the `iv` (initialization vector) to `gcm_iv`. This `iv` copying is our biggest concern. Is `iv` from the hardware, or is it handled in a way where it could be uninitialized?
3. **Include files**: The `skbuff.h` change adds the `decrypt_failed` bitfield, and the code copies this bit. The key here is checking how `sk_buff` is initialized. If it's zeroed on allocation, or if fields are cleared, it's safe. I need to investigate `__alloc_skb` or how the bitfields near `decrypt_failed` are initialized. I also need to check how the IV is handled.
Okay, my initial thoughts are: The code is *careful* to select based on the TLS version. The most significant KMSAN concern is with the Mellanox driver's handling of the `iv` and `decrypt_failed` flag within the `sk_buff`. Specifically, in the mlx5 driver:
1. How is `iv` initialized? If the hardware provides this data, is there any possibility it's uninitialized if something goes wrong?
2. Is `sk_buff` zeroed before use? I *strongly suspect* this is the case, but it needs verification. If it's *not* zeroed, then `decrypt_failed` (and other bitfields) could be
|
| 49/3 | 2026/09/20 01:06 | tool | grepper |
0mArgs:{
"Expression": "skb_copy_decrypted"
}Results: {
"Output": "include/linux/skbuff.h=1710=static inline bool skb_is_decrypted(const struct sk_buff *skb)\n--\ninclude/linux/skbuff.h-1718-\ninclude/linux/skbuff.h:1719:static inline void skb_copy_decrypted(struct sk_buff *to,\ninclude/linux/skbuff.h-1720-\t\t\t\t const struct sk_buff *from)\n--\nnet/ipv4/tcp_input.c=5773=tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,\n--\nnet/ipv4/tcp_input.c-5836-\t\tmemcpy(nskb-\u003ecb, skb-\u003ecb, sizeof(skb-\u003ecb));\nnet/ipv4/tcp_input.c:5837:\t\tskb_copy_decrypted(nskb, skb);\nnet/ipv4/tcp_input.c-5838-\t\tTCP_SKB_CB(nskb)-\u003eseq = TCP_SKB_CB(nskb)-\u003eend_seq = start;\n--\nnet/ipv4/tcp_output.c=1844=int tcp_fragment(struct sock *sk, enum tcp_queue tcp_queue,\n--\nnet/ipv4/tcp_output.c-1880-\t\treturn -ENOMEM; /* We'll just try again later. */\nnet/ipv4/tcp_output.c:1881:\tskb_copy_decrypted(buff, skb);\nnet/ipv4/tcp_output.c-1882-\tmptcp_skb_ext_copy(buff, skb);\n--\nnet/ipv4/tcp_output.c=2408=static int tso_fragment(struct sock *sk, struct sk_buff *skb, unsigned int len,\n--\nnet/ipv4/tcp_output.c-2420-\t\treturn -ENOMEM;\nnet/ipv4/tcp_output.c:2421:\tskb_copy_decrypted(buff, skb);\nnet/ipv4/tcp_output.c-2422-\tmptcp_skb_ext_copy(buff, skb);\n--\nnet/ipv4/tcp_output.c=2692=static int tcp_mtu_probe(struct sock *sk)\n--\nnet/ipv4/tcp_output.c-2772-\tskb = tcp_send_head(sk);\nnet/ipv4/tcp_output.c:2773:\tskb_copy_decrypted(nskb, skb);\nnet/ipv4/tcp_output.c-2774-\tmptcp_skb_ext_copy(nskb, skb);\n--\nnet/tls/tls_strp.c=342=static int tls_strp_copyin(read_descriptor_t *desc, struct sk_buff *in_skb,\n--\nnet/tls/tls_strp.c-353-\tif (!skb-\u003elen)\nnet/tls/tls_strp.c:354:\t\tskb_copy_decrypted(skb, in_skb);\nnet/tls/tls_strp.c-355-\telse\n"
} |
| 50/3 | 2026/09/20 01:06 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 75119 output: 32 thoughts: 23 Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience: **Analyzing sk_buff Initialization** Okay, so I'm digging into `net/core/skbuff.c` again. I want to trace the initialization of the `sk_buff`'s bitfields. I need to understand exactly where these crucial control flags and offsets are first set within the `sk_buff` structure. It's time to follow the trail from allocation all the way through to initial configuration. This is likely going to involve looking at the more low-level allocation functions, the socket creation routines, and any helper functions related to buffer construction. I'm assuming that the more esoteric bitfields get initialized earlier, closer to the actual raw buffer, while higher-level flags might be set during the more typical socket operations. Time to fire up the source code browser! |
| 51/3 | 2026/09/20 01:07 | tool | grepper |
0mArgs:{
"Expression": "decrypted\\b",
"PathPrefix": "net/core/skbuff.c"
}Results: {
"Output": "net/core/skbuff.c=4287=int skb_shift(struct sk_buff *tgt, struct sk_buff *skb, int shiftlen)\n--\nnet/core/skbuff.c-4299-\tDEBUG_NET_WARN_ON_ONCE(tgt-\u003epp_recycle != skb-\u003epp_recycle);\nnet/core/skbuff.c:4300:\tDEBUG_NET_WARN_ON_ONCE(skb_cmp_decrypted(tgt, skb));\nnet/core/skbuff.c-4301-\n"
} |
| 52/3 | 2026/09/20 01:07 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 75505 output: 38 thoughts: 1611 Thoughts: **Deep Dive into TLS Offload and Rekeying** Okay, let's break this down. First, I'm starting with the low-level memory initialization in `__alloc_skb`, where `memset` is used to zero out the `skb` header up to `headers_start`. This ensures all bitfields are initialized to a known state, a crucial starting point for any network packet manipulation. Now, let's pivot to `tcp_write_collapse_fence` and `tcp_rtx_queue_tail`. We see a conditional check for an existing `skb` and, if one exists, the `eor` flag is set within the `TCP_SKB_CB`. It's a fundamental piece of functionality for ensuring proper packet handling, especially during retransmissions. Next, I'm examining `include/net/tls.h`. The reordering of `struct cipher_context` and `union tls_crypto_context` is interesting, hinting at optimization or restructuring within TLS context. The introduction of the `rekey` struct in both `tls_offload_context_tx`, `tls_context` and `tls_offload_context_rx` is a clear indication that we're dealing with a rekeying mechanism. The added flags in `enum tls_context_flags` further emphasize this, with `tls_sw_ctx_tx`, `tls_tx_cipher_ctx`, and `tls_is_skb_tx_device_offloaded` all checking or utilizing the `rekey` context. And the check of `skb_is_decrypted` is a hint regarding how encrypted data should be handled by the offload context. Then, moving to `net/sched/sch_generic.c`, the `dequeue_skb` function is where the decryption check comes in. If a packet has already been decrypted using the offload context, this flag is set. Now, into `net/tls/tls_device.c`. `tls_device_free_ctx` is key here. It shows that in case of HW offload, there's explicit deallocation and zeroing of the `rekey` members and the contexts. I am paying close attention to the `memzero_explicit` calls – these are *essential* for security to prevent sensitive data leaks. The cautious deallocation of `old_aead_recv` in the RX context highlights a defensive programming approach, minimizing potential security vulnerabilities. Let's switch to `tls_device_init`. Allocating `dummy_page` with `__GFP_ZERO` is incredibly important! It guarantees that the initial content is zeroed, which is crucial for the identity mapping used for the record type fallback and ensuring security. Using `__GFP_ZERO` is a very good practice! Now for the really interesting part: `tls_device_record_close`. The critical part here is the conditional logic based on the `pfrag` and the appending of the TLS tail (tag or content_type + tag). If `skb_page_frag_refill` fails, it falls back to the `dummy_page` allocated and initialized earlier! The comments state that the device is supposed to fill in the tag. This is where I started to realize that the *hardware* is going to replace a *placeholder* tag with the *actual authentication tag*. This suggests *TX hardware offload*! And by the look of it, the code *doesn't* protect against the case where the packet gets retransmitted on fallback. Let's dig deeper in `tls_device_fallback.c` by reading `tls_enc_skb` and `tls_enc_record` to see how the tag is handled in the fallback case. I have to read into the `tls_device_fallback.c` file for the details, because right now, this is where it gets interesting! |
| 53/3 | 2026/09/20 01:07 | tool | read-file |
0mArgs:{
"File": "net/tls/tls_device_fallback.c",
"FirstLine": 55,
"LineCount": 50
}Results: {
"Contents": " 55:\t\n 56:\t\tcipher_desc = get_cipher_desc(prot-\u003ecipher_type);\n 57:\t\tDEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc-\u003eoffloadable);\n 58:\t\n 59:\t\tbuf_size = prot-\u003eprepend_size;\n 60:\t\tlen = min_t(int, *in_len, buf_size);\n 61:\t\n 62:\t\tmemcpy_from_scatterwalk(buf, in, len);\n 63:\t\tmemcpy_to_scatterwalk(out, buf, len);\n 64:\t\n 65:\t\t*in_len -= len;\n 66:\t\tif (!*in_len)\n 67:\t\t\treturn 0;\n 68:\t\n 69:\t\tlen = buf[4] | (buf[3] \u003c\u003c 8);\n 70:\t\tif (prot-\u003eversion != TLS_1_3_VERSION)\n 71:\t\t\tlen -= cipher_desc-\u003eiv;\n 72:\t\n 73:\t\ttls_make_aad(aad, len - cipher_desc-\u003etag, (char *)\u0026rcd_sn, buf[0], prot);\n 74:\t\n 75:\t\tif (prot-\u003eversion == TLS_1_3_VERSION) {\n 76:\t\t\tvoid *iv_src = crypto_info_iv(\u0026tls_ctx-\u003ecrypto_send.info,\n 77:\t\t\t\t\t\t cipher_desc);\n 78:\t\n 79:\t\t\tmemcpy(iv + cipher_desc-\u003esalt, iv_src, cipher_desc-\u003eiv);\n 80:\t\t} else {\n 81:\t\t\tmemcpy(iv + cipher_desc-\u003esalt, buf + TLS_HEADER_SIZE,\n 82:\t\t\t cipher_desc-\u003eiv);\n 83:\t\t}\n 84:\t\n 85:\t\ttls_xor_iv_with_seq(prot, iv, (char *)\u0026rcd_sn);\n 86:\t\n 87:\t\tsg_init_table(sg_in, ARRAY_SIZE(sg_in));\n 88:\t\tsg_init_table(sg_out, ARRAY_SIZE(sg_out));\n 89:\t\tsg_set_buf(sg_in, aad, prot-\u003eaad_size);\n 90:\t\tsg_set_buf(sg_out, aad, prot-\u003eaad_size);\n 91:\t\tscatterwalk_get_sglist(in, sg_in + 1);\n 92:\t\tscatterwalk_get_sglist(out, sg_out + 1);\n 93:\t\n 94:\t\t*in_len -= len;\n 95:\t\tif (*in_len \u003c 0) {\n 96:\t\t\t*in_len += cipher_desc-\u003etag;\n 97:\t\t\t/* the input buffer doesn't contain the entire record.\n 98:\t\t\t * trim len accordingly. The resulting authentication tag\n 99:\t\t\t * will contain garbage, but we don't care, so we won't\n 100:\t\t\t * include any of it in the output skb\n 101:\t\t\t * Note that we assume the output buffer length\n 102:\t\t\t * is larger then input buffer length + tag size\n 103:\t\t\t */\n 104:\t\t\tif (*in_len \u003c 0)\n"
} |
| 54/3 | 2026/09/20 01:07 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 78804 output: 37 thoughts: 482 Thoughts: **Analyzing `do_tls_getsockopt_conf` for Potential Vulnerabilities** Okay, let's dive into `do_tls_getsockopt_conf` within `net/tls/tls_main.c`. My immediate focus is triggered by guideline item 1, the one about kernel structures potentially leaking uninitialized data to user space. Always a prime target for scrutiny. Specifically, I need to see what the diff changed in `do_tls_getsockopt_conf`. The code snippet shows a clear modification, particularly around line 459. Previously, a simple assignment of `crypto_info` and `cctx` was being done for the transmit path. Now, there's a more complex selection logic. We're now using `tls_tx_cipher_ctx(ctx)` to select `cctx`, which is good. We also see that `crypto_info` may now be pulled from `tls_offload_ctx_tx(ctx)->rekey.crypto_send.info` if a rekey is in progress, otherwise it remains as `ctx->crypto_send.info`. The comments state that this change is intended to ensure `getsockopt` reports the IV/rec_seq used by `sendmsg`, accounting for pending rekeys. That makes sense. The comment also states that `lock_sock` is held so there is no race. Now, I'll need to examine how these structures are populated and, most importantly, if there's any scenario where these `crypto_info` or `cctx` structures, or any of their member fields, *could* be partially initialized before being copied to user space. It is key to understand if the rekey path and the standard path have consistent initialization strategies. I'll need to trace the data flow from these structures to the user space output, paying close attention to any uses of `copy_to_user`, `put_user`, or similar functions that could inadvertently expose kernel data. It is important to know if the rekey path initialization is always guaranteed to be complete before the data is copied out to the user space. |
| 55/3 | 2026/09/20 01:07 | tool | read-file |
0mArgs:{
"File": "net/tls/tls_main.c",
"FirstLine": 440,
"LineCount": 80
}Results: {
"Contents": " 440:\t\n 441:\tstatic int do_tls_getsockopt_conf(struct sock *sk, sockopt_t *opt, int tx)\n 442:\t{\n 443:\t\tint rc = 0;\n 444:\t\tconst struct tls_cipher_desc *cipher_desc;\n 445:\t\tstruct tls_context *ctx = tls_get_ctx(sk);\n 446:\t\tstruct tls_crypto_info *crypto_info;\n 447:\t\tstruct cipher_context *cctx;\n 448:\t\tint len = opt-\u003eoptlen;\n 449:\t\n 450:\t\tif (!opt-\u003eiter_out.ubuf || len \u003c sizeof(*crypto_info)) {\n 451:\t\t\trc = -EINVAL;\n 452:\t\t\tgoto out;\n 453:\t\t}\n 454:\t\n 455:\t\tif (!ctx) {\n 456:\t\t\trc = -EBUSY;\n 457:\t\t\tgoto out;\n 458:\t\t}\n 459:\t\n 460:\t\t/* get user crypto info */\n 461:\t\tif (tx) {\n 462:\t\t\t/* Select the cipher context via the same accessor the data path\n 463:\t\t\t * uses, so getsockopt reports the IV/rec_seq that sendmsg encrypts\n 464:\t\t\t * with (the pending rekey's while one is in flight, else the\n 465:\t\t\t * active key). crypto_info has no accessor; select it the same way.\n 466:\t\t\t * lock_sock is held, so rekey.cipher_ctx cannot change under us.\n 467:\t\t\t */\n 468:\t\t\tcctx = tls_tx_cipher_ctx(ctx);\n 469:\t\t\tif (ctx-\u003erekey.cipher_ctx)\n 470:\t\t\t\tcrypto_info = \u0026tls_offload_ctx_tx(ctx)-\u003erekey.crypto_send.info;\n 471:\t\t\telse\n 472:\t\t\t\tcrypto_info = \u0026ctx-\u003ecrypto_send.info;\n 473:\t\t} else {\n 474:\t\t\tcrypto_info = \u0026ctx-\u003ecrypto_recv.info;\n 475:\t\t\tcctx = \u0026ctx-\u003erx;\n 476:\t\t}\n 477:\t\n 478:\t\tif (!TLS_CRYPTO_INFO_READY(crypto_info)) {\n 479:\t\t\trc = -EBUSY;\n 480:\t\t\tgoto out;\n 481:\t\t}\n 482:\t\n 483:\t\tif (len == sizeof(*crypto_info)) {\n 484:\t\t\tif (copy_to_iter(crypto_info, sizeof(*crypto_info),\n 485:\t\t\t\t\t \u0026opt-\u003eiter_out) != sizeof(*crypto_info))\n 486:\t\t\t\trc = -EFAULT;\n 487:\t\t\tgoto out;\n 488:\t\t}\n 489:\t\n 490:\t\tcipher_desc = get_cipher_desc(crypto_info-\u003ecipher_type);\n 491:\t\tif (!cipher_desc || len != cipher_desc-\u003ecrypto_info) {\n 492:\t\t\trc = -EINVAL;\n 493:\t\t\tgoto out;\n 494:\t\t}\n 495:\t\n 496:\t\tmemcpy(crypto_info_iv(crypto_info, cipher_desc),\n 497:\t\t cctx-\u003eiv + cipher_desc-\u003esalt, cipher_desc-\u003eiv);\n 498:\t\tmemcpy(crypto_info_rec_seq(crypto_info, cipher_desc),\n 499:\t\t cctx-\u003erec_seq, cipher_desc-\u003erec_seq);\n 500:\t\n 501:\t\tif (copy_to_iter(crypto_info, cipher_desc-\u003ecrypto_info,\n 502:\t\t\t\t \u0026opt-\u003eiter_out) != cipher_desc-\u003ecrypto_info)\n 503:\t\t\trc = -EFAULT;\n 504:\t\n 505:\tout:\n 506:\t\treturn rc;\n 507:\t}\n 508:\t\n 509:\tstatic int do_tls_getsockopt_tx_zc(struct sock *sk, sockopt_t *opt)\n 510:\t{\n 511:\t\tstruct tls_context *ctx = tls_get_ctx(sk);\n 512:\t\tunsigned int value;\n 513:\t\tint len = opt-\u003eoptlen;\n 514:\t\n 515:\t\tif (len != sizeof(value))\n 516:\t\t\treturn -EINVAL;\n 517:\t\n 518:\t\tvalue = ctx-\u003ezerocopy_sendfile;\n 519:\t\tif (copy_to_iter(\u0026value, sizeof(value), \u0026opt-\u003eiter_out) != sizeof(value))\n"
} |
| 56/3 | 2026/09/20 01:07 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 81006 output: 27 thoughts: 173 Thoughts: **Deep Dive into `do_tls_getsockopt_conf` and Rekeying Logic** Okay, let's dissect this `do_tls_getsockopt_conf` function. I need to understand the flow here, especially the rekeying process. The critical question burning in my mind is: where does `crypto_info` get its data when `ctx->rekey.cipher_ctx` is true? That's the key to understanding this. Ah, it appears to be pulled from `crypto_info = &tls_offload_ctx_tx(ctx)->rekey.crypto_send.info;`. Right, that points me towards needing to understand the initialization of `tls_offload_ctx_tx(ctx)->rekey.crypto_send.info`. Time to backtrack. Let's trace where `rekey.crypto_send` gets written to within `net/tls/tls_device.c`. That's where the heart of this initialization lies, I reckon. I'm now going to focus my attention on the lines in `net/tls/tls_device.c` involving `unsafe_memcpy(&offload_ctx->rekey.crypto_send.info, new_crypto_info, ...)` and its surroundings. That should reveal how the `crypto_send.info` structure is populated during rekeying and give me the context I need to properly evaluate this socket option. This is how I'll get to the bottom of the security implications. |
| 57/3 | 2026/09/20 01:07 | tool | grepper |
0mArgs:{
"Expression": "crypto_send",
"PathPrefix": "net/tls/"
}Results: {
"Output": "net/tls/tls_device.c=1596=static int tls_device_start_rekey(struct sock *sk,\n--\nnet/tls/tls_device.c-1741-\nnet/tls/tls_device.c:1742:\tunsafe_memcpy(\u0026offload_ctx-\u003erekey.crypto_send.info, new_crypto_info,\nnet/tls/tls_device.c-1743-\t\t cipher_desc-\u003ecrypto_info,\n--\nnet/tls/tls_device.c=1750=static int tls_device_complete_rekey(struct sock *sk, struct tls_context *ctx,\n--\nnet/tls/tls_device.c-1760-\nnet/tls/tls_device.c:1761:\tcipher_desc = get_cipher_desc(offload_ctx-\u003erekey.crypto_send.info.cipher_type);\nnet/tls/tls_device.c-1762-\tDEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc-\u003eoffloadable);\n--\nnet/tls/tls_device.c-1808-\t */\nnet/tls/tls_device.c:1809:\tkey = crypto_info_key(\u0026offload_ctx-\u003erekey.crypto_send.info, cipher_desc);\nnet/tls/tls_device.c-1810-\tnew_aead = tls_device_build_rekey_aead(cipher_desc, key, CRYPTO_ALG_ASYNC);\n--\nnet/tls/tls_device.c-1815-\nnet/tls/tls_device.c:1816:\t/* crypto_send.info.rec_seq is frozen at setsockopt time; the SW context\nnet/tls/tls_device.c-1817-\t * advanced rekey.tx.rec_seq for every record it sent, so hand the NIC the\n--\nnet/tls/tls_device.c-1819-\t */\nnet/tls/tls_device.c:1820:\tmemcpy(crypto_info_rec_seq(\u0026offload_ctx-\u003erekey.crypto_send.info, cipher_desc),\nnet/tls/tls_device.c-1821-\t offload_ctx-\u003erekey.tx.rec_seq, cipher_desc-\u003erec_seq);\nnet/tls/tls_device.c-1822-\nnet/tls/tls_device.c:1823:\trc = tls_device_dev_add_tx(sk, netdev, \u0026offload_ctx-\u003erekey.crypto_send.info,\nnet/tls/tls_device.c-1824-\t\t\t\t tcp_sk(sk)-\u003ewrite_seq);\n--\nnet/tls/tls_device.c-1841-\t cipher_desc-\u003erec_seq);\nnet/tls/tls_device.c:1842:\tunsafe_memcpy(\u0026ctx-\u003ecrypto_send.info,\nnet/tls/tls_device.c:1843:\t\t \u0026offload_ctx-\u003erekey.crypto_send.info,\nnet/tls/tls_device.c-1844-\t\t cipher_desc-\u003ecrypto_info,\n--\nnet/tls/tls_device.c=2041=int tls_set_device_offload(struct sock *sk,\n--\nnet/tls/tls_device.c-2055-\nnet/tls/tls_device.c:2056:\tcrypto_info = \u0026ctx-\u003ecrypto_send.info;\nnet/tls/tls_device.c-2057-\tsrc_crypto_info = new_crypto_info ?: crypto_info;\n--\nnet/tls/tls_device_fallback.c=40=static int tls_enc_record(struct tls_context *tls_ctx,\n--\nnet/tls/tls_device_fallback.c-75-\tif (prot-\u003eversion == TLS_1_3_VERSION) {\nnet/tls/tls_device_fallback.c:76:\t\tvoid *iv_src = crypto_info_iv(\u0026tls_ctx-\u003ecrypto_send.info,\nnet/tls/tls_device_fallback.c-77-\t\t\t\t\t cipher_desc);\n--\nnet/tls/tls_device_fallback.c=291=static void fill_sg_out(struct scatterlist sg_out[3], void *buf,\n--\nnet/tls/tls_device_fallback.c-299-\tconst struct tls_cipher_desc *cipher_desc =\nnet/tls/tls_device_fallback.c:300:\t\tget_cipher_desc(tls_ctx-\u003ecrypto_send.info.cipher_type);\nnet/tls/tls_device_fallback.c-301-\n--\nnet/tls/tls_device_fallback.c=309=static struct sk_buff *tls_enc_skb(struct tls_context *tls_ctx,\n--\nnet/tls/tls_device_fallback.c-327-\nnet/tls/tls_device_fallback.c:328:\tcipher_desc = get_cipher_desc(tls_ctx-\u003ecrypto_send.info.cipher_type);\nnet/tls/tls_device_fallback.c-329-\tDEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc-\u003eoffloadable);\n--\nnet/tls/tls_device_fallback.c-340-\tiv = buf;\nnet/tls/tls_device_fallback.c:341:\tsalt = crypto_info_salt(\u0026tls_ctx-\u003ecrypto_send.info, cipher_desc);\nnet/tls/tls_device_fallback.c-342-\tmemcpy(iv, salt, cipher_desc-\u003esalt);\n--\nnet/tls/tls_main.c=323=void tls_ctx_free(struct sock *sk, struct tls_context *ctx)\n--\nnet/tls/tls_main.c-327-\nnet/tls/tls_main.c:328:\tmemzero_explicit(\u0026ctx-\u003ecrypto_send, sizeof(ctx-\u003ecrypto_send));\nnet/tls/tls_main.c-329-\tmemzero_explicit(\u0026ctx-\u003ecrypto_recv, sizeof(ctx-\u003ecrypto_recv));\n--\nnet/tls/tls_main.c=441=static int do_tls_getsockopt_conf(struct sock *sk, sockopt_t *opt, int tx)\n--\nnet/tls/tls_main.c-469-\t\tif (ctx-\u003erekey.cipher_ctx)\nnet/tls/tls_main.c:470:\t\t\tcrypto_info = \u0026tls_offload_ctx_tx(ctx)-\u003erekey.crypto_send.info;\nnet/tls/tls_main.c-471-\t\telse\nnet/tls/tls_main.c:472:\t\t\tcrypto_info = \u0026ctx-\u003ecrypto_send.info;\nnet/tls/tls_main.c-473-\t} else {\n--\nnet/tls/tls_main.c=646=static int do_tls_setsockopt_conf(struct sock *sk, sockptr_t optval,\n--\nnet/tls/tls_main.c-673-\tif (tx) {\nnet/tls/tls_main.c:674:\t\tcrypto_ctx = \u0026ctx-\u003ecrypto_send;\nnet/tls/tls_main.c-675-\t\talt_crypto_info = \u0026ctx-\u003ecrypto_recv.info;\n--\nnet/tls/tls_main.c-677-\t\tcrypto_ctx = \u0026ctx-\u003ecrypto_recv;\nnet/tls/tls_main.c:678:\t\talt_crypto_info = \u0026ctx-\u003ecrypto_send.info;\nnet/tls/tls_main.c-679-\t}\n--\nnet/tls/tls_sw.c=2566=int tls_sw_ctx_init(struct sock *sk, int tx,\n--\nnet/tls/tls_sw.c-2597-\t\tsw_ctx_tx = ctx-\u003epriv_ctx_tx;\nnet/tls/tls_sw.c:2598:\t\tcrypto_info = \u0026ctx-\u003ecrypto_send.info;\nnet/tls/tls_sw.c-2599-\t\taead = \u0026sw_ctx_tx-\u003eaead_send;\n--\nnet/tls/tls_sw.c=2684=void tls_sw_ctx_finalize(struct sock *sk, int tx,\n--\nnet/tls/tls_sw.c-2693-\tif (tx) {\nnet/tls/tls_sw.c:2694:\t\tcrypto_info = \u0026ctx-\u003ecrypto_send.info;\nnet/tls/tls_sw.c-2695-\t\tcctx = \u0026ctx-\u003etx;\n"
} |
| 58/3 | 2026/09/20 01:07 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 83674 output: 32 thoughts: 93 Thoughts: Here's my thought process, as it stands: **Debugging a Rekey Initialization** Okay, so I'm digging into this rekey issue in `tls_device_start_rekey`. First things first, I need to understand how the `offload_ctx->rekey` variable is getting initialized. This is a critical piece of the puzzle, and a potential point of failure. Let's trace back. I need to figure out where `offload_ctx` is even allocated and initialized. The flow is likely somewhere earlier in the setup. Aha, here's a likely candidate: `tls_set_device_offload_initial` is where it seems to happen. Specifically, it looks like `offload_ctx = alloc_offload_ctx_tx(ctx);` is the key line. Now I need to understand what `alloc_offload_ctx_tx` actually *does*. That's where the important details are likely hiding. Let's delve into that function definition. |
| 59/3 | 2026/09/20 01:07 | tool | grepper |
0mArgs:{
"Expression": "alloc_offload_ctx_tx",
"PathPrefix": "net/tls/"
}Results: {
"Output": "net/tls/tls_device.c=1491=static void tls_device_attach(struct tls_context *ctx, struct sock *sk,\n--\nnet/tls/tls_device.c-1506-\nnet/tls/tls_device.c:1507:static struct tls_offload_context_tx *alloc_offload_ctx_tx(struct tls_context *ctx)\nnet/tls/tls_device.c-1508-{\n--\nnet/tls/tls_device.c=1957=static int tls_set_device_offload_initial(struct sock *sk,\n--\nnet/tls/tls_device.c-1982-\nnet/tls/tls_device.c:1983:\toffload_ctx = alloc_offload_ctx_tx(ctx);\nnet/tls/tls_device.c-1984-\tif (!offload_ctx) {\n--\nnet/tls/tls_device.c=2041=int tls_set_device_offload(struct sock *sk,\n--\nnet/tls/tls_device.c-2071-\t/* Initial install: a HW TX context must not already exist, otherwise\nnet/tls/tls_device.c:2072:\t * alloc_offload_ctx_tx() below would silently overwrite it.\nnet/tls/tls_device.c-2073-\t */\n"
} |
| 60/3 | 2026/09/20 01:07 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 84369 output: 38 thoughts: 26 Thoughts: Here's my summary, as though I'm reading and thinking about the code: **Diving into `alloc_offload_ctx_tx`** Okay, time to dig into `alloc_offload_ctx_tx` within `net/tls/tls_device.c`. I need to refresh my memory on how this offload context allocation works, especially for the TX path. The name itself is pretty descriptive, but I want to understand the specifics of what resources are being allocated and initialized here. I'm guessing this function is central to setting up the necessary state for hardware-accelerated TLS transmit operations. Let's see how it handles things like the crypto session, any DMA mappings, and how it interfaces with the device's specific offload engine. I'll be looking closely for the interactions with any relevant data structures, particularly the `tls_device` and associated contexts, and how it ensures thread safety if that's relevant. Ultimately, I'm trying to pinpoint the critical path: what needs to be in place for an outbound TLS packet to actually hit the hardware and accelerate its transport. |
| 61/3 | 2026/09/20 01:07 | tool | read-file |
0mArgs:{
"File": "net/tls/tls_device.c",
"FirstLine": 1505,
"LineCount": 30
}Results: {
"Contents": "1505:\t}\n1506:\t\n1507:\tstatic struct tls_offload_context_tx *alloc_offload_ctx_tx(struct tls_context *ctx)\n1508:\t{\n1509:\t\tstruct tls_offload_context_tx *offload_ctx;\n1510:\t\t__be64 rcd_sn;\n1511:\t\n1512:\t\toffload_ctx = kzalloc_obj(*offload_ctx);\n1513:\t\tif (!offload_ctx)\n1514:\t\t\treturn NULL;\n1515:\t\n1516:\t\tINIT_WORK(\u0026offload_ctx-\u003edestruct_work, tls_device_tx_del_task);\n1517:\t\tINIT_LIST_HEAD(\u0026offload_ctx-\u003erecords_list);\n1518:\t\tspin_lock_init(\u0026offload_ctx-\u003elock);\n1519:\t\tsg_init_table(offload_ctx-\u003esg_tx_data,\n1520:\t\t\t ARRAY_SIZE(offload_ctx-\u003esg_tx_data));\n1521:\t\n1522:\t\t/* start at rec_seq - 1 to account for the start marker record */\n1523:\t\tmemcpy(\u0026rcd_sn, ctx-\u003etx.rec_seq, sizeof(rcd_sn));\n1524:\t\toffload_ctx-\u003eunacked_record_sn = be64_to_cpu(rcd_sn) - 1;\n1525:\t\n1526:\t\toffload_ctx-\u003ectx = ctx;\n1527:\t\n1528:\t\treturn offload_ctx;\n1529:\t}\n1530:\t\n1531:\t/* Build a fresh AEAD tfm for the rekey with the given key, so it can be\n1532:\t * swapped in only on success. Re-keying a live tfm in place is not atomic:\n1533:\t * a failed crypto_aead_setkey() leaves it with CRYPTO_TFM_NEED_KEY set,\n1534:\t * destroying the previous key. Returns an ERR_PTR() on failure.\n"
} |
| 62/3 | 2026/09/20 01:07 | llm | kmsan-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 85219 output: 37 thoughts: 242 Thoughts: **My Analysis of the Offload Context Rekeying Process** Okay, so I'm digging into this `alloc_offload_ctx_tx` function. The first thing that jumps out is that it uses `kzalloc_obj(*offload_ctx)`. That means the *entire* `offload_ctx` struct is zero-initialized, including `offload_ctx->rekey`. Good to know, keeps things clean initially. Now, when the rekey operation is finished, I see a `memzero_explicit(&offload_ctx->rekey, sizeof(offload_ctx->rekey));`. Makes sense, we're explicitly clearing that sensitive data, which is crucial for security. We don't want any lingering keys. Then, looking at the `tls_device_start_rekey` function, I see the copy of `new_crypto_info` into `offload_ctx->rekey.crypto_send.info`. The critical aspect of this operation appears to be `unsafe_memcpy` which immediately raises a flag for me. It's a bit worrying that this involves a direct memory copy, but it is clear that there are checks in place via `cipher_desc->crypto_info` and I suppose this is a design choice. Still, it warrants careful attention. Further, `memzero_explicit(new_crypto_info, cipher_desc->crypto_info);` is called to clear the buffer that previously held this information. Another very good thing, no lingering. Okay, so where does `new_crypto_info` come from? It was copied from userspace in `do_tls_setsockopt_conf`, which I need to investigate in `net/tls/tls_main.c`. That's the next step to confirm this is safe, that the user data is properly validated before copying and not leaking any information. Need to see *how* that user-provided `new_crypto_info` is handled there. |
| 63/3 | 2026/09/20 01:08 | tool | read-file |
0mArgs:{
"File": "net/tls/tls_main.c",
"FirstLine": 645,
"LineCount": 80
}Results: {
"Contents": " 645:\t\n 646:\tstatic int do_tls_setsockopt_conf(struct sock *sk, sockptr_t optval,\n 647:\t\t\t\t\t unsigned int optlen, int tx)\n 648:\t{\n 649:\t\tstruct tls_crypto_info *crypto_info, *alt_crypto_info;\n 650:\t\tstruct tls_crypto_info *old_crypto_info = NULL;\n 651:\t\tstruct tls_context *ctx = tls_get_ctx(sk);\n 652:\t\tconst struct tls_cipher_desc *cipher_desc;\n 653:\t\tunion tls_crypto_context *crypto_ctx;\n 654:\t\tunion tls_crypto_context tmp = {};\n 655:\t\tbool update = false;\n 656:\t\tint rc = 0;\n 657:\t\tint conf;\n 658:\t\n 659:\t\t/* TLS and sockmap are mutually exclusive. A socket already in a\n 660:\t\t * sockmap (i.e. with a psock attached) cannot be upgraded to TLS.\n 661:\t\t * sockmap rejects TLS sockets already (see sk_psock_init()).\n 662:\t\t */\n 663:\t\trcu_read_lock();\n 664:\t\tif (sk_psock(sk)) {\n 665:\t\t\trcu_read_unlock();\n 666:\t\t\treturn -EINVAL;\n 667:\t\t}\n 668:\t\trcu_read_unlock();\n 669:\t\n 670:\t\tif (sockptr_is_null(optval) || (optlen \u003c sizeof(*crypto_info)))\n 671:\t\t\treturn -EINVAL;\n 672:\t\n 673:\t\tif (tx) {\n 674:\t\t\tcrypto_ctx = \u0026ctx-\u003ecrypto_send;\n 675:\t\t\talt_crypto_info = \u0026ctx-\u003ecrypto_recv.info;\n 676:\t\t} else {\n 677:\t\t\tcrypto_ctx = \u0026ctx-\u003ecrypto_recv;\n 678:\t\t\talt_crypto_info = \u0026ctx-\u003ecrypto_send.info;\n 679:\t\t}\n 680:\t\n 681:\t\tcrypto_info = \u0026crypto_ctx-\u003einfo;\n 682:\t\n 683:\t\tif (TLS_CRYPTO_INFO_READY(crypto_info)) {\n 684:\t\t\t/* Currently we only support setting crypto info more\n 685:\t\t\t * than one time for TLS 1.3\n 686:\t\t\t */\n 687:\t\t\tif (crypto_info-\u003eversion != TLS_1_3_VERSION) {\n 688:\t\t\t\tTLS_INC_STATS(sock_net(sk), tx ? LINUX_MIB_TLSTXREKEYERROR\n 689:\t\t\t\t\t\t\t : LINUX_MIB_TLSRXREKEYERROR);\n 690:\t\t\t\treturn -EBUSY;\n 691:\t\t\t}\n 692:\t\n 693:\t\t\tupdate = true;\n 694:\t\t\told_crypto_info = crypto_info;\n 695:\t\t\tcrypto_info = \u0026tmp.info;\n 696:\t\t\tcrypto_ctx = \u0026tmp;\n 697:\t\t}\n 698:\t\n 699:\t\trc = copy_from_sockptr(crypto_info, optval, sizeof(*crypto_info));\n 700:\t\tif (rc) {\n 701:\t\t\trc = -EFAULT;\n 702:\t\t\tgoto err_crypto_info;\n 703:\t\t}\n 704:\t\n 705:\t\tif (update) {\n 706:\t\t\t/* Ensure that TLS version and ciphers are not modified */\n 707:\t\t\tif (crypto_info-\u003eversion != old_crypto_info-\u003eversion ||\n 708:\t\t\t crypto_info-\u003ecipher_type != old_crypto_info-\u003ecipher_type)\n 709:\t\t\t\trc = -EINVAL;\n 710:\t\t} else {\n 711:\t\t\trc = validate_crypto_info(crypto_info, alt_crypto_info);\n 712:\t\t}\n 713:\t\tif (rc)\n 714:\t\t\tgoto err_crypto_info;\n 715:\t\n 716:\t\tcipher_desc = get_cipher_desc(crypto_info-\u003ecipher_type);\n 717:\t\tif (!cipher_desc) {\n 718:\t\t\trc = -EINVAL;\n 719:\t\t\tgoto err_crypto_info;\n 720:\t\t}\n 721:\t\n 722:\t\tif (optlen != cipher_desc-\u003ecrypto_info) {\n 723:\t\t\trc = -EINVAL;\n 724:\t\t\tgoto err_crypto_info;\n"
} |
| 64/3 | 2026/09/20 01:08 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 87137 output: 38 thoughts: 899 Thoughts: **Analyzing TLS Crypto Context Initialization and Usage**
Okay, let's break this down. Line 654 immediately caught my eye. The `union tls_crypto_context tmp = {};` initialization is crucial. It's zeroed out, and then `copy_from_sockptr` overwrites *only* the user-provided data within it, leaving any padding zeroed. This is good practice to ensure unused parts are always initialized to a known value.
Then, in `tls_device_start_rekey`, the `unsafe_memcpy` to `offload_ctx->rekey.crypto_send.info` is safe because `offload_ctx` was allocated with `kzalloc_obj`, so any trailing bytes beyond `cipher_desc->crypto_info` are already zero. And, the `copy_to_iter` in `do_tls_getsockopt_conf` only copies the expected `cipher_desc->crypto_info` bytes to userspace. So far, so good – we're seeing consistent zeroing and controlled copy sizes.
Now, let's move to the RX side and `tls_device_deferred_dev_add_rx`. Here, `crypto_ctx` is copied from `tls_ctx->crypto_recv` (which has been initialized earlier, and we haven't seen a problem yet), updated with `tls_ctx->rx.rec_seq`, passed to `tls_device_dev_add_rx`, and *then* explicitly zeroed. The explicit zeroing here helps prevent information leakage, which is good.
But, hold on. Looking at `tls_device_reencrypt`... That `orig_buf = kmalloc(rxm->full_len + prot->prepend_size, sk->sk_allocation);`... is `orig_buf` being read before a write? I need to analyze this function much more carefully because this is where the bug often lurks. I need to make sure that the `orig_buf` isn't read before it's filled.
|
| 65/3 | 2026/09/20 01:08 | tool | read-file |
0mArgs:{
"File": "net/tls/tls_device.c",
"FirstLine": 1220,
"LineCount": 80
}Results: {
"Contents": "1220:\t\tstruct scatterlist sg[1];\n1221:\t\tstruct strp_msg *rxm;\n1222:\t\tchar *orig_buf, *buf;\n1223:\t\n1224:\t\tcipher_desc = get_cipher_desc(tls_ctx-\u003ecrypto_recv.info.cipher_type);\n1225:\t\tDEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc-\u003eoffloadable);\n1226:\t\n1227:\t\trxm = strp_msg(tls_strp_msg(sw_ctx));\n1228:\t\torig_buf = kmalloc(rxm-\u003efull_len + prot-\u003eprepend_size,\n1229:\t\t\t\t sk-\u003esk_allocation);\n1230:\t\tif (!orig_buf)\n1231:\t\t\treturn -ENOMEM;\n1232:\t\tbuf = orig_buf;\n1233:\t\n1234:\t\terr = tls_strp_msg_cow(sw_ctx);\n1235:\t\tif (unlikely(err))\n1236:\t\t\tgoto free_buf;\n1237:\t\n1238:\t\tskb = tls_strp_msg(sw_ctx);\n1239:\t\trxm = strp_msg(skb);\n1240:\t\toffset = rxm-\u003eoffset;\n1241:\t\n1242:\t\tsg_init_table(sg, 1);\n1243:\t\tsg_set_buf(\u0026sg[0], buf, rxm-\u003efull_len + prot-\u003eprepend_size);\n1244:\t\terr = skb_copy_bits(skb, offset, buf, prot-\u003eprepend_size);\n1245:\t\tif (err)\n1246:\t\t\tgoto free_buf;\n1247:\t\n1248:\t\t/* We are interested only in the decrypted data not the auth */\n1249:\t\terr = decrypt_skb(sk, sg);\n1250:\t\tif (err != -EBADMSG)\n1251:\t\t\tgoto free_buf;\n1252:\t\telse\n1253:\t\t\terr = 0;\n1254:\t\n1255:\t\tdata_len = rxm-\u003efull_len - cipher_desc-\u003etag;\n1256:\t\n1257:\t\tif (skb_pagelen(skb) \u003e offset) {\n1258:\t\t\tcopy = min_t(int, skb_pagelen(skb) - offset, data_len);\n1259:\t\n1260:\t\t\tif (skb-\u003edecrypted || skb-\u003edecrypt_failed) {\n1261:\t\t\t\terr = skb_store_bits(skb, offset, buf, copy);\n1262:\t\t\t\tif (err)\n1263:\t\t\t\t\tgoto free_buf;\n1264:\t\t\t}\n1265:\t\n1266:\t\t\toffset += copy;\n1267:\t\t\tbuf += copy;\n1268:\t\t}\n1269:\t\n1270:\t\tpos = skb_pagelen(skb);\n1271:\t\tskb_walk_frags(skb, skb_iter) {\n1272:\t\t\tint frag_pos;\n1273:\t\n1274:\t\t\t/* Practically all frags must belong to msg if reencrypt\n1275:\t\t\t * is needed with current strparser and coalescing logic,\n1276:\t\t\t * but strparser may \"get optimized\", so let's be safe.\n1277:\t\t\t */\n1278:\t\t\tif (pos + skb_iter-\u003elen \u003c= offset)\n1279:\t\t\t\tgoto done_with_frag;\n1280:\t\t\tif (pos \u003e= data_len + rxm-\u003eoffset)\n1281:\t\t\t\tbreak;\n1282:\t\n1283:\t\t\tfrag_pos = offset - pos;\n1284:\t\t\tcopy = min_t(int, skb_iter-\u003elen - frag_pos,\n1285:\t\t\t\t data_len + rxm-\u003eoffset - offset);\n1286:\t\n1287:\t\t\tif (skb_iter-\u003edecrypted || skb_iter-\u003edecrypt_failed) {\n1288:\t\t\t\terr = skb_store_bits(skb_iter, frag_pos, buf, copy);\n1289:\t\t\t\tif (err)\n1290:\t\t\t\t\tgoto free_buf;\n1291:\t\t\t}\n1292:\t\n1293:\t\t\toffset += copy;\n1294:\t\t\tbuf += copy;\n1295:\tdone_with_frag:\n1296:\t\t\tpos += skb_iter-\u003elen;\n1297:\t\t}\n1298:\t\n1299:\tfree_buf:\n"
} |
| 66/3 | 2026/09/20 01:08 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 89729 output: 28 thoughts: 67 Thoughts: **Deconstructing the Decryption Process** Okay, so I'm looking at this `decrypt_skb(sk, sg)` function and it seems the output, `sg`, gets populated with the decrypted data. `sg` points to a buffer (`buf`) that's of a specific length: `rxm->full_len + prot->prepend_size`. That's important. I need to understand how *exactly* the decryption happens. I can see the parameters: a socket structure (`sk`) and a scatter-gather list (`sg`), the latter of which will be populated with the decrypted data. But *how* does it work? I think I need to dig into the `decrypt_skb` function itself to truly grasp this. Let me see what `decrypt_skb` actually *does*. |
| 67/3 | 2026/09/20 01:08 | tool | grepper |
0mArgs:{
"Expression": "decrypt_skb",
"PathPrefix": "net/tls/"
}Results: {
"Output": "net/tls/tls.h=197=int tls_process_cmsg(struct sock *sk, struct msghdr *msg,\nnet/tls/tls.h-198-\t\t unsigned char *record_type);\nnet/tls/tls.h:199:int decrypt_skb(struct sock *sk, struct scatterlist *sgout);\nnet/tls/tls.h-200-\n--\nnet/tls/tls_device.c=1213=tls_device_reencrypt(struct sock *sk, struct tls_context *tls_ctx)\n--\nnet/tls/tls_device.c-1248-\t/* We are interested only in the decrypted data not the auth */\nnet/tls/tls_device.c:1249:\terr = decrypt_skb(sk, sg);\nnet/tls/tls_device.c-1250-\tif (err != -EBADMSG)\n--\nnet/tls/tls_sw.c=1575=static int tls_rx_one_record(struct sock *sk, struct msghdr *msg,\n--\nnet/tls/tls_sw.c-1598-\nnet/tls/tls_sw.c:1599:int decrypt_skb(struct sock *sk, struct scatterlist *sgout)\nnet/tls/tls_sw.c-1600-{\n"
} |
| 68/3 | 2026/09/20 01:08 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 90342 output: 38 thoughts: 22 Thoughts: Here's my summary, as though I'm reading and thinking about the code: **Diving into `decrypt_skb`** Okay, time to get into the heart of the matter – I need to really understand how the decryption process is handled in `net/tls/tls_sw.c`, specifically the `decrypt_skb` function. Let's see how it's actually implemented. I'm looking for the key components. I need to understand how the Secure Sockets Layer (SSL) and Transport Layer Security (TLS) decryption process happens here, packet by packet. What are the key stages? Is it a hardware offload being used or is it being software implemented here? What are the key data structures being manipulated, and how is the context of the TLS connection managed during the decryption? I know this part is critical for performance and security. I'm particularly interested in how they're handling potential vulnerabilities or attack vectors during decryption. Any clever optimizations? Let's go. |
| 69/3 | 2026/09/20 01:08 | tool | read-file |
0mArgs:{
"File": "net/tls/tls_sw.c",
"FirstLine": 1598,
"LineCount": 40
}Results: {
"Contents": "1598:\t\n1599:\tint decrypt_skb(struct sock *sk, struct scatterlist *sgout)\n1600:\t{\n1601:\t\tstruct tls_decrypt_arg darg = { .zc = true, };\n1602:\t\n1603:\t\treturn tls_decrypt_sg(sk, NULL, sgout, \u0026darg);\n1604:\t}\n1605:\t\n1606:\t/* All records returned from a recvmsg() call must have the same type.\n1607:\t * 0 is not a valid content type. Use it as \"no type reported, yet\".\n1608:\t */\n1609:\tstatic int tls_record_content_type(struct msghdr *msg, struct tls_msg *tlm,\n1610:\t\t\t\t\t u8 *control)\n1611:\t{\n1612:\t\tint err;\n1613:\t\n1614:\t\tif (!*control) {\n1615:\t\t\t*control = tlm-\u003econtrol;\n1616:\t\t\tif (!*control)\n1617:\t\t\t\treturn -EBADMSG;\n1618:\t\n1619:\t\t\terr = put_cmsg(msg, SOL_TLS, TLS_GET_RECORD_TYPE,\n1620:\t\t\t\t sizeof(*control), control);\n1621:\t\t\tif (*control != TLS_RECORD_TYPE_DATA) {\n1622:\t\t\t\tif (err || msg-\u003emsg_flags \u0026 MSG_CTRUNC)\n1623:\t\t\t\t\treturn -EIO;\n1624:\t\t\t}\n1625:\t\t} else if (*control != tlm-\u003econtrol) {\n1626:\t\t\treturn 0;\n1627:\t\t}\n1628:\t\n1629:\t\treturn 1;\n1630:\t}\n1631:\t\n1632:\t/* The deferred announce is fired once on reader exit by\n1633:\t * tls_rx_reader_release().\n1634:\t */\n1635:\tstatic void tls_rx_rec_done(struct tls_sw_context_rx *ctx)\n1636:\t{\n1637:\t\ttls_strp_msg_consume(\u0026ctx-\u003estrp);\n"
} |
| 70/3 | 2026/09/20 01:08 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 91196 output: 29 thoughts: 18 Thoughts: Okay, let's dive into this. Here's how I'd approach summarizing the prompt, given my expertise: **Examining `tls_decrypt_sg` and `sgout`** Alright, so I need to figure out what `tls_decrypt_sg` does, specifically how it interacts with `sgout`. Given the names, my initial assumption is that we're dealing with a TLS (Transport Layer Security) decryption function, and `sgout` likely represents some form of output, potentially related to scatter/gather I/O. So, the question is, *how* does this function handle the decrypted data, and *where* does that decrypted data end up? Is `sgout` an existing buffer or is it allocated dynamically? Are there any preconditions on `sgout`? What about error handling; is there any indication how `tls_decrypt_sg` signals success/failure or if `sgout` is truncated in any situation? I'll need to look closely at the implementation to understand the memory management and data flow. Knowing the context – is this a kernel driver, a user-space library, embedded system code? – is crucial to understand the implications of the operation. I'm essentially reverse-engineering the thought process by breaking the function and output into smaller pieces. |
| 71/3 | 2026/09/20 01:08 | tool | grepper |
0mArgs:{
"Expression": "tls_decrypt_sg",
"PathPrefix": "net/tls/"
}Results: {
"Output": "net/tls/tls_sw.c=1233=tls_alloc_clrtxt_skb(struct sock *sk, struct sk_buff *skb,\n--\nnet/tls/tls_sw.c-1274- */\nnet/tls/tls_sw.c:1275:static int tls_decrypt_sg(struct sock *sk, struct iov_iter *out_iov,\nnet/tls/tls_sw.c-1276-\t\t\t struct scatterlist *out_sg,\n--\nnet/tls/tls_sw.c=1450=tls_decrypt_sw(struct sock *sk, struct tls_context *tls_ctx,\n--\nnet/tls/tls_sw.c-1457-\nnet/tls/tls_sw.c:1458:\terr = tls_decrypt_sg(sk, \u0026msg-\u003emsg_iter, NULL, darg);\nnet/tls/tls_sw.c-1459-\tif (err \u003c 0) {\n--\nnet/tls/tls_sw.c=1599=int decrypt_skb(struct sock *sk, struct scatterlist *sgout)\n--\nnet/tls/tls_sw.c-1602-\nnet/tls/tls_sw.c:1603:\treturn tls_decrypt_sg(sk, NULL, sgout, \u0026darg);\nnet/tls/tls_sw.c-1604-}\n"
} |
| 72/3 | 2026/09/20 01:08 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 91875 output: 38 thoughts: 23 Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience: **Diving into tls_decrypt_sg** Okay, time to dissect `tls_decrypt_sg` within `net/tls/tls_sw.c`. I'm going to start by confirming my understanding of the larger context – this is where the software TLS implementation handles decryption, specifically with scatter-gather lists. I want to see how they're handling the data chunks. I'm expecting to see calls related to cipher setup (if it isn't hardware accelerated), key schedules, and then the actual decryption loop, handling potential padding and MAC verification. I'm very interested in seeing how the SG list traversal is optimized, especially regarding cache line alignment and memory access patterns. Does the implementation attempt any prefetching? Are they using any tricks with the cipher that leverage the scatter-gather capabilities efficiently? This part of the code is critical for performance, particularly on high-bandwidth connections, so I'm hoping to see well-thought-out data handling. I'll be looking out for any potential vulnerabilities, too. Any integer overflows in the length calculations? Improper bounds checking in the SG list traversal? Let's get reading. |
| 73/3 | 2026/09/20 01:08 | tool | read-file |
0mArgs:{
"File": "net/tls/tls_sw.c",
"FirstLine": 1275,
"LineCount": 80
}Results: {
"Contents": "1275:\tstatic int tls_decrypt_sg(struct sock *sk, struct iov_iter *out_iov,\n1276:\t\t\t\t struct scatterlist *out_sg,\n1277:\t\t\t\t struct tls_decrypt_arg *darg)\n1278:\t{\n1279:\t\tstruct tls_context *tls_ctx = tls_get_ctx(sk);\n1280:\t\tstruct tls_sw_context_rx *ctx = tls_sw_ctx_rx(tls_ctx);\n1281:\t\tstruct tls_prot_info *prot = \u0026tls_ctx-\u003eprot_info;\n1282:\t\tint n_sgin, n_sgout, aead_size, err, pages = 0;\n1283:\t\tstruct sk_buff *skb = tls_strp_msg(ctx);\n1284:\t\tconst struct strp_msg *rxm = strp_msg(skb);\n1285:\t\tconst struct tls_msg *tlm = tls_msg(skb);\n1286:\t\tstruct aead_request *aead_req;\n1287:\t\tstruct scatterlist *sgin = NULL;\n1288:\t\tstruct scatterlist *sgout = NULL;\n1289:\t\tconst int data_len = rxm-\u003efull_len - prot-\u003eoverhead_size;\n1290:\t\tint tail_pages = !!prot-\u003etail_size;\n1291:\t\tstruct tls_decrypt_ctx *dctx;\n1292:\t\tstruct sk_buff *clear_skb;\n1293:\t\tint iv_offset = 0;\n1294:\t\tu8 *mem;\n1295:\t\n1296:\t\tn_sgin = skb_nsg(skb, rxm-\u003eoffset + prot-\u003eprepend_size,\n1297:\t\t\t\t rxm-\u003efull_len - prot-\u003eprepend_size);\n1298:\t\tif (n_sgin \u003c 1)\n1299:\t\t\treturn n_sgin ?: -EBADMSG;\n1300:\t\n1301:\t\tif (darg-\u003ezc \u0026\u0026 (out_iov || out_sg)) {\n1302:\t\t\tclear_skb = NULL;\n1303:\t\n1304:\t\t\tif (out_iov)\n1305:\t\t\t\tn_sgout = 1 + tail_pages +\n1306:\t\t\t\t\tiov_iter_npages_cap(out_iov, INT_MAX, data_len);\n1307:\t\t\telse\n1308:\t\t\t\tn_sgout = sg_nents(out_sg);\n1309:\t\t} else {\n1310:\t\t\tdarg-\u003ezc = false;\n1311:\t\n1312:\t\t\tclear_skb = tls_alloc_clrtxt_skb(sk, skb, rxm-\u003efull_len);\n1313:\t\t\tif (!clear_skb)\n1314:\t\t\t\treturn -ENOMEM;\n1315:\t\n1316:\t\t\tn_sgout = 1 + skb_shinfo(clear_skb)-\u003enr_frags;\n1317:\t\t}\n1318:\t\n1319:\t\t/* Increment to accommodate AAD */\n1320:\t\tn_sgin = n_sgin + 1;\n1321:\t\n1322:\t\t/* Allocate a single block of memory which contains\n1323:\t\t * aead_req || tls_decrypt_ctx.\n1324:\t\t * Both structs are variable length.\n1325:\t\t */\n1326:\t\taead_size = sizeof(*aead_req) + crypto_aead_reqsize(ctx-\u003eaead_recv);\n1327:\t\taead_size = ALIGN(aead_size, __alignof__(*dctx));\n1328:\t\tmem = kmalloc(aead_size + struct_size(dctx, sg, size_add(n_sgin, n_sgout)),\n1329:\t\t\t sk-\u003esk_allocation);\n1330:\t\tif (!mem) {\n1331:\t\t\terr = -ENOMEM;\n1332:\t\t\tgoto exit_free_skb;\n1333:\t\t}\n1334:\t\n1335:\t\t/* Segment the allocated memory */\n1336:\t\taead_req = (struct aead_request *)mem;\n1337:\t\tdctx = (struct tls_decrypt_ctx *)(mem + aead_size);\n1338:\t\tdctx-\u003esk = sk;\n1339:\t\tsgin = \u0026dctx-\u003esg[0];\n1340:\t\tsgout = \u0026dctx-\u003esg[n_sgin];\n1341:\t\n1342:\t\t/* For CCM based ciphers, first byte of nonce+iv is a constant */\n1343:\t\tswitch (prot-\u003ecipher_type) {\n1344:\t\tcase TLS_CIPHER_AES_CCM_128:\n1345:\t\t\tdctx-\u003eiv[0] = TLS_AES_CCM_IV_B0_BYTE;\n1346:\t\t\tiv_offset = 1;\n1347:\t\t\tbreak;\n1348:\t\tcase TLS_CIPHER_SM4_CCM:\n1349:\t\t\tdctx-\u003eiv[0] = TLS_SM4_CCM_IV_B0_BYTE;\n1350:\t\t\tiv_offset = 1;\n1351:\t\t\tbreak;\n1352:\t\t}\n1353:\t\n1354:\t\t/* Prepare IV */\n"
} |
| 74/3 | 2026/09/20 01:08 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 93593 output: 33 thoughts: 1390 Thoughts: **Analyzing the TLS Re-encryption and Rekeying Logic** Okay, so I'm looking at these TLS code changes. The function `tls_decrypt_sg` is already established, which is good. The core change in `tls_device_reencrypt` seems focused on the memory allocation and buffer handling. It's swapped from using `TLS_HEADER_SIZE + cipher_desc->iv` to `prot->prepend_size` for `kmalloc` and `sg_set_buf`. Makes sense, optimizing based on the protocol's specific prepend size. It also now checks for `skb->decrypt_failed` which makes the logic more robust. The general flow is still there: `prot->prepend_size` bytes are copied from the SKB into `buf`, the crypto engine decrypts data directly into `buf`, and then `skb_store_bits` stores `buf` back to the SKB. Now, let's consider `tls_device_reencrypt_old_key`. This function appears to handle key rekeying. It saves the current AEAD, IV, and record sequence number, swaps them with the old key's values, performs the re-encryption, restores the original values, and then increments the old record sequence number. All of this seems correct. The use of `memcpy` with properly sized arrays is also critical for safety. Hold on, I need to check how `tls_device_commit_rekey_marker` works. It handles committing the rekey marker. It iterates through a list of records, removes any start markers at the head, updates the unacknowledged record sequence number, and adds a new start marker. Seems alright. The critical piece here is the `start_marker_record`. It's initialized in `tls_set_device_offload_rekey`. I noticed the odd line `kmalloc_obj(*offload_ctx->rekey.start_marker);`. That is suspicious. It looks like it's trying to allocate memory based on a dereferenced pointer! What is this structure? I better check the definition of `struct tls_record_info` in `include/net/tls.h` to see what is going on. |
| 75/3 | 2026/09/20 01:09 | tool | grepper |
0mArgs:{
"Expression": "struct tls_record_info",
"PathPrefix": "include/net/tls.h"
}Results: {
"Output": "include/net/tls.h=131=struct tls_sw_context_rx {\n--\ninclude/net/tls.h-149-\ninclude/net/tls.h:150:struct tls_record_info {\ninclude/net/tls.h-151-\tstruct list_head list;\n--\ninclude/net/tls.h=175=struct tls_offload_context_tx {\n--\ninclude/net/tls.h-178-\tstruct list_head records_list;\ninclude/net/tls.h:179:\tstruct tls_record_info *open_record;\ninclude/net/tls.h:180:\tstruct tls_record_info *retransmit_hint;\ninclude/net/tls.h-181-\tu64 hint_record_sn;\n--\ninclude/net/tls.h-192-\t\tunion tls_crypto_context crypto_send; /* Crypto for new key */\ninclude/net/tls.h:193:\t\tstruct tls_record_info *start_marker;\ninclude/net/tls.h-194-\t} rekey;\n--\ninclude/net/tls.h=366=struct tls_offload_context_rx {\n--\ninclude/net/tls.h-403-\ninclude/net/tls.h:404:struct tls_record_info *tls_get_record(struct tls_offload_context_tx *context,\ninclude/net/tls.h-405-\t\t\t\t u32 seq, u64 *p_record_sn);\ninclude/net/tls.h-406-\ninclude/net/tls.h:407:static inline bool tls_record_is_start_marker(struct tls_record_info *rec)\ninclude/net/tls.h-408-{\n--\ninclude/net/tls.h-411-\ninclude/net/tls.h:412:static inline u32 tls_record_start_seq(struct tls_record_info *rec)\ninclude/net/tls.h-413-{\n"
} |
| 76/3 | 2026/09/20 01:09 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 96008 output: 35 thoughts: 26 Thoughts: Here's a summary of my immediate thought process, based on the prompt: **Analyzing `tls_record_info` and `tls_record_is_start_marker`** Okay, so we're diving into the TLS record handling code. My first instinct is to pull up the definition of `struct tls_record_info`. I need to refresh my memory on its structure – what fields does it contain? What kind of metadata does it store about a TLS record? I'm particularly interested in anything that might relate to the beginning of a record, given the second function we are considering. Next, I need to understand `tls_record_is_start_marker`. Given the naming, my initial assumption is this function probably tests a flag or field within `tls_record_info` to determine if this particular record marks the beginning of something significant - like a new handshake phase or a new application data block. Is it a simple boolean check, or a more involved calculation based on other record metadata? I need to figure out what constitutes this "start" condition. It could be based on record type, sequence number, length, or some other contextual information. Understanding this function's logic is critical for understanding the overall flow of the TLS record processing. I also need to think about how this interacts with the higher level TLS state machine; how does this “start marker” feed into the parsing or processing of the record data itself? |
| 77/3 | 2026/09/20 01:09 | tool | read-file |
0mArgs:{
"File": "include/net/tls.h",
"FirstLine": 149,
"LineCount": 20
}Results: {
"Contents": " 149:\t\n 150:\tstruct tls_record_info {\n 151:\t\tstruct list_head list;\n 152:\t\tu32 end_seq;\n 153:\t\tint len;\n 154:\t\tint num_frags;\n 155:\t\tskb_frag_t frags[MAX_SKB_FRAGS];\n 156:\t};\n 157:\t\n 158:\tstruct cipher_context {\n 159:\t\tchar iv[TLS_MAX_IV_SIZE + TLS_MAX_SALT_SIZE];\n 160:\t\tchar rec_seq[TLS_MAX_REC_SEQ_SIZE];\n 161:\t};\n 162:\t\n 163:\tunion tls_crypto_context {\n 164:\t\tstruct tls_crypto_info info;\n 165:\t\tunion {\n 166:\t\t\tstruct tls12_crypto_info_aes_gcm_128 aes_gcm_128;\n 167:\t\t\tstruct tls12_crypto_info_aes_gcm_256 aes_gcm_256;\n 168:\t\t\tstruct tls12_crypto_info_chacha20_poly1305 chacha20_poly1305;\n"
} |
| 78/3 | 2026/09/20 01:09 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 96706 output: 35 thoughts: 15 Thoughts: Here's a summary of my thought process, as if I were the one thinking it: **Diving into `tls_record_is_start_marker`** Okay, so let's get right into examining this `tls_record_is_start_marker` function, shall we? I need to analyze how this piece of code handles identifying the start of a TLS record. I assume it's dealing with some sort of parsing or demultiplexing within the TLS protocol stack. The term "marker" implies a specific indicator – perhaps a field or pattern in the incoming data stream that signals the beginning of a new TLS record. I'm immediately thinking about different approaches. Is it a simple byte check? Does it need to account for variable-length fields? Are there potential corner cases or malformed inputs I should consider? I hope it's well-documented! Time to see what it does. I want to look at its implementation very carefully. |
| 79/3 | 2026/09/20 01:09 | tool | read-file |
0mArgs:{
"File": "include/net/tls.h",
"FirstLine": 405,
"LineCount": 20
}Results: {
"Contents": " 405:\t\t\t\t\t u32 seq, u64 *p_record_sn);\n 406:\t\n 407:\tstatic inline bool tls_record_is_start_marker(struct tls_record_info *rec)\n 408:\t{\n 409:\t\treturn rec-\u003elen == 0;\n 410:\t}\n 411:\t\n 412:\tstatic inline u32 tls_record_start_seq(struct tls_record_info *rec)\n 413:\t{\n 414:\t\treturn rec-\u003eend_seq - rec-\u003elen;\n 415:\t}\n 416:\t\n 417:\tstruct sk_buff *\n 418:\ttls_validate_xmit_skb(struct sock *sk, struct net_device *dev,\n 419:\t\t\t struct sk_buff *skb);\n 420:\tstruct sk_buff *\n 421:\ttls_validate_xmit_skb_sw(struct sock *sk, struct net_device *dev,\n 422:\t\t\t\t struct sk_buff *skb);\n 423:\tstruct sk_buff *\n 424:\ttls_validate_xmit_skb_rekey(struct sock *sk, struct net_device *dev,\n"
} |
| 80/3 | 2026/09/20 01:09 | llm | kmsan-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 97297 output: 37 thoughts: 218 Thoughts: **My Analysis of the TLS Record Handling** Okay, so I'm looking at this `tls_record_is_start_marker` function, and it's basically doing a simple check: `rec->len == 0`. Seems straightforward. Then, I jump into `tls_device_add_start_marker`. Ah, right, this function is responsible for adding a start marker record. I see the code initializing `start_marker_record` and setting `end_seq`, `len` (to 0, of course), and `num_frags` to zero before adding it to the `offload_ctx->records_list`. The setup seems clean enough. The write sequence number is being set to `end_seq`, which makes sense as the sequence of writes will be defined up until that point. But hold on, this `start_marker_record` gets added to a list. This raises an immediate question in my mind, a point for follow-up analysis. What happens to this `start_marker_record` later? How is it destroyed? And critically, how is it *properly* cleaned up when the `offload_ctx` is eventually freed, avoiding any memory corruption or leaks? I'd better go investigate the `destroy_record` function to see how these records are managed. I've got a feeling I need to carefully scrutinize the RCU semantics here too. |
| 81/3 | 2026/09/20 01:10 | tool | read-file |
0mArgs:{
"File": "net/tls/tls_device.c",
"FirstLine": 330,
"LineCount": 20
}Results: {
"Contents": " 330:\t\tint i;\n 331:\t\n 332:\t\tfor (i = 0; i \u003c record-\u003enum_frags; i++)\n 333:\t\t\t__skb_frag_unref(\u0026record-\u003efrags[i], false);\n 334:\t\tkfree(record);\n 335:\t}\n 336:\t\n 337:\tstatic void delete_all_records(struct tls_offload_context_tx *offload_ctx)\n 338:\t{\n 339:\t\tstruct tls_record_info *info, *temp;\n 340:\t\n 341:\t\tlist_for_each_entry_safe(info, temp, \u0026offload_ctx-\u003erecords_list, list) {\n 342:\t\t\tlist_del(\u0026info-\u003elist);\n 343:\t\t\tdestroy_record(info);\n 344:\t\t}\n 345:\t\n 346:\t\toffload_ctx-\u003eretransmit_hint = NULL;\n 347:\t}\n 348:\t\n 349:\tstatic void tls_device_commit_rekey_marker(struct sock *sk,\n"
} |
| 82/3 | 2026/09/20 01:10 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 98158 output: 29 thoughts: 75 Thoughts: **Analyzing a Potential Vulnerability** Okay, so I'm looking at this `destroy_record` function, and something's bugging me. Specifically, that `num_frags` value. I'm seeing that when it's zero, in the case of a marker, we're calling `kfree(record)` directly, without ever touching `record->frags`. That seems… risky. We're potentially freeing memory that might still be in use. But wait, hold on. What about `tls_get_record`? Does that function, when dealing with a start marker, interact with `frags`? If it does, we're probably okay, or at least this particular vulnerability isn't as critical as I first thought. Let me quickly double-check that by examining the `tls_get_record` function to determine if it uses `frags` on a start marker. I need to make sure I haven't missed something crucial here. |
| 83/3 | 2026/09/20 01:10 | tool | grepper |
0mArgs:{
"Expression": "tls_get_record",
"PathPrefix": "net/tls/"
}Results: {
"Output": "net/tls/tls_device.c=900=void tls_device_splice_eof(struct socket *sock)\n--\nnet/tls/tls_device.c-924-\nnet/tls/tls_device.c:925:struct tls_record_info *tls_get_record(struct tls_offload_context_tx *context,\nnet/tls/tls_device.c-926-\t\t\t\t u32 seq, u64 *p_record_sn)\n--\nnet/tls/tls_device.c-982-}\nnet/tls/tls_device.c:983:EXPORT_SYMBOL(tls_get_record);\nnet/tls/tls_device.c-984-\n--\nnet/tls/tls_device_fallback.c=227=static int fill_sg_in(struct scatterlist *sg_in,\n--\nnet/tls/tls_device_fallback.c-242-\tspin_lock_irqsave(\u0026ctx-\u003elock, flags);\nnet/tls/tls_device_fallback.c:243:\trecord = tls_get_record(ctx, tcp_seq, rcd_sn);\nnet/tls/tls_device_fallback.c-244-\tif (!record) {\n"
} |
| 84/3 | 2026/09/20 01:10 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 98786 output: 37 thoughts: 23 Thoughts: Here's my attempt at summarizing the thought process, tailored for an expert audience: **Diving into the TLS Implementation** Okay, time to dig into this. I need to refresh my memory on the specifics of how the TLS record layer is handled in this codebase. Let's start by zeroing in on `tls_get_record`. That function call in `net/tls/tls_device.c` should be the entry point for understanding how incoming TLS records are retrieved. I'm hoping to quickly identify the data structures and control flow involved in fetching the next available TLS record from the underlying transport. This is likely where the decryption and record boundary determination will happen, or at least be initiated. I want to see if any obvious vulnerabilities jump out, or opportunities for optimization or fuzzing. |
| 85/3 | 2026/09/20 01:10 | tool | read-file |
0mArgs:{
"File": "net/tls/tls_device.c",
"FirstLine": 940,
"LineCount": 45
}Results: {
"Contents": " 940:\t\t\t\treturn NULL;\n 941:\t\t\t/* send the start_marker record if seq number is before the\n 942:\t\t\t * tls offload start marker sequence number. This record is\n 943:\t\t\t * required to handle TCP packets which are before TLS offload\n 944:\t\t\t * started.\n 945:\t\t\t * And if it's not start marker, look if this seq number\n 946:\t\t\t * belongs to the list.\n 947:\t\t\t */\n 948:\t\t\tif (likely(!tls_record_is_start_marker(info))) {\n 949:\t\t\t\t/* we have the first record, get the last record to see\n 950:\t\t\t\t * if this seq number belongs to the list.\n 951:\t\t\t\t */\n 952:\t\t\t\tlast = list_last_entry(\u0026context-\u003erecords_list,\n 953:\t\t\t\t\t\t struct tls_record_info, list);\n 954:\t\n 955:\t\t\t\tif (!between(seq, tls_record_start_seq(info),\n 956:\t\t\t\t\t last-\u003eend_seq))\n 957:\t\t\t\t\treturn NULL;\n 958:\t\t\t}\n 959:\t\t\trecord_sn = context-\u003eunacked_record_sn;\n 960:\t\t}\n 961:\t\n 962:\t\t/* We just need the _rcu for the READ_ONCE() */\n 963:\t\trcu_read_lock();\n 964:\t\tlist_for_each_entry_from_rcu(info, \u0026context-\u003erecords_list, list) {\n 965:\t\t\tif (before(seq, info-\u003eend_seq)) {\n 966:\t\t\t\tif (!context-\u003eretransmit_hint ||\n 967:\t\t\t\t after(info-\u003eend_seq,\n 968:\t\t\t\t\t context-\u003eretransmit_hint-\u003eend_seq)) {\n 969:\t\t\t\t\tcontext-\u003ehint_record_sn = record_sn;\n 970:\t\t\t\t\tcontext-\u003eretransmit_hint = info;\n 971:\t\t\t\t}\n 972:\t\t\t\t*p_record_sn = record_sn;\n 973:\t\t\t\tgoto exit_rcu_unlock;\n 974:\t\t\t}\n 975:\t\t\trecord_sn++;\n 976:\t\t}\n 977:\t\tinfo = NULL;\n 978:\t\n 979:\texit_rcu_unlock:\n 980:\t\trcu_read_unlock();\n 981:\t\treturn info;\n 982:\t}\n 983:\tEXPORT_SYMBOL(tls_get_record);\n 984:\t\n"
} |
| 86/3 | 2026/09/20 01:10 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 99683 output: 39 thoughts: 34 Thoughts: Okay, here's my take on those thoughts, framed as my own: **Digging into that TLS Device Fallback** Right, let's zero in on `net/tls/tls_device_fallback.c` again. I need to refresh my memory on the `fill_sg_in` function. I seem to recall some odd behavior around the fallback mechanism – it's always tricky when dealing with those layers of abstraction. Let's start by revisiting the lines around 243. I have a feeling something in that area needs a closer look. I'll bet there's some subtle interaction with the scatter-gather lists that's causing the problem. Always the scatter-gather lists. They are such a pain. |
| 87/3 | 2026/09/20 01:10 | tool | read-file |
0mArgs:{
"File": "net/tls/tls_device_fallback.c",
"FirstLine": 240,
"LineCount": 40
}Results: {
"Contents": " 240:\t\tint i;\n 241:\t\n 242:\t\tspin_lock_irqsave(\u0026ctx-\u003elock, flags);\n 243:\t\trecord = tls_get_record(ctx, tcp_seq, rcd_sn);\n 244:\t\tif (!record) {\n 245:\t\t\tspin_unlock_irqrestore(\u0026ctx-\u003elock, flags);\n 246:\t\t\treturn -EINVAL;\n 247:\t\t}\n 248:\t\n 249:\t\t*sync_size = tcp_seq - tls_record_start_seq(record);\n 250:\t\tif (*sync_size \u003c 0) {\n 251:\t\t\tint is_start_marker = tls_record_is_start_marker(record);\n 252:\t\n 253:\t\t\tspin_unlock_irqrestore(\u0026ctx-\u003elock, flags);\n 254:\t\t\t/* This should only occur if the relevant record was\n 255:\t\t\t * already acked. In that case it should be ok\n 256:\t\t\t * to drop the packet and avoid retransmission.\n 257:\t\t\t *\n 258:\t\t\t * There is a corner case where the packet contains\n 259:\t\t\t * both an acked and a non-acked record.\n 260:\t\t\t * We currently don't handle that case and rely\n 261:\t\t\t * on TCP to retransmit a packet that doesn't contain\n 262:\t\t\t * already acked payload.\n 263:\t\t\t */\n 264:\t\t\tif (!is_start_marker)\n 265:\t\t\t\t*sync_size = 0;\n 266:\t\t\treturn -EINVAL;\n 267:\t\t}\n 268:\t\n 269:\t\tremaining = *sync_size;\n 270:\t\tfor (i = 0; remaining \u003e 0; i++) {\n 271:\t\t\tskb_frag_t *frag = \u0026record-\u003efrags[i];\n 272:\t\n 273:\t\t\t__skb_frag_ref(frag);\n 274:\t\t\tsg_set_page(sg_in + i, skb_frag_page(frag),\n 275:\t\t\t\t skb_frag_size(frag), skb_frag_off(frag));\n 276:\t\n 277:\t\t\tremaining -= skb_frag_size(frag);\n 278:\t\n 279:\t\t\tif (remaining \u003c 0)\n"
} |
| 88/3 | 2026/09/20 01:10 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 100544 output: 39 thoughts: 1455 Thoughts: **Analyzing TLS Cipher Context Initialization and Rekeying** Okay, so I'm digging into how the TLS cipher context is set up and rekeyed, specifically in the software path. The first thing I'm noticing is that if `*sync_size` is negative, we get an `-EINVAL` for the start marker. If it's zero, the loop that handles the fragments (`frags`) never executes for the start marker, which could be an issue. So, the `frags` are never read for start_marker. I should definitely keep this in mind as I continue. I want to check how the context is actually initialized and finalized, so I'm looking at `tls_sw_ctx_init` and `tls_sw_ctx_finalize` in `net/tls/tls_sw.c`. The code seems to differentiate between transmit and receive paths, getting a `cipher_desc`, calling `init_prot_info`, and fetching the key. The finalization function, `tls_sw_ctx_finalize`, copies IV, salt, and sequence number information. Crucially, if `new_crypto_info` is provided, it performs a `memcpy` to update the crypto info. If it's the receiver path, it calls `tls_finish_key_update` as well. So far, so good. Now, let's look at `tls_device_init_rekey_sw`. This seems to be where the software context is initialized for rekeying. The code here retrieves the `cipher_desc` and then, most importantly, *zeros* the `sw_ctx` using `memset`. Then, it calls `tls_sw_ctx_tx_init` and builds the AEAD. That `memset` is crucial; it means any stale data is wiped out. This seems to be consistent, but I have to watch this to ensure it's applied correctly. Finally, I'm checking the `tls_enc_skb` function within `net/tls/tls_device_fallback.c`. I'm trying to see if this buffer (`buf`) is a scatterlist or how it's used. Based on the code, `buf` gets allocated with the necessary size for the salt, IV, AAD size, the `sync_size`, and the tag. The salt, IV, and a dummy buffer are then set up with pointers. But I need to verify that `buf` is read or used as a scatterlist. Time to look into `fill_sg_out`! |
| 89/3 | 2026/09/20 01:10 | tool | read-file |
0mArgs:{
"File": "net/tls/tls_device_fallback.c",
"FirstLine": 290,
"LineCount": 45
}Results: {
"Contents": " 290:\t\n 291:\tstatic void fill_sg_out(struct scatterlist sg_out[3], void *buf,\n 292:\t\t\t\tstruct tls_context *tls_ctx,\n 293:\t\t\t\tstruct sk_buff *nskb,\n 294:\t\t\t\tint tcp_payload_offset,\n 295:\t\t\t\tint payload_len,\n 296:\t\t\t\tint sync_size,\n 297:\t\t\t\tvoid *dummy_buf)\n 298:\t{\n 299:\t\tconst struct tls_cipher_desc *cipher_desc =\n 300:\t\t\tget_cipher_desc(tls_ctx-\u003ecrypto_send.info.cipher_type);\n 301:\t\n 302:\t\tsg_set_buf(\u0026sg_out[0], dummy_buf, sync_size);\n 303:\t\tsg_set_buf(\u0026sg_out[1], nskb-\u003edata + tcp_payload_offset, payload_len);\n 304:\t\t/* Add room for authentication tag produced by crypto */\n 305:\t\tdummy_buf += sync_size;\n 306:\t\tsg_set_buf(\u0026sg_out[2], dummy_buf, cipher_desc-\u003etag);\n 307:\t}\n 308:\t\n 309:\tstatic struct sk_buff *tls_enc_skb(struct tls_context *tls_ctx,\n 310:\t\t\t\t\t struct scatterlist sg_out[3],\n 311:\t\t\t\t\t struct scatterlist *sg_in,\n 312:\t\t\t\t\t struct sk_buff *skb,\n 313:\t\t\t\t\t s32 sync_size, u64 rcd_sn)\n 314:\t{\n 315:\t\tstruct tls_offload_context_tx *ctx = tls_offload_ctx_tx(tls_ctx);\n 316:\t\tint tcp_payload_offset = skb_tcp_all_headers(skb);\n 317:\t\tint payload_len = skb-\u003elen - tcp_payload_offset;\n 318:\t\tconst struct tls_cipher_desc *cipher_desc;\n 319:\t\tvoid *buf, *iv, *aad, *dummy_buf, *salt;\n 320:\t\tstruct aead_request *aead_req;\n 321:\t\tstruct sk_buff *nskb = NULL;\n 322:\t\tint buf_len;\n 323:\t\n 324:\t\taead_req = tls_alloc_aead_request(ctx-\u003eaead_send, GFP_ATOMIC);\n 325:\t\tif (!aead_req)\n 326:\t\t\treturn NULL;\n 327:\t\n 328:\t\tcipher_desc = get_cipher_desc(tls_ctx-\u003ecrypto_send.info.cipher_type);\n 329:\t\tDEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc-\u003eoffloadable);\n 330:\t\n 331:\t\taead_request_set_ad(aead_req, tls_ctx-\u003eprot_info.aad_size);\n 332:\t\n 333:\t\tbuf_len = cipher_desc-\u003esalt + cipher_desc-\u003eiv +\n 334:\t\t\t tls_ctx-\u003eprot_info.aad_size +\n"
} |
| 90/3 | 2026/09/20 01:10 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 103410 output: 50 thoughts: 514 Thoughts: **My Current Thoughts on the Encryption Process** Okay, so I'm focusing on the data flow and how it interacts with the Secure Network Protocol (SKB) and specifically, the AEAD encryption with KTLS. From what I understand, `dummy_buf` is acting as the output buffer for the encrypted data and the authentication tag. It's essentially where the AEAD encryption spits out its result, which makes sense. `sg_out[0]` and `sg_out[2]` appear to hold that output. I'm focusing on the output. Now, looking at `complete_skb`, this function clearly handles the final assembly of the SKB. It's copying headers, adjusting the length, and then copying data. Specifically, the encrypted payload, which, from my understanding, the AEAD encryption puts directly into `nskb->data + tcp_payload_offset` – the output from `sg_out[1]`. `nskb` is the "new skb", the updated one. The interesting part is the `mlx5` integration. I'm seeing `mlx5/core/en_accel/ktls_txrx.c` pop up. This is where things get interesting. The `EXTRACT_INFO_FIELDS` macro suggests that the driver is pulling out specific fields like `salt`, `rec_seq`, and `iv` (Initialization Vector). And then in `fill_static_params`, it looks like these values, along with other parameters, are being set up for the hardware offload. And wait a second, there's a check for `crypto_info->crypto_info.version`, this is determining if it's TLS 1.2 or 1.3, which is important because of the initialization vector! For 1.3, it seems like the IV is being combined with the salt in a specific way using `gcm_iv + salt_sz`. The crucial question now is how `union mlx5e_crypto_info` is defined and used. I need to understand how these crypto parameters are passed to the hardware for encryption. That's the missing piece. I need to dig deeper into the definition and usage of that union. That will provide a complete picture of the crypto parameters and how they are used, for both TLS versions. |
| 91/3 | 2026/09/20 01:10 | tool | grepper |
0mArgs:{
"Expression": "union mlx5e_crypto_info",
"PathPrefix": "drivers/net/ethernet/mellanox/mlx5/core/en_accel/"
}Results: {
"Output": "drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c=45=struct mlx5e_ktls_offload_context_rx {\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c:46:\tunion mlx5e_crypto_info crypto_info;\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c-47-\tstruct accel_rule rule;\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c=90=struct mlx5e_ktls_offload_context_tx {\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-96-\tstruct list_head list_node; /* member of the pool */\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c:97:\tunion mlx5e_crypto_info crypto_info;\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-98-\tstruct tls_offload_context_tx *tx_ctx;\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c=26=fill_static_params(struct mlx5_wqe_tls_static_params_seg *params,\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c:27:\t\t union mlx5e_crypto_info *crypto_info,\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c-28-\t\t u32 key_id, u32 resync_tcp_sn)\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c=82=mlx5e_ktls_build_static_params(struct mlx5e_set_tls_static_params_wqe *wqe,\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c-83-\t\t\t u16 pc, u32 sqn,\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c:84:\t\t\t union mlx5e_crypto_info *crypto_info,\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_txrx.c-85-\t\t\t u32 tis_tir_num, u32 key_id, u32 resync_tcp_sn,\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h=28=void mlx5e_ktls_rx_resync(struct net_device *netdev, struct sock *sk, u32 seq, u8 *rcd_sn);\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h-29-\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h:30:union mlx5e_crypto_info {\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h-31-\tstruct tls_crypto_info crypto_info;\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h=79=mlx5e_ktls_build_static_params(struct mlx5e_set_tls_static_params_wqe *wqe,\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h-80-\t\t\t u16 pc, u32 sqn,\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h:81:\t\t\t union mlx5e_crypto_info *crypto_info,\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_utils.h-82-\t\t\t u32 tis_tir_num, u32 key_id, u32 resync_tcp_sn,\n"
} |
| 92/3 | 2026/09/20 01:10 | llm | kmsan-evaluator |
1mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 105595 output: 50 thoughts: 53 Thoughts: **Analyzing mlx5e_crypto_info Usage** Alright, let's dive in. I need to trace the usage of that `union mlx5e_crypto_info` structure within the mlx5 driver, specifically in the receive and transmit paths. My immediate thought is to check `mlx5e_ktls_add_rx` and `mlx5e_ktls_add_tx`. That seems like the most likely place to start, given the nature of the structure. I'm assuming it's involved in key exchange or configuration related to TLS processing. Let's see how it's actually being used within those functions... |
| 93/3 | 2026/09/20 01:11 | tool | grepper |
0mArgs:{
"Expression": "crypto_info",
"PathPrefix": "drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c"
}Results: {
"Output": "drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c=90=struct mlx5e_ktls_offload_context_tx {\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-96-\tstruct list_head list_node; /* member of the pool */\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c:97:\tunion mlx5e_crypto_info crypto_info;\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-98-\tstruct tls_offload_context_tx *tx_ctx;\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c=454=int mlx5e_ktls_add_tx(struct net_device *netdev, struct sock *sk,\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c:455:\t\t struct tls_crypto_info *crypto_info, u32 start_offload_tcp_sn)\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-456-{\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-471-\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c:472:\tswitch (crypto_info-\u003ecipher_type) {\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-473-\tcase TLS_CIPHER_AES_GCM_128:\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c:474:\t\tpriv_tx-\u003ecrypto_info.crypto_info_128 =\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c:475:\t\t\t*(struct tls12_crypto_info_aes_gcm_128 *)crypto_info;\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-476-\t\tbreak;\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-477-\tcase TLS_CIPHER_AES_GCM_256:\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c:478:\t\tpriv_tx-\u003ecrypto_info.crypto_info_256 =\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c:479:\t\t\t*(struct tls12_crypto_info_aes_gcm_256 *)crypto_info;\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-480-\t\tbreak;\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-482-\t\tWARN_ONCE(1, \"Unsupported cipher type %u\\n\",\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c:483:\t\t\t crypto_info-\u003ecipher_type);\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-484-\t\terr = -EOPNOTSUPP;\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-487-\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c:488:\tdek = mlx5_ktls_create_key(priv-\u003etls-\u003edek_pool, crypto_info);\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-489-\tif (IS_ERR(dek)) {\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c=549=post_static_params(struct mlx5e_txqsq *sq,\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-558-\twqe = MLX5E_TLS_FETCH_SET_STATIC_PARAMS_WQE(sq, pi);\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c:559:\tmlx5e_ktls_build_static_params(wqe, sq-\u003epc, sq-\u003esqn, \u0026priv_tx-\u003ecrypto_info,\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-560-\t\t\t\t priv_tx-\u003etisn,\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c=677=tx_post_resync_params(struct mlx5e_txqsq *sq,\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-685-\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c:686:\tswitch (priv_tx-\u003ecrypto_info.crypto_info.cipher_type) {\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-687-\tcase TLS_CIPHER_AES_GCM_128: {\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c:688:\t\tstruct tls12_crypto_info_aes_gcm_128 *info = \u0026priv_tx-\u003ecrypto_info.crypto_info_128;\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-689-\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-694-\tcase TLS_CIPHER_AES_GCM_256: {\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c:695:\t\tstruct tls12_crypto_info_aes_gcm_256 *info = \u0026priv_tx-\u003ecrypto_info.crypto_info_256;\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-696-\n--\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-702-\t\tWARN_ONCE(1, \"Unsupported cipher type %u\\n\",\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c:703:\t\t\t priv_tx-\u003ecrypto_info.crypto_info.cipher_type);\ndrivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_tx.c-704-\t\treturn;\n"
} |
| 94/3 | 2026/09/20 01:11 | llm | kmsan-evaluator |
0mModel:gemini-3.8-flash Args: null Results: null Tokens: input: 107746 output: 589 thoughts: 5507 Thoughts: **KMSAN Assessment: No Uninitialized Memory Risks Detected**
Alright, let's break this down. My goal here is to determine if this patch introduces any uninitialized memory exposures that could be exploited. I'm looking for situations where kernel structures are exposed to userspace with uninitialized data, or where conditional logic might be based on uninitialized variables, leading to potential information leaks or other vulnerabilities. I'll need to scrutinize any memory copies, allocations, and related data flows introduced or modified by this patch.
First, I've gone over the code and reviewed the critical areas. The primary focus of this patch appears to be implementing TLS 1.3 key rekeying within the existing offload framework, specifically for Mellanox (mlx5) and some other devices. This involves changes to:
* **Data Structures:** New fields for rekeying state, including context structures (`struct tls_offload_context_tx/rx`, `struct tls_context`, `struct tls_record_info`).
* **Functions:** Modifications to existing functions (`fill_static_params`, `tls_device_free_ctx`, `tls_device_sendmsg`, `tls_device_complete_rekey`, etc.) and new functions for rekey handling and key management (`tls_device_rekey`, `tls_device_rx_resync_new_rec`, etc.)
* **Socket Option:** Handling for TLS socket options (`getsockopt`, `setsockopt`).
* **SKB Updates:** A new bitfield `decrypt_failed` in `struct sk_buff`.
Let's carefully evaluate against the four KMSAN criteria.
1. **Kernel Structures to Userspace:** The most likely candidate is the `getsockopt` code path. In `do_tls_getsockopt_conf`, when rekeying is active, a `crypto_info` structure is copied to userspace. I've traced the life cycle of this structure:
* It's initially allocated and zeroed out (using `kzalloc_obj`).
* User-provided data is copied into a temporary zero-initialized structure (`tmp` which is zero initialized).
* The relevant fields are then copied to the target `offload_ctx->rekey.crypto_send.info`.
* When copied out via `getsockopt`, the function copies a specific number of bytes from the `crypto_info` structure to userspace.
* The IV and sequence number are overwritten with `cctx->iv + cipher_desc->salt` and `cctx->rec_seq` before `copy_to_iter`.
* The `rekey` fields are initialized when new keys are used (e.g. `tls_device_copy_rekey_iv_seq` copies everything from user space), and zeroed out when rekeying completes (`memzero_explicit(&offload_ctx->rekey, ...)`)
In summary, only initialized fields are exposed. No uninitialized memory exposure.
2. **Conditional Logic/Branching:** There's a lot of conditional logic related to rekeying, checking for states like `TLS_TX_REKEY_PENDING`, `TLS_TX_REKEY_READY`, `TLS_TX_REKEY_FAILED` and such. I've looked at all these branches, and they are based on explicitly initialized state flags and values. There's no situation where an uninitialized variable directly controls a branch that could lead to a leak or memory access.
3. **Complex Data Structure Allocation/Initialization:** All new allocations use `kzalloc_obj` (which zero-initializes the allocated memory) or `alloc_page(GFP_KERNEL | __GFP_ZERO)` to initialize `dummy_page`. I have reviewed the usage of `struct tls_record_info`, `struct tls_offload_context_tx` and `rx` and confirmed that all the relevant fields are initialized before being used. The author's comments about `dummy_page` explicitly addresses uninitialized memory during page initialization!
4. **Bounds Checks/Length Logic:** The code has several checks for sequence numbers, lengths, etc. But I could not find a situation where an uninitialized variable controls a length or a bounds check that could allow access to uninitialized bytes of existing buffers.
Now, on the topic of KMSAN vs. KASAN and memory safety, I see that this patch primarily involves:
* Concurrency and Locking.
* Memory Management/Deallocation (AEAD contexts, `sk_buff`).
* Data Structure state transitions.
Concurrency, UAF, double-free, and list corruption are all very real risks here. Race conditions and synchronization are core to the rekey process and need to
|
| 95/3 | 2026/09/20 01:11 | tool | set-results |
0mArgs:{
"KMSANReasoning": "The patch series implements TLS 1.3 hardware offload and key rotation (KeyUpdate) support for kTLS (in net/tls and driver offloads like mlx5).\n\nAnalysis of memory initialization and safety:\n1. Heap/Page Allocations:\n - All newly added context structures (`tls_offload_context_tx`, `tls_offload_context_rx`) continue to be allocated via kzalloc (`kzalloc_obj`), guaranteeing zero-initialization of newly added fields (e.g. `rekey`).\n - The start marker record (`offload_ctx-\u003erekey.start_marker`) is allocated and its fields (`end_seq`, `len = 0`, `num_frags = 0`) are explicitly initialized prior to being queued on `records_list`; its `frags` are not accessed when `num_frags == 0`.\n - The fallback page `dummy_page` in `tls_device_init()` is explicitly allocated with `__GFP_ZERO` and initialized with an identity map specifically to avoid any uninitialized byte leak.\n2. User-Space Info Leaks (getsockopt):\n - In `do_tls_getsockopt_conf()`, reading the rekey crypto info accesses `offload_ctx-\u003erekey.crypto_send.info`. This union is originally populated from user input into a zero-initialized union (`tmp = {}`) and copied into the kzalloc'd `offload_ctx`. When returned via `copy_to_iter()`, strictly `cipher_desc-\u003ecrypto_info` bytes are copied, leaving no uninitialized padding or fields.\n3. sk_buff Bitfield:\n - The new `decrypt_failed` bitfield in `struct sk_buff` resides in the header area zeroed out by `__alloc_skb()` upon allocation, and is maintained via `skb_copy_decrypted()`.\n4. KASAN vs KMSAN Applicability:\n - The risks introduced by this patch series pertain to asynchronous state machine transitions, concurrent RCU dereferences / reader-writer locking (`tx_lock`, `device_offload_lock`, `lock_sock`), and lifetime management of crypto AEAD transforms during fallback / rekeying.\n - Any bugs arising from these changes (e.g. use-after-free of AEAD transforms, double frees, list corruption, or deadlock) are detected by KASAN and LOCKDEP.\n - There are no risks of uninitialized memory reads or information leaks that would require KMSAN detection.\n\nTherefore, a dedicated KMSAN fuzzing session is not justified.",
"NeedsKMSAN": false
}Results: {
"KMSANReasoning": "The patch series implements TLS 1.3 hardware offload and key rotation (KeyUpdate) support for kTLS (in net/tls and driver offloads like mlx5).\n\nAnalysis of memory initialization and safety:\n1. Heap/Page Allocations:\n - All newly added context structures (`tls_offload_context_tx`, `tls_offload_context_rx`) continue to be allocated via kzalloc (`kzalloc_obj`), guaranteeing zero-initialization of newly added fields (e.g. `rekey`).\n - The start marker record (`offload_ctx-\u003erekey.start_marker`) is allocated and its fields (`end_seq`, `len = 0`, `num_frags = 0`) are explicitly initialized prior to being queued on `records_list`; its `frags` are not accessed when `num_frags == 0`.\n - The fallback page `dummy_page` in `tls_device_init()` is explicitly allocated with `__GFP_ZERO` and initialized with an identity map specifically to avoid any uninitialized byte leak.\n2. User-Space Info Leaks (getsockopt):\n - In `do_tls_getsockopt_conf()`, reading the rekey crypto info accesses `offload_ctx-\u003erekey.crypto_send.info`. This union is originally populated from user input into a zero-initialized union (`tmp = {}`) and copied into the kzalloc'd `offload_ctx`. When returned via `copy_to_iter()`, strictly `cipher_desc-\u003ecrypto_info` bytes are copied, leaving no uninitialized padding or fields.\n3. sk_buff Bitfield:\n - The new `decrypt_failed` bitfield in `struct sk_buff` resides in the header area zeroed out by `__alloc_skb()` upon allocation, and is maintained via `skb_copy_decrypted()`.\n4. KASAN vs KMSAN Applicability:\n - The risks introduced by this patch series pertain to asynchronous state machine transitions, concurrent RCU dereferences / reader-writer locking (`tx_lock`, `device_offload_lock`, `lock_sock`), and lifetime management of crypto AEAD transforms during fallback / rekeying.\n - Any bugs arising from these changes (e.g. use-after-free of AEAD transforms, double frees, list corruption, or deadlock) are detected by KASAN and LOCKDEP.\n - There are no risks of uninitialized memory reads or information leaks that would require KMSAN detection.\n\nTherefore, a dedicated KMSAN fuzzing session is not justified.",
"NeedsKMSAN": false
} |
| Total Calls | Total Tokens | Avg Tokens | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|
| Total Calls | Total Duration (Seconds) | Avg Duration (Seconds) |
|---|