| 1 | |
| 2 |
|
| 3 | ==================
|
| 4 | Kernel TLS offload
|
| 5 | ==================
|
| 6 |
|
| 7 | Kernel TLS operation
|
| 8 | ====================
|
| 9 |
|
| 10 | Linux kernel provides TLS connection offload infrastructure. Once a TCP
|
| 11 | connection is in ``ESTABLISHED`` state user space can enable the TLS Upper
|
| 12 | Layer Protocol (ULP) and install the cryptographic connection state.
|
| 13 | For details regarding the user-facing interface refer to the TLS
|
| 14 | documentation in :ref:`Documentation/networking/tls.rst <kernel_tls>`.
|
| 15 |
|
| 16 | ``ktls`` can operate in two modes:
|
| 17 |
|
| 18 | * Software crypto mode (``TLS_SW``) - CPU handles the cryptography.
|
| 19 | In most basic cases only crypto operations synchronous with the CPU
|
| 20 | can be used, but depending on calling context CPU may utilize
|
| 21 | asynchronous crypto accelerators. The use of accelerators introduces extra
|
| 22 | latency on socket reads (decryption only starts when a read syscall
|
| 23 | is made) and additional I/O load on the system.
|
| 24 | * Packet-based NIC offload mode (``TLS_HW``) - the NIC handles crypto
|
| 25 | on a packet by packet basis, provided the packets arrive in order.
|
| 26 | This mode integrates best with the kernel stack and is described in detail
|
| 27 | in the remaining part of this document
|
| 28 | (``ethtool`` flags ``tls-hw-tx-offload`` and ``tls-hw-rx-offload``).
|
| 29 |
|
| 30 | The operation mode is selected automatically based on device configuration,
|
| 31 | offload opt-in or opt-out on per-connection basis is not currently supported.
|
| 32 |
|
| 33 | TX
|
| 34 | --
|
| 35 |
|
| 36 | At a high level user write requests are turned into a scatter list, the TLS ULP
|
| 37 | intercepts them, inserts record framing, performs encryption (in ``TLS_SW``
|
| 38 | mode) and then hands the modified scatter list to the TCP layer. From this
|
| 39 | point on the TCP stack proceeds as normal.
|
| 40 |
|
| 41 | In ``TLS_HW`` mode the encryption is not performed in the TLS ULP.
|
| 42 | Instead packets reach a device driver, the driver will mark the packets
|
| 43 | for crypto offload based on the socket the packet is attached to,
|
| 44 | and send them to the device for encryption and transmission.
|
| 45 |
|
| 46 | RX
|
| 47 | --
|
| 48 |
|
| 49 | On the receive side, if the device handled decryption and authentication
|
| 50 | successfully, the driver will set the decrypted bit in the associated
|
| 51 | :c:type:`struct sk_buff <sk_buff>`. The packets reach the TCP stack and
|
| 52 | are handled normally. ``ktls`` is informed when data is queued to the socket
|
| 53 | and the ``strparser`` mechanism is used to delineate the records. Upon read
|
| 54 | request, records are retrieved from the socket and passed to decryption routine.
|
| 55 | If device decrypted all the segments of the record the decryption is skipped,
|
| 56 | otherwise software path handles decryption.
|
| 57 |
|
| 58 | .. kernel-figure:: tls-offload-layers.svg
|
| 59 | :alt: TLS offload layers
|
| 60 | :align: center
|
| 61 | :figwidth: 28em
|
| 62 |
|
| 63 | Layers of Kernel TLS stack
|
| 64 |
|
| 65 | Device configuration
|
| 66 | ====================
|
| 67 |
|
| 68 | During driver initialization device sets the ``NETIF_F_HW_TLS_RX`` and
|
| 69 | ``NETIF_F_HW_TLS_TX`` features and installs its
|
| 70 | :c:type:`struct tlsdev_ops <tlsdev_ops>`
|
| 71 | pointer in the :c:member:`tlsdev_ops` member of the
|
| 72 | :c:type:`struct net_device <net_device>`.
|
| 73 |
|
| 74 | When TLS cryptographic connection state is installed on a ``ktls`` socket
|
| 75 | (note that it is done twice, once for RX and once for TX direction,
|
| 76 | and the two are completely independent), the kernel checks if the underlying
|
| 77 | network device is offload-capable and attempts the offload. In case offload
|
| 78 | fails the connection is handled entirely in software using the same mechanism
|
| 79 | as if the offload was never tried.
|
| 80 |
|
| 81 | Offload request is performed via the :c:member:`tls_dev_add` callback of
|
| 82 | :c:type:`struct tlsdev_ops <tlsdev_ops>`:
|
| 83 |
|
| 84 | .. code-block:: c
|
| 85 |
|
| 86 | int (*tls_dev_add)(struct net_device *netdev, struct sock *sk,
|
| 87 | enum tls_offload_ctx_dir direction,
|
| 88 | struct tls_crypto_info *crypto_info,
|
| 89 | u32 start_offload_tcp_sn);
|
| 90 |
|
| 91 | ``direction`` indicates whether the cryptographic information is for
|
| 92 | the received or transmitted packets. Driver uses the ``sk`` parameter
|
| 93 | to retrieve the connection 5-tuple and socket family (IPv4 vs IPv6).
|
| 94 | Cryptographic information in ``crypto_info`` includes the key, iv, salt
|
| 95 | as well as TLS record sequence number. ``start_offload_tcp_sn`` indicates
|
| 96 | which TCP sequence number corresponds to the beginning of the record with
|
| 97 | sequence number from ``crypto_info``. The driver can add its state
|
| 98 | at the end of kernel structures (see :c:member:`driver_state` members
|
| 99 | in ``include/net/tls.h``) to avoid additional allocations and pointer
|
| 100 | dereferences.
|
| 101 |
|
| 102 | When the offloaded connection is destroyed the core calls
|
| 103 | the :c:member:`tls_dev_del` callback so the driver can release per-direction
|
| 104 | state:
|
| 105 |
|
| 106 | .. code-block:: c
|
| 107 |
|
| 108 | void (*tls_dev_del)(struct net_device *netdev,
|
| 109 | struct tls_context *ctx,
|
| 110 | enum tls_offload_ctx_dir direction);
|
| 111 |
|
| 112 | ``tls_dev_del`` is mandatory whenever ``tls_dev_add`` is provided.
|
| 113 |
|
| 114 | The third TLS device callback is :c:member:`tls_dev_resync`, called by the core
|
| 115 | to synchronize the TCP stream with the record boundaries:
|
| 116 |
|
| 117 | .. code-block:: c
|
| 118 |
|
| 119 | int (*tls_dev_resync)(struct net_device *netdev,
|
| 120 | struct sock *sk, u32 seq, u8 *rcd_sn,
|
| 121 | enum tls_offload_ctx_dir direction);
|
| 122 |
|
| 123 | See the `Resync handling`_ section for details.
|
| 124 |
|
| 125 | TX
|
| 126 | --
|
| 127 |
|
| 128 | After TX state is installed, the stack guarantees that the first segment
|
| 129 | of the stream will start exactly at the ``start_offload_tcp_sn`` sequence
|
| 130 | number, simplifying TCP sequence number matching.
|
| 131 |
|
| 132 | TX offload being fully initialized does not imply that all segments passing
|
| 133 | through the driver and which belong to the offloaded socket will be after
|
| 134 | the expected sequence number and will have kernel record information.
|
| 135 | In particular, already encrypted data may have been queued to the socket
|
| 136 | before installing the connection state in the kernel.
|
| 137 |
|
| 138 | RX
|
| 139 | --
|
| 140 |
|
| 141 | In the RX direction, the local networking stack has little control over
|
| 142 | segmentation, so the initial records' TCP sequence number may be anywhere
|
| 143 | inside the segment.
|
| 144 |
|
| 145 | Normal operation
|
| 146 | ================
|
| 147 |
|
| 148 | At the minimum the device maintains the following state for each connection, in
|
| 149 | each direction:
|
| 150 |
|
| 151 | * crypto secrets (key, iv, salt)
|
| 152 | * crypto processing state (partial blocks, partial authentication tag, etc.)
|
| 153 | * record metadata (sequence number, processing offset and length)
|
| 154 | * expected TCP sequence number
|
| 155 |
|
| 156 | There are no guarantees on record length or record segmentation. In particular
|
| 157 | segments may start at any point of a record and contain any number of records.
|
| 158 | Assuming segments are received in order, the device should be able to perform
|
| 159 | crypto operations and authentication regardless of segmentation. For this
|
| 160 | to be possible, the device has to keep a small amount of segment-to-segment
|
| 161 | state. This includes at least:
|
| 162 |
|
| 163 | * partial headers (if a segment carried only a part of the TLS header)
|
| 164 | * partial data block
|
| 165 | * partial authentication tag (all data had been seen but part of the
|
| 166 | authentication tag has to be written or read from the subsequent segment)
|
| 167 |
|
| 168 | Record reassembly is not necessary for TLS offload. If the packets arrive
|
| 169 | in order the device should be able to handle them separately and make
|
| 170 | forward progress.
|
| 171 |
|
| 172 | TX
|
| 173 | --
|
| 174 |
|
| 175 | The kernel stack performs record framing reserving space for the authentication
|
| 176 | tag and populating all other TLS header and tailer fields.
|
| 177 |
|
| 178 | Both the device and the driver maintain expected TCP sequence numbers
|
| 179 | due to the possibility of retransmissions and the lack of software fallback
|
| 180 | once the packet reaches the device.
|
| 181 | For segments passed in order, the driver marks the packets with
|
| 182 | a connection identifier (note that a 5-tuple lookup is insufficient to identify
|
| 183 | packets requiring HW offload, see the :ref:`5tuple_problems` section)
|
| 184 | and hands them to the device. The device identifies the packet as requiring
|
| 185 | TLS handling and confirms the sequence number matches its expectation.
|
| 186 | The device performs encryption and authentication of the record data.
|
| 187 | It replaces the authentication tag and TCP checksum with correct values.
|
| 188 |
|
| 189 | RX
|
| 190 | --
|
| 191 |
|
| 192 | Before a packet is DMAed to the host (but after NIC's embedded switching
|
| 193 | and packet transformation functions) the device validates the Layer 4
|
| 194 | checksum and performs a 5-tuple lookup to find any TLS connection the packet
|
| 195 | may belong to (technically a 4-tuple
|
| 196 | lookup is sufficient - IP addresses and TCP port numbers, as the protocol
|
| 197 | is always TCP). If the packet is matched to a connection, the device confirms
|
| 198 | if the TCP sequence number is the expected one and proceeds to TLS handling
|
| 199 | (record delineation, decryption, authentication for each record in the packet).
|
| 200 | The device leaves the record framing unmodified, the stack takes care of record
|
| 201 | decapsulation. Device indicates successful handling of TLS offload in the
|
| 202 | per-packet context (descriptor) passed to the host.
|
| 203 |
|
| 204 | Upon reception of a TLS offloaded packet, the driver sets
|
| 205 | the :c:member:`decrypted` mark in :c:type:`struct sk_buff <sk_buff>`
|
| 206 | corresponding to the segment. Networking stack makes sure decrypted
|
| 207 | and non-decrypted segments do not get coalesced (e.g. by GRO or socket layer)
|
| 208 | and takes care of partial decryption.
|
| 209 |
|
| 210 | Resync handling
|
| 211 | ===============
|
| 212 |
|
| 213 | In presence of packet drops or network packet reordering, the device may lose
|
| 214 | synchronization with the TLS stream, and require a resync with the kernel's
|
| 215 | TCP stack.
|
| 216 |
|
| 217 | Note that resync is only attempted for connections which were successfully
|
| 218 | added to the device table and are in TLS_HW mode. For example,
|
| 219 | if the table was full when cryptographic state was installed in the kernel,
|
| 220 | such connection will never get offloaded. Therefore the resync request
|
| 221 | does not carry any cryptographic connection state.
|
| 222 |
|
| 223 | TX
|
| 224 | --
|
| 225 |
|
| 226 | Segments transmitted from an offloaded socket can get out of sync
|
| 227 | in similar ways to the receive side-retransmissions - local drops
|
| 228 | are possible, though network reorders are not. There are currently
|
| 229 | two mechanisms for dealing with out of order segments.
|
| 230 |
|
| 231 | Crypto state rebuilding
|
| 232 | ~~~~~~~~~~~~~~~~~~~~~~~
|
| 233 |
|
| 234 | Whenever an out of order segment is transmitted the driver provides
|
| 235 | the device with enough information to perform cryptographic operations.
|
| 236 | This means most likely that the part of the record preceding the current
|
| 237 | segment has to be passed to the device as part of the packet context,
|
| 238 | together with its TCP sequence number and TLS record number. The device
|
| 239 | can then initialize its crypto state, process and discard the preceding
|
| 240 | data (to be able to insert the authentication tag) and move onto handling
|
| 241 | the actual packet.
|
| 242 |
|
| 243 | In this mode depending on the implementation the driver can either ask
|
| 244 | for a continuation with the crypto state and the new sequence number
|
| 245 | (next expected segment is the one after the out of order one), or continue
|
| 246 | with the previous stream state - assuming that the out of order segment
|
| 247 | was just a retransmission. The former is simpler, and does not require
|
| 248 | retransmission detection therefore it is the recommended method until
|
| 249 | such time it is proven inefficient.
|
| 250 |
|
| 251 | Next record sync
|
| 252 | ~~~~~~~~~~~~~~~~
|
| 253 |
|
| 254 | Whenever an out of order segment is detected the driver requests
|
| 255 | that the ``ktls`` software fallback code encrypt it. If the segment's
|
| 256 | sequence number is lower than expected the driver assumes retransmission
|
| 257 | and doesn't change device state. If the segment is in the future, it
|
| 258 | may imply a local drop, the driver asks the stack to sync the device
|
| 259 | to the next record state and falls back to software.
|
| 260 |
|
| 261 | Resync request is indicated with:
|
| 262 |
|
| 263 | .. code-block:: c
|
| 264 |
|
| 265 | void tls_offload_tx_resync_request(struct sock *sk, u32 got_seq, u32 exp_seq)
|
| 266 |
|
| 267 | Until resync is complete driver should not access its expected TCP
|
| 268 | sequence number (as it will be updated from a different context).
|
| 269 | Following helper should be used to test if resync is complete:
|
| 270 |
|
| 271 | .. code-block:: c
|
| 272 |
|
| 273 | bool tls_offload_tx_resync_pending(struct sock *sk)
|
| 274 |
|
| 275 | Next time ``ktls`` pushes a record it will first send its TCP sequence number
|
| 276 | and TLS record number to the driver via the ``tls_dev_resync`` callback.
|
| 277 | The stack will also make sure that the new record will start on a segment
|
| 278 | boundary (like it does when the connection is initially added).
|
| 279 |
|
| 280 | RX
|
| 281 | --
|
| 282 |
|
| 283 | A small amount of RX reorder events may not require a full resynchronization.
|
| 284 | In particular the device should not lose synchronization
|
| 285 | when record boundary can be recovered:
|
| 286 |
|
| 287 | .. kernel-figure:: tls-offload-reorder-good.svg
|
| 288 | :alt: reorder of non-header segment
|
| 289 | :align: center
|
| 290 |
|
| 291 | Reorder of non-header segment
|
| 292 |
|
| 293 | Green segments are successfully decrypted, blue ones are passed
|
| 294 | as received on wire, red stripes mark start of new records.
|
| 295 |
|
| 296 | In above case segment 1 is received and decrypted successfully.
|
| 297 | Segment 2 was dropped so 3 arrives out of order. The device knows
|
| 298 | the next record starts inside 3, based on record length in segment 1.
|
| 299 | Segment 3 is passed untouched, because due to lack of data from segment 2
|
| 300 | the remainder of the previous record inside segment 3 cannot be handled.
|
| 301 | The device can, however, collect the authentication algorithm's state
|
| 302 | and partial block from the new record in segment 3 and when 4 and 5
|
| 303 | arrive continue decryption. Finally when 2 arrives it's completely outside
|
| 304 | of expected window of the device so it's passed as is without special
|
| 305 | handling. ``ktls`` software fallback handles the decryption of record
|
| 306 | spanning segments 1, 2 and 3. The device did not get out of sync,
|
| 307 | even though two segments did not get decrypted.
|
| 308 |
|
| 309 | Kernel synchronization may be necessary if the lost segment contained
|
| 310 | a record header and arrived after the next record header has already passed:
|
| 311 |
|
| 312 | .. kernel-figure:: tls-offload-reorder-bad.svg
|
| 313 | :alt: reorder of header segment
|
| 314 | :align: center
|
| 315 |
|
| 316 | Reorder of segment with a TLS header
|
| 317 |
|
| 318 | In this example segment 2 gets dropped, and it contains a record header.
|
| 319 | Device can only detect that segment 4 also contains a TLS header
|
| 320 | if it knows the length of the previous record from segment 2. In this case
|
| 321 | the device will lose synchronization with the stream.
|
| 322 |
|
| 323 | Stream scan resynchronization
|
| 324 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 325 |
|
| 326 | When the device gets out of sync and the stream reaches TCP sequence
|
| 327 | numbers more than a max size record past the expected TCP sequence number,
|
| 328 | the device starts scanning for a known header pattern. For example
|
| 329 | for TLS 1.2 and TLS 1.3 subsequent bytes of value ``0x03 0x03`` occur
|
| 330 | in the SSL/TLS version field of the header. Once pattern is matched
|
| 331 | the device continues attempting parsing headers at expected locations
|
| 332 | (based on the length fields at guessed locations).
|
| 333 | Whenever the expected location does not contain a valid header the scan
|
| 334 | is restarted.
|
| 335 |
|
| 336 | When the header is matched the device sends a confirmation request
|
| 337 | to the kernel, asking if the guessed location is correct (if a TLS record
|
| 338 | really starts there), and which record sequence number the given header had.
|
| 339 |
|
| 340 | The asynchronous resync process is coordinated on the kernel side using
|
| 341 | struct tls_offload_resync_async, which tracks and manages the resync request.
|
| 342 |
|
| 343 | Helper functions to manage struct tls_offload_resync_async:
|
| 344 |
|
| 345 | ``tls_offload_rx_resync_async_request_start()``
|
| 346 | Initializes an asynchronous resync attempt by specifying the sequence range to
|
| 347 | monitor and resetting internal state in the struct.
|
| 348 |
|
| 349 | ``tls_offload_rx_resync_async_request_end()``
|
| 350 | Retains the device's guessed TCP sequence number for comparison with current or
|
| 351 | future logged ones. It also clears the RESYNC_REQ_ASYNC flag from the resync
|
| 352 | request, indicating that the device has submitted its guessed sequence number.
|
| 353 |
|
| 354 | ``tls_offload_rx_resync_async_request_cancel()``
|
| 355 | Cancels any in-progress resync attempt, clearing the request state.
|
| 356 |
|
| 357 | When the kernel processes an RX segment that begins a new TLS record, it
|
| 358 | examines the current status of the asynchronous resynchronization request.
|
| 359 |
|
| 360 | If the device is still waiting to provide its guessed TCP sequence number
|
| 361 | (the async state), the kernel records the sequence number of this segment so
|
| 362 | that it can later be compared once the device's guess becomes available.
|
| 363 |
|
| 364 | If the device has already submitted its guessed sequence number (the non-async
|
| 365 | state), the kernel now tries to match that guess against the sequence numbers of
|
| 366 | all TLS record headers that have been logged since the resync request
|
| 367 | started.
|
| 368 |
|
| 369 | The kernel confirms the guessed location was correct and tells the device
|
| 370 | the record sequence number via the ``tls_dev_resync`` callback. Meanwhile,
|
| 371 | the device had been parsing and counting all records since the just-confirmed
|
| 372 | one, it adds the number of records it had seen to the record number provided
|
| 373 | by the kernel.
|
| 374 | At this point the device is in sync and can resume decryption at next
|
| 375 | segment boundary.
|
| 376 |
|
| 377 | In a pathological case the device may latch onto a sequence of matching
|
| 378 | headers and never hear back from the kernel (there is no negative
|
| 379 | confirmation from the kernel). The implementation may choose to periodically
|
| 380 | restart scan. Given how unlikely falsely-matching stream is, however,
|
| 381 | periodic restart is not deemed necessary.
|
| 382 |
|
| 383 | Special care has to be taken if the confirmation request is passed
|
| 384 | asynchronously to the packet stream and record may get processed
|
| 385 | by the kernel before the confirmation request.
|
| 386 |
|
| 387 | Stack-driven resynchronization
|
| 388 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 389 |
|
| 390 | The driver may also request the stack to perform resynchronization
|
| 391 | whenever it sees the records are no longer getting decrypted.
|
| 392 | If the connection is configured in this mode the stack automatically
|
| 393 | schedules resynchronization after it has received two completely encrypted
|
| 394 | records.
|
| 395 |
|
| 396 | The stack waits for the socket to drain and informs the device about
|
| 397 | the next expected record number and its TCP sequence number via the
|
| 398 | ``tls_dev_resync`` callback. If the
|
| 399 | records continue to be received fully encrypted stack retries the
|
| 400 | synchronization with an exponential back off (first after 2 encrypted
|
| 401 | records, then after 4 records, after 8, after 16... up until every
|
| 402 | 128 records).
|
| 403 |
|
| 404 | Rekey
|
| 405 | =====
|
| 406 |
|
| 407 | Offload does not currently support TLS 1.3, therefore key rotation
|
| 408 | is not a concern for offloaded connections at this point.
|
| 409 |
|
| 410 | Error handling
|
| 411 | ==============
|
| 412 |
|
| 413 | TX
|
| 414 | --
|
| 415 |
|
| 416 | Packets may be redirected or rerouted by the stack to a different
|
| 417 | device than the selected TLS offload device. The stack will handle
|
| 418 | such condition using the :c:func:`sk_validate_xmit_skb` helper
|
| 419 | (TLS offload code installs :c:func:`tls_validate_xmit_skb` at this hook).
|
| 420 | Offload maintains information about all records until the data is
|
| 421 | fully acknowledged, so if skbs reach the wrong device they can be handled
|
| 422 | by software fallback.
|
| 423 |
|
| 424 | Any device TLS offload handling error on the transmission side must result
|
| 425 | in the packet being dropped. For example if a packet got out of order
|
| 426 | due to a bug in the stack or the device, reached the device and can't
|
| 427 | be encrypted such packet must be dropped.
|
| 428 |
|
| 429 | RX
|
| 430 | --
|
| 431 |
|
| 432 | If the device encounters any problems with TLS offload on the receive
|
| 433 | side it should pass the packet to the host's networking stack as it was
|
| 434 | received on the wire.
|
| 435 |
|
| 436 | For example authentication failure for any record in the segment should
|
| 437 | result in passing the unmodified packet to the software fallback. This means
|
| 438 | packets should not be modified "in place". Splitting segments to handle partial
|
| 439 | decryption is not advised. In other words either all records in the packet
|
| 440 | had been handled successfully and authenticated or the packet has to be passed
|
| 441 | to the host's stack as it was on the wire (recovering original packet in the
|
| 442 | driver if device provides precise error is sufficient).
|
| 443 |
|
| 444 | The Linux networking stack does not provide a way of reporting per-packet
|
| 445 | decryption and authentication errors, packets with errors must simply not
|
| 446 | have the :c:member:`decrypted` mark set.
|
| 447 |
|
| 448 | A packet should also not be handled by the TLS offload if it contains
|
| 449 | incorrect checksums.
|
| 450 |
|
| 451 | Performance metrics
|
| 452 | ===================
|
| 453 |
|
| 454 | TLS offload can be characterized by the following basic metrics:
|
| 455 |
|
| 456 | * max connection count
|
| 457 | * connection installation rate
|
| 458 | * connection installation latency
|
| 459 | * total cryptographic performance
|
| 460 |
|
| 461 | Note that each TCP connection requires a TLS session in both directions,
|
| 462 | the performance may be reported treating each direction separately.
|
| 463 |
|
| 464 | Max connection count
|
| 465 | --------------------
|
| 466 |
|
| 467 | The number of connections device can support can be exposed via
|
| 468 | ``devlink resource`` API.
|
| 469 |
|
| 470 | Total cryptographic performance
|
| 471 | -------------------------------
|
| 472 |
|
| 473 | Offload performance may depend on segment and record size.
|
| 474 |
|
| 475 | Overload of the cryptographic subsystem of the device should not have
|
| 476 | significant performance impact on non-offloaded streams.
|
| 477 |
|
| 478 | Statistics
|
| 479 | ==========
|
| 480 |
|
| 481 | Following minimum set of TLS-related statistics should be reported
|
| 482 | by the driver:
|
| 483 |
|
| 484 | * ``rx_tls_decrypted_packets`` - number of successfully decrypted RX packets
|
| 485 | which were part of a TLS stream.
|
| 486 | * ``rx_tls_decrypted_bytes`` - number of TLS payload bytes in RX packets
|
| 487 | which were successfully decrypted.
|
| 488 | * ``rx_tls_ctx`` - number of TLS RX HW offload contexts added to device for
|
| 489 | decryption.
|
| 490 | * ``rx_tls_del`` - number of TLS RX HW offload contexts deleted from device
|
| 491 | (connection has finished).
|
| 492 | * ``rx_tls_resync_req_pkt`` - number of received TLS packets with a resync
|
| 493 | request.
|
| 494 | * ``rx_tls_resync_req_start`` - number of times the TLS async resync request
|
| 495 | was started.
|
| 496 | * ``rx_tls_resync_req_end`` - number of times the TLS async resync request
|
| 497 | properly ended with providing the HW tracked tcp-seq.
|
| 498 | * ``rx_tls_resync_req_skip`` - number of times the TLS async resync request
|
| 499 | procedure was started but not properly ended.
|
| 500 | * ``rx_tls_resync_res_ok`` - number of times the TLS resync response call to
|
| 501 | the driver was successfully handled.
|
| 502 | * ``rx_tls_resync_res_skip`` - number of times the TLS resync response call to
|
| 503 | the driver was terminated unsuccessfully.
|
| 504 | * ``rx_tls_err`` - number of RX packets which were part of a TLS stream
|
| 505 | but were not decrypted due to unexpected error in the state machine.
|
| 506 | * ``tx_tls_encrypted_packets`` - number of TX packets passed to the device
|
| 507 | for encryption of their TLS payload.
|
| 508 | * ``tx_tls_encrypted_bytes`` - number of TLS payload bytes in TX packets
|
| 509 | passed to the device for encryption.
|
| 510 | * ``tx_tls_ctx`` - number of TLS TX HW offload contexts added to device for
|
| 511 | encryption.
|
| 512 | * ``tx_tls_ooo`` - number of TX packets which were part of a TLS stream
|
| 513 | but did not arrive in the expected order.
|
| 514 | * ``tx_tls_skip_no_sync_data`` - number of TX packets which were part of
|
| 515 | a TLS stream and arrived out-of-order, but skipped the HW offload routine
|
| 516 | and went to the regular transmit flow as they were retransmissions of the
|
| 517 | connection handshake.
|
| 518 | * ``tx_tls_drop_no_sync_data`` - number of TX packets which were part of
|
| 519 | a TLS stream dropped, because they arrived out of order and associated
|
| 520 | record could not be found.
|
| 521 | * ``tx_tls_drop_bypass_req`` - number of TX packets which were part of a TLS
|
| 522 | stream dropped, because they contain both data that has been encrypted by
|
| 523 | software and data that expects hardware crypto offload.
|
| 524 |
|
| 525 | Notable corner cases, exceptions and additional requirements
|
| 526 | ============================================================
|
| 527 |
|
| 528 | .. _5tuple_problems:
|
| 529 |
|
| 530 | 5-tuple matching limitations
|
| 531 | ----------------------------
|
| 532 |
|
| 533 | The device can only recognize received packets based on the 5-tuple
|
| 534 | of the socket. Current ``ktls`` implementation will not offload sockets
|
| 535 | routed through software interfaces such as those used for tunneling
|
| 536 | or virtual networking. However, many packet transformations performed
|
| 537 | by the networking stack (most notably any BPF logic) do not require
|
| 538 | any intermediate software device, therefore a 5-tuple match may
|
| 539 | consistently miss at the device level. In such cases the device
|
| 540 | should still be able to perform TX offload (encryption) and should
|
| 541 | fallback cleanly to software decryption (RX).
|
| 542 |
|
| 543 | Out of order
|
| 544 | ------------
|
| 545 |
|
| 546 | Introducing extra processing in NICs should not cause packets to be
|
| 547 | transmitted or received out of order, for example pure ACK packets
|
| 548 | should not be reordered with respect to data segments.
|
| 549 |
|
| 550 | Ingress reorder
|
| 551 | ---------------
|
| 552 |
|
| 553 | A device is permitted to perform packet reordering for consecutive
|
| 554 | TCP segments (i.e. placing packets in the correct order) but any form
|
| 555 | of additional buffering is disallowed.
|
| 556 |
|
| 557 | Coexistence with standard networking offload features
|
| 558 | -----------------------------------------------------
|
| 559 |
|
| 560 | Offloaded ``ktls`` sockets should support standard TCP stack features
|
| 561 | transparently. Enabling device TLS offload should not cause any difference
|
| 562 | in packets as seen on the wire.
|
| 563 |
|
| 564 | Transport layer transparency
|
| 565 | ----------------------------
|
| 566 |
|
| 567 | For the purpose of simplifying TLS offload, the device should not modify any
|
| 568 | packet headers.
|
| 569 |
|
| 570 | The device should not depend on any packet headers beyond what is strictly
|
| 571 | necessary for TLS offload.
|
| 572 |
|
| 573 | Segment drops
|
| 574 | -------------
|
| 575 |
|
| 576 | Dropping packets is acceptable only in the event of catastrophic
|
| 577 | system errors and should never be used as an error handling mechanism
|
| 578 | in cases arising from normal operation. In other words, reliance
|
| 579 | on TCP retransmissions to handle corner cases is not acceptable.
|
| 580 |
|
| 581 | TLS device features
|
| 582 | -------------------
|
| 583 |
|
| 584 | Drivers should ignore the changes to the TLS device feature flags.
|
| 585 | These flags will be acted upon accordingly by the core ``ktls`` code.
|
| 586 | TLS device feature flags only control adding of new TLS connection
|
| 587 | offloads, old connections will remain active after flags are cleared.
|
| 588 |
|
| 589 | TLS encryption cannot be offloaded to devices without checksum calculation
|
| 590 | offload. Hence, TLS TX device feature flag requires TX csum offload being set.
|
| 591 | Disabling the latter implies clearing the former. Disabling TX checksum offload
|
| 592 | should not affect old connections, and drivers should make sure checksum
|
| 593 | calculation does not break for them.
|
| 594 | Similarly, device-offloaded TLS decryption implies doing RXCSUM. If the user
|
| 595 | does not want to enable RX csum offload, TLS RX device feature is disabled
|
| 596 | as well.
|