| 1 | |
| 2 |
|
| 3 | ============
|
| 4 | Timestamping
|
| 5 | ============
|
| 6 |
|
| 7 |
|
| 8 | 1. Control Interfaces
|
| 9 | =====================
|
| 10 |
|
| 11 | The interfaces for receiving network packages timestamps are:
|
| 12 |
|
| 13 | SO_TIMESTAMP
|
| 14 | Generates a timestamp for each incoming packet in (not necessarily
|
| 15 | monotonic) system time. Reports the timestamp via recvmsg() in a
|
| 16 | control message in usec resolution.
|
| 17 | SO_TIMESTAMP is defined as SO_TIMESTAMP_NEW or SO_TIMESTAMP_OLD
|
| 18 | based on the architecture type and time_t representation of libc.
|
| 19 | Control message format is in struct __kernel_old_timeval for
|
| 20 | SO_TIMESTAMP_OLD and in struct __kernel_sock_timeval for
|
| 21 | SO_TIMESTAMP_NEW options respectively.
|
| 22 |
|
| 23 | SO_TIMESTAMPNS
|
| 24 | Same timestamping mechanism as SO_TIMESTAMP, but reports the
|
| 25 | timestamp as struct timespec in nsec resolution.
|
| 26 | SO_TIMESTAMPNS is defined as SO_TIMESTAMPNS_NEW or SO_TIMESTAMPNS_OLD
|
| 27 | based on the architecture type and time_t representation of libc.
|
| 28 | Control message format is in struct timespec for SO_TIMESTAMPNS_OLD
|
| 29 | and in struct __kernel_timespec for SO_TIMESTAMPNS_NEW options
|
| 30 | respectively.
|
| 31 |
|
| 32 | IP_MULTICAST_LOOP + SO_TIMESTAMP[NS]
|
| 33 | Only for multicast:approximate transmit timestamp obtained by
|
| 34 | reading the looped packet receive timestamp.
|
| 35 |
|
| 36 | SO_TIMESTAMPING
|
| 37 | Generates timestamps on reception, transmission or both. Supports
|
| 38 | multiple timestamp sources, including hardware. Supports generating
|
| 39 | timestamps for stream sockets.
|
| 40 |
|
| 41 |
|
| 42 | 1.1 SO_TIMESTAMP (also SO_TIMESTAMP_OLD and SO_TIMESTAMP_NEW)
|
| 43 | -------------------------------------------------------------
|
| 44 |
|
| 45 | This socket option enables timestamping of datagrams on the reception
|
| 46 | path. Because the destination socket, if any, is not known early in
|
| 47 | the network stack, the feature has to be enabled for all packets. The
|
| 48 | same is true for all early receive timestamp options.
|
| 49 |
|
| 50 | For interface details, see `man 7 socket`.
|
| 51 |
|
| 52 | Always use SO_TIMESTAMP_NEW timestamp to always get timestamp in
|
| 53 | struct __kernel_sock_timeval format.
|
| 54 |
|
| 55 | SO_TIMESTAMP_OLD returns incorrect timestamps after the year 2038
|
| 56 | on 32 bit machines.
|
| 57 |
|
| 58 | 1.2 SO_TIMESTAMPNS (also SO_TIMESTAMPNS_OLD and SO_TIMESTAMPNS_NEW)
|
| 59 | -------------------------------------------------------------------
|
| 60 |
|
| 61 | This option is identical to SO_TIMESTAMP except for the returned data type.
|
| 62 | Its struct timespec allows for higher resolution (ns) timestamps than the
|
| 63 | timeval of SO_TIMESTAMP (ms).
|
| 64 |
|
| 65 | Always use SO_TIMESTAMPNS_NEW timestamp to always get timestamp in
|
| 66 | struct __kernel_timespec format.
|
| 67 |
|
| 68 | SO_TIMESTAMPNS_OLD returns incorrect timestamps after the year 2038
|
| 69 | on 32 bit machines.
|
| 70 |
|
| 71 | 1.3 SO_TIMESTAMPING (also SO_TIMESTAMPING_OLD and SO_TIMESTAMPING_NEW)
|
| 72 | ----------------------------------------------------------------------
|
| 73 |
|
| 74 | Supports multiple types of timestamp requests. As a result, this
|
| 75 | socket option takes a bitmap of flags, not a boolean. In::
|
| 76 |
|
| 77 | err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val));
|
| 78 |
|
| 79 | val is an integer with any of the following bits set. Setting other
|
| 80 | bit returns EINVAL and does not change the current state.
|
| 81 |
|
| 82 | The socket option configures timestamp generation for individual
|
| 83 | sk_buffs (1.3.1), timestamp reporting to the socket's error
|
| 84 | queue (1.3.2) and options (1.3.3). Timestamp generation can also
|
| 85 | be enabled for individual sendmsg calls using cmsg (1.3.4).
|
| 86 |
|
| 87 |
|
| 88 | 1.3.1 Timestamp Generation
|
| 89 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 90 |
|
| 91 | Some bits are requests to the stack to try to generate timestamps. Any
|
| 92 | combination of them is valid. Changes to these bits apply to newly
|
| 93 | created packets, not to packets already in the stack. As a result, it
|
| 94 | is possible to selectively request timestamps for a subset of packets
|
| 95 | (e.g., for sampling) by embedding an send() call within two setsockopt
|
| 96 | calls, one to enable timestamp generation and one to disable it.
|
| 97 | Timestamps may also be generated for reasons other than being
|
| 98 | requested by a particular socket, such as when receive timestamping is
|
| 99 | enabled system wide, as explained earlier.
|
| 100 |
|
| 101 | SOF_TIMESTAMPING_RX_HARDWARE:
|
| 102 | Request rx timestamps generated by the network adapter.
|
| 103 |
|
| 104 | SOF_TIMESTAMPING_RX_SOFTWARE:
|
| 105 | Request rx timestamps when data enters the kernel. These timestamps
|
| 106 | are generated just after a device driver hands a packet to the
|
| 107 | kernel receive stack.
|
| 108 |
|
| 109 | SOF_TIMESTAMPING_TX_HARDWARE:
|
| 110 | Request tx timestamps generated by the network adapter. This flag
|
| 111 | can be enabled via both socket options and control messages.
|
| 112 |
|
| 113 | SOF_TIMESTAMPING_TX_SOFTWARE:
|
| 114 | Request tx timestamps when data leaves the kernel. These timestamps
|
| 115 | are generated in the device driver as close as possible, but always
|
| 116 | prior to, passing the packet to the network interface. Hence, they
|
| 117 | require driver support and may not be available for all devices.
|
| 118 | This flag can be enabled via both socket options and control messages.
|
| 119 |
|
| 120 | SOF_TIMESTAMPING_TX_SCHED:
|
| 121 | Request tx timestamps prior to entering the packet scheduler. Kernel
|
| 122 | transmit latency is, if long, often dominated by queuing delay. The
|
| 123 | difference between this timestamp and one taken at
|
| 124 | SOF_TIMESTAMPING_TX_SOFTWARE will expose this latency independent
|
| 125 | of protocol processing. The latency incurred in protocol
|
| 126 | processing, if any, can be computed by subtracting a userspace
|
| 127 | timestamp taken immediately before send() from this timestamp. On
|
| 128 | machines with virtual devices where a transmitted packet travels
|
| 129 | through multiple devices and, hence, multiple packet schedulers,
|
| 130 | a timestamp is generated at each layer. This allows for fine
|
| 131 | grained measurement of queuing delay. This flag can be enabled
|
| 132 | via both socket options and control messages.
|
| 133 |
|
| 134 | SOF_TIMESTAMPING_TX_ACK:
|
| 135 | Request tx timestamps when all data in the send buffer has been
|
| 136 | acknowledged. This only makes sense for reliable protocols. It is
|
| 137 | currently only implemented for TCP. For that protocol, it may
|
| 138 | over-report measurement, because the timestamp is generated when all
|
| 139 | data up to and including the buffer at send() was acknowledged: the
|
| 140 | cumulative acknowledgment. The mechanism ignores SACK and FACK.
|
| 141 | This flag can be enabled via both socket options and control messages.
|
| 142 |
|
| 143 | SOF_TIMESTAMPING_TX_COMPLETION:
|
| 144 | Request tx timestamps on packet tx completion. The completion
|
| 145 | timestamp is generated by the kernel when it receives packet a
|
| 146 | completion report from the hardware. Hardware may report multiple
|
| 147 | packets at once, and completion timestamps reflect the timing of the
|
| 148 | report and not actual tx time. This flag can be enabled via both
|
| 149 | socket options and control messages.
|
| 150 |
|
| 151 |
|
| 152 | 1.3.2 Timestamp Reporting
|
| 153 | ^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 154 |
|
| 155 | The other three bits control which timestamps will be reported in a
|
| 156 | generated control message. Changes to the bits take immediate
|
| 157 | effect at the timestamp reporting locations in the stack. Timestamps
|
| 158 | are only reported for packets that also have the relevant timestamp
|
| 159 | generation request set.
|
| 160 |
|
| 161 | SOF_TIMESTAMPING_SOFTWARE:
|
| 162 | Report any software timestamps when available.
|
| 163 |
|
| 164 | SOF_TIMESTAMPING_SYS_HARDWARE:
|
| 165 | This option is deprecated and ignored.
|
| 166 |
|
| 167 | SOF_TIMESTAMPING_RAW_HARDWARE:
|
| 168 | Report hardware timestamps as generated by
|
| 169 | SOF_TIMESTAMPING_TX_HARDWARE or SOF_TIMESTAMPING_RX_HARDWARE
|
| 170 | when available.
|
| 171 |
|
| 172 |
|
| 173 | 1.3.3 Timestamp Options
|
| 174 | ^^^^^^^^^^^^^^^^^^^^^^^
|
| 175 |
|
| 176 | The interface supports the options
|
| 177 |
|
| 178 | SOF_TIMESTAMPING_OPT_ID:
|
| 179 | Generate a unique identifier along with each packet. A process can
|
| 180 | have multiple concurrent timestamping requests outstanding. Packets
|
| 181 | can be reordered in the transmit path, for instance in the packet
|
| 182 | scheduler. In that case timestamps will be queued onto the error
|
| 183 | queue out of order from the original send() calls. It is not always
|
| 184 | possible to uniquely match timestamps to the original send() calls
|
| 185 | based on timestamp order or payload inspection alone, then.
|
| 186 |
|
| 187 | This option associates each packet at send() with a unique
|
| 188 | identifier and returns that along with the timestamp. The identifier
|
| 189 | is derived from a per-socket u32 counter (that wraps). For datagram
|
| 190 | sockets, the counter increments with each sent packet. For stream
|
| 191 | sockets, it increments with every byte. For stream sockets, also set
|
| 192 | SOF_TIMESTAMPING_OPT_ID_TCP, see the section below.
|
| 193 |
|
| 194 | The counter starts at zero. It is initialized the first time that
|
| 195 | the socket option is enabled. It is reset each time the option is
|
| 196 | enabled after having been disabled. Resetting the counter does not
|
| 197 | change the identifiers of existing packets in the system.
|
| 198 |
|
| 199 | This option is implemented only for transmit timestamps. There, the
|
| 200 | timestamp is always looped along with a struct sock_extended_err.
|
| 201 | The option modifies field ee_data to pass an id that is unique
|
| 202 | among all possibly concurrently outstanding timestamp requests for
|
| 203 | that socket.
|
| 204 |
|
| 205 | The process can optionally override the default generated ID, by
|
| 206 | passing a specific ID with control message SCM_TS_OPT_ID (not
|
| 207 | supported for TCP sockets)::
|
| 208 |
|
| 209 | struct msghdr *msg;
|
| 210 | ...
|
| 211 | cmsg = CMSG_FIRSTHDR(msg);
|
| 212 | cmsg->cmsg_level = SOL_SOCKET;
|
| 213 | cmsg->cmsg_type = SCM_TS_OPT_ID;
|
| 214 | cmsg->cmsg_len = CMSG_LEN(sizeof(__u32));
|
| 215 | *((__u32 *) CMSG_DATA(cmsg)) = opt_id;
|
| 216 | err = sendmsg(fd, msg, 0);
|
| 217 |
|
| 218 |
|
| 219 | SOF_TIMESTAMPING_OPT_ID_TCP:
|
| 220 | Pass this modifier along with SOF_TIMESTAMPING_OPT_ID for new TCP
|
| 221 | timestamping applications. SOF_TIMESTAMPING_OPT_ID defines how the
|
| 222 | counter increments for stream sockets, but its starting point is
|
| 223 | not entirely trivial. This option fixes that.
|
| 224 |
|
| 225 | For stream sockets, if SOF_TIMESTAMPING_OPT_ID is set, this should
|
| 226 | always be set too. On datagram sockets the option has no effect.
|
| 227 |
|
| 228 | A reasonable expectation is that the counter is reset to zero with
|
| 229 | the system call, so that a subsequent write() of N bytes generates
|
| 230 | a timestamp with counter N-1. SOF_TIMESTAMPING_OPT_ID_TCP
|
| 231 | implements this behavior under all conditions.
|
| 232 |
|
| 233 | SOF_TIMESTAMPING_OPT_ID without modifier often reports the same,
|
| 234 | especially when the socket option is set when no data is in
|
| 235 | transmission. If data is being transmitted, it may be off by the
|
| 236 | length of the output queue (SIOCOUTQ).
|
| 237 |
|
| 238 | The difference is due to being based on snd_una versus write_seq.
|
| 239 | snd_una is the offset in the stream acknowledged by the peer. This
|
| 240 | depends on factors outside of process control, such as network RTT.
|
| 241 | write_seq is the last byte written by the process. This offset is
|
| 242 | not affected by external inputs.
|
| 243 |
|
| 244 | The difference is subtle and unlikely to be noticed when configured
|
| 245 | at initial socket creation, when no data is queued or sent. But
|
| 246 | SOF_TIMESTAMPING_OPT_ID_TCP behavior is more robust regardless of
|
| 247 | when the socket option is set.
|
| 248 |
|
| 249 | SOF_TIMESTAMPING_OPT_CMSG:
|
| 250 | Support recv() cmsg for all timestamped packets. Control messages
|
| 251 | are already supported unconditionally on all packets with receive
|
| 252 | timestamps and on IPv6 packets with transmit timestamp. This option
|
| 253 | extends them to IPv4 packets with transmit timestamp. One use case
|
| 254 | is to correlate packets with their egress device, by enabling socket
|
| 255 | option IP_PKTINFO simultaneously.
|
| 256 |
|
| 257 |
|
| 258 | SOF_TIMESTAMPING_OPT_TSONLY:
|
| 259 | Applies to transmit timestamps only. Makes the kernel return the
|
| 260 | timestamp as a cmsg alongside an empty packet, as opposed to
|
| 261 | alongside the original packet. This reduces the amount of memory
|
| 262 | charged to the socket's receive budget (SO_RCVBUF) and delivers
|
| 263 | the timestamp even if sysctl net.core.tstamp_allow_data is 0.
|
| 264 | This option disables SOF_TIMESTAMPING_OPT_CMSG.
|
| 265 |
|
| 266 | SOF_TIMESTAMPING_OPT_STATS:
|
| 267 | Optional stats that are obtained along with the transmit timestamps.
|
| 268 | It must be used together with SOF_TIMESTAMPING_OPT_TSONLY. When the
|
| 269 | transmit timestamp is available, the stats are available in a
|
| 270 | separate control message of type SCM_TIMESTAMPING_OPT_STATS, as a
|
| 271 | list of TLVs (struct nlattr) of types. These stats allow the
|
| 272 | application to associate various transport layer stats with
|
| 273 | the transmit timestamps, such as how long a certain block of
|
| 274 | data was limited by peer's receiver window.
|
| 275 |
|
| 276 | SOF_TIMESTAMPING_OPT_PKTINFO:
|
| 277 | Enable the SCM_TIMESTAMPING_PKTINFO control message for incoming
|
| 278 | packets with hardware timestamps. The message contains struct
|
| 279 | scm_ts_pktinfo, which supplies the index of the real interface which
|
| 280 | received the packet and its length at layer 2. A valid (non-zero)
|
| 281 | interface index will be returned only if CONFIG_NET_RX_BUSY_POLL is
|
| 282 | enabled and the driver is using NAPI. The struct contains also two
|
| 283 | other fields, but they are reserved and undefined.
|
| 284 |
|
| 285 | SOF_TIMESTAMPING_OPT_TX_SWHW:
|
| 286 | Request both hardware and software timestamps for outgoing packets
|
| 287 | when SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE
|
| 288 | are enabled at the same time. If both timestamps are generated,
|
| 289 | two separate messages will be looped to the socket's error queue,
|
| 290 | each containing just one timestamp.
|
| 291 |
|
| 292 | SOF_TIMESTAMPING_OPT_RX_FILTER:
|
| 293 | Filter out spurious receive timestamps: report a receive timestamp
|
| 294 | only if the matching timestamp generation flag is enabled.
|
| 295 |
|
| 296 | Receive timestamps are generated early in the ingress path, before a
|
| 297 | packet's destination socket is known. If any socket enables receive
|
| 298 | timestamps, packets for all socket will receive timestamped packets.
|
| 299 | Including those that request timestamp reporting with
|
| 300 | SOF_TIMESTAMPING_SOFTWARE and/or SOF_TIMESTAMPING_RAW_HARDWARE, but
|
| 301 | do not request receive timestamp generation. This can happen when
|
| 302 | requesting transmit timestamps only.
|
| 303 |
|
| 304 | Receiving spurious timestamps is generally benign. A process can
|
| 305 | ignore the unexpected non-zero value. But it makes behavior subtly
|
| 306 | dependent on other sockets. This flag isolates the socket for more
|
| 307 | deterministic behavior.
|
| 308 |
|
| 309 | New applications are encouraged to pass SOF_TIMESTAMPING_OPT_ID to
|
| 310 | disambiguate timestamps and SOF_TIMESTAMPING_OPT_TSONLY to operate
|
| 311 | regardless of the setting of sysctl net.core.tstamp_allow_data.
|
| 312 |
|
| 313 | An exception is when a process needs additional cmsg data, for
|
| 314 | instance SOL_IP/IP_PKTINFO to detect the egress network interface.
|
| 315 | Then pass option SOF_TIMESTAMPING_OPT_CMSG. This option depends on
|
| 316 | having access to the contents of the original packet, so cannot be
|
| 317 | combined with SOF_TIMESTAMPING_OPT_TSONLY.
|
| 318 |
|
| 319 |
|
| 320 | 1.3.4. Enabling timestamps via control messages
|
| 321 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 322 |
|
| 323 | In addition to socket options, timestamp generation can be requested
|
| 324 | per write via cmsg, only for SOF_TIMESTAMPING_TX_* (see Section 1.3.1).
|
| 325 | Using this feature, applications can sample timestamps per sendmsg()
|
| 326 | without paying the overhead of enabling and disabling timestamps via
|
| 327 | setsockopt::
|
| 328 |
|
| 329 | struct msghdr *msg;
|
| 330 | ...
|
| 331 | cmsg = CMSG_FIRSTHDR(msg);
|
| 332 | cmsg->cmsg_level = SOL_SOCKET;
|
| 333 | cmsg->cmsg_type = SO_TIMESTAMPING;
|
| 334 | cmsg->cmsg_len = CMSG_LEN(sizeof(__u32));
|
| 335 | *((__u32 *) CMSG_DATA(cmsg)) = SOF_TIMESTAMPING_TX_SCHED |
|
| 336 | SOF_TIMESTAMPING_TX_SOFTWARE |
|
| 337 | SOF_TIMESTAMPING_TX_ACK;
|
| 338 | err = sendmsg(fd, msg, 0);
|
| 339 |
|
| 340 | The SOF_TIMESTAMPING_TX_* flags set via cmsg will override
|
| 341 | the SOF_TIMESTAMPING_TX_* flags set via setsockopt.
|
| 342 |
|
| 343 | Moreover, applications must still enable timestamp reporting via
|
| 344 | setsockopt to receive timestamps::
|
| 345 |
|
| 346 | __u32 val = SOF_TIMESTAMPING_SOFTWARE |
|
| 347 | SOF_TIMESTAMPING_OPT_ID /* or any other flag */;
|
| 348 | err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val));
|
| 349 |
|
| 350 |
|
| 351 | 1.4 Bytestream Timestamps
|
| 352 | -------------------------
|
| 353 |
|
| 354 | The SO_TIMESTAMPING interface supports timestamping of bytes in a
|
| 355 | bytestream. Each request is interpreted as a request for when the
|
| 356 | entire contents of the buffer has passed a timestamping point. That
|
| 357 | is, for streams option SOF_TIMESTAMPING_TX_SOFTWARE will record
|
| 358 | when all bytes have reached the device driver, regardless of how
|
| 359 | many packets the data has been converted into.
|
| 360 |
|
| 361 | In general, bytestreams have no natural delimiters and therefore
|
| 362 | correlating a timestamp with data is non-trivial. A range of bytes
|
| 363 | may be split across segments, any segments may be merged (possibly
|
| 364 | coalescing sections of previously segmented buffers associated with
|
| 365 | independent send() calls). Segments can be reordered and the same
|
| 366 | byte range can coexist in multiple segments for protocols that
|
| 367 | implement retransmissions.
|
| 368 |
|
| 369 | It is essential that all timestamps implement the same semantics,
|
| 370 | regardless of these possible transformations, as otherwise they are
|
| 371 | incomparable. Handling "rare" corner cases differently from the
|
| 372 | simple case (a 1:1 mapping from buffer to skb) is insufficient
|
| 373 | because performance debugging often needs to focus on such outliers.
|
| 374 |
|
| 375 | In practice, timestamps can be correlated with segments of a
|
| 376 | bytestream consistently, if both semantics of the timestamp and the
|
| 377 | timing of measurement are chosen correctly. This challenge is no
|
| 378 | different from deciding on a strategy for IP fragmentation. There, the
|
| 379 | definition is that only the first fragment is timestamped. For
|
| 380 | bytestreams, we chose that a timestamp is generated only when all
|
| 381 | bytes have passed a point. SOF_TIMESTAMPING_TX_ACK as defined is easy to
|
| 382 | implement and reason about. An implementation that has to take into
|
| 383 | account SACK would be more complex due to possible transmission holes
|
| 384 | and out of order arrival.
|
| 385 |
|
| 386 | On the host, TCP can also break the simple 1:1 mapping from buffer to
|
| 387 | skbuff as a result of Nagle, cork, autocork, segmentation and GSO. The
|
| 388 | implementation ensures correctness in all cases by tracking the
|
| 389 | individual last byte passed to send(), even if it is no longer the
|
| 390 | last byte after an skbuff extend or merge operation. It stores the
|
| 391 | relevant sequence number in skb_shinfo(skb)->tskey. Because an skbuff
|
| 392 | has only one such field, only one timestamp can be generated.
|
| 393 |
|
| 394 | In rare cases, a timestamp request can be missed if two requests are
|
| 395 | collapsed onto the same skb. A process can detect this situation by
|
| 396 | enabling SOF_TIMESTAMPING_OPT_ID and comparing the byte offset at
|
| 397 | send time with the value returned for each timestamp. It can prevent
|
| 398 | the situation by always flushing the TCP stack in between requests,
|
| 399 | for instance by enabling TCP_NODELAY and disabling TCP_CORK and
|
| 400 | autocork. After linux-4.7, a better way to prevent coalescing is
|
| 401 | to use MSG_EOR flag at sendmsg() time.
|
| 402 |
|
| 403 | These precautions ensure that the timestamp is generated only when all
|
| 404 | bytes have passed a timestamp point, assuming that the network stack
|
| 405 | itself does not reorder the segments. The stack indeed tries to avoid
|
| 406 | reordering. The one exception is under administrator control: it is
|
| 407 | possible to construct a packet scheduler configuration that delays
|
| 408 | segments from the same stream differently. Such a setup would be
|
| 409 | unusual.
|
| 410 |
|
| 411 |
|
| 412 | 2 Data Interfaces
|
| 413 | ==================
|
| 414 |
|
| 415 | Timestamps are read using the ancillary data feature of recvmsg().
|
| 416 | See `man 3 cmsg` for details of this interface. The socket manual
|
| 417 | page (`man 7 socket`) describes how timestamps generated with
|
| 418 | SO_TIMESTAMP and SO_TIMESTAMPNS records can be retrieved.
|
| 419 |
|
| 420 |
|
| 421 | 2.1 SCM_TIMESTAMPING records
|
| 422 | ----------------------------
|
| 423 |
|
| 424 | These timestamps are returned in a control message with cmsg_level
|
| 425 | SOL_SOCKET, cmsg_type SCM_TIMESTAMPING, and payload of type
|
| 426 |
|
| 427 | For SO_TIMESTAMPING_OLD::
|
| 428 |
|
| 429 | struct scm_timestamping {
|
| 430 | struct timespec ts[3];
|
| 431 | };
|
| 432 |
|
| 433 | For SO_TIMESTAMPING_NEW::
|
| 434 |
|
| 435 | struct scm_timestamping64 {
|
| 436 | struct __kernel_timespec ts[3];
|
| 437 |
|
| 438 | Always use SO_TIMESTAMPING_NEW timestamp to always get timestamp in
|
| 439 | struct scm_timestamping64 format.
|
| 440 |
|
| 441 | SO_TIMESTAMPING_OLD returns incorrect timestamps after the year 2038
|
| 442 | on 32 bit machines.
|
| 443 |
|
| 444 | The structure can return up to three timestamps. This is a legacy
|
| 445 | feature. At least one field is non-zero at any time. Most timestamps
|
| 446 | are passed in ts[0]. Hardware timestamps are passed in ts[2].
|
| 447 |
|
| 448 | ts[1] used to hold hardware timestamps converted to system time.
|
| 449 | Instead, expose the hardware clock device on the NIC directly as
|
| 450 | a HW PTP clock source, to allow time conversion in userspace and
|
| 451 | optionally synchronize system time with a userspace PTP stack such
|
| 452 | as linuxptp. For the PTP clock API, see Documentation/driver-api/ptp.rst.
|
| 453 |
|
| 454 | Note that if the SO_TIMESTAMP or SO_TIMESTAMPNS option is enabled
|
| 455 | together with SO_TIMESTAMPING using SOF_TIMESTAMPING_SOFTWARE, a false
|
| 456 | software timestamp will be generated in the recvmsg() call and passed
|
| 457 | in ts[0] when a real software timestamp is missing. This happens also
|
| 458 | on hardware transmit timestamps.
|
| 459 |
|
| 460 | 2.1.1 Transmit timestamps with MSG_ERRQUEUE
|
| 461 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 462 |
|
| 463 | For transmit timestamps the outgoing packet is looped back to the
|
| 464 | socket's error queue with the send timestamp(s) attached. A process
|
| 465 | receives the timestamps by calling recvmsg() with flag MSG_ERRQUEUE
|
| 466 | set and with a msg_control buffer sufficiently large to receive the
|
| 467 | relevant metadata structures. The recvmsg call returns the original
|
| 468 | outgoing data packet with two ancillary messages attached.
|
| 469 |
|
| 470 | A message of cm_level SOL_IP(V6) and cm_type IP(V6)_RECVERR
|
| 471 | embeds a struct sock_extended_err. This defines the error type. For
|
| 472 | timestamps, the ee_errno field is ENOMSG. The other ancillary message
|
| 473 | will have cm_level SOL_SOCKET and cm_type SCM_TIMESTAMPING. This
|
| 474 | embeds the struct scm_timestamping.
|
| 475 |
|
| 476 |
|
| 477 | 2.1.1.2 Timestamp types
|
| 478 | ~~~~~~~~~~~~~~~~~~~~~~~
|
| 479 |
|
| 480 | The semantics of the three struct timespec are defined by field
|
| 481 | ee_info in the extended error structure. It contains a value of
|
| 482 | type SCM_TSTAMP_* to define the actual timestamp passed in
|
| 483 | scm_timestamping.
|
| 484 |
|
| 485 | The SCM_TSTAMP_* types are 1:1 matches to the SOF_TIMESTAMPING_*
|
| 486 | control fields discussed previously, with one exception. For legacy
|
| 487 | reasons, SCM_TSTAMP_SND is equal to zero and can be set for both
|
| 488 | SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE. It
|
| 489 | is the first if ts[2] is non-zero, the second otherwise, in which
|
| 490 | case the timestamp is stored in ts[0].
|
| 491 |
|
| 492 |
|
| 493 | 2.1.1.3 Fragmentation
|
| 494 | ~~~~~~~~~~~~~~~~~~~~~
|
| 495 |
|
| 496 | Fragmentation of outgoing datagrams is rare, but is possible, e.g., by
|
| 497 | explicitly disabling PMTU discovery. If an outgoing packet is fragmented,
|
| 498 | then only the first fragment is timestamped and returned to the sending
|
| 499 | socket.
|
| 500 |
|
| 501 |
|
| 502 | 2.1.1.4 Packet Payload
|
| 503 | ~~~~~~~~~~~~~~~~~~~~~~
|
| 504 |
|
| 505 | The calling application is often not interested in receiving the whole
|
| 506 | packet payload that it passed to the stack originally: the socket
|
| 507 | error queue mechanism is just a method to piggyback the timestamp on.
|
| 508 | In this case, the application can choose to read datagrams with a
|
| 509 | smaller buffer, possibly even of length 0. The payload is truncated
|
| 510 | accordingly. Until the process calls recvmsg() on the error queue,
|
| 511 | however, the full packet is queued, taking up budget from SO_RCVBUF.
|
| 512 |
|
| 513 |
|
| 514 | 2.1.1.5 Blocking Read
|
| 515 | ~~~~~~~~~~~~~~~~~~~~~
|
| 516 |
|
| 517 | Reading from the error queue is always a non-blocking operation. To
|
| 518 | block waiting on a timestamp, use poll or select. poll() will return
|
| 519 | POLLERR in pollfd.revents if any data is ready on the error queue.
|
| 520 | There is no need to pass this flag in pollfd.events. This flag is
|
| 521 | ignored on request. See also `man 2 poll`.
|
| 522 |
|
| 523 |
|
| 524 | 2.1.2 Receive timestamps
|
| 525 | ^^^^^^^^^^^^^^^^^^^^^^^^
|
| 526 |
|
| 527 | On reception, there is no reason to read from the socket error queue.
|
| 528 | The SCM_TIMESTAMPING ancillary data is sent along with the packet data
|
| 529 | on a normal recvmsg(). Since this is not a socket error, it is not
|
| 530 | accompanied by a message SOL_IP(V6)/IP(V6)_RECVERROR. In this case,
|
| 531 | the meaning of the three fields in struct scm_timestamping is
|
| 532 | implicitly defined. ts[0] holds a software timestamp if set, ts[1]
|
| 533 | is again deprecated and ts[2] holds a hardware timestamp if set.
|
| 534 |
|
| 535 |
|
| 536 | 3. Hardware Timestamping configuration: ETHTOOL_MSG_TSCONFIG_SET/GET
|
| 537 | ====================================================================
|
| 538 |
|
| 539 | Hardware time stamping must also be initialized for each device driver
|
| 540 | that is expected to do hardware time stamping. The parameter is defined in
|
| 541 | include/uapi/linux/net_tstamp.h as::
|
| 542 |
|
| 543 | struct hwtstamp_config {
|
| 544 | int flags; /* no flags defined right now, must be zero */
|
| 545 | int tx_type; /* HWTSTAMP_TX_* */
|
| 546 | int rx_filter; /* HWTSTAMP_FILTER_* */
|
| 547 | };
|
| 548 |
|
| 549 | Desired behavior is passed into the kernel and to a specific device by
|
| 550 | calling the tsconfig netlink socket ``ETHTOOL_MSG_TSCONFIG_SET``.
|
| 551 | The ``ETHTOOL_A_TSCONFIG_TX_TYPES``, ``ETHTOOL_A_TSCONFIG_RX_FILTERS`` and
|
| 552 | ``ETHTOOL_A_TSCONFIG_HWTSTAMP_FLAGS`` netlink attributes are then used to set
|
| 553 | the struct hwtstamp_config accordingly.
|
| 554 |
|
| 555 | The ``ETHTOOL_A_TSCONFIG_HWTSTAMP_PROVIDER`` netlink nested attribute is used
|
| 556 | to select the source of the hardware time stamping. It is composed of an index
|
| 557 | for the device source and a qualifier for the type of time stamping.
|
| 558 |
|
| 559 | Drivers are free to use a more permissive configuration than the requested
|
| 560 | configuration. It is expected that drivers should only implement directly the
|
| 561 | most generic mode that can be supported. For example if the hardware can
|
| 562 | support HWTSTAMP_FILTER_PTP_V2_EVENT, then it should generally always upscale
|
| 563 | HWTSTAMP_FILTER_PTP_V2_L2_SYNC, and so forth, as HWTSTAMP_FILTER_PTP_V2_EVENT
|
| 564 | is more generic (and more useful to applications).
|
| 565 |
|
| 566 | A driver which supports hardware time stamping shall update the struct
|
| 567 | with the actual, possibly more permissive configuration. If the
|
| 568 | requested packets cannot be time stamped, then nothing should be
|
| 569 | changed and ERANGE shall be returned (in contrast to EINVAL, which
|
| 570 | indicates that SIOCSHWTSTAMP is not supported at all).
|
| 571 |
|
| 572 | Only a processes with admin rights may change the configuration. User
|
| 573 | space is responsible to ensure that multiple processes don't interfere
|
| 574 | with each other and that the settings are reset.
|
| 575 |
|
| 576 | Any process can read the actual configuration by requesting tsconfig netlink
|
| 577 | socket ``ETHTOOL_MSG_TSCONFIG_GET``.
|
| 578 |
|
| 579 | The legacy configuration is the use of the ioctl(SIOCSHWTSTAMP) with a pointer
|
| 580 | to a struct ifreq whose ifr_data points to a struct hwtstamp_config.
|
| 581 | The tx_type and rx_filter are hints to the driver what it is expected to do.
|
| 582 | If the requested fine-grained filtering for incoming packets is not
|
| 583 | supported, the driver may time stamp more than just the requested types
|
| 584 | of packets. ioctl(SIOCGHWTSTAMP) is used in the same way as the
|
| 585 | ioctl(SIOCSHWTSTAMP). However, this has not been implemented in all drivers.
|
| 586 |
|
| 587 | ::
|
| 588 |
|
| 589 | /* possible values for hwtstamp_config->tx_type */
|
| 590 | enum {
|
| 591 | /*
|
| 592 | * no outgoing packet will need hardware time stamping;
|
| 593 | * should a packet arrive which asks for it, no hardware
|
| 594 | * time stamping will be done
|
| 595 | */
|
| 596 | HWTSTAMP_TX_OFF,
|
| 597 |
|
| 598 | /*
|
| 599 | * enables hardware time stamping for outgoing packets;
|
| 600 | * the sender of the packet decides which are to be
|
| 601 | * time stamped by setting SOF_TIMESTAMPING_TX_SOFTWARE
|
| 602 | * before sending the packet
|
| 603 | */
|
| 604 | HWTSTAMP_TX_ON,
|
| 605 | };
|
| 606 |
|
| 607 | /* possible values for hwtstamp_config->rx_filter */
|
| 608 | enum {
|
| 609 | /* time stamp no incoming packet at all */
|
| 610 | HWTSTAMP_FILTER_NONE,
|
| 611 |
|
| 612 | /* time stamp any incoming packet */
|
| 613 | HWTSTAMP_FILTER_ALL,
|
| 614 |
|
| 615 | /* return value: time stamp all packets requested plus some others */
|
| 616 | HWTSTAMP_FILTER_SOME,
|
| 617 |
|
| 618 | /* PTP v1, UDP, any kind of event packet */
|
| 619 | HWTSTAMP_FILTER_PTP_V1_L4_EVENT,
|
| 620 |
|
| 621 | /* for the complete list of values, please check
|
| 622 | * the include file include/uapi/linux/net_tstamp.h
|
| 623 | */
|
| 624 | };
|
| 625 |
|
| 626 | 3.1 Hardware Timestamping Implementation: Device Drivers
|
| 627 | --------------------------------------------------------
|
| 628 |
|
| 629 | A driver which supports hardware time stamping must support the
|
| 630 | ndo_hwtstamp_set NDO and update the supplied struct hwtstamp_config with
|
| 631 | the actual values as described in the section on SIOCSHWTSTAMP. It
|
| 632 | should also support ndo_hwtstamp_get NDO to retrieve configuration.
|
| 633 |
|
| 634 | Time stamps for received packets must be stored in the skb. To get a pointer
|
| 635 | to the shared time stamp structure of the skb call skb_hwtstamps(). Then
|
| 636 | set the time stamps in the structure::
|
| 637 |
|
| 638 | struct skb_shared_hwtstamps {
|
| 639 | /* hardware time stamp transformed into duration
|
| 640 | * since arbitrary point in time
|
| 641 | */
|
| 642 | ktime_t hwtstamp;
|
| 643 | };
|
| 644 |
|
| 645 | Time stamps for outgoing packets are to be generated as follows:
|
| 646 |
|
| 647 | - In hard_start_xmit(), check if (skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP)
|
| 648 | is set no-zero. If yes, then the driver is expected to do hardware time
|
| 649 | stamping.
|
| 650 | - If this is possible for the skb and requested, then declare
|
| 651 | that the driver is doing the time stamping by setting the flag
|
| 652 | SKBTX_IN_PROGRESS in skb_shinfo(skb)->tx_flags , e.g. with::
|
| 653 |
|
| 654 | skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
|
| 655 |
|
| 656 | You might want to keep a pointer to the associated skb for the next step
|
| 657 | and not free the skb. A driver not supporting hardware time stamping doesn't
|
| 658 | do that. A driver must never touch sk_buff::tstamp! It is used to store
|
| 659 | software generated time stamps by the network subsystem.
|
| 660 | - Driver should call skb_tx_timestamp() as close to passing sk_buff to hardware
|
| 661 | as possible. skb_tx_timestamp() provides a software time stamp if requested
|
| 662 | and hardware timestamping is not possible (SKBTX_IN_PROGRESS not set).
|
| 663 | - As soon as the driver has sent the packet and/or obtained a
|
| 664 | hardware time stamp for it, it passes the time stamp back by
|
| 665 | calling skb_tstamp_tx() with the original skb, the raw
|
| 666 | hardware time stamp. skb_tstamp_tx() clones the original skb and
|
| 667 | adds the timestamps, therefore the original skb has to be freed now.
|
| 668 | If obtaining the hardware time stamp somehow fails, then the driver
|
| 669 | should not fall back to software time stamping. The rationale is that
|
| 670 | this would occur at a later time in the processing pipeline than other
|
| 671 | software time stamping and therefore could lead to unexpected deltas
|
| 672 | between time stamps.
|
| 673 |
|
| 674 | 3.2 Special considerations for stacked PTP Hardware Clocks
|
| 675 | ----------------------------------------------------------
|
| 676 |
|
| 677 | There are situations when there may be more than one PHC (PTP Hardware Clock)
|
| 678 | in the data path of a packet. The kernel has no explicit mechanism to allow the
|
| 679 | user to select which PHC to use for timestamping Ethernet frames. Instead, the
|
| 680 | assumption is that the outermost PHC is always the most preferable, and that
|
| 681 | kernel drivers collaborate towards achieving that goal. Currently there are 3
|
| 682 | cases of stacked PHCs, detailed below:
|
| 683 |
|
| 684 | 3.2.1 DSA (Distributed Switch Architecture) switches
|
| 685 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 686 |
|
| 687 | These are Ethernet switches which have one of their ports connected to an
|
| 688 | (otherwise completely unaware) host Ethernet interface, and perform the role of
|
| 689 | a port multiplier with optional forwarding acceleration features. Each DSA
|
| 690 | switch port is visible to the user as a standalone (virtual) network interface,
|
| 691 | and its network I/O is performed, under the hood, indirectly through the host
|
| 692 | interface (redirecting to the host port on TX, and intercepting frames on RX).
|
| 693 |
|
| 694 | When a DSA switch is attached to a host port, PTP synchronization has to
|
| 695 | suffer, since the switch's variable queuing delay introduces a path delay
|
| 696 | jitter between the host port and its PTP partner. For this reason, some DSA
|
| 697 | switches include a timestamping clock of their own, and have the ability to
|
| 698 | perform network timestamping on their own MAC, such that path delays only
|
| 699 | measure wire and PHY propagation latencies. Timestamping DSA switches are
|
| 700 | supported in Linux and expose the same ABI as any other network interface (save
|
| 701 | for the fact that the DSA interfaces are in fact virtual in terms of network
|
| 702 | I/O, they do have their own PHC). It is typical, but not mandatory, for all
|
| 703 | interfaces of a DSA switch to share the same PHC.
|
| 704 |
|
| 705 | By design, PTP timestamping with a DSA switch does not need any special
|
| 706 | handling in the driver for the host port it is attached to. However, when the
|
| 707 | host port also supports PTP timestamping, DSA will take care of intercepting
|
| 708 | the ``.ndo_eth_ioctl`` calls towards the host port, and block attempts to enable
|
| 709 | hardware timestamping on it. This is because the SO_TIMESTAMPING API does not
|
| 710 | allow the delivery of multiple hardware timestamps for the same packet, so
|
| 711 | anybody else except for the DSA switch port must be prevented from doing so.
|
| 712 |
|
| 713 | In the generic layer, DSA provides the following infrastructure for PTP
|
| 714 | timestamping:
|
| 715 |
|
| 716 | - ``.port_txtstamp()``: a hook called prior to the transmission of
|
| 717 | packets with a hardware TX timestamping request from user space.
|
| 718 | This is required for two-step timestamping, since the hardware
|
| 719 | timestamp becomes available after the actual MAC transmission, so the
|
| 720 | driver must be prepared to correlate the timestamp with the original
|
| 721 | packet so that it can re-enqueue the packet back into the socket's
|
| 722 | error queue. To save the packet for when the timestamp becomes
|
| 723 | available, the driver can call ``skb_clone_sk`` , save the clone pointer
|
| 724 | in skb->cb and enqueue a tx skb queue. Typically, a switch will have a
|
| 725 | PTP TX timestamp register (or sometimes a FIFO) where the timestamp
|
| 726 | becomes available. In case of a FIFO, the hardware might store
|
| 727 | key-value pairs of PTP sequence ID/message type/domain number and the
|
| 728 | actual timestamp. To perform the correlation correctly between the
|
| 729 | packets in a queue waiting for timestamping and the actual timestamps,
|
| 730 | drivers can use a BPF classifier (``ptp_classify_raw``) to identify
|
| 731 | the PTP transport type, and ``ptp_parse_header`` to interpret the PTP
|
| 732 | header fields. There may be an IRQ that is raised upon this
|
| 733 | timestamp's availability, or the driver might have to poll after
|
| 734 | invoking ``dev_queue_xmit()`` towards the host interface.
|
| 735 | One-step TX timestamping do not require packet cloning, since there is
|
| 736 | no follow-up message required by the PTP protocol (because the
|
| 737 | TX timestamp is embedded into the packet by the MAC), and therefore
|
| 738 | user space does not expect the packet annotated with the TX timestamp
|
| 739 | to be re-enqueued into its socket's error queue.
|
| 740 |
|
| 741 | - ``.port_rxtstamp()``: On RX, the BPF classifier is run by DSA to
|
| 742 | identify PTP event messages (any other packets, including PTP general
|
| 743 | messages, are not timestamped). The original (and only) timestampable
|
| 744 | skb is provided to the driver, for it to annotate it with a timestamp,
|
| 745 | if that is immediately available, or defer to later. On reception,
|
| 746 | timestamps might either be available in-band (through metadata in the
|
| 747 | DSA header, or attached in other ways to the packet), or out-of-band
|
| 748 | (through another RX timestamping FIFO). Deferral on RX is typically
|
| 749 | necessary when retrieving the timestamp needs a sleepable context. In
|
| 750 | that case, it is the responsibility of the DSA driver to call
|
| 751 | ``netif_rx()`` on the freshly timestamped skb.
|
| 752 |
|
| 753 | 3.2.2 Ethernet PHYs
|
| 754 | ^^^^^^^^^^^^^^^^^^^
|
| 755 |
|
| 756 | These are devices that typically fulfill a Layer 1 role in the network stack,
|
| 757 | hence they do not have a representation in terms of a network interface as DSA
|
| 758 | switches do. However, PHYs may be able to detect and timestamp PTP packets, for
|
| 759 | performance reasons: timestamps taken as close as possible to the wire have the
|
| 760 | potential to yield a more stable and precise synchronization.
|
| 761 |
|
| 762 | A PHY driver that supports PTP timestamping must create a ``struct
|
| 763 | mii_timestamper`` and add a pointer to it in ``phydev->mii_ts``. The presence
|
| 764 | of this pointer will be checked by the networking stack.
|
| 765 |
|
| 766 | Since PHYs do not have network interface representations, the timestamping and
|
| 767 | ethtool ioctl operations for them need to be mediated by their respective MAC
|
| 768 | driver. Therefore, as opposed to DSA switches, modifications need to be done
|
| 769 | to each individual MAC driver for PHY timestamping support. This entails:
|
| 770 |
|
| 771 | - Checking, in ``.ndo_eth_ioctl``, whether ``phy_has_hwtstamp(netdev->phydev)``
|
| 772 | is true or not. If it is, then the MAC driver should not process this request
|
| 773 | but instead pass it on to the PHY using ``phy_mii_ioctl()``.
|
| 774 |
|
| 775 | - On RX, special intervention may or may not be needed, depending on the
|
| 776 | function used to deliver skb's up the network stack. In the case of plain
|
| 777 | ``netif_rx()`` and similar, MAC drivers must check whether
|
| 778 | ``skb_defer_rx_timestamp(skb)`` is necessary or not - and if it is, don't
|
| 779 | call ``netif_rx()`` at all. If ``CONFIG_NETWORK_PHY_TIMESTAMPING`` is
|
| 780 | enabled, and ``skb->dev->phydev->mii_ts`` exists, its ``.rxtstamp()`` hook
|
| 781 | will be called now, to determine, using logic very similar to DSA, whether
|
| 782 | deferral for RX timestamping is necessary. Again like DSA, it becomes the
|
| 783 | responsibility of the PHY driver to send the packet up the stack when the
|
| 784 | timestamp is available.
|
| 785 |
|
| 786 | For other skb receive functions, such as ``napi_gro_receive`` and
|
| 787 | ``netif_receive_skb``, the stack automatically checks whether
|
| 788 | ``skb_defer_rx_timestamp()`` is necessary, so this check is not needed inside
|
| 789 | the driver.
|
| 790 |
|
| 791 | - On TX, again, special intervention might or might not be needed. The
|
| 792 | function that calls the ``mii_ts->txtstamp()`` hook is named
|
| 793 | ``skb_clone_tx_timestamp()``. This function can either be called directly
|
| 794 | (case in which explicit MAC driver support is indeed needed), but the
|
| 795 | function also piggybacks from the ``skb_tx_timestamp()`` call, which many MAC
|
| 796 | drivers already perform for software timestamping purposes. Therefore, if a
|
| 797 | MAC supports software timestamping, it does not need to do anything further
|
| 798 | at this stage.
|
| 799 |
|
| 800 | 3.2.3 MII bus snooping devices
|
| 801 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 802 |
|
| 803 | These perform the same role as timestamping Ethernet PHYs, save for the fact
|
| 804 | that they are discrete devices and can therefore be used in conjunction with
|
| 805 | any PHY even if it doesn't support timestamping. In Linux, they are
|
| 806 | discoverable and attachable to a ``struct phy_device`` through Device Tree, and
|
| 807 | for the rest, they use the same mii_ts infrastructure as those. See
|
| 808 | Documentation/devicetree/bindings/ptp/timestamper.txt for more details.
|
| 809 |
|
| 810 | 3.2.4 Other caveats for MAC drivers
|
| 811 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 812 |
|
| 813 | The use of stacked PHCs may uncover MAC driver bugs which were impossible to
|
| 814 | trigger without them. One example has to do with this line of code, already
|
| 815 | presented earlier::
|
| 816 |
|
| 817 | skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS;
|
| 818 |
|
| 819 | Any TX timestamping logic, be it a plain MAC driver, a DSA switch driver, a PHY
|
| 820 | driver or a MII bus snooping device driver, should set this flag.
|
| 821 | But a MAC driver that is unaware of PHC stacking might get tripped up by
|
| 822 | somebody other than itself setting this flag, and deliver a duplicate
|
| 823 | timestamp.
|
| 824 | For example, a typical driver design for TX timestamping might be to split the
|
| 825 | transmission part into 2 portions:
|
| 826 |
|
| 827 | 1. "TX": checks whether PTP timestamping has been previously enabled through
|
| 828 | the ``.ndo_eth_ioctl`` ("``priv->hwtstamp_tx_enabled == true``") and the
|
| 829 | current skb requires a TX timestamp ("``skb_shinfo(skb)->tx_flags &
|
| 830 | SKBTX_HW_TSTAMP``"). If this is true, it sets the
|
| 831 | "``skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS``" flag. Note: as
|
| 832 | described above, in the case of a stacked PHC system, this condition should
|
| 833 | never trigger, as this MAC is certainly not the outermost PHC. But this is
|
| 834 | not where the typical issue is. Transmission proceeds with this packet.
|
| 835 |
|
| 836 | 2. "TX confirmation": Transmission has finished. The driver checks whether it
|
| 837 | is necessary to collect any TX timestamp for it. Here is where the typical
|
| 838 | issues are: the MAC driver takes a shortcut and only checks whether
|
| 839 | "``skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS``" was set. With a stacked
|
| 840 | PHC system, this is incorrect because this MAC driver is not the only entity
|
| 841 | in the TX data path who could have enabled SKBTX_IN_PROGRESS in the first
|
| 842 | place.
|
| 843 |
|
| 844 | The correct solution for this problem is for MAC drivers to have a compound
|
| 845 | check in their "TX confirmation" portion, not only for
|
| 846 | "``skb_shinfo(skb)->tx_flags & SKBTX_IN_PROGRESS``", but also for
|
| 847 | "``priv->hwtstamp_tx_enabled == true``". Because the rest of the system ensures
|
| 848 | that PTP timestamping is not enabled for anything other than the outermost PHC,
|
| 849 | this enhanced check will avoid delivering a duplicated TX timestamp to user
|
| 850 | space.
|