| 1 | |
| 2 |
|
| 3 | ======================
|
| 4 | USB4 and Thunderbolt
|
| 5 | ======================
|
| 6 | USB4 is the public specification based on Thunderbolt 3 protocol with
|
| 7 | some differences at the register level among other things. Connection
|
| 8 | manager is an entity running on the host router (host controller)
|
| 9 | responsible for enumerating routers and establishing tunnels. A
|
| 10 | connection manager can be implemented either in firmware or software.
|
| 11 | Typically PCs come with a firmware connection manager for Thunderbolt 3
|
| 12 | and early USB4 capable systems. Apple systems on the other hand use
|
| 13 | software connection manager and the later USB4 compliant devices follow
|
| 14 | the suit.
|
| 15 |
|
| 16 | The Linux Thunderbolt driver supports both and can detect at runtime which
|
| 17 | connection manager implementation is to be used. To be on the safe side the
|
| 18 | software connection manager in Linux also advertises security level
|
| 19 | ``user`` which means PCIe tunneling is disabled by default. The
|
| 20 | documentation below applies to both implementations with the exception that
|
| 21 | the software connection manager only supports ``user`` security level and
|
| 22 | is expected to be accompanied with an IOMMU based DMA protection.
|
| 23 |
|
| 24 | Security levels and how to use them
|
| 25 | -----------------------------------
|
| 26 | The interface presented here is not meant for end users. Instead there
|
| 27 | should be a userspace tool that handles all the low-level details, keeps
|
| 28 | a database of the authorized devices and prompts users for new connections.
|
| 29 |
|
| 30 | More details about the sysfs interface for Thunderbolt devices can be
|
| 31 | found in Documentation/ABI/testing/sysfs-bus-thunderbolt.
|
| 32 |
|
| 33 | Those users who just want to connect any device without any sort of
|
| 34 | manual work can add following line to
|
| 35 | ``/etc/udev/rules.d/99-local.rules``::
|
| 36 |
|
| 37 | ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{authorized}="1"
|
| 38 |
|
| 39 | This will authorize all devices automatically when they appear. However,
|
| 40 | keep in mind that this bypasses the security levels and makes the system
|
| 41 | vulnerable to DMA attacks.
|
| 42 |
|
| 43 | Starting with Intel Falcon Ridge Thunderbolt controller there are 4
|
| 44 | security levels available. Intel Titan Ridge added one more security level
|
| 45 | (usbonly). The reason for these is the fact that the connected devices can
|
| 46 | be DMA masters and thus read contents of the host memory without CPU and OS
|
| 47 | knowing about it. There are ways to prevent this by setting up an IOMMU but
|
| 48 | it is not always available for various reasons.
|
| 49 |
|
| 50 | Some USB4 systems have a BIOS setting to disable PCIe tunneling. This is
|
| 51 | treated as another security level (nopcie).
|
| 52 |
|
| 53 | The security levels are as follows:
|
| 54 |
|
| 55 | none
|
| 56 | All devices are automatically connected by the firmware. No user
|
| 57 | approval is needed. In BIOS settings this is typically called
|
| 58 | *Legacy mode*.
|
| 59 |
|
| 60 | user
|
| 61 | User is asked whether the device is allowed to be connected.
|
| 62 | Based on the device identification information available through
|
| 63 | ``/sys/bus/thunderbolt/devices``, the user then can make the decision.
|
| 64 | In BIOS settings this is typically called *Unique ID*.
|
| 65 |
|
| 66 | secure
|
| 67 | User is asked whether the device is allowed to be connected. In
|
| 68 | addition to UUID the device (if it supports secure connect) is sent
|
| 69 | a challenge that should match the expected one based on a random key
|
| 70 | written to the ``key`` sysfs attribute. In BIOS settings this is
|
| 71 | typically called *One time saved key*.
|
| 72 |
|
| 73 | dponly
|
| 74 | The firmware automatically creates tunnels for Display Port and
|
| 75 | USB. No PCIe tunneling is done. In BIOS settings this is
|
| 76 | typically called *Display Port Only*.
|
| 77 |
|
| 78 | usbonly
|
| 79 | The firmware automatically creates tunnels for the USB controller and
|
| 80 | Display Port in a dock. All PCIe links downstream of the dock are
|
| 81 | removed.
|
| 82 |
|
| 83 | nopcie
|
| 84 | PCIe tunneling is disabled/forbidden from the BIOS. Available in some
|
| 85 | USB4 systems.
|
| 86 |
|
| 87 | The current security level can be read from
|
| 88 | ``/sys/bus/thunderbolt/devices/domainX/security`` where ``domainX`` is
|
| 89 | the Thunderbolt domain the host controller manages. There is typically
|
| 90 | one domain per Thunderbolt host controller.
|
| 91 |
|
| 92 | If the security level reads as ``user`` or ``secure`` the connected
|
| 93 | device must be authorized by the user before PCIe tunnels are created
|
| 94 | (e.g the PCIe device appears).
|
| 95 |
|
| 96 | Each Thunderbolt device plugged in will appear in sysfs under
|
| 97 | ``/sys/bus/thunderbolt/devices``. The device directory carries
|
| 98 | information that can be used to identify the particular device,
|
| 99 | including its name and UUID.
|
| 100 |
|
| 101 | Authorizing devices when security level is ``user`` or ``secure``
|
| 102 | -----------------------------------------------------------------
|
| 103 | When a device is plugged in it will appear in sysfs as follows::
|
| 104 |
|
| 105 | /sys/bus/thunderbolt/devices/0-1/authorized - 0
|
| 106 | /sys/bus/thunderbolt/devices/0-1/device - 0x8004
|
| 107 | /sys/bus/thunderbolt/devices/0-1/device_name - Thunderbolt to FireWire Adapter
|
| 108 | /sys/bus/thunderbolt/devices/0-1/vendor - 0x1
|
| 109 | /sys/bus/thunderbolt/devices/0-1/vendor_name - Apple, Inc.
|
| 110 | /sys/bus/thunderbolt/devices/0-1/unique_id - e0376f00-0300-0100-ffff-ffffffffffff
|
| 111 |
|
| 112 | The ``authorized`` attribute reads 0 which means no PCIe tunnels are
|
| 113 | created yet. The user can authorize the device by simply entering::
|
| 114 |
|
| 115 | # echo 1 > /sys/bus/thunderbolt/devices/0-1/authorized
|
| 116 |
|
| 117 | This will create the PCIe tunnels and the device is now connected.
|
| 118 |
|
| 119 | If the device supports secure connect, and the domain security level is
|
| 120 | set to ``secure``, it has an additional attribute ``key`` which can hold
|
| 121 | a random 32-byte value used for authorization and challenging the device in
|
| 122 | future connects::
|
| 123 |
|
| 124 | /sys/bus/thunderbolt/devices/0-3/authorized - 0
|
| 125 | /sys/bus/thunderbolt/devices/0-3/device - 0x305
|
| 126 | /sys/bus/thunderbolt/devices/0-3/device_name - AKiTiO Thunder3 PCIe Box
|
| 127 | /sys/bus/thunderbolt/devices/0-3/key -
|
| 128 | /sys/bus/thunderbolt/devices/0-3/vendor - 0x41
|
| 129 | /sys/bus/thunderbolt/devices/0-3/vendor_name - inXtron
|
| 130 | /sys/bus/thunderbolt/devices/0-3/unique_id - dc010000-0000-8508-a22d-32ca6421cb16
|
| 131 |
|
| 132 | Notice the key is empty by default.
|
| 133 |
|
| 134 | If the user does not want to use secure connect they can just ``echo 1``
|
| 135 | to the ``authorized`` attribute and the PCIe tunnels will be created in
|
| 136 | the same way as in the ``user`` security level.
|
| 137 |
|
| 138 | If the user wants to use secure connect, the first time the device is
|
| 139 | plugged a key needs to be created and sent to the device::
|
| 140 |
|
| 141 | # key=$(openssl rand -hex 32)
|
| 142 | # echo $key > /sys/bus/thunderbolt/devices/0-3/key
|
| 143 | # echo 1 > /sys/bus/thunderbolt/devices/0-3/authorized
|
| 144 |
|
| 145 | Now the device is connected (PCIe tunnels are created) and in addition
|
| 146 | the key is stored on the device NVM.
|
| 147 |
|
| 148 | Next time the device is plugged in the user can verify (challenge) the
|
| 149 | device using the same key::
|
| 150 |
|
| 151 | # echo $key > /sys/bus/thunderbolt/devices/0-3/key
|
| 152 | # echo 2 > /sys/bus/thunderbolt/devices/0-3/authorized
|
| 153 |
|
| 154 | If the challenge the device returns back matches the one we expect based
|
| 155 | on the key, the device is connected and the PCIe tunnels are created.
|
| 156 | However, if the challenge fails no tunnels are created and error is
|
| 157 | returned to the user.
|
| 158 |
|
| 159 | If the user still wants to connect the device they can either approve
|
| 160 | the device without a key or write a new key and write 1 to the
|
| 161 | ``authorized`` file to get the new key stored on the device NVM.
|
| 162 |
|
| 163 | De-authorizing devices
|
| 164 | ----------------------
|
| 165 | It is possible to de-authorize devices by writing ``0`` to their
|
| 166 | ``authorized`` attribute. This requires support from the connection
|
| 167 | manager implementation and can be checked by reading domain
|
| 168 | ``deauthorization`` attribute. If it reads ``1`` then the feature is
|
| 169 | supported.
|
| 170 |
|
| 171 | When a device is de-authorized the PCIe tunnel from the parent device
|
| 172 | PCIe downstream (or root) port to the device PCIe upstream port is torn
|
| 173 | down. This is essentially the same thing as PCIe hot-remove and the PCIe
|
| 174 | toplogy in question will not be accessible anymore until the device is
|
| 175 | authorized again. If there is storage such as NVMe or similar involved,
|
| 176 | there is a risk for data loss if the filesystem on that storage is not
|
| 177 | properly shut down. You have been warned!
|
| 178 |
|
| 179 | DMA protection utilizing IOMMU
|
| 180 | ------------------------------
|
| 181 | Recent systems from 2018 and forward with Thunderbolt ports may natively
|
| 182 | support IOMMU. This means that Thunderbolt security is handled by an IOMMU
|
| 183 | so connected devices cannot access memory regions outside of what is
|
| 184 | allocated for them by drivers. When Linux is running on such system it
|
| 185 | automatically enables IOMMU if not enabled by the user already. These
|
| 186 | systems can be identified by reading ``1`` from
|
| 187 | ``/sys/bus/thunderbolt/devices/domainX/iommu_dma_protection`` attribute.
|
| 188 |
|
| 189 | The driver does not do anything special in this case but because DMA
|
| 190 | protection is handled by the IOMMU, security levels (if set) are
|
| 191 | redundant. For this reason some systems ship with security level set to
|
| 192 | ``none``. Other systems have security level set to ``user`` in order to
|
| 193 | support downgrade to older OS, so users who want to automatically
|
| 194 | authorize devices when IOMMU DMA protection is enabled can use the
|
| 195 | following ``udev`` rule::
|
| 196 |
|
| 197 | ACTION=="add", SUBSYSTEM=="thunderbolt", ATTRS{iommu_dma_protection}=="1", ATTR{authorized}=="0", ATTR{authorized}="1"
|
| 198 |
|
| 199 | Upgrading NVM on Thunderbolt device, host or retimer
|
| 200 | ----------------------------------------------------
|
| 201 | Since most of the functionality is handled in firmware running on a
|
| 202 | host controller or a device, it is important that the firmware can be
|
| 203 | upgraded to the latest where possible bugs in it have been fixed.
|
| 204 | Typically OEMs provide this firmware from their support site.
|
| 205 |
|
| 206 | Currently, recommended method of updating firmware is through "fwupd" tool.
|
| 207 | It uses LVFS (Linux Vendor Firmware Service) portal by default to get the
|
| 208 | latest firmware from hardware vendors and updates connected devices if found
|
| 209 | compatible. For details refer to: https://github.com/fwupd/fwupd.
|
| 210 |
|
| 211 | Before you upgrade firmware on a device, host or retimer, please make
|
| 212 | sure it is a suitable upgrade. Failing to do that may render the device
|
| 213 | in a state where it cannot be used properly anymore without special
|
| 214 | tools!
|
| 215 |
|
| 216 | Host NVM upgrade on Apple Macs is not supported.
|
| 217 |
|
| 218 | Fwupd is installed by default. If you don't have it on your system, simply
|
| 219 | use your distro package manager to get it.
|
| 220 |
|
| 221 | To see possible updates through fwupd, you need to plug in a Thunderbolt
|
| 222 | device so that the host controller appears. It does not matter which
|
| 223 | device is connected (unless you are upgrading NVM on a device - then you
|
| 224 | need to connect that particular device).
|
| 225 |
|
| 226 | Note an OEM-specific method to power the controller up ("force power") may
|
| 227 | be available for your system in which case there is no need to plug in a
|
| 228 | Thunderbolt device.
|
| 229 |
|
| 230 | Updating firmware using fwupd is straightforward - refer to official
|
| 231 | readme on fwupd github.
|
| 232 |
|
| 233 | If firmware image is written successfully, the device shortly disappears.
|
| 234 | Once it comes back, the driver notices it and initiates a full power
|
| 235 | cycle. After a while device appears again and this time it should be
|
| 236 | fully functional.
|
| 237 |
|
| 238 | Device of interest should display new version under "Current version"
|
| 239 | and "Update State: Success" in fwupd's interface.
|
| 240 |
|
| 241 | Upgrading firmware manually
|
| 242 | ---------------------------------------------------------------
|
| 243 | If possible, use fwupd to updated the firmware. However, if your device OEM
|
| 244 | has not uploaded the firmware to LVFS, but it is available for download
|
| 245 | from their side, you can use method below to directly upgrade the
|
| 246 | firmware.
|
| 247 |
|
| 248 | Manual firmware update can be done with 'dd' tool. To update firmware
|
| 249 | using this method, you need to write it to the non-active parts of NVM
|
| 250 | of the host or device. Example on how to update Intel NUC6i7KYK
|
| 251 | (Skull Canyon) Thunderbolt controller NVM::
|
| 252 |
|
| 253 | # dd if=KYK_TBT_FW_0018.bin of=/sys/bus/thunderbolt/devices/0-0/nvm_non_active0/nvmem
|
| 254 |
|
| 255 | Once the operation completes we can trigger NVM authentication and
|
| 256 | upgrade process as follows::
|
| 257 |
|
| 258 | # echo 1 > /sys/bus/thunderbolt/devices/0-0/nvm_authenticate
|
| 259 |
|
| 260 | If no errors are returned, device should behave as described in previous
|
| 261 | section.
|
| 262 |
|
| 263 | We can verify that the new NVM firmware is active by running the following
|
| 264 | commands::
|
| 265 |
|
| 266 | # cat /sys/bus/thunderbolt/devices/0-0/nvm_authenticate
|
| 267 | 0x0
|
| 268 | # cat /sys/bus/thunderbolt/devices/0-0/nvm_version
|
| 269 | 18.0
|
| 270 |
|
| 271 | If ``nvm_authenticate`` contains anything other than 0x0 it is the error
|
| 272 | code from the last authentication cycle, which means the authentication
|
| 273 | of the NVM image failed.
|
| 274 |
|
| 275 | Note names of the NVMem devices ``nvm_activeN`` and ``nvm_non_activeN``
|
| 276 | depend on the order they are registered in the NVMem subsystem. N in
|
| 277 | the name is the identifier added by the NVMem subsystem.
|
| 278 |
|
| 279 | Upgrading on-board retimer NVM when there is no cable connected
|
| 280 | ---------------------------------------------------------------
|
| 281 | If the platform supports, it may be possible to upgrade the retimer NVM
|
| 282 | firmware even when there is nothing connected to the USB4
|
| 283 | ports. When this is the case the ``usb4_portX`` devices have two special
|
| 284 | attributes: ``offline`` and ``rescan``. The way to upgrade the firmware
|
| 285 | is to first put the USB4 port into offline mode::
|
| 286 |
|
| 287 | # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/offline
|
| 288 |
|
| 289 | This step makes sure the port does not respond to any hotplug events,
|
| 290 | and also ensures the retimers are powered on. The next step is to scan
|
| 291 | for the retimers::
|
| 292 |
|
| 293 | # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/rescan
|
| 294 |
|
| 295 | This enumerates and adds the on-board retimers. Now retimer NVM can be
|
| 296 | upgraded in the same way than with cable connected (see previous
|
| 297 | section). However, the retimer is not disconnected as we are in offline
|
| 298 | mode, so after writing ``1`` to ``nvm_authenticate`` one should wait for
|
| 299 | 5 or more seconds before running rescan again::
|
| 300 |
|
| 301 | # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/rescan
|
| 302 |
|
| 303 | This point if everything went fine, the port can be put back to
|
| 304 | functional state again::
|
| 305 |
|
| 306 | # echo 0 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/offline
|
| 307 |
|
| 308 | Upgrading NVM when host controller is in safe mode
|
| 309 | --------------------------------------------------
|
| 310 | If the existing NVM is not properly authenticated (or is missing) the
|
| 311 | host controller goes into safe mode which means that the only available
|
| 312 | functionality is flashing a new NVM image. When in this mode, reading
|
| 313 | ``nvm_version`` fails with ``ENODATA`` and the device identification
|
| 314 | information is missing.
|
| 315 |
|
| 316 | To recover from this mode, one needs to flash a valid NVM image to the
|
| 317 | host controller in the same way it is done in the previous chapter.
|
| 318 |
|
| 319 | Tunneling events
|
| 320 | ----------------
|
| 321 | The driver sends ``KOBJ_CHANGE`` events to userspace when there is a
|
| 322 | tunneling change in the ``thunderbolt_domain``. The notification carries
|
| 323 | following environment variables::
|
| 324 |
|
| 325 | TUNNEL_EVENT=<EVENT>
|
| 326 | TUNNEL_DETAILS=0:12 <-> 1:20 (USB3)
|
| 327 |
|
| 328 | Possible values for ``<EVENT>`` are:
|
| 329 |
|
| 330 | activated
|
| 331 | The tunnel was activated (created).
|
| 332 |
|
| 333 | changed
|
| 334 | There is a change in this tunnel. For example bandwidth allocation was
|
| 335 | changed.
|
| 336 |
|
| 337 | deactivated
|
| 338 | The tunnel was torn down.
|
| 339 |
|
| 340 | low bandwidth
|
| 341 | The tunnel is not getting optimal bandwidth.
|
| 342 |
|
| 343 | insufficient bandwidth
|
| 344 | There is not enough bandwidth for the current tunnel requirements.
|
| 345 |
|
| 346 | The ``TUNNEL_DETAILS`` is only provided if the tunnel is known. For
|
| 347 | example, in case of Firmware Connection Manager this is missing or does
|
| 348 | not provide full tunnel information. In case of Software Connection Manager
|
| 349 | this includes full tunnel details. The format currently matches what the
|
| 350 | driver uses when logging. This may change over time.
|
| 351 |
|
| 352 | Networking over Thunderbolt cable
|
| 353 | ---------------------------------
|
| 354 | Thunderbolt technology allows software communication between two hosts
|
| 355 | connected by a Thunderbolt cable.
|
| 356 |
|
| 357 | It is possible to tunnel any kind of traffic over a Thunderbolt link but
|
| 358 | currently we only support Apple ThunderboltIP protocol.
|
| 359 |
|
| 360 | If the other host is running Windows or macOS, the only thing you need to
|
| 361 | do is to connect a Thunderbolt cable between the two hosts; the
|
| 362 | ``thunderbolt-net`` driver is loaded automatically. If the other host is
|
| 363 | also Linux you should load ``thunderbolt-net`` manually on one host (it
|
| 364 | does not matter which one)::
|
| 365 |
|
| 366 | # modprobe thunderbolt-net
|
| 367 |
|
| 368 | This triggers module load on the other host automatically. If the driver
|
| 369 | is built-in to the kernel image, there is no need to do anything.
|
| 370 |
|
| 371 | The driver will create one virtual ethernet interface per Thunderbolt
|
| 372 | port which are named like ``thunderbolt0`` and so on. From this point
|
| 373 | you can either use standard userspace tools like ``ip`` to
|
| 374 | configure the interface or let your GUI handle it automatically.
|
| 375 |
|
| 376 | Streaming data directly over Thunderbolt cable
|
| 377 | ----------------------------------------------
|
| 378 | In addition to Thunderbolt networking (aka. USB4NET) Linux supports
|
| 379 | streaming data directly over a cable as well (aka. USB4STREAM). This is
|
| 380 | possible through ``thunderbolt-stream`` driver.
|
| 381 |
|
| 382 | Similarly to ``thunderbolt-net`` you load the driver first on one end::
|
| 383 |
|
| 384 | host1 # modprobe thunderbolt-stream
|
| 385 |
|
| 386 | Then you configure it via ``ConfigFS``::
|
| 387 |
|
| 388 | host1 # cd /sys/kernel/config/thunderbolt/stream
|
| 389 | host1 # mkdir -p 0-1.0/data
|
| 390 | host1 # cd 0-1.0
|
| 391 | host1 # echo -1 > data/in_hopid
|
| 392 | host1 # echo -1 > data/out_hopid
|
| 393 |
|
| 394 | This information is automatically announced to the other side via
|
| 395 | XDomain properties so if you have cable connected the other side knows
|
| 396 | that there is a stream named ``data`` available and can configure it for
|
| 397 | you automatically::
|
| 398 |
|
| 399 | host2 # cd /sys/kernel/config/thunderbolt/stream
|
| 400 | host2 # mkdir -p 0-3.0/data
|
| 401 |
|
| 402 | Here we used auto-configuration but you can configure it manually too.
|
| 403 | In that case you need to fill ``in_hopid`` and ``out_hopid`` accordingly.
|
| 404 | If you set them to ``-1`` the next available HopID is used which is
|
| 405 | typically what we want.
|
| 406 |
|
| 407 | Once they are configured you can use ``/dev/tbstreamX`` on both sides to
|
| 408 | transfer data::
|
| 409 |
|
| 410 | host2 # cat /dev/tbstream0
|
| 411 | host1 # dmesg > /dev/tbstream0
|
| 412 |
|
| 413 | Once you are done with the stream you can remove them::
|
| 414 |
|
| 415 | host2 # cd /sys/kernel/config/thunderbolt/stream
|
| 416 | host2 # rmdir -p 0-1.0/data
|
| 417 | host1 # cd /sys/kernel/config/thunderbolt/stream
|
| 418 | host1 # rmdir -p 0-3.0/data
|
| 419 |
|
| 420 | Since streams are essentially files you can use any existing application
|
| 421 | that supports ``read(2)`` and ``write(2)`` in some form.
|
| 422 |
|
| 423 | It is possible to have more than one stream and you can have both stream
|
| 424 | and ``thunderbolt-net`` in use simultaneously. For example we can create
|
| 425 | two streams with name ``control`` and ``data`` like this::
|
| 426 |
|
| 427 | host1 # cd /sys/kernel/config/thunderbolt/stream
|
| 428 | host1 # mkdir 0-1.0
|
| 429 | host1 # cd 0-1.0
|
| 430 | host1 # mkdir control
|
| 431 | host1 # mkdir data
|
| 432 |
|
| 433 | Then you have ``/dev/tbstream0`` for ``control`` and ``/dev/tbstream1``
|
| 434 | for ``data``. Before you can use them you need to configure them as
|
| 435 | shown above for the one stream case.
|
| 436 |
|
| 437 | Forcing power
|
| 438 | -------------
|
| 439 | Many OEMs include a method that can be used to force the power of a
|
| 440 | Thunderbolt controller to an "On" state even if nothing is connected.
|
| 441 | If supported by your machine this will be exposed by the WMI bus with
|
| 442 | a sysfs attribute called "force_power", see
|
| 443 | Documentation/ABI/testing/sysfs-platform-intel-wmi-thunderbolt for details.
|
| 444 |
|
| 445 | Note: it's currently not possible to query the force power state of a platform.
|