| 1 | |
| 2 |
|
| 3 | ===================================================
|
| 4 | FSP (Foundation Security Processor) and Secure Boot
|
| 5 | ===================================================
|
| 6 | This document describes the role of the FSP in the GPU boot sequence on
|
| 7 | Hopper and Blackwell GPUs, and how it differs from the earlier Ampere boot
|
| 8 | flow. It also provides a brief overview of the PRC (Product Reconfiguration
|
| 9 | Control) protocol used to query device configuration through FSP. As with
|
| 10 | other documents in this directory, the information is subject to change and
|
| 11 | is intended to help developers understand the corresponding kernel code.
|
| 12 |
|
| 13 | What is FSP?
|
| 14 | ============
|
| 15 | The Foundation Security Processor (FSP) is the GPU's Internal Root of Trust
|
| 16 | (IROT). It is a dedicated security processor that boots from immutable ROM
|
| 17 | (Boot ROM) inside the GPU and is responsible for establishing the Chain of
|
| 18 | Trust before any other firmware is allowed to run.
|
| 19 |
|
| 20 | FSP runs independently of the host CPU and starts executing as soon as the
|
| 21 | GPU is powered on. By the time the nova-core driver is loaded, FSP has
|
| 22 | already completed its own secure boot and is ready to accept commands from
|
| 23 | the driver.
|
| 24 |
|
| 25 | Simplified boot flow (Hopper/Blackwell)
|
| 26 | =======================================
|
| 27 | Starting with Hopper, the boot flow is significantly simplified compared to
|
| 28 | earlier GPU generations like Ampere.
|
| 29 |
|
| 30 | On an **Ampere** GPU, the boot verification chain involves multiple Falcon
|
| 31 | engines and multiple ucode stages (see falcon.rst for details)::
|
| 32 |
|
| 33 | Hardware BROM (SEC2)
|
| 34 | -> HS Booter (SEC2)
|
| 35 | -> LS GSP-RM (GSP)
|
| 36 |
|
| 37 | The driver must extract ucode from VBIOS, manage SEC2 and GSP, and
|
| 38 | orchestrate the Booter to load GSP-RM. This involves FWSEC-FRTS, devinit,
|
| 39 | and the Booter stages.
|
| 40 |
|
| 41 | On **Hopper/Blackwell** GPUs, FSP replaces this multi-stage process with a
|
| 42 | single message-driven interface::
|
| 43 |
|
| 44 | FSP (hardware root of trust, boots from ROM)
|
| 45 | -> FMC (Falcon Microcontroller, verified by FSP)
|
| 46 | -> GSP-RM (verified and loaded by FMC)
|
| 47 |
|
| 48 | The driver only needs to:
|
| 49 |
|
| 50 | 1. Wait for FSP to complete its own secure boot (polling a scratch register).
|
| 51 | 2. Send a Chain of Trust (COT) message to FSP with the FMC firmware location,
|
| 52 | cryptographic signatures, and GSP boot parameters.
|
| 53 | 3. FSP authenticates the FMC firmware and boots it, FMC in turn loads GSP-RM.
|
| 54 |
|
| 55 | There is no SEC2 involvement, no Booter ucode, and no FWSEC-FRTS stage. The
|
| 56 | entire secure boot is driven by a single FSP message exchange.
|
| 57 |
|
| 58 | Chain of Trust (COT) protocol
|
| 59 | =============================
|
| 60 | The Chain of Trust establishes a cryptographically enforced boot sequence,
|
| 61 | ensuring the GPU reaches a known, trusted state.
|
| 62 |
|
| 63 | The driver communicates with FSP using a message queue (Falcon MSGQ
|
| 64 | interface). Each message consists of an MCTP (Management Component Transport
|
| 65 | Protocol) transport header and an NVDM (NVIDIA Vendor Defined Message) header,
|
| 66 | followed by a protocol-specific payload.
|
| 67 |
|
| 68 | For Chain of Trust, the payload includes:
|
| 69 |
|
| 70 | - The system memory address of the FMC firmware image.
|
| 71 | - Cryptographic material: a SHA-384 hash, RSA-3K public key, and RSA-3K
|
| 72 | signature extracted from the FMC ELF firmware.
|
| 73 | - FRTS (Firmware Runtime Services) region information (vidmem offset and size).
|
| 74 | - The system memory address of the GSP boot arguments structure.
|
| 75 |
|
| 76 | FSP verifies the signature against the provided public key and hash, and if
|
| 77 | verification succeeds, boots the FMC. The FMC then authenticates and launches
|
| 78 | GSP-RM.
|
| 79 |
|
| 80 | The message flow is::
|
| 81 |
|
| 82 | nova-core FSP
|
| 83 | | |
|
| 84 | | 1. Poll scratch register |
|
| 85 | | (wait for FSP boot complete) |
|
| 86 | | |
|
| 87 | | 2. COT message ------------> |
|
| 88 | | (FMC addr, signatures, |
|
| 89 | | boot params) |
|
| 90 | | |
|
| 91 | | |--- Verify FMC signature
|
| 92 | | |--- Boot FMC
|
| 93 | | |--- FMC loads GSP-RM
|
| 94 | | |
|
| 95 | | 3. COT response <------------ |
|
| 96 | | (success/error) |
|
| 97 | | |
|
| 98 |
|
| 99 | FSP message format
|
| 100 | ==================
|
| 101 | All FSP messages share a common header format consisting of two 32-bit words:
|
| 102 |
|
| 103 | **MCTP header** (Management Component Transport Protocol):
|
| 104 |
|
| 105 | - Bit 31: SOM (Start of Message)
|
| 106 | - Bit 30: EOM (End of Message)
|
| 107 | - Bits 29:28: Packet sequence number
|
| 108 | - Bits 23:16: Source Endpoint ID
|
| 109 |
|
| 110 | **NVDM header** (NVIDIA Vendor Defined Message):
|
| 111 |
|
| 112 | - Bits 6:0: MCTP message type (0x7e = vendor-defined PCI)
|
| 113 | - Bits 23:8: PCI vendor ID (0x10de = NVIDIA)
|
| 114 | - Bits 31:24: NVDM type (0x14 = COT, 0x13 = PRC, 0x15 = FSP response)
|
| 115 |
|
| 116 | PRC (Product Reconfiguration Control) protocol
|
| 117 | ===============================================
|
| 118 | PRC is an API system exposed through FSP's Management Partition that allows
|
| 119 | querying and modifying device configuration without firmware updates.
|
| 120 |
|
| 121 | Configuration parameters are called "knobs". Each knob has a unique object
|
| 122 | ID and controls a specific device behavior. Examples include vGPU mode, ECC
|
| 123 | enable, confidential computing mode, and NVLINK configuration.
|
| 124 |
|
| 125 | Each knob has two values:
|
| 126 |
|
| 127 | - **Active**: the currently effective value for this boot cycle.
|
| 128 | - **Persistent**: the value stored in InfoROM, applied on subsequent boots.
|
| 129 |
|
| 130 | The nova-core driver uses PRC to read the vGPU mode knob (object ID 0x29)
|
| 131 | during early boot, before firmware loading, to determine whether the GPU
|
| 132 | should operate in vGPU mode.
|
| 133 |
|
| 134 | The PRC message format follows the same MCTP/NVDM header structure as COT,
|
| 135 | with NVDM type 0x13. The payload contains:
|
| 136 |
|
| 137 | - A sub-command (e.g., 0x0c for read).
|
| 138 | - Flags indicating which value to read (bit 0 = persistent, bit 1 = active).
|
| 139 | - The knob object ID.
|
| 140 |
|
| 141 | The response includes the common FSP response header (with error status)
|
| 142 | followed by the knob's 16-bit state value.
|