| 1 | |
| 2 |
|
| 3 | ==========
|
| 4 | VBIOS
|
| 5 | ==========
|
| 6 | This document describes the layout of the VBIOS image which is a series of concatenated
|
| 7 | images in the ROM of the GPU. The VBIOS is mirrored onto the BAR 0 space and is read
|
| 8 | by both Boot ROM firmware (also known as IFR or init-from-rom firmware) on the GPU to
|
| 9 | bootstrap various microcontrollers (PMU, SEC, GSP) with critical initialization before
|
| 10 | the driver loads, as well as by the nova-core driver in the kernel to boot the GSP.
|
| 11 |
|
| 12 | The format of the images in the ROM follow the "BIOS Specification" part of the
|
| 13 | PCI specification, with Nvidia-specific extensions. The ROM images of type FwSec
|
| 14 | are the ones that contain Falcon ucode and what we are mainly looking for.
|
| 15 |
|
| 16 | As an example, the following are the different image types that can be found in the
|
| 17 | VBIOS of an Ampere GA102 GPU which is supported by the nova-core driver.
|
| 18 |
|
| 19 | - PciAt Image (Type 0x00) - This is the standard PCI BIOS image, whose name
|
| 20 | likely comes from the "IBM PC/AT" architecture.
|
| 21 |
|
| 22 | - EFI Image (Type 0x03) - This is the EFI BIOS image. It contains the UEFI GOP
|
| 23 | driver that is used to display UEFI graphics output.
|
| 24 |
|
| 25 | - First FwSec Image (Type 0xE0) - The first FwSec image (Secure Firmware)
|
| 26 |
|
| 27 | - Second FwSec Image (Type 0xE0) - The second FwSec image (Secure Firmware)
|
| 28 | contains various microcodes (also known as an applications) that do a range
|
| 29 | of different functions. The FWSEC ucode is run in heavy-secure mode and
|
| 30 | typically runs directly on the GSP (it could be running on a different
|
| 31 | designated processor in future generations but as of Ampere, it is the GSP).
|
| 32 | This firmware then loads other firmware ucodes onto the PMU and SEC2
|
| 33 | microcontrollers for gfw initialization after GPU reset and before the driver
|
| 34 | loads (see devinit.rst). The DEVINIT ucode is itself another ucode that is
|
| 35 | stored in this ROM partition.
|
| 36 |
|
| 37 | Once located, the Falcon ucodes have "Application Interfaces" in their data
|
| 38 | memory (DMEM). For FWSEC, the application interface we use for FWSEC is the
|
| 39 | "DMEM mapper" interface which is configured to run the "FRTS" command. This
|
| 40 | command carves out the WPR2 (Write-Protected Region) in VRAM. It then places
|
| 41 | important power-management data, called 'FRTS', into this region. The WPR2
|
| 42 | region is only accessible to heavy-secure ucode.
|
| 43 |
|
| 44 | .. note::
|
| 45 | It is not clear why FwSec has 2 different partitions in the ROM, but they both
|
| 46 | are of type 0xE0 and can be identified as such. This could be subject to change
|
| 47 | in future generations.
|
| 48 |
|
| 49 | IFR Header
|
| 50 | ----------
|
| 51 | On Kepler and later GPUs, the ROM begins with an Init-from-ROM (IFR) header
|
| 52 | rather than a standard PCI ROM signature (0xAA55). The driver must parse the
|
| 53 | IFR header to find where the PCI ROM images actually start.
|
| 54 |
|
| 55 | Init-from-ROM (IFR) is a special GPU feature used for power management
|
| 56 | on some Nvidia GPUs. It references data in the VBIOS for its operation,
|
| 57 | but for drivers the important piece is a header that precedes the
|
| 58 | VBIOS PCI Expansion ROM.
|
| 59 |
|
| 60 | Most such GPUs do not need to parse the IFR header in order to find the
|
| 61 | VBIOS, but the Nvidia GA100 is the exception. GA100 lacks a display engine,
|
| 62 | so the PRAMIN method (which reads the VBIOS from VRAM via display hardware)
|
| 63 | is unavailable, forcing the driver to read the ROM directly via PROM.
|
| 64 | On other similar GPUs, either PRAMIN succeeds before PROM is tried, or the
|
| 65 | IFR hardware has already applied the ROM offset so that PROM reads
|
| 66 | transparently skip the IFR header.
|
| 67 |
|
| 68 | The driver should first check for the standard 0xAA55 signature at offset 0.
|
| 69 | If found, there is no IFR header and the PCI ROM images start at
|
| 70 | offset 0. If not found, check for the IFR signature and parse the header to
|
| 71 | determine the PCI ROM image offset.
|
| 72 |
|
| 73 | Fixed Header Format
|
| 74 | ~~~~~~~~~~~~~~~~~~~
|
| 75 |
|
| 76 | The IFR header begins with four 32-bit words at fixed offsets::
|
| 77 |
|
| 78 | Offset Name Fields
|
| 79 | ------ ------- ------
|
| 80 | 0x00 FIXED0 bits 31:0 - Signature (must be 0x4947564E, ASCII "NVGI")
|
| 81 | 0x04 FIXED1 bit 31 - Reserved
|
| 82 | bits 30:16 - FIXED_DATA_SIZE Fixed data size (offset to extended section)
|
| 83 | bits 15:8 - VERSIONSW Software version
|
| 84 | bits 7:0 - Reserved
|
| 85 | 0x08 FIXED2 bit 31 - Reserved
|
| 86 | bits 30:20 - Reserved (zero)
|
| 87 | bits 19:0 - TOTAL_DATA_SIZE Total data size
|
| 88 |
|
| 89 | Finding the PCI ROM Image Offset
|
| 90 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 91 |
|
| 92 | The method to find this offset depends on `VERSIONSW`.
|
| 93 |
|
| 94 | - **Version 1 and 2**: Read `FIXED_DATA_SIZE` from `FIXED1` to get the extended
|
| 95 | section offset. The PCI ROM image is the 32-bit value at `FIXED_DATA_SIZE + 4`.
|
| 96 |
|
| 97 | - **Version 3**: Read `TOTAL_DATA_SIZE` from `FIXED2`. The 32-bit value at that
|
| 98 | offset is a flash status offset. Add 4096 to get the ROM directory offset,
|
| 99 | `ROM_DIRECTORY_OFFSET`. The ROM directory must have signature 0x44524652
|
| 100 | (ASCII "RFRD"). The PCI ROM image offset is the 32-bit value at
|
| 101 | `ROM_DIRECTORY_OFFSET + 8`.
|
| 102 |
|
| 103 | The PCI ROM image offset must be 4-byte aligned. All offsets are relative to the
|
| 104 | start of ROM (BAR0 + 0x300000).
|
| 105 |
|
| 106 | VBIOS ROM Layout
|
| 107 | ----------------
|
| 108 | The VBIOS (PCI Expansion ROM) is a series of concatenated images laid out as
|
| 109 | follows. On GPUs with an IFR header, this layout begins at the image offset
|
| 110 | determined by parsing the IFR header. On older GPUs, it begins at offset 0::
|
| 111 |
|
| 112 | +----------------------------------------------------------------------------+
|
| 113 | | VBIOS (Starting at ROM_OFFSET: 0x300000 + IFR image offset) |
|
| 114 | +----------------------------------------------------------------------------+
|
| 115 | | +-----------------------------------------------+ |
|
| 116 | | | PciAt Image (Type 0x00) | |
|
| 117 | | +-----------------------------------------------+ |
|
| 118 | | | +-------------------+ | |
|
| 119 | | | | ROM Header | | |
|
| 120 | | | | (Signature 0xAA55)| | |
|
| 121 | | | +-------------------+ | |
|
| 122 | | | | rom header's pci_data_struct_offset | |
|
| 123 | | | | points to the PCIR structure | |
|
| 124 | | | V | |
|
| 125 | | | +-------------------+ | |
|
| 126 | | | | PCIR Structure | | |
|
| 127 | | | | (Signature "PCIR")| | |
|
| 128 | | | | last_image: 0x80 | | |
|
| 129 | | | | image_len: size | | |
|
| 130 | | | | in 512-byte units | | |
|
| 131 | | | +-------------------+ | |
|
| 132 | | | | | |
|
| 133 | | | | NPDE immediately follows PCIR | |
|
| 134 | | | V | |
|
| 135 | | | +-------------------+ | |
|
| 136 | | | | NPDE Structure | | |
|
| 137 | | | | (Signature "NPDE")| | |
|
| 138 | | | | last_image: 0x00 | | |
|
| 139 | | | +-------------------+ | |
|
| 140 | | | | |
|
| 141 | | | +-------------------+ | |
|
| 142 | | | | BIT Header | (Signature scanning | |
|
| 143 | | | | (Signature "BIT") | provides the location | |
|
| 144 | | | +-------------------+ of the BIT table) | |
|
| 145 | | | | header is | |
|
| 146 | | | | followed by a table of tokens | |
|
| 147 | | | V one of which is for falcon data. | |
|
| 148 | | | +-------------------+ | |
|
| 149 | | | | BIT Tokens | | |
|
| 150 | | | | ______________ | | |
|
| 151 | | | | | Falcon Data | | | |
|
| 152 | | | | | Token (0x70)|---+------------>------------+--+ |
|
| 153 | | | | +-------------+ | falcon_data_ptr() | | |
|
| 154 | | | +-------------------+ | V |
|
| 155 | | +-----------------------------------------------+ | |
|
| 156 | | (no gap between images) | |
|
| 157 | | +-----------------------------------------------+ | |
|
| 158 | | | EFI Image (Type 0x03) | | |
|
| 159 | | +-----------------------------------------------+ | |
|
| 160 | | | Contains the UEFI GOP driver (Graphics Output)| | |
|
| 161 | | | +-------------------+ | | |
|
| 162 | | | | ROM Header | | | |
|
| 163 | | | +-------------------+ | | |
|
| 164 | | | | PCIR Structure | | | |
|
| 165 | | | +-------------------+ | | |
|
| 166 | | | | NPDE Structure | | | |
|
| 167 | | | +-------------------+ | | |
|
| 168 | | | | Image data | | | |
|
| 169 | | | +-------------------+ | | |
|
| 170 | | +-----------------------------------------------+ | |
|
| 171 | | (no gap between images) | |
|
| 172 | | +-----------------------------------------------+ | |
|
| 173 | | | First FwSec Image (Type 0xE0) | | |
|
| 174 | | +-----------------------------------------------+ | |
|
| 175 | | | +-------------------+ | | |
|
| 176 | | | | ROM Header | | | |
|
| 177 | | | +-------------------+ | | |
|
| 178 | | | | PCIR Structure | | | |
|
| 179 | | | +-------------------+ | | |
|
| 180 | | | | NPDE Structure | | | |
|
| 181 | | | +-------------------+ | | |
|
| 182 | | | | Image data | | | |
|
| 183 | | | +-------------------+ | | |
|
| 184 | | +-----------------------------------------------+ | |
|
| 185 | | (no gap between images) | |
|
| 186 | | +-----------------------------------------------+ | |
|
| 187 | | | Second FwSec Image (Type 0xE0) | | |
|
| 188 | | +-----------------------------------------------+ | |
|
| 189 | | | +-------------------+ | | |
|
| 190 | | | | ROM Header | | | |
|
| 191 | | | +-------------------+ | | |
|
| 192 | | | | PCIR Structure | | | |
|
| 193 | | | +-------------------+ | | |
|
| 194 | | | | NPDE Structure | | | |
|
| 195 | | | +-------------------+ | | |
|
| 196 | | | | | |
|
| 197 | | | +-------------------+ | | |
|
| 198 | | | | PMU Lookup Table | <- falcon_data_offset <----+ |
|
| 199 | | | | +-------------+ | pmu_lookup_table | |
|
| 200 | | | | | Entry 0x85 | | | |
|
| 201 | | | | | FWSEC_PROD | | | |
|
| 202 | | | | +-------------+ | | |
|
| 203 | | | +-------------------+ | |
|
| 204 | | | | | |
|
| 205 | | | | points to | |
|
| 206 | | | V | |
|
| 207 | | | +-------------------+ | |
|
| 208 | | | | FalconUCodeDescV3 | <- falcon_ucode_offset | |
|
| 209 | | | | (FWSEC Firmware) | fwsec_header() | |
|
| 210 | | | +-------------------+ | |
|
| 211 | | | | immediately followed by... | |
|
| 212 | | | V | |
|
| 213 | | | +----------------------------+ | |
|
| 214 | | | | Signatures + FWSEC Ucode | | |
|
| 215 | | | | fwsec_sigs(), fwsec_ucode()| | |
|
| 216 | | | +----------------------------+ | |
|
| 217 | | +-----------------------------------------------+ |
|
| 218 | | |
|
| 219 | +----------------------------------------------------------------------------+
|
| 220 |
|
| 221 | .. note::
|
| 222 | This diagram is created based on an GA-102 Ampere GPU as an example and could
|
| 223 | vary for future or other GPUs.
|
| 224 |
|
| 225 | .. note::
|
| 226 | For more explanations of acronyms, see the detailed descriptions in `vbios.rs`.
|
| 227 |
|
| 228 | Falcon data Lookup
|
| 229 | ------------------
|
| 230 | A key part of the VBIOS extraction code (vbios.rs) is to find the location of the
|
| 231 | Falcon data in the VBIOS which contains the PMU lookup table. This lookup table is
|
| 232 | used to find the required Falcon ucode based on an application ID.
|
| 233 |
|
| 234 | The location of the PMU lookup table is found by scanning the BIT (`BIOS Information Table`_)
|
| 235 | tokens for a token with the Falcon data token id (0x70) which indicates the
|
| 236 | offset of the same from the start of the VBIOS image. Unfortunately, the offset
|
| 237 | does not account for the EFI image located between the PciAt and FwSec images.
|
| 238 | The `vbios.rs` code compensates for this with appropriate arithmetic.
|
| 239 |
|
| 240 | |