| 1 | .. _embargoed_hardware_issues:
|
| 2 |
|
| 3 | Embargoed hardware issues
|
| 4 | =========================
|
| 5 |
|
| 6 | Scope
|
| 7 | -----
|
| 8 |
|
| 9 | Hardware issues which result in security problems are a different category
|
| 10 | of security bugs than pure software bugs which only affect the Linux
|
| 11 | kernel.
|
| 12 |
|
| 13 | Hardware issues like Meltdown, Spectre, L1TF etc. must be treated
|
| 14 | differently because they usually affect all Operating Systems ("OS") and
|
| 15 | therefore need coordination across different OS vendors, distributions,
|
| 16 | silicon vendors, hardware integrators, and other parties. For some of the
|
| 17 | issues, software mitigations can depend on microcode or firmware updates,
|
| 18 | which need further coordination.
|
| 19 |
|
| 20 | .. _Contact:
|
| 21 |
|
| 22 | Contact
|
| 23 | -------
|
| 24 |
|
| 25 | The Linux kernel hardware security team is separate from the regular Linux
|
| 26 | kernel security team.
|
| 27 |
|
| 28 | The team only handles developing fixes for embargoed hardware security
|
| 29 | issues. Reports of pure software security bugs in the Linux kernel are not
|
| 30 | handled by this team and the reporter will be guided to contact the regular
|
| 31 | Linux kernel security team (:ref:`Documentation/admin-guide/
|
| 32 | <securitybugs>`) instead.
|
| 33 |
|
| 34 | The team can be contacted by email at <[email protected]>. This
|
| 35 | is a private list of security officers who will help you coordinate a fix
|
| 36 | according to our documented process.
|
| 37 |
|
| 38 | The list is encrypted and email to the list can be sent by either PGP or
|
| 39 | S/MIME encrypted and must be signed with the reporter's PGP key or S/MIME
|
| 40 | certificate. The list's PGP key and S/MIME certificate are available from
|
| 41 | the following URLs:
|
| 42 |
|
| 43 | - PGP: https://www.kernel.org/static/files/hardware-security.asc
|
| 44 | - S/MIME: https://www.kernel.org/static/files/hardware-security.crt
|
| 45 |
|
| 46 | While hardware security issues are often handled by the affected silicon
|
| 47 | vendor, we welcome contact from researchers or individuals who have
|
| 48 | identified a potential hardware flaw.
|
| 49 |
|
| 50 | Hardware security officers
|
| 51 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 52 |
|
| 53 | The current team of hardware security officers:
|
| 54 |
|
| 55 | - Linus Torvalds (Linux Foundation Fellow)
|
| 56 | - Greg Kroah-Hartman (Linux Foundation Fellow)
|
| 57 | - Thomas Gleixner (Linux Foundation Fellow)
|
| 58 |
|
| 59 | Operation of mailing-lists
|
| 60 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 61 |
|
| 62 | The encrypted mailing-lists which are used in our process are hosted on
|
| 63 | Linux Foundation's IT infrastructure. By providing this service, members
|
| 64 | of Linux Foundation's IT operations personnel technically have the
|
| 65 | ability to access the embargoed information, but are obliged to
|
| 66 | confidentiality by their employment contract. Linux Foundation IT
|
| 67 | personnel are also responsible for operating and managing the rest of
|
| 68 | kernel.org's infrastructure.
|
| 69 |
|
| 70 | The Linux Foundation's current director of IT Project infrastructure is
|
| 71 | Konstantin Ryabitsev.
|
| 72 |
|
| 73 |
|
| 74 | Non-disclosure agreements
|
| 75 | -------------------------
|
| 76 |
|
| 77 | The Linux kernel hardware security team is not a formal body and therefore
|
| 78 | unable to enter into any non-disclosure agreements. The kernel community
|
| 79 | is aware of the sensitive nature of such issues and offers a Memorandum of
|
| 80 | Understanding instead.
|
| 81 |
|
| 82 |
|
| 83 | Memorandum of Understanding
|
| 84 | ---------------------------
|
| 85 |
|
| 86 | The Linux kernel community has a deep understanding of the requirement to
|
| 87 | keep hardware security issues under embargo for coordination between
|
| 88 | different OS vendors, distributors, silicon vendors, and other parties.
|
| 89 |
|
| 90 | The Linux kernel community has successfully handled hardware security
|
| 91 | issues in the past and has the necessary mechanisms in place to allow
|
| 92 | community compliant development under embargo restrictions.
|
| 93 |
|
| 94 | The Linux kernel community has a dedicated hardware security team for
|
| 95 | initial contact, which oversees the process of handling such issues under
|
| 96 | embargo rules.
|
| 97 |
|
| 98 | The hardware security team identifies the developers (domain experts) who
|
| 99 | will form the initial response team for a particular issue. The initial
|
| 100 | response team can bring in further developers (domain experts) to address
|
| 101 | the issue in the best technical way.
|
| 102 |
|
| 103 | All involved developers pledge to adhere to the embargo rules and to keep
|
| 104 | the received information confidential. Violation of the pledge will lead to
|
| 105 | immediate exclusion from the current issue and removal from all related
|
| 106 | mailing lists. In addition, the hardware security team will also exclude
|
| 107 | the offender from future issues. The impact of this consequence is a highly
|
| 108 | effective deterrent in our community. In case a violation happens the
|
| 109 | hardware security team will inform the involved parties immediately. If you
|
| 110 | or anyone else becomes aware of a potential violation, please report it
|
| 111 | immediately to the Hardware security officers.
|
| 112 |
|
| 113 |
|
| 114 | Process
|
| 115 | ^^^^^^^
|
| 116 |
|
| 117 | Due to the globally distributed nature of Linux kernel development,
|
| 118 | face-to-face meetings are almost impossible to address hardware security
|
| 119 | issues. Phone conferences are hard to coordinate due to time zones and
|
| 120 | other factors and should be only used when absolutely necessary. Encrypted
|
| 121 | email has been proven to be the most effective and secure communication
|
| 122 | method for these types of issues.
|
| 123 |
|
| 124 | Start of Disclosure
|
| 125 | """""""""""""""""""
|
| 126 |
|
| 127 | Disclosure starts by emailing the Linux kernel hardware security team per
|
| 128 | the Contact section above. This initial contact should contain a
|
| 129 | description of the problem and a list of any known affected silicon. If
|
| 130 | your organization builds or distributes the affected hardware, we encourage
|
| 131 | you to also consider what other hardware could be affected. The disclosing
|
| 132 | party is responsible for contacting the affected silicon vendors in a
|
| 133 | timely manner.
|
| 134 |
|
| 135 | The hardware security team will provide an incident-specific encrypted
|
| 136 | mailing list which will be used for initial discussion with the reporter,
|
| 137 | further disclosure, and coordination of fixes.
|
| 138 |
|
| 139 | The hardware security team will provide the disclosing party a list of
|
| 140 | developers (domain experts) who should be informed initially about the
|
| 141 | issue after confirming with the developers that they will adhere to this
|
| 142 | Memorandum of Understanding and the documented process. These developers
|
| 143 | form the initial response team and will be responsible for handling the
|
| 144 | issue after initial contact. The hardware security team is supporting the
|
| 145 | response team, but is not necessarily involved in the mitigation
|
| 146 | development process.
|
| 147 |
|
| 148 | While individual developers might be covered by a non-disclosure agreement
|
| 149 | via their employer, they cannot enter individual non-disclosure agreements
|
| 150 | in their role as Linux kernel developers. They will, however, agree to
|
| 151 | adhere to this documented process and the Memorandum of Understanding.
|
| 152 |
|
| 153 | The disclosing party should provide a list of contacts for all other
|
| 154 | entities who have already been, or should be, informed about the issue.
|
| 155 | This serves several purposes:
|
| 156 |
|
| 157 | - The list of disclosed entities allows communication across the
|
| 158 | industry, e.g. other OS vendors, HW vendors, etc.
|
| 159 |
|
| 160 | - The disclosed entities can be contacted to name experts who should
|
| 161 | participate in the mitigation development.
|
| 162 |
|
| 163 | - If an expert who is required to handle an issue is employed by a listed
|
| 164 | entity or member of an listed entity, then the response teams can
|
| 165 | request the disclosure of that expert from that entity. This ensures
|
| 166 | that the expert is also part of the entity's response team.
|
| 167 |
|
| 168 | Disclosure
|
| 169 | """"""""""
|
| 170 |
|
| 171 | The disclosing party provides detailed information to the initial response
|
| 172 | team via the specific encrypted mailing-list.
|
| 173 |
|
| 174 | From our experience, the technical documentation of these issues is usually
|
| 175 | a sufficient starting point, and further technical clarification is best
|
| 176 | done via email.
|
| 177 |
|
| 178 | Mitigation development
|
| 179 | """"""""""""""""""""""
|
| 180 |
|
| 181 | The initial response team sets up an encrypted mailing-list or repurposes
|
| 182 | an existing one if appropriate.
|
| 183 |
|
| 184 | Using a mailing list is close to the normal Linux development process and
|
| 185 | has been successfully used to develop mitigations for various hardware
|
| 186 | security issues in the past.
|
| 187 |
|
| 188 | The mailing list operates in the same way as normal Linux development.
|
| 189 | Patches are posted, discussed, and reviewed and if agreed upon, applied to
|
| 190 | a non-public Git repository which is only accessible to the participating
|
| 191 | developers via a secure connection. The repository contains the main
|
| 192 | development branch against the mainline kernel and backport branches for
|
| 193 | stable kernel versions as necessary.
|
| 194 |
|
| 195 | The initial response team will identify further experts from the Linux
|
| 196 | kernel developer community as needed. Any involved party can suggest
|
| 197 | further experts to be included, each of which will be subject to the same
|
| 198 | requirements outlined above.
|
| 199 |
|
| 200 | Bringing in experts can happen at any time in the development process and
|
| 201 | needs to be handled in a timely manner.
|
| 202 |
|
| 203 | If an expert is employed by or a member of an entity on the disclosure list
|
| 204 | provided by the disclosing party, then participation will be requested from
|
| 205 | the relevant entity.
|
| 206 |
|
| 207 | If not, then the disclosing party will be informed about the experts'
|
| 208 | participation. The experts are covered by the Memorandum of Understanding
|
| 209 | and the disclosing party is requested to acknowledge their participation.
|
| 210 | In the case where the disclosing party has a compelling reason to object,
|
| 211 | any objection must to be raised within five working days and resolved with
|
| 212 | the incident team immediately. If the disclosing party does not react
|
| 213 | within five working days this is taken as silent acknowledgment.
|
| 214 |
|
| 215 | After the incident team acknowledges or resolves an objection, the expert
|
| 216 | is disclosed and brought into the development process.
|
| 217 |
|
| 218 | List participants may not communicate about the issue outside of the
|
| 219 | private mailing list. List participants may not use any shared resources
|
| 220 | (e.g. employer build farms, CI systems, etc) when working on patches.
|
| 221 |
|
| 222 | Early access
|
| 223 | """"""""""""
|
| 224 |
|
| 225 | The patches discussed and developed on the list can neither be distributed
|
| 226 | to any individual who is not a member of the response team nor to any other
|
| 227 | organization.
|
| 228 |
|
| 229 | To allow the affected silicon vendors to work with their internal teams and
|
| 230 | industry partners on testing, validation, and logistics, the following
|
| 231 | exception is provided:
|
| 232 |
|
| 233 | Designated representatives of the affected silicon vendors are
|
| 234 | allowed to hand over the patches at any time to the silicon
|
| 235 | vendor’s response team. The representative must notify the kernel
|
| 236 | response team about the handover. The affected silicon vendor must
|
| 237 | have and maintain their own documented security process for any
|
| 238 | patches shared with their response team that is consistent with
|
| 239 | this policy.
|
| 240 |
|
| 241 | The silicon vendor’s response team can distribute these patches to
|
| 242 | their industry partners and to their internal teams under the
|
| 243 | silicon vendor’s documented security process. Feedback from the
|
| 244 | industry partners goes back to the silicon vendor and is
|
| 245 | communicated by the silicon vendor to the kernel response team.
|
| 246 |
|
| 247 | The handover to the silicon vendor’s response team removes any
|
| 248 | responsibility or liability from the kernel response team regarding
|
| 249 | premature disclosure, which happens due to the involvement of the
|
| 250 | silicon vendor’s internal teams or industry partners. The silicon
|
| 251 | vendor guarantees this release of liability by agreeing to this
|
| 252 | process.
|
| 253 |
|
| 254 | Coordinated release
|
| 255 | """""""""""""""""""
|
| 256 |
|
| 257 | The involved parties will negotiate the date and time when the embargo
|
| 258 | ends. At that point, the prepared mitigations are published into the
|
| 259 | relevant kernel trees. There is no pre-notification process: the
|
| 260 | mitigations are published in public and available to everyone at the same
|
| 261 | time.
|
| 262 |
|
| 263 | While we understand that hardware security issues need coordinated embargo
|
| 264 | time, the embargo time should be constrained to the minimum time that is
|
| 265 | required for all involved parties to develop, test, and prepare their
|
| 266 | mitigations. Extending embargo time artificially to meet conference talk
|
| 267 | dates or other non-technical reasons creates more work and burden for the
|
| 268 | involved developers and response teams as the patches need to be kept up to
|
| 269 | date in order to follow the ongoing upstream kernel development, which
|
| 270 | might create conflicting changes.
|
| 271 |
|
| 272 | CVE assignment
|
| 273 | """"""""""""""
|
| 274 |
|
| 275 | Neither the hardware security team nor the initial response team assign
|
| 276 | CVEs, nor are CVEs required for the development process. If CVEs are
|
| 277 | provided by the disclosing party they can be used for documentation
|
| 278 | purposes.
|
| 279 |
|
| 280 | Process ambassadors
|
| 281 | -------------------
|
| 282 |
|
| 283 | For assistance with this process we have established ambassadors in various
|
| 284 | organizations, who can answer questions about or provide guidance on the
|
| 285 | reporting process and further handling. Ambassadors are not involved in the
|
| 286 | disclosure of a particular issue, unless requested by a response team or by
|
| 287 | an involved disclosed party. The current ambassadors list:
|
| 288 |
|
| 289 | ============= ========================================================
|
| 290 | AMD Tom Lendacky <[email protected]>
|
| 291 | Ampere Darren Hart <[email protected]>
|
| 292 | ARM Catalin Marinas <[email protected]>
|
| 293 | IBM Power Madhavan Srinivasan <[email protected]>
|
| 294 | IBM Z Christian Borntraeger <[email protected]>
|
| 295 | Intel Tony Luck <[email protected]>
|
| 296 | Qualcomm Trilok Soni <[email protected]>
|
| 297 | RISC-V Palmer Dabbelt <[email protected]>
|
| 298 | Samsung Javier González <[email protected]>
|
| 299 |
|
| 300 | Microsoft James Morris <[email protected]>
|
| 301 | Xen Andrew Cooper <[email protected]>
|
| 302 |
|
| 303 | Canonical John Johansen <[email protected]>
|
| 304 | Debian Ben Hutchings <[email protected]>
|
| 305 | Oracle Konrad Rzeszutek Wilk <[email protected]>
|
| 306 | Red Hat Josh Poimboeuf <[email protected]>
|
| 307 | SUSE Jiri Kosina <[email protected]>
|
| 308 |
|
| 309 | Google Kees Cook <[email protected]>
|
| 310 |
|
| 311 | LLVM Nick Desaulniers <[email protected]>
|
| 312 | ============= ========================================================
|
| 313 |
|
| 314 | If you want your organization to be added to the ambassadors list, please
|
| 315 | contact the hardware security team. The nominated ambassador has to
|
| 316 | understand and support our process fully and is ideally well-connected in
|
| 317 | the Linux kernel community.
|
| 318 |
|
| 319 | Encrypted mailing-lists
|
| 320 | -----------------------
|
| 321 |
|
| 322 | We use encrypted mailing lists for communication. The operating principle
|
| 323 | of these lists is that email sent to the list is encrypted either with the
|
| 324 | list's PGP key or with the list's S/MIME certificate. The mailing list
|
| 325 | software decrypts the email and re-encrypts it individually for each
|
| 326 | subscriber with the subscriber's PGP key or S/MIME certificate. Details
|
| 327 | about the mailing list software and the setup that is used to ensure the
|
| 328 | security of the lists and protection of the data can be found here:
|
| 329 | https://korg.wiki.kernel.org/userdoc/remail.
|
| 330 |
|
| 331 | List keys
|
| 332 | ^^^^^^^^^
|
| 333 |
|
| 334 | For initial contact see the :ref:`Contact` section above. For incident
|
| 335 | specific mailing lists, the key and S/MIME certificate are conveyed to the
|
| 336 | subscribers by email sent from the specific list.
|
| 337 |
|
| 338 | Subscription to incident-specific lists
|
| 339 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 340 |
|
| 341 | Subscription to incident-specific lists is handled by the response teams.
|
| 342 | Disclosed parties who want to participate in the communication send a list
|
| 343 | of potential experts to the response team so the response team can validate
|
| 344 | subscription requests.
|
| 345 |
|
| 346 | Each subscriber needs to send a subscription request to the response team
|
| 347 | by email. The email must be signed with the subscriber's PGP key or S/MIME
|
| 348 | certificate. If a PGP key is used, it must be available from a public key
|
| 349 | server and is ideally connected to the Linux kernel's PGP web of trust. See
|
| 350 | also: https://www.kernel.org/signature.html.
|
| 351 |
|
| 352 | The response team verifies that the subscriber request is valid and adds
|
| 353 | the subscriber to the list. After subscription the subscriber will receive
|
| 354 | email from the mailing-list which is signed either with the list's PGP key
|
| 355 | or the list's S/MIME certificate. The subscriber's email client can extract
|
| 356 | the PGP key or the S/MIME certificate from the signature so the subscriber
|
| 357 | can send encrypted email to the list.
|
| 358 |
|