| 1 | |
| 2 |
|
| 3 | =========================================
|
| 4 | Automated testing of the DRM subsystem
|
| 5 | =========================================
|
| 6 |
|
| 7 | Introduction
|
| 8 | ============
|
| 9 |
|
| 10 | Making sure that changes to the core or drivers don't introduce regressions can
|
| 11 | be very time-consuming when lots of different hardware configurations need to
|
| 12 | be tested. Moreover, it isn't practical for each person interested in this
|
| 13 | testing to have to acquire and maintain what can be a considerable amount of
|
| 14 | hardware.
|
| 15 |
|
| 16 | Also, it is desirable for developers to check for regressions in their code by
|
| 17 | themselves, instead of relying on the maintainers to find them and then
|
| 18 | reporting back.
|
| 19 |
|
| 20 | There are facilities in gitlab.freedesktop.org to automatically test Mesa that
|
| 21 | can be used as well for testing the DRM subsystem. This document explains how
|
| 22 | people interested in testing it can use this shared infrastructure to save
|
| 23 | quite some time and effort.
|
| 24 |
|
| 25 |
|
| 26 | Relevant files
|
| 27 | ==============
|
| 28 |
|
| 29 | drivers/gpu/drm/ci/gitlab-ci.yml
|
| 30 | --------------------------------
|
| 31 |
|
| 32 | This is the root configuration file for GitLab CI. Among other less interesting
|
| 33 | bits, it specifies the specific version of the scripts to be used. There are
|
| 34 | some variables that can be modified to change the behavior of the pipeline:
|
| 35 |
|
| 36 | DRM_CI_PROJECT_PATH
|
| 37 | Repository that contains the Mesa software infrastructure for CI
|
| 38 |
|
| 39 | DRM_CI_COMMIT_SHA
|
| 40 | A particular revision to use from that repository
|
| 41 |
|
| 42 | UPSTREAM_REPO
|
| 43 | URL to git repository containing the target branch
|
| 44 |
|
| 45 | TARGET_BRANCH
|
| 46 | Branch to which this branch is to be merged into
|
| 47 |
|
| 48 | IGT_VERSION
|
| 49 | Revision of igt-gpu-tools being used, from
|
| 50 | https://gitlab.freedesktop.org/drm/igt-gpu-tools
|
| 51 |
|
| 52 | drivers/gpu/drm/ci/testlist.txt
|
| 53 | -------------------------------
|
| 54 |
|
| 55 | IGT tests to be run on all drivers (unless mentioned in a driver's \*-skips.txt
|
| 56 | file, see below).
|
| 57 |
|
| 58 | drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-fails.txt
|
| 59 | ----------------------------------------------------------
|
| 60 |
|
| 61 | Lists the known failures for a given driver on a specific hardware revision.
|
| 62 |
|
| 63 | drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-flakes.txt
|
| 64 | -----------------------------------------------------------
|
| 65 |
|
| 66 | Lists the tests that for a given driver on a specific hardware revision are
|
| 67 | known to behave unreliably. These tests won't cause a job to fail regardless of
|
| 68 | the result. They will still be run.
|
| 69 |
|
| 70 | Each new flake entry must be associated with a link to the email reporting the
|
| 71 | bug to the author of the affected driver or the relevant GitLab issue. The entry
|
| 72 | must also include the board name or Device Tree name, the first kernel version
|
| 73 | affected, the IGT version used for tests, and an approximation of the failure rate.
|
| 74 |
|
| 75 | They should be provided under the following format::
|
| 76 |
|
| 77 | # Bug Report: $LORE_URL_OR_GITLAB_ISSUE
|
| 78 | # Board Name: broken-board.dtb
|
| 79 | # Linux Version: 6.6-rc1
|
| 80 | # IGT Version: 1.28-gd2af13d9f
|
| 81 | # Failure Rate: 100
|
| 82 | flaky-test
|
| 83 |
|
| 84 | Use the appropriate link below to create a GitLab issue:
|
| 85 | amdgpu driver: https://gitlab.freedesktop.org/drm/amd/-/issues
|
| 86 | i915 driver: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues
|
| 87 | msm driver: https://gitlab.freedesktop.org/drm/msm/-/issues
|
| 88 | xe driver: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues
|
| 89 |
|
| 90 | drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-skips.txt
|
| 91 | -----------------------------------------------------------
|
| 92 |
|
| 93 | Lists the tests that won't be run for a given driver on a specific hardware
|
| 94 | revision. These are usually tests that interfere with the running of the test
|
| 95 | list due to hanging the machine, causing OOM, taking too long, etc.
|
| 96 |
|
| 97 |
|
| 98 | How to enable automated testing on your tree
|
| 99 | ============================================
|
| 100 |
|
| 101 | 1. Create a Linux tree in https://gitlab.freedesktop.org/ if you don't have one
|
| 102 | yet, by forking https://gitlab.freedesktop.org/drm/kernel (this allows GitLab
|
| 103 | to internally track that these are the same git objects).
|
| 104 |
|
| 105 | 2. In your kernel repo's configuration (eg.
|
| 106 | https://gitlab.freedesktop.org/janedoe/linux/-/settings/ci_cd), change the
|
| 107 | CI/CD configuration file from .gitlab-ci.yml to
|
| 108 | drivers/gpu/drm/ci/gitlab-ci.yml.
|
| 109 |
|
| 110 | 3. Request to be added to the drm/ci-ok group so that your user has the
|
| 111 | necessary privileges to run the CI on https://gitlab.freedesktop.org/drm/ci-ok
|
| 112 |
|
| 113 | 4. Next time you push to this repository, you will see a CI pipeline being
|
| 114 | created (eg. https://gitlab.freedesktop.org/janedoe/linux/-/pipelines)
|
| 115 |
|
| 116 | 5. The various jobs will be run and when the pipeline is finished, all jobs
|
| 117 | should be green unless a regression has been found.
|
| 118 |
|
| 119 | 6. Warnings in the pipeline indicate that lockdep
|
| 120 | (see Documentation/locking/lockdep-design.rst) issues have been detected
|
| 121 | during the tests.
|
| 122 |
|
| 123 |
|
| 124 | How to update test expectations
|
| 125 | ===============================
|
| 126 |
|
| 127 | If your changes to the code fix any tests, you will have to remove one or more
|
| 128 | lines from one or more of the files in
|
| 129 | drivers/gpu/drm/ci/${DRIVER_NAME}_*_fails.txt, for each of the test platforms
|
| 130 | affected by the change.
|
| 131 |
|
| 132 |
|
| 133 | How to expand coverage
|
| 134 | ======================
|
| 135 |
|
| 136 | If your code changes make it possible to run more tests (by solving reliability
|
| 137 | issues, for example), you can remove tests from the flakes and/or skips lists,
|
| 138 | and then the expected results if there are any known failures.
|
| 139 |
|
| 140 | If there is a need for updating the version of IGT being used (maybe you have
|
| 141 | added more tests to it), update the IGT_VERSION variable at the top of the
|
| 142 | gitlab-ci.yml file.
|
| 143 |
|
| 144 |
|
| 145 | How to test your changes to the scripts
|
| 146 | =======================================
|
| 147 |
|
| 148 | For testing changes to the scripts in the drm-ci repo, change the
|
| 149 | DRM_CI_PROJECT_PATH and DRM_CI_COMMIT_SHA variables in
|
| 150 | drivers/gpu/drm/ci/gitlab-ci.yml to match your fork of the project (eg.
|
| 151 | janedoe/drm-ci). This fork needs to be in https://gitlab.freedesktop.org/.
|
| 152 |
|
| 153 |
|
| 154 | How to incorporate external fixes in your testing
|
| 155 | =================================================
|
| 156 |
|
| 157 | Often, regressions in other trees will prevent testing changes local to the
|
| 158 | tree under test. These fixes will be automatically merged in during the build
|
| 159 | jobs from a branch in the target tree that is named as
|
| 160 | ${TARGET_BRANCH}-external-fixes.
|
| 161 |
|
| 162 | If the pipeline is not in a merge request and a branch with the same name
|
| 163 | exists in the local tree, commits from that branch will be merged in as well.
|
| 164 |
|
| 165 |
|
| 166 | How to deal with automated testing labs that may be down
|
| 167 | ========================================================
|
| 168 |
|
| 169 | If a hardware farm is down and thus causing pipelines to fail that would
|
| 170 | otherwise pass, one can disable all jobs that would be submitted to that farm
|
| 171 | by editing the file at
|
| 172 | https://gitlab.freedesktop.org/gfx-ci/lab-status/-/blob/main/lab-status.yml.
|