| 1 | |
| 2 |
|
| 3 | Written by: Neil Brown
|
| 4 | Please see MAINTAINERS file for where to send questions.
|
| 5 |
|
| 6 | Overlay Filesystem
|
| 7 | ==================
|
| 8 |
|
| 9 | This document describes a prototype for a new approach to providing
|
| 10 | overlay-filesystem functionality in Linux (sometimes referred to as
|
| 11 | union-filesystems). An overlay-filesystem tries to present a
|
| 12 | filesystem which is the result of overlaying one filesystem on top
|
| 13 | of the other.
|
| 14 |
|
| 15 |
|
| 16 | Overlay objects
|
| 17 | ---------------
|
| 18 |
|
| 19 | The overlay filesystem approach is 'hybrid', because the objects that
|
| 20 | appear in the filesystem do not always appear to belong to that filesystem.
|
| 21 | In many cases, an object accessed in the union will be indistinguishable
|
| 22 | from accessing the corresponding object from the original filesystem.
|
| 23 | This is most obvious from the 'st_dev' field returned by stat(2).
|
| 24 |
|
| 25 | While directories will report an st_dev from the overlay-filesystem,
|
| 26 | non-directory objects may report an st_dev from the lower filesystem or
|
| 27 | upper filesystem that is providing the object. Similarly st_ino will
|
| 28 | only be unique when combined with st_dev, and both of these can change
|
| 29 | over the lifetime of a non-directory object. Many applications and
|
| 30 | tools ignore these values and will not be affected.
|
| 31 |
|
| 32 | In the special case of all overlay layers on the same underlying
|
| 33 | filesystem, all objects will report an st_dev from the overlay
|
| 34 | filesystem and st_ino from the underlying filesystem. This will
|
| 35 | make the overlay mount more compliant with filesystem scanners and
|
| 36 | overlay objects will be distinguishable from the corresponding
|
| 37 | objects in the original filesystem.
|
| 38 |
|
| 39 | On 64bit systems, even if all overlay layers are not on the same
|
| 40 | underlying filesystem, the same compliant behavior could be achieved
|
| 41 | with the "xino" feature. The "xino" feature composes a unique object
|
| 42 | identifier from the real object st_ino and an underlying fsid number.
|
| 43 | The "xino" feature uses the high inode number bits for fsid, because the
|
| 44 | underlying filesystems rarely use the high inode number bits. In case
|
| 45 | the underlying inode number does overflow into the high xino bits, overlay
|
| 46 | filesystem will fall back to the non xino behavior for that inode.
|
| 47 |
|
| 48 | The "xino" feature can be enabled with the "-o xino=on" overlay mount option.
|
| 49 | If all underlying filesystems support NFS file handles, the value of st_ino
|
| 50 | for overlay filesystem objects is not only unique, but also persistent over
|
| 51 | the lifetime of the filesystem. The "-o xino=auto" overlay mount option
|
| 52 | enables the "xino" feature only if the persistent st_ino requirement is met.
|
| 53 |
|
| 54 | The following table summarizes what can be expected in different overlay
|
| 55 | configurations.
|
| 56 |
|
| 57 | Inode properties
|
| 58 | ````````````````
|
| 59 |
|
| 60 | +--------------+------------+------------+-----------------+----------------+
|
| 61 | |Configuration | Persistent | Uniform | st_ino == d_ino | d_ino == i_ino |
|
| 62 | | | st_ino | st_dev | | [*] |
|
| 63 | +==============+=====+======+=====+======+========+========+========+=======+
|
| 64 | | | dir | !dir | dir | !dir | dir | !dir | dir | !dir |
|
| 65 | +--------------+-----+------+-----+------+--------+--------+--------+-------+
|
| 66 | | All layers | Y | Y | Y | Y | Y | Y | Y | Y |
|
| 67 | | on same fs | | | | | | | | |
|
| 68 | +--------------+-----+------+-----+------+--------+--------+--------+-------+
|
| 69 | | Layers not | N | N | Y | N | N | Y | N | Y |
|
| 70 | | on same fs, | | | | | | | | |
|
| 71 | | xino=off | | | | | | | | |
|
| 72 | +--------------+-----+------+-----+------+--------+--------+--------+-------+
|
| 73 | | xino=on/auto | Y | Y | Y | Y | Y | Y | Y | Y |
|
| 74 | +--------------+-----+------+-----+------+--------+--------+--------+-------+
|
| 75 | | xino=on/auto,| N | N | Y | N | N | Y | N | Y |
|
| 76 | | ino overflow | | | | | | | | |
|
| 77 | +--------------+-----+------+-----+------+--------+--------+--------+-------+
|
| 78 |
|
| 79 | [*] nfsd v3 readdirplus verifies d_ino == i_ino. i_ino is exposed via several
|
| 80 | /proc files, such as /proc/locks and /proc/self/fdinfo/<fd> of an inotify
|
| 81 | file descriptor.
|
| 82 |
|
| 83 | Upper and Lower
|
| 84 | ---------------
|
| 85 |
|
| 86 | An overlay filesystem combines two filesystems - an 'upper' filesystem
|
| 87 | and a 'lower' filesystem. When a name exists in both filesystems, the
|
| 88 | object in the 'upper' filesystem is visible while the object in the
|
| 89 | 'lower' filesystem is either hidden or, in the case of directories,
|
| 90 | merged with the 'upper' object.
|
| 91 |
|
| 92 | It would be more correct to refer to an upper and lower 'directory
|
| 93 | tree' rather than 'filesystem' as it is quite possible for both
|
| 94 | directory trees to be in the same filesystem and there is no
|
| 95 | requirement that the root of a filesystem be given for either upper or
|
| 96 | lower.
|
| 97 |
|
| 98 | A wide range of filesystems supported by Linux can be the lower filesystem,
|
| 99 | but not all filesystems that are mountable by Linux have the features
|
| 100 | needed for OverlayFS to work. The lower filesystem does not need to be
|
| 101 | writable. The lower filesystem can even be another overlayfs. The upper
|
| 102 | filesystem will normally be writable and if it is it must support the
|
| 103 | creation of trusted.* and/or user.* extended attributes, and must provide
|
| 104 | valid d_type in readdir responses, so NFS is not suitable.
|
| 105 |
|
| 106 | A read-only overlay of two read-only filesystems may use any
|
| 107 | filesystem type.
|
| 108 |
|
| 109 | Directories
|
| 110 | -----------
|
| 111 |
|
| 112 | Overlaying mainly involves directories. If a given name appears in both
|
| 113 | upper and lower filesystems and refers to a non-directory in either,
|
| 114 | then the lower object is hidden - the name refers only to the upper
|
| 115 | object.
|
| 116 |
|
| 117 | Where both upper and lower objects are directories, a merged directory
|
| 118 | is formed.
|
| 119 |
|
| 120 | At mount time, the two directories given as mount options "lowerdir" and
|
| 121 | "upperdir" are combined into a merged directory::
|
| 122 |
|
| 123 | mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,\
|
| 124 | workdir=/work /merged
|
| 125 |
|
| 126 | The "workdir" needs to be a directory on the same filesystem as upperdir.
|
| 127 |
|
| 128 | Then whenever a lookup is requested in such a merged directory, the
|
| 129 | lookup is performed in each actual directory and the combined result
|
| 130 | is cached in the dentry belonging to the overlay filesystem. If both
|
| 131 | actual lookups find directories, both are stored and a merged
|
| 132 | directory is created, otherwise only one is stored: the upper if it
|
| 133 | exists, else the lower.
|
| 134 |
|
| 135 | Only the lists of names from directories are merged. Other content
|
| 136 | such as metadata and extended attributes are reported for the upper
|
| 137 | directory only. These attributes of the lower directory are hidden.
|
| 138 |
|
| 139 | whiteouts and opaque directories
|
| 140 | --------------------------------
|
| 141 |
|
| 142 | In order to support rm and rmdir without changing the lower
|
| 143 | filesystem, an overlay filesystem needs to record in the upper filesystem
|
| 144 | that files have been removed. This is done using whiteouts and opaque
|
| 145 | directories (non-directories are always opaque).
|
| 146 |
|
| 147 | A whiteout is created as a character device with 0/0 device number or
|
| 148 | as a zero-size regular file with the xattr "trusted.overlay.whiteout".
|
| 149 |
|
| 150 | When a whiteout is found in the upper level of a merged directory, any
|
| 151 | matching name in the lower level is ignored, and the whiteout itself
|
| 152 | is also hidden.
|
| 153 |
|
| 154 | A directory is made opaque by setting the xattr "trusted.overlay.opaque"
|
| 155 | to "y". Where the upper filesystem contains an opaque directory, any
|
| 156 | directory in the lower filesystem with the same name is ignored.
|
| 157 |
|
| 158 | An opaque directory should not contain any whiteouts, because they do not
|
| 159 | serve any purpose. A merge directory containing regular files with the xattr
|
| 160 | "trusted.overlay.whiteout", should be additionally marked by setting the xattr
|
| 161 | "trusted.overlay.opaque" to "x" on the merge directory itself.
|
| 162 | This is needed to avoid the overhead of checking the "trusted.overlay.whiteout"
|
| 163 | on all entries during readdir in the common case.
|
| 164 |
|
| 165 | readdir
|
| 166 | -------
|
| 167 |
|
| 168 | When a 'readdir' request is made on a merged directory, the upper and
|
| 169 | lower directories are each read and the name lists merged in the
|
| 170 | obvious way (upper is read first, then lower - entries that already
|
| 171 | exist are not re-added). This merged name list is cached in the
|
| 172 | 'struct file' and so remains as long as the file is kept open. If the
|
| 173 | directory is opened and read by two processes at the same time, they
|
| 174 | will each have separate caches. A seekdir to the start of the
|
| 175 | directory (offset 0) followed by a readdir will cause the cache to be
|
| 176 | discarded and rebuilt.
|
| 177 |
|
| 178 | This means that changes to the merged directory do not appear while a
|
| 179 | directory is being read. This is unlikely to be noticed by many
|
| 180 | programs.
|
| 181 |
|
| 182 | seek offsets are assigned sequentially when the directories are read.
|
| 183 | Thus if:
|
| 184 |
|
| 185 | - read part of a directory
|
| 186 | - remember an offset, and close the directory
|
| 187 | - re-open the directory some time later
|
| 188 | - seek to the remembered offset
|
| 189 |
|
| 190 | there may be little correlation between the old and new locations in
|
| 191 | the list of filenames, particularly if anything has changed in the
|
| 192 | directory.
|
| 193 |
|
| 194 | Readdir on directories that are not merged is simply handled by the
|
| 195 | underlying directory (upper or lower).
|
| 196 |
|
| 197 | renaming directories
|
| 198 | --------------------
|
| 199 |
|
| 200 | When renaming a directory that is on the lower layer or merged (i.e. the
|
| 201 | directory was not created on the upper layer to start with) overlayfs can
|
| 202 | handle it in two different ways:
|
| 203 |
|
| 204 | 1. return EXDEV error: this error is returned by rename(2) when trying to
|
| 205 | move a file or directory across filesystem boundaries. Hence
|
| 206 | applications are usually prepared to handle this error (mv(1) for example
|
| 207 | recursively copies the directory tree). This is the default behavior.
|
| 208 |
|
| 209 | 2. If the "redirect_dir" feature is enabled, then the directory will be
|
| 210 | copied up (but not the contents). Then the "trusted.overlay.redirect"
|
| 211 | extended attribute is set to the path of the original location from the
|
| 212 | root of the overlay. Finally the directory is moved to the new
|
| 213 | location.
|
| 214 |
|
| 215 | There are several ways to tune the "redirect_dir" feature.
|
| 216 |
|
| 217 | Kernel config options:
|
| 218 |
|
| 219 | - OVERLAY_FS_REDIRECT_DIR:
|
| 220 | If this is enabled, then redirect_dir is turned on by default.
|
| 221 | - OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW:
|
| 222 | If this is enabled, then redirects are always followed by default. Enabling
|
| 223 | this results in a less secure configuration. Enable this option only when
|
| 224 | worried about backward compatibility with kernels that have the redirect_dir
|
| 225 | feature and follow redirects even if turned off.
|
| 226 |
|
| 227 | Module options (can also be changed through /sys/module/overlay/parameters/):
|
| 228 |
|
| 229 | - "redirect_dir=BOOL":
|
| 230 | See OVERLAY_FS_REDIRECT_DIR kernel config option above.
|
| 231 | - "redirect_always_follow=BOOL":
|
| 232 | See OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW kernel config option above.
|
| 233 | - "redirect_max=NUM":
|
| 234 | The maximum number of bytes in an absolute redirect (default is 256).
|
| 235 |
|
| 236 | Mount options:
|
| 237 |
|
| 238 | - "redirect_dir=on":
|
| 239 | Redirects are enabled.
|
| 240 | - "redirect_dir=follow":
|
| 241 | Redirects are not created, but followed.
|
| 242 | - "redirect_dir=nofollow":
|
| 243 | Redirects are not created and not followed.
|
| 244 | - "redirect_dir=off":
|
| 245 | If "redirect_always_follow" is enabled in the kernel/module config,
|
| 246 | this "off" translates to "follow", otherwise it translates to "nofollow".
|
| 247 |
|
| 248 | When the NFS export feature is enabled, every copied up directory is
|
| 249 | indexed by the file handle of the lower inode and a file handle of the
|
| 250 | upper directory is stored in a "trusted.overlay.upper" extended attribute
|
| 251 | on the index entry. On lookup of a merged directory, if the upper
|
| 252 | directory does not match the file handle stores in the index, that is an
|
| 253 | indication that multiple upper directories may be redirected to the same
|
| 254 | lower directory. In that case, lookup returns an error and warns about
|
| 255 | a possible inconsistency.
|
| 256 |
|
| 257 | Because lower layer redirects cannot be verified with the index, enabling
|
| 258 | NFS export support on an overlay filesystem with no upper layer requires
|
| 259 | turning off redirect follow (e.g. "redirect_dir=nofollow").
|
| 260 |
|
| 261 |
|
| 262 | Non-directories
|
| 263 | ---------------
|
| 264 |
|
| 265 | Objects that are not directories (files, symlinks, device-special
|
| 266 | files etc.) are presented either from the upper or lower filesystem as
|
| 267 | appropriate. When a file in the lower filesystem is accessed in a way
|
| 268 | that requires write-access, such as opening for write access, changing
|
| 269 | some metadata etc., the file is first copied from the lower filesystem
|
| 270 | to the upper filesystem (copy_up). Note that creating a hard-link
|
| 271 | also requires copy_up, though of course creation of a symlink does
|
| 272 | not.
|
| 273 |
|
| 274 | The copy_up may turn out to be unnecessary, for example if the file is
|
| 275 | opened for read-write but the data is not modified.
|
| 276 |
|
| 277 | The copy_up process first makes sure that the containing directory
|
| 278 | exists in the upper filesystem - creating it and any parents as
|
| 279 | necessary. It then creates the object with the same metadata (owner,
|
| 280 | mode, mtime, symlink-target etc.) and then if the object is a file, the
|
| 281 | data is copied from the lower to the upper filesystem. Finally any
|
| 282 | extended attributes are copied up.
|
| 283 |
|
| 284 | Once the copy_up is complete, the overlay filesystem simply
|
| 285 | provides direct access to the newly created file in the upper
|
| 286 | filesystem - future operations on the file are barely noticed by the
|
| 287 | overlay filesystem (though an operation on the name of the file such as
|
| 288 | rename or unlink will of course be noticed and handled).
|
| 289 |
|
| 290 |
|
| 291 | Permission model
|
| 292 | ----------------
|
| 293 |
|
| 294 | An overlay filesystem stashes credentials that will be used when
|
| 295 | accessing lower or upper filesystems.
|
| 296 |
|
| 297 | In the old mount api the credentials of the task calling mount(2) are
|
| 298 | stashed. In the new mount api the credentials of the task creating the
|
| 299 | superblock through FSCONFIG_CMD_CREATE command of fsconfig(2) are
|
| 300 | stashed.
|
| 301 |
|
| 302 | Starting with kernel v6.15 it is possible to use the "override_creds"
|
| 303 | mount option which will cause the credentials of the calling task to be
|
| 304 | recorded. Note that "override_creds" is only meaningful when used with
|
| 305 | the new mount api as the old mount api combines setting options and
|
| 306 | superblock creation in a single mount(2) syscall.
|
| 307 |
|
| 308 | Permission checking in the overlay filesystem follows these principles:
|
| 309 |
|
| 310 | 1) permission check SHOULD return the same result before and after copy up
|
| 311 |
|
| 312 | 2) task creating the overlay mount MUST NOT gain additional privileges
|
| 313 |
|
| 314 | 3) task[*] MAY gain additional privileges through the overlay,
|
| 315 | compared to direct access on underlying lower or upper filesystems
|
| 316 |
|
| 317 | This is achieved by performing two permission checks on each access:
|
| 318 |
|
| 319 | a) check if current task is allowed access based on local DAC (owner,
|
| 320 | group, mode and posix acl), as well as MAC checks
|
| 321 |
|
| 322 | b) check if stashed credentials would be allowed real operation on lower or
|
| 323 | upper layer based on underlying filesystem permissions, again including
|
| 324 | MAC checks
|
| 325 |
|
| 326 | Check (a) ensures consistency (1) since owner, group, mode and posix acls
|
| 327 | are copied up. On the other hand it can result in server enforced
|
| 328 | permissions (used by NFS, for example) being ignored (3).
|
| 329 |
|
| 330 | Check (b) ensures that no task gains permissions to underlying layers that
|
| 331 | the stashed credentials do not have (2). This also means that it is possible
|
| 332 | to create setups where the consistency rule (1) does not hold; normally,
|
| 333 | however, the stashed credentials will have sufficient privileges to
|
| 334 | perform all operations.
|
| 335 |
|
| 336 | Another way to demonstrate this model is drawing parallels between::
|
| 337 |
|
| 338 | mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,... /merged
|
| 339 |
|
| 340 | and::
|
| 341 |
|
| 342 | cp -a /lower /upper
|
| 343 | mount --bind /upper /merged
|
| 344 |
|
| 345 | The resulting access permissions should be the same. The difference is in
|
| 346 | the time of copy (on-demand vs. up-front).
|
| 347 |
|
| 348 |
|
| 349 | Idmapped mounts
|
| 350 | ---------------
|
| 351 |
|
| 352 | The overlay mount itself can be turned into an idmapped mount by applying an
|
| 353 | idmapping to it with mount_setattr(2) and MOUNT_ATTR_IDMAP, just like for
|
| 354 | other filesystems that support idmapped mounts.
|
| 355 |
|
| 356 | The mount idmapping only changes how ownership and permissions of the overlay
|
| 357 | inodes are presented to and interpreted for the caller. It does not change
|
| 358 | how overlayfs accesses the underlying layers: those are still accessed with
|
| 359 | the stashed mounter's credentials through their own mounts, which may
|
| 360 | themselves be idmapped. The overlay mount idmapping and any layer idmapping
|
| 361 | compose, an underlying id is first mapped according to the relevant layer
|
| 362 | idmapping and then according to the overlay mount idmapping.
|
| 363 |
|
| 364 |
|
| 365 | Multiple lower layers
|
| 366 | ---------------------
|
| 367 |
|
| 368 | Multiple lower layers can now be given using the colon (":") as a
|
| 369 | separator character between the directory names. For example::
|
| 370 |
|
| 371 | mount -t overlay overlay -olowerdir=/lower1:/lower2:/lower3 /merged
|
| 372 |
|
| 373 | As the example shows, "upperdir=" and "workdir=" may be omitted. In
|
| 374 | that case the overlay will be read-only.
|
| 375 |
|
| 376 | The specified lower directories will be stacked beginning from the
|
| 377 | rightmost one and going left. In the above example lower1 will be the
|
| 378 | top, lower2 the middle and lower3 the bottom layer.
|
| 379 |
|
| 380 | Note: directory names containing colons can be provided as lower layer by
|
| 381 | escaping the colons with a single backslash. For example::
|
| 382 |
|
| 383 | mount -t overlay overlay -olowerdir=/a\:lower\:\:dir /merged
|
| 384 |
|
| 385 | Since kernel version v6.8, directory names containing colons can also
|
| 386 | be configured as lower layer using the "lowerdir+" mount options and the
|
| 387 | fsconfig syscall from new mount api. For example::
|
| 388 |
|
| 389 | fsconfig(fs_fd, FSCONFIG_SET_STRING, "lowerdir+", "/a:lower::dir", 0);
|
| 390 |
|
| 391 | In the latter case, colons in lower layer directory names will be escaped
|
| 392 | as an octal characters (\072) when displayed in /proc/self/mountinfo.
|
| 393 |
|
| 394 | Metadata only copy up
|
| 395 | ---------------------
|
| 396 |
|
| 397 | When the "metacopy" feature is enabled, overlayfs will only copy
|
| 398 | up metadata (as opposed to whole file), when a metadata specific operation
|
| 399 | like chown/chmod is performed. An upper file in this state is marked with
|
| 400 | "trusted.overlayfs.metacopy" xattr which indicates that the upper file
|
| 401 | contains no data. The data will be copied up later when file is opened for
|
| 402 | WRITE operation. After the lower file's data is copied up,
|
| 403 | the "trusted.overlayfs.metacopy" xattr is removed from the upper file.
|
| 404 |
|
| 405 | In other words, this is delayed data copy up operation and data is copied
|
| 406 | up when there is a need to actually modify data.
|
| 407 |
|
| 408 | There are multiple ways to enable/disable this feature. A config option
|
| 409 | CONFIG_OVERLAY_FS_METACOPY can be set/unset to enable/disable this feature
|
| 410 | by default. Or one can enable/disable it at module load time with module
|
| 411 | parameter metacopy=on/off. Lastly, there is also a per mount option
|
| 412 | metacopy=on/off to enable/disable this feature per mount.
|
| 413 |
|
| 414 | Do not use metacopy=on with untrusted upper/lower directories. Otherwise
|
| 415 | it is possible that an attacker can create a handcrafted file with
|
| 416 | appropriate REDIRECT and METACOPY xattrs, and gain access to file on lower
|
| 417 | pointed by REDIRECT. This should not be possible on local system as setting
|
| 418 | "trusted." xattrs will require CAP_SYS_ADMIN. But it should be possible
|
| 419 | for untrusted layers like from a pen drive.
|
| 420 |
|
| 421 | Note: redirect_dir={off|nofollow|follow[*]} and nfs_export=on mount options
|
| 422 | conflict with metacopy=on, and will result in an error.
|
| 423 |
|
| 424 | [*] redirect_dir=follow only conflicts with metacopy=on if upperdir=... is
|
| 425 | given.
|
| 426 |
|
| 427 |
|
| 428 | Data-only lower layers
|
| 429 | ----------------------
|
| 430 |
|
| 431 | With "metacopy" feature enabled, an overlayfs regular file may be a composition
|
| 432 | of information from up to three different layers:
|
| 433 |
|
| 434 | 1) metadata from a file in the upper layer
|
| 435 |
|
| 436 | 2) st_ino and st_dev object identifier from a file in a lower layer
|
| 437 |
|
| 438 | 3) data from a file in another lower layer (further below)
|
| 439 |
|
| 440 | The "lower data" file can be on any lower layer, except from the top most
|
| 441 | lower layer.
|
| 442 |
|
| 443 | Below the topmost lower layer, any number of lowermost layers may be defined
|
| 444 | as "data-only" lower layers, using double colon ("::") separators.
|
| 445 | A normal lower layer is not allowed to be below a data-only layer, so single
|
| 446 | colon separators are not allowed to the right of double colon ("::") separators.
|
| 447 |
|
| 448 |
|
| 449 | For example::
|
| 450 |
|
| 451 | mount -t overlay overlay -olowerdir=/l1:/l2:/l3::/do1::/do2 /merged
|
| 452 |
|
| 453 | The paths of files in the "data-only" lower layers are not visible in the
|
| 454 | merged overlayfs directories and the metadata and st_ino/st_dev of files
|
| 455 | in the "data-only" lower layers are not visible in overlayfs inodes.
|
| 456 |
|
| 457 | Only the data of the files in the "data-only" lower layers may be visible
|
| 458 | when a "metacopy" file in one of the lower layers above it, has a "redirect"
|
| 459 | to the absolute path of the "lower data" file in the "data-only" lower layer.
|
| 460 |
|
| 461 | Instead of explicitly enabling "metacopy=on" it is sufficient to specify at
|
| 462 | least one data-only layer to enable redirection of data to a data-only layer.
|
| 463 | In this case other forms of metacopy are rejected. Note: this way, data-only
|
| 464 | layers may be used together with "userxattr", in which case careful attention
|
| 465 | must be given to privileges needed to change the "user.overlay.redirect" xattr
|
| 466 | to prevent misuse.
|
| 467 |
|
| 468 | Since kernel version v6.8, "data-only" lower layers can also be added using
|
| 469 | the "datadir+" mount options and the fsconfig syscall from new mount api.
|
| 470 | For example::
|
| 471 |
|
| 472 | fsconfig(fs_fd, FSCONFIG_SET_STRING, "lowerdir+", "/l1", 0);
|
| 473 | fsconfig(fs_fd, FSCONFIG_SET_STRING, "lowerdir+", "/l2", 0);
|
| 474 | fsconfig(fs_fd, FSCONFIG_SET_STRING, "lowerdir+", "/l3", 0);
|
| 475 | fsconfig(fs_fd, FSCONFIG_SET_STRING, "datadir+", "/do1", 0);
|
| 476 | fsconfig(fs_fd, FSCONFIG_SET_STRING, "datadir+", "/do2", 0);
|
| 477 |
|
| 478 |
|
| 479 | Specifying layers via file descriptors
|
| 480 | --------------------------------------
|
| 481 |
|
| 482 | Since kernel v6.13, overlayfs supports specifying layers via file descriptors in
|
| 483 | addition to specifying them as paths. This feature is available for the
|
| 484 | "datadir+", "lowerdir+", "upperdir", and "workdir+" mount options with the
|
| 485 | fsconfig syscall from the new mount api::
|
| 486 |
|
| 487 | fsconfig(fs_fd, FSCONFIG_SET_FD, "lowerdir+", NULL, fd_lower1);
|
| 488 | fsconfig(fs_fd, FSCONFIG_SET_FD, "lowerdir+", NULL, fd_lower2);
|
| 489 | fsconfig(fs_fd, FSCONFIG_SET_FD, "lowerdir+", NULL, fd_lower3);
|
| 490 | fsconfig(fs_fd, FSCONFIG_SET_FD, "datadir+", NULL, fd_data1);
|
| 491 | fsconfig(fs_fd, FSCONFIG_SET_FD, "datadir+", NULL, fd_data2);
|
| 492 | fsconfig(fs_fd, FSCONFIG_SET_FD, "workdir", NULL, fd_work);
|
| 493 | fsconfig(fs_fd, FSCONFIG_SET_FD, "upperdir", NULL, fd_upper);
|
| 494 |
|
| 495 |
|
| 496 | fs-verity support
|
| 497 | -----------------
|
| 498 |
|
| 499 | During metadata copy up of a lower file, if the source file has
|
| 500 | fs-verity enabled and overlay verity support is enabled, then the
|
| 501 | digest of the lower file is added to the "trusted.overlay.metacopy"
|
| 502 | xattr. This is then used to verify the content of the lower file
|
| 503 | each the time the metacopy file is opened.
|
| 504 |
|
| 505 | When a layer containing verity xattrs is used, it means that any such
|
| 506 | metacopy file in the upper layer is guaranteed to match the content
|
| 507 | that was in the lower at the time of the copy-up. If at any time
|
| 508 | (during a mount, after a remount, etc) such a file in the lower is
|
| 509 | replaced or modified in any way, access to the corresponding file in
|
| 510 | overlayfs will result in EIO errors (either on open, due to overlayfs
|
| 511 | digest check, or from a later read due to fs-verity) and a detailed
|
| 512 | error is printed to the kernel logs. For more details of how fs-verity
|
| 513 | file access works, see :ref:`Documentation/filesystems/fsverity.rst
|
| 514 | <accessing_verity_files>`.
|
| 515 |
|
| 516 | Verity can be used as a general robustness check to detect accidental
|
| 517 | changes in the overlayfs directories in use. But, with additional care
|
| 518 | it can also give more powerful guarantees. For example, if the upper
|
| 519 | layer is fully trusted (by using dm-verity or something similar), then
|
| 520 | an untrusted lower layer can be used to supply validated file content
|
| 521 | for all metacopy files. If additionally the untrusted lower
|
| 522 | directories are specified as "Data-only", then they can only supply
|
| 523 | such file content, and the entire mount can be trusted to match the
|
| 524 | upper layer.
|
| 525 |
|
| 526 | This feature is controlled by the "verity" mount option, which
|
| 527 | supports these values:
|
| 528 |
|
| 529 | - "off":
|
| 530 | The metacopy digest is never generated or used. This is the
|
| 531 | default if verity option is not specified.
|
| 532 | - "on":
|
| 533 | Whenever a metacopy file specifies an expected digest, the
|
| 534 | corresponding data file must match the specified digest. When
|
| 535 | generating a metacopy file the verity digest will be set in it
|
| 536 | based on the source file (if it has one).
|
| 537 | - "require":
|
| 538 | Same as "on", but additionally all metacopy files must specify a
|
| 539 | digest (or EIO is returned on open). This means metadata copy up
|
| 540 | will only be used if the data file has fs-verity enabled,
|
| 541 | otherwise a full copy-up is used.
|
| 542 |
|
| 543 | Sharing and copying layers
|
| 544 | --------------------------
|
| 545 |
|
| 546 | Lower layers may be shared among several overlay mounts and that is indeed
|
| 547 | a very common practice. An overlay mount may use the same lower layer
|
| 548 | path as another overlay mount and it may use a lower layer path that is
|
| 549 | beneath or above the path of another overlay lower layer path.
|
| 550 |
|
| 551 | Using an upper layer path and/or a workdir path that are already used by
|
| 552 | another overlay mount is not allowed and may fail with EBUSY. Using
|
| 553 | partially overlapping paths is not allowed and may fail with EBUSY.
|
| 554 | If files are accessed from two overlayfs mounts which share or overlap the
|
| 555 | upper layer and/or workdir path, the behavior of the overlay is undefined,
|
| 556 | though it will not result in a crash or deadlock.
|
| 557 |
|
| 558 | Mounting an overlay using an upper layer path, where the upper layer path
|
| 559 | was previously used by another mounted overlay in combination with a
|
| 560 | different lower layer path, is allowed, unless the "index" or "metacopy"
|
| 561 | features are enabled.
|
| 562 |
|
| 563 | With the "index" feature, on the first time mount, an NFS file
|
| 564 | handle of the lower layer root directory, along with the UUID of the lower
|
| 565 | filesystem, are encoded and stored in the "trusted.overlay.origin" extended
|
| 566 | attribute on the upper layer root directory. On subsequent mount attempts,
|
| 567 | the lower root directory file handle and lower filesystem UUID are compared
|
| 568 | to the stored origin in upper root directory. On failure to verify the
|
| 569 | lower root origin, mount will fail with ESTALE. An overlayfs mount with
|
| 570 | "index" enabled will fail with EOPNOTSUPP if the lower filesystem
|
| 571 | does not support NFS export, lower filesystem does not have a valid UUID or
|
| 572 | if the upper filesystem does not support extended attributes.
|
| 573 |
|
| 574 | For the "metacopy" feature, there is no verification mechanism at
|
| 575 | mount time. So if same upper is mounted with different set of lower, mount
|
| 576 | probably will succeed but expect the unexpected later on. So don't do it.
|
| 577 |
|
| 578 | It is quite a common practice to copy overlay layers to a different
|
| 579 | directory tree on the same or different underlying filesystem, and even
|
| 580 | to a different machine. With the "index" feature, trying to mount
|
| 581 | the copied layers will fail the verification of the lower root file handle.
|
| 582 |
|
| 583 | Nesting overlayfs mounts
|
| 584 | ------------------------
|
| 585 |
|
| 586 | It is possible to use a lower directory that is stored on an overlayfs
|
| 587 | mount. For regular files this does not need any special care. However, files
|
| 588 | that have overlayfs attributes, such as whiteouts or "overlay.*" xattrs, will
|
| 589 | be interpreted by the underlying overlayfs mount and stripped out. In order to
|
| 590 | allow the second overlayfs mount to see the attributes they must be escaped.
|
| 591 |
|
| 592 | Overlayfs specific xattrs are escaped by using a special prefix of
|
| 593 | "overlay.overlay.". So, a file with a "trusted.overlay.overlay.metacopy" xattr
|
| 594 | in the lower dir will be exposed as a regular file with a
|
| 595 | "trusted.overlay.metacopy" xattr in the overlayfs mount. This can be nested by
|
| 596 | repeating the prefix multiple time, as each instance only removes one prefix.
|
| 597 |
|
| 598 | A lower dir with a regular whiteout will always be handled by the overlayfs
|
| 599 | mount, so to support storing an effective whiteout file in an overlayfs mount an
|
| 600 | alternative form of whiteout is supported. This form is a regular, zero-size
|
| 601 | file with the "overlay.whiteout" xattr set, inside a directory with the
|
| 602 | "overlay.opaque" xattr set to "x" (see `whiteouts and opaque directories`_).
|
| 603 | These alternative whiteouts are never created by overlayfs, but can be used by
|
| 604 | userspace tools (like containers) that generate lower layers.
|
| 605 | These alternative whiteouts can be escaped using the standard xattr escape
|
| 606 | mechanism in order to properly nest to any depth.
|
| 607 |
|
| 608 | Non-standard behavior
|
| 609 | ---------------------
|
| 610 |
|
| 611 | Current version of overlayfs can act as a mostly POSIX compliant
|
| 612 | filesystem.
|
| 613 |
|
| 614 | This is the list of cases that overlayfs doesn't currently handle:
|
| 615 |
|
| 616 | a) POSIX mandates updating st_atime for reads. This is currently not
|
| 617 | done in the case when the file resides on a lower layer.
|
| 618 |
|
| 619 | b) If a file residing on a lower layer is opened for read-only and then
|
| 620 | memory mapped with MAP_SHARED, then subsequent changes to the file are not
|
| 621 | reflected in the memory mapping.
|
| 622 |
|
| 623 | c) If a file residing on a lower layer is being executed, then opening that
|
| 624 | file for write or truncating the file will not be denied with ETXTBSY.
|
| 625 |
|
| 626 | The following options allow overlayfs to act more like a standards
|
| 627 | compliant filesystem:
|
| 628 |
|
| 629 | redirect_dir
|
| 630 | ````````````
|
| 631 |
|
| 632 | Enabled with the mount option or module option: "redirect_dir=on" or with
|
| 633 | the kernel config option CONFIG_OVERLAY_FS_REDIRECT_DIR=y.
|
| 634 |
|
| 635 | If this feature is disabled, then rename(2) on a lower or merged directory
|
| 636 | will fail with EXDEV ("Invalid cross-device link").
|
| 637 |
|
| 638 | index
|
| 639 | `````
|
| 640 |
|
| 641 | Enabled with the mount option or module option "index=on" or with the
|
| 642 | kernel config option CONFIG_OVERLAY_FS_INDEX=y.
|
| 643 |
|
| 644 | If this feature is disabled and a file with multiple hard links is copied
|
| 645 | up, then this will "break" the link. Changes will not be propagated to
|
| 646 | other names referring to the same inode.
|
| 647 |
|
| 648 | xino
|
| 649 | ````
|
| 650 |
|
| 651 | Enabled with the mount option "xino=auto" or "xino=on", with the module
|
| 652 | option "xino_auto=on" or with the kernel config option
|
| 653 | CONFIG_OVERLAY_FS_XINO_AUTO=y. Also implicitly enabled by using the same
|
| 654 | underlying filesystem for all layers making up the overlay.
|
| 655 |
|
| 656 | If this feature is disabled or the underlying filesystem doesn't have
|
| 657 | enough free bits in the inode number, then overlayfs will not be able to
|
| 658 | guarantee that the values of st_ino and st_dev returned by stat(2) and the
|
| 659 | value of d_ino returned by readdir(3) will act like on a normal filesystem.
|
| 660 | E.g. the value of st_dev may be different for two objects in the same
|
| 661 | overlay filesystem and the value of st_ino for filesystem objects may not be
|
| 662 | persistent and could change even while the overlay filesystem is mounted, as
|
| 663 | summarized in the `Inode properties`_ table above.
|
| 664 |
|
| 665 |
|
| 666 | Changes to underlying filesystems
|
| 667 | ---------------------------------
|
| 668 |
|
| 669 | Changes to the underlying filesystems while part of a mounted overlay
|
| 670 | filesystem are not allowed. If the underlying filesystem is changed,
|
| 671 | the behavior of the overlay is undefined, though it will not result in
|
| 672 | a crash or deadlock.
|
| 673 |
|
| 674 | Offline changes, when the overlay is not mounted, are allowed to the
|
| 675 | upper tree. Offline changes to the lower tree are only allowed if the
|
| 676 | "metacopy", "index", "xino" and "redirect_dir" features
|
| 677 | have not been used. If the lower tree is modified and any of these
|
| 678 | features has been used, the behavior of the overlay is undefined,
|
| 679 | though it will not result in a crash or deadlock.
|
| 680 |
|
| 681 | When the overlay NFS export feature is enabled, overlay filesystems
|
| 682 | behavior on offline changes of the underlying lower layer is different
|
| 683 | than the behavior when NFS export is disabled.
|
| 684 |
|
| 685 | On every copy_up, an NFS file handle of the lower inode, along with the
|
| 686 | UUID of the lower filesystem, are encoded and stored in an extended
|
| 687 | attribute "trusted.overlay.origin" on the upper inode.
|
| 688 |
|
| 689 | When the NFS export feature is enabled, a lookup of a merged directory,
|
| 690 | that found a lower directory at the lookup path or at the path pointed
|
| 691 | to by the "trusted.overlay.redirect" extended attribute, will verify
|
| 692 | that the found lower directory file handle and lower filesystem UUID
|
| 693 | match the origin file handle that was stored at copy_up time. If a
|
| 694 | found lower directory does not match the stored origin, that directory
|
| 695 | will not be merged with the upper directory.
|
| 696 |
|
| 697 |
|
| 698 |
|
| 699 | NFS export
|
| 700 | ----------
|
| 701 |
|
| 702 | When the underlying filesystems supports NFS export and the "nfs_export"
|
| 703 | feature is enabled, an overlay filesystem may be exported to NFS.
|
| 704 |
|
| 705 | With the "nfs_export" feature, on copy_up of any lower object, an index
|
| 706 | entry is created under the index directory. The index entry name is the
|
| 707 | hexadecimal representation of the copy up origin file handle. For a
|
| 708 | non-directory object, the index entry is a hard link to the upper inode.
|
| 709 | For a directory object, the index entry has an extended attribute
|
| 710 | "trusted.overlay.upper" with an encoded file handle of the upper
|
| 711 | directory inode.
|
| 712 |
|
| 713 | When encoding a file handle from an overlay filesystem object, the
|
| 714 | following rules apply:
|
| 715 |
|
| 716 | 1. For a non-upper object, encode a lower file handle from lower inode
|
| 717 | 2. For an indexed object, encode a lower file handle from copy_up origin
|
| 718 | 3. For a pure-upper object and for an existing non-indexed upper object,
|
| 719 | encode an upper file handle from upper inode
|
| 720 |
|
| 721 | The encoded overlay file handle includes:
|
| 722 |
|
| 723 | - Header including path type information (e.g. lower/upper)
|
| 724 | - UUID of the underlying filesystem
|
| 725 | - Underlying filesystem encoding of underlying inode
|
| 726 |
|
| 727 | This encoding format is identical to the encoding format file handles that
|
| 728 | are stored in extended attribute "trusted.overlay.origin".
|
| 729 |
|
| 730 | When decoding an overlay file handle, the following steps are followed:
|
| 731 |
|
| 732 | 1. Find underlying layer by UUID and path type information.
|
| 733 | 2. Decode the underlying filesystem file handle to underlying dentry.
|
| 734 | 3. For a lower file handle, lookup the handle in index directory by name.
|
| 735 | 4. If a whiteout is found in index, return ESTALE. This represents an
|
| 736 | overlay object that was deleted after its file handle was encoded.
|
| 737 | 5. For a non-directory, instantiate a disconnected overlay dentry from the
|
| 738 | decoded underlying dentry, the path type and index inode, if found.
|
| 739 | 6. For a directory, use the connected underlying decoded dentry, path type
|
| 740 | and index, to lookup a connected overlay dentry.
|
| 741 |
|
| 742 | Decoding a non-directory file handle may return a disconnected dentry.
|
| 743 | copy_up of that disconnected dentry will create an upper index entry with
|
| 744 | no upper alias.
|
| 745 |
|
| 746 | When overlay filesystem has multiple lower layers, a middle layer
|
| 747 | directory may have a "redirect" to lower directory. Because middle layer
|
| 748 | "redirects" are not indexed, a lower file handle that was encoded from the
|
| 749 | "redirect" origin directory, cannot be used to find the middle or upper
|
| 750 | layer directory. Similarly, a lower file handle that was encoded from a
|
| 751 | descendant of the "redirect" origin directory, cannot be used to
|
| 752 | reconstruct a connected overlay path. To mitigate the cases of
|
| 753 | directories that cannot be decoded from a lower file handle, these
|
| 754 | directories are copied up on encode and encoded as an upper file handle.
|
| 755 | On an overlay filesystem with no upper layer this mitigation cannot be
|
| 756 | used NFS export in this setup requires turning off redirect follow (e.g.
|
| 757 | "redirect_dir=nofollow").
|
| 758 |
|
| 759 | The overlay filesystem does not support non-directory connectable file
|
| 760 | handles, so exporting with the 'subtree_check' exportfs configuration will
|
| 761 | cause failures to lookup files over NFS.
|
| 762 |
|
| 763 | When the NFS export feature is enabled, all directory index entries are
|
| 764 | verified on mount time to check that upper file handles are not stale.
|
| 765 | This verification may cause significant overhead in some cases.
|
| 766 |
|
| 767 | Note: the mount options index=off,nfs_export=on are conflicting for a
|
| 768 | read-write mount and will result in an error.
|
| 769 |
|
| 770 | Note: the mount option uuid=off can be used to replace UUID of the underlying
|
| 771 | filesystem in file handles with null, in order to relax the UUID checks. This
|
| 772 | can be useful in case the underlying disk is copied and the UUID of this copy
|
| 773 | is changed. This is only applicable if all lower directories are on
|
| 774 | the same filesystem, otherwise it will fallback to normal behaviour.
|
| 775 |
|
| 776 |
|
| 777 | UUID and fsid
|
| 778 | -------------
|
| 779 |
|
| 780 | The UUID of overlayfs instance itself and the fsid reported by statfs(2) are
|
| 781 | controlled by the "uuid" mount option, which supports these values:
|
| 782 |
|
| 783 | - "null":
|
| 784 | UUID of overlayfs is null. fsid is taken from upper most filesystem.
|
| 785 | - "off":
|
| 786 | UUID of overlayfs is null. fsid is taken from upper most filesystem.
|
| 787 | UUID of underlying layers is ignored and null used instead.
|
| 788 | - "on":
|
| 789 | UUID of overlayfs is generated and used to report a unique fsid.
|
| 790 | UUID is stored in xattr "trusted.overlay.uuid", making overlayfs fsid
|
| 791 | unique and persistent. This option requires an overlayfs with upper
|
| 792 | filesystem that supports xattrs.
|
| 793 | - "auto": (default)
|
| 794 | UUID is taken from xattr "trusted.overlay.uuid" if it exists.
|
| 795 | Upgrade to "uuid=on" on first time mount of new overlay filesystem that
|
| 796 | meets the prerequisites.
|
| 797 | Downgrade to "uuid=null" for existing overlay filesystems that were never
|
| 798 | mounted with "uuid=on".
|
| 799 |
|
| 800 |
|
| 801 | Durability and copy up
|
| 802 | ----------------------
|
| 803 |
|
| 804 | The fsync(2) system call ensures that the data and metadata of a file
|
| 805 | are safely written to the backing storage, which is expected to
|
| 806 | guarantee the existence of the information post system crash.
|
| 807 |
|
| 808 | Without an fsync(2) call, there is no guarantee that the observed
|
| 809 | data after a system crash will be either the old or the new data, but
|
| 810 | in practice, the observed data after crash is often the old or new data
|
| 811 | or a mix of both.
|
| 812 |
|
| 813 | When an overlayfs file is modified for the first time, copy up will
|
| 814 | create a copy of the lower file and its parent directories in the upper
|
| 815 | layer. Since the Linux filesystem API does not enforce any particular
|
| 816 | ordering on storing changes without explicit fsync(2) calls, in case
|
| 817 | of a system crash, the upper file could end up with no data at all
|
| 818 | (i.e. zeros), which would be an unusual outcome. To avoid this
|
| 819 | experience, overlayfs calls fsync(2) on the upper file before completing
|
| 820 | data copy up with rename(2) or link(2) to make the copy up "atomic".
|
| 821 |
|
| 822 | By default, overlayfs does not explicitly call fsync(2) on copied up
|
| 823 | directories or on metadata-only copy up, so it provides no guarantee to
|
| 824 | persist the user's modification unless the user calls fsync(2).
|
| 825 | The fsync during copy up only guarantees that if a copy up is observed
|
| 826 | after a crash, the observed data is not zeroes or intermediate values
|
| 827 | from the copy up staging area.
|
| 828 |
|
| 829 | On traditional local filesystems with a single journal (e.g. ext4, xfs),
|
| 830 | fsync on a file also persists the parent directory changes, because they
|
| 831 | are usually modified in the same transaction, so metadata durability during
|
| 832 | data copy up effectively comes for free. Overlayfs further limits risk by
|
| 833 | disallowing network filesystems as upper layer.
|
| 834 |
|
| 835 | Overlayfs can be tuned to prefer performance or durability when storing
|
| 836 | to the underlying upper layer. This is controlled by the "fsync" mount
|
| 837 | option, which supports these values:
|
| 838 |
|
| 839 | - "auto": (default)
|
| 840 | Call fsync(2) on upper file before completion of data copy up.
|
| 841 | No explicit fsync(2) on directory or metadata-only copy up.
|
| 842 | - "strict":
|
| 843 | Call fsync(2) on upper file and directories before completion of any
|
| 844 | copy up.
|
| 845 | - "volatile": [*]
|
| 846 | Prefer performance over durability (see `Volatile mount`_)
|
| 847 |
|
| 848 | [*] The mount option "volatile" is an alias to "fsync=volatile".
|
| 849 |
|
| 850 |
|
| 851 | Volatile mount
|
| 852 | --------------
|
| 853 |
|
| 854 | This is enabled with the "volatile" mount option. Volatile mounts are not
|
| 855 | guaranteed to survive a crash. It is strongly recommended that volatile
|
| 856 | mounts are only used if data written to the overlay can be recreated
|
| 857 | without significant effort.
|
| 858 |
|
| 859 | The advantage of mounting with the "volatile" option is that all forms of
|
| 860 | sync calls to the upper filesystem are omitted.
|
| 861 |
|
| 862 | In order to avoid giving a false sense of safety, the syncfs (and fsync)
|
| 863 | semantics of volatile mounts are slightly different than that of the rest of
|
| 864 | VFS. If any writeback error occurs on the upperdir's filesystem after a
|
| 865 | volatile mount takes place, all sync functions will return an error. Once this
|
| 866 | condition is reached, the filesystem will not recover, and every subsequent sync
|
| 867 | call will return an error, even if the upperdir has not experienced a new error
|
| 868 | since the last sync call.
|
| 869 |
|
| 870 | When overlay is mounted with "volatile" option, the directory
|
| 871 | "$workdir/work/incompat/volatile" is created. During next mount, overlay
|
| 872 | checks for this directory and refuses to mount if present. This is a strong
|
| 873 | indicator that the user should discard upper and work directories and create
|
| 874 | fresh ones. In very limited cases where the user knows that the system has
|
| 875 | not crashed and contents of upperdir are intact, the "volatile" directory
|
| 876 | can be removed.
|
| 877 |
|
| 878 |
|
| 879 | User xattr
|
| 880 | ----------
|
| 881 |
|
| 882 | The "-o userxattr" mount option forces overlayfs to use the
|
| 883 | "user.overlay." xattr namespace instead of "trusted.overlay.". This is
|
| 884 | useful for unprivileged mounting of overlayfs.
|
| 885 |
|
| 886 |
|
| 887 | Testsuite
|
| 888 | ---------
|
| 889 |
|
| 890 | There's a testsuite originally developed by David Howells and currently
|
| 891 | maintained by Amir Goldstein at:
|
| 892 |
|
| 893 | https://github.com/amir73il/unionmount-testsuite.git
|
| 894 |
|
| 895 | Run as root::
|
| 896 |
|
| 897 | # cd unionmount-testsuite
|
| 898 | # ./run --ov --verify
|