Merge branch 'vfs-scale-working' of git://git.kernel.org/pub/scm/linux/kernel/git/npiggin/linux-npiggin

b4a45f5f · Linus Torvalds <[email protected]> · 2011-01-07 08:56:33 -08:00

* 'vfs-scale-working' of git://git.kernel.org/pub/scm/linux/kernel/git/npiggin/linux-npiggin: (57 commits)
  fs: scale mntget/mntput
  fs: rename vfsmount counter helpers
  fs: implement faster dentry memcmp
  fs: prefetch inode data in dcache lookup
  fs: improve scalability of pseudo filesystems
  fs: dcache per-inode inode alias locking
  fs: dcache per-bucket dcache hash locking
  bit_spinlock: add required includes
  kernel: add bl_list
  xfs: provide simple rcu-walk ACL implementation
  btrfs: provide simple rcu-walk ACL implementation
  ext2,3,4: provide simple rcu-walk ACL implementation
  fs: provide simple rcu-walk generic_check_acl implementation
  fs: provide rcu-walk aware permission i_ops
  fs: rcu-walk aware d_revalidate method
  fs: cache optimise dentry and inode for rcu-walk
  fs: dcache reduce branches in lookup path
  fs: dcache remove d_mounted
  fs: fs_struct use seqlock
  fs: rcu-walk for path lookup
  ...

Commit: b4a45f5fe8078bfc10837dbd5b98735058bc4698
Parents: 01539ba2 · b3e19d92

Merge commit. Changes below are compared with the first parent: 01539ba2.

Changes

212 files changed · +4774 −2014

File	Status	Add	Del	Content
Documentation/filesystems/Locking	modified	+16	−13	View file
Documentation/filesystems/dentry-locking.txt	deleted	+0	−174	View before deletion
Documentation/filesystems/path-lookup.txt	added	+382	−0	View file
Documentation/filesystems/porting	modified	+68	−1	View file
Documentation/filesystems/vfs.txt	modified	+57	−17	View file
arch/ia64/kernel/perfmon.c	modified	+3	−3	View file
arch/powerpc/platforms/cell/spufs/inode.c	modified	+12	−6	View file
drivers/infiniband/hw/ipath/ipath_fs.c	modified	+2	−6	View file
drivers/infiniband/hw/qib/qib_fs.c	modified	+1	−4	View file
drivers/mtd/mtdchar.c	modified	+1	−1	View file
drivers/staging/autofs/root.c	modified	+5	−2	View file
drivers/staging/pohmelfs/inode.c	modified	+8	−1	View file
drivers/staging/pohmelfs/path_entry.c	modified	+12	−5	View file
drivers/staging/smbfs/cache.c	modified	+8	−8	View file
drivers/staging/smbfs/dir.c	modified	+33	−17	View file
drivers/staging/smbfs/file.c	modified	+4	−1	View file
drivers/staging/smbfs/inode.c	modified	+8	−1	View file
drivers/usb/core/inode.c	modified	+7	−5	View file
fs/9p/acl.c	modified	+4	−1	View file
fs/9p/acl.h	modified	+1	−1	View file
fs/9p/vfs_dentry.c	modified	+2	−2	View file
fs/9p/vfs_inode.c	modified	+23	−16	View file
fs/adfs/dir.c	modified	+8	−5	View file
fs/adfs/super.c	modified	+9	−2	View file
fs/affs/amigaffs.c	modified	+2	−2	View file
fs/affs/namei.c	modified	+41	−27	View file
fs/affs/super.c	modified	+9	−2	View file
fs/afs/dir.c	modified	+7	−3	View file
fs/afs/internal.h	modified	+1	−1	View file
fs/afs/security.c	modified	+5	−2	View file
fs/afs/super.c	modified	+9	−1	View file
fs/anon_inodes.c	modified	+3	−3	View file
fs/autofs4/autofs_i.h	modified	+11	−10	View file
fs/autofs4/expire.c	modified	+77	−64	View file
fs/autofs4/inode.c	modified	+1	−1	View file
fs/autofs4/root.c	modified	+57	−34	View file
fs/autofs4/waitq.c	modified	+18	−5	View file
fs/bad_inode.c	modified	+4	−1	View file
fs/befs/linuxvfs.c	modified	+8	−2	View file
fs/bfs/inode.c	modified	+8	−1	View file
fs/block_dev.c	modified	+8	−1	View file
fs/btrfs/acl.c	modified	+13	−8	View file
fs/btrfs/ctree.h	modified	+1	−1	View file
fs/btrfs/export.c	modified	+2	−2	View file
fs/btrfs/inode.c	modified	+12	−5	View file
fs/ceph/dir.c	modified	+18	−10	View file
fs/ceph/inode.c	modified	+26	−12	View file
fs/ceph/mds_client.c	modified	+1	−1	View file
fs/ceph/super.h	modified	+1	−1	View file
fs/cifs/cifsfs.c	modified	+13	−3	View file
fs/cifs/dir.c	modified	+44	−33	View file
fs/cifs/inode.c	modified	+7	−7	View file
fs/cifs/link.c	modified	+2	−2	View file
fs/cifs/readdir.c	modified	+3	−3	View file
fs/coda/cache.c	modified	+2	−2	View file
fs/coda/dir.c	modified	+14	−6	View file
fs/coda/inode.c	modified	+8	−1	View file
fs/coda/pioctl.c	modified	+4	−2	View file
fs/configfs/configfs_internal.h	modified	+2	−2	View file
fs/configfs/dir.c	modified	+12	−12	View file
fs/configfs/inode.c	modified	+2	−6	View file
fs/dcache.c	modified	+981	−394	View file
fs/ecryptfs/dentry.c	modified	+7	−2	View file
fs/ecryptfs/inode.c	modified	+7	−5	View file
fs/ecryptfs/main.c	modified	+2	−2	View file
fs/ecryptfs/super.c	modified	+11	−1	View file
fs/efs/super.c	modified	+8	−1	View file
fs/exofs/super.c	modified	+8	−1	View file
fs/exportfs/expfs.c	modified	+8	−6	View file
fs/ext2/acl.c	modified	+9	−2	View file
fs/ext2/acl.h	modified	+1	−1	View file
fs/ext2/super.c	modified	+8	−1	View file
fs/ext3/acl.c	modified	+9	−2	View file
fs/ext3/acl.h	modified	+1	−1	View file
fs/ext3/super.c	modified	+8	−1	View file
fs/ext4/acl.c	modified	+9	−2	View file
fs/ext4/acl.h	modified	+1	−1	View file
fs/ext4/super.c	modified	+8	−1	View file
fs/fat/inode.c	modified	+10	−3	View file
fs/fat/namei_msdos.c	modified	+13	−10	View file
fs/fat/namei_vfat.c	modified	+35	−20	View file
fs/filesystems.c	modified	+3	−0	View file
fs/freevxfs/vxfs_inode.c	modified	+8	−1	View file
fs/fs_struct.c	modified	+26	−10	View file
fs/fuse/dir.c	modified	+13	−5	View file
fs/fuse/inode.c	modified	+10	−3	View file
fs/generic_acl.c	modified	+13	−7	View file
fs/gfs2/acl.c	modified	+4	−1	View file
fs/gfs2/acl.h	modified	+1	−1	View file
fs/gfs2/dentry.c	modified	+16	−6	View file
fs/gfs2/export.c	modified	+2	−2	View file
fs/gfs2/file.c	modified	+1	−1	View file
fs/gfs2/inode.c	modified	+2	−2	View file
fs/gfs2/inode.h	modified	+1	−1	View file
fs/gfs2/ops_fstype.c	modified	+1	−1	View file
fs/gfs2/ops_inode.c	modified	+12	−8	View file
fs/gfs2/super.c	modified	+8	−1	View file
fs/hfs/dir.c	modified	+1	−1	View file
fs/hfs/hfs_fs.h	modified	+6	−2	View file
fs/hfs/string.c	modified	+9	−8	View file
fs/hfs/super.c	modified	+9	−2	View file
fs/hfs/sysdep.c	modified	+6	−1	View file
fs/hfsplus/dir.c	modified	+1	−1	View file
fs/hfsplus/hfsplus_fs.h	modified	+6	−2	View file
fs/hfsplus/super.c	modified	+10	−2	View file
fs/hfsplus/unicode.c	modified	+11	−7	View file
fs/hostfs/hostfs_kern.c	modified	+25	−19	View file
fs/hpfs/dentry.c	modified	+17	−10	View file
fs/hpfs/namei.c	modified	+1	−1	View file
fs/hpfs/super.c	modified	+8	−1	View file
fs/hppfs/hppfs.c	modified	+8	−1	View file
fs/hugetlbfs/inode.c	modified	+8	−1	View file
fs/inode.c	modified	+32	−18	View file
fs/internal.h	modified	+1	−0	View file
fs/isofs/inode.c	modified	+74	−57	View file
fs/isofs/namei.c	modified	+3	−2	View file
fs/jffs2/acl.c	modified	+4	−1	View file
fs/jffs2/acl.h	modified	+1	−1	View file
fs/jffs2/super.c	modified	+8	−1	View file
fs/jfs/acl.c	modified	+6	−2	View file
fs/jfs/jfs_acl.h	modified	+1	−1	View file
fs/jfs/namei.c	modified	+48	−15	View file
fs/jfs/super.c	modified	+10	−2	View file
fs/libfs.c	modified	+39	−24	View file
fs/locks.c	modified	+1	−1	View file
fs/logfs/dir.c	modified	+4	−2	View file
fs/logfs/inode.c	modified	+8	−1	View file
fs/minix/inode.c	modified	+8	−1	View file
fs/minix/namei.c	modified	+1	−1	View file
fs/namei.c	modified	+682	−175	View file
fs/namespace.c	modified	+235	−56	View file
fs/ncpfs/dir.c	modified	+38	−50	View file
fs/ncpfs/inode.c	modified	+10	−2	View file
fs/ncpfs/ncplib_kernel.h	modified	+8	−8	View file
fs/nfs/dir.c	modified	+18	−14	View file
fs/nfs/getroot.c	modified	+6	−4	View file
fs/nfs/inode.c	modified	+8	−1	View file
fs/nfs/namespace.c	modified	+13	−4	View file
fs/nfs/unlink.c	modified	+1	−1	View file
fs/nfsd/vfs.c	modified	+2	−3	View file
fs/nilfs2/inode.c	modified	+7	−3	View file
fs/nilfs2/nilfs.h	modified	+1	−1	View file
fs/nilfs2/super.c	modified	+10	−2	View file
fs/notify/fsnotify.c	modified	+5	−3	View file
fs/ntfs/inode.c	modified	+8	−1	View file
fs/ocfs2/acl.c	modified	+6	−2	View file
fs/ocfs2/acl.h	modified	+1	−1	View file
fs/ocfs2/dcache.c	modified	+14	−6	View file
fs/ocfs2/dlmfs/dlmfs.c	modified	+8	−1	View file
fs/ocfs2/export.c	modified	+2	−2	View file
fs/ocfs2/file.c	modified	+5	−2	View file
fs/ocfs2/file.h	modified	+1	−1	View file
fs/ocfs2/namei.c	modified	+5	−5	View file
fs/ocfs2/super.c	modified	+8	−1	View file
fs/openpromfs/inode.c	modified	+8	−1	View file
fs/pipe.c	modified	+8	−4	View file
fs/pnode.c	modified	+2	−2	View file
fs/proc/base.c	modified	+37	−16	View file
fs/proc/generic.c	modified	+2	−2	View file
fs/proc/inode.c	modified	+8	−1	View file
fs/proc/proc_sysctl.c	modified	+21	−10	View file
fs/qnx4/inode.c	modified	+8	−1	View file
fs/reiserfs/super.c	modified	+8	−1	View file
fs/reiserfs/xattr.c	modified	+13	−5	View file
fs/romfs/super.c	modified	+8	−1	View file
fs/squashfs/super.c	modified	+8	−1	View file
fs/super.c	modified	+3	−2	View file
fs/sysfs/dir.c	modified	+7	−3	View file
fs/sysfs/inode.c	modified	+8	−3	View file
fs/sysfs/sysfs.h	modified	+1	−1	View file
fs/sysv/inode.c	modified	+8	−1	View file
fs/sysv/namei.c	modified	+3	−2	View file
fs/sysv/super.c	modified	+1	−1	View file
fs/ubifs/super.c	modified	+9	−1	View file
fs/udf/super.c	modified	+8	−1	View file
fs/ufs/super.c	modified	+8	−1	View file
fs/xfs/linux-2.6/xfs_acl.c	modified	+9	−2	View file
fs/xfs/xfs_acl.h	modified	+1	−1	View file
fs/xfs/xfs_iget.c	modified	+12	−1	View file
include/linux/bit_spinlock.h	modified	+4	−0	View file
include/linux/coda_linux.h	modified	+1	−1	View file
include/linux/dcache.h	modified	+128	−115	View file
include/linux/fs.h	modified	+39	−24	View file
include/linux/fs_struct.h	modified	+3	−0	View file
include/linux/fsnotify.h	modified	+0	−2	View file
include/linux/fsnotify_backend.h	modified	+7	−4	View file
include/linux/generic_acl.h	modified	+1	−1	View file
include/linux/list_bl.h	added	+144	−0	View file
include/linux/mount.h	modified	+17	−36	View file
include/linux/namei.h	modified	+10	−6	View file
include/linux/ncp_fs.h	modified	+2	−2	View file
include/linux/nfs_fs.h	modified	+1	−1	View file
include/linux/path.h	modified	+2	−0	View file
include/linux/posix_acl.h	modified	+19	−0	View file
include/linux/rculist_bl.h	added	+127	−0	View file
include/linux/reiserfs_xattr.h	modified	+1	−1	View file
include/linux/security.h	modified	+7	−1	View file
include/linux/seqlock.h	modified	+73	−7	View file
include/linux/slab.h	modified	+0	−2	View file
ipc/mqueue.c	modified	+8	−1	View file
kernel/cgroup.c	modified	+39	−15	View file
mm/filemap.c	modified	+0	−3	View file
mm/shmem.c	modified	+8	−1	View file
mm/slab.c	modified	+1	−31	View file
mm/slob.c	modified	+0	−5	View file
mm/slub.c	modified	+0	−40	View file
mm/util.c	modified	+0	−21	View file
net/socket.c	modified	+20	−4	View file
net/sunrpc/rpc_pipe.c	modified	+11	−3	View file
security/security.c	modified	+9	−0	View file
security/selinux/selinuxfs.c	modified	+10	−6	View file
security/tomoyo/realpath.c	modified	+1	−0	View file

Only the first 600 diff lines are shown. Download the complete patch.

Documentation/filesystems/Locking

View file

diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
index 33fa3e5..977d891 100644
--- a/Documentation/filesystems/Locking
+++ b/Documentation/filesystems/Locking
@@ -9,22 +9,25 @@ be able to use diff(1).
 
 --------------------------- dentry_operations --------------------------
 prototypes:
-	int (*d_revalidate)(struct dentry *, int);
-	int (*d_hash) (struct dentry *, struct qstr *);
-	int (*d_compare) (struct dentry *, struct qstr *, struct qstr *);
+	int (*d_revalidate)(struct dentry *, struct nameidata *);
+	int (*d_hash)(const struct dentry *, const struct inode *,
+			struct qstr *);
+	int (*d_compare)(const struct dentry *, const struct inode *,
+			const struct dentry *, const struct inode *,
+			unsigned int, const char *, const struct qstr *);
 	int (*d_delete)(struct dentry *);
 	void (*d_release)(struct dentry *);
 	void (*d_iput)(struct dentry *, struct inode *);
 	char *(*d_dname)((struct dentry *dentry, char *buffer, int buflen);
 
 locking rules:
-		dcache_lock	rename_lock	->d_lock	may block
-d_revalidate:	no		no		no		yes
-d_hash		no		no		no		yes
-d_compare:	no		yes		no		no 
-d_delete:	yes		no		yes		no
-d_release:	no		no		no		yes
-d_iput:		no		no		no		yes
+		rename_lock	->d_lock	may block	rcu-walk
+d_revalidate:	no		no		yes (ref-walk)	maybe
+d_hash		no		no		no		maybe
+d_compare:	yes		no		no		maybe
+d_delete:	no		yes		no		no
+d_release:	no		no		yes		no
+d_iput:		no		no		yes		no
 d_dname:	no		no		no		no
 
 --------------------------- inode_operations --------------------------- 
@@ -44,8 +47,8 @@ ata *);
 	void * (*follow_link) (struct dentry *, struct nameidata *);
 	void (*put_link) (struct dentry *, struct nameidata *, void *);
 	void (*truncate) (struct inode *);
-	int (*permission) (struct inode *, int, struct nameidata *);
-	int (*check_acl)(struct inode *, int);
+	int (*permission) (struct inode *, int, unsigned int);
+	int (*check_acl)(struct inode *, int, unsigned int);
 	int (*setattr) (struct dentry *, struct iattr *);
 	int (*getattr) (struct vfsmount *, struct dentry *, struct kstat *);
 	int (*setxattr) (struct dentry *, const char *,const void *,size_t,int);
@@ -73,7 +76,7 @@ follow_link:	no
 put_link:	no
 truncate:	yes		(see below)
 setattr:	yes
-permission:	no
+permission:	no (may not block if called in rcu-walk mode)
 check_acl:	no
 getattr:	no
 setxattr:	yes

Documentation/filesystems/dentry-locking.txt

View before deletion

diff --git a/Documentation/filesystems/dentry-locking.txt b/Documentation/filesystems/dentry-locking.txt
deleted file mode 100644
index 79334ed..0000000
--- a/Documentation/filesystems/dentry-locking.txt
+++ /dev/null
@@ -1,174 +0,0 @@
-RCU-based dcache locking model
-==============================
-
-On many workloads, the most common operation on dcache is to look up a
-dentry, given a parent dentry and the name of the child. Typically,
-for every open(), stat() etc., the dentry corresponding to the
-pathname will be looked up by walking the tree starting with the first
-component of the pathname and using that dentry along with the next
-component to look up the next level and so on. Since it is a frequent
-operation for workloads like multiuser environments and web servers,
-it is important to optimize this path.
-
-Prior to 2.5.10, dcache_lock was acquired in d_lookup and thus in
-every component during path look-up. Since 2.5.10 onwards, fast-walk
-algorithm changed this by holding the dcache_lock at the beginning and
-walking as many cached path component dentries as possible. This
-significantly decreases the number of acquisition of
-dcache_lock. However it also increases the lock hold time
-significantly and affects performance in large SMP machines. Since
-2.5.62 kernel, dcache has been using a new locking model that uses RCU
-to make dcache look-up lock-free.
-
-The current dcache locking model is not very different from the
-existing dcache locking model. Prior to 2.5.62 kernel, dcache_lock
-protected the hash chain, d_child, d_alias, d_lru lists as well as
-d_inode and several other things like mount look-up. RCU-based changes
-affect only the way the hash chain is protected. For everything else
-the dcache_lock must be taken for both traversing as well as
-updating. The hash chain updates too take the dcache_lock.  The
-significant change is the way d_lookup traverses the hash chain, it
-doesn't acquire the dcache_lock for this and rely on RCU to ensure
-that the dentry has not been *freed*.
-
-
-Dcache locking details
-======================
-
-For many multi-user workloads, open() and stat() on files are very
-frequently occurring operations. Both involve walking of path names to
-find the dentry corresponding to the concerned file. In 2.4 kernel,
-dcache_lock was held during look-up of each path component. Contention
-and cache-line bouncing of this global lock caused significant
-scalability problems. With the introduction of RCU in Linux kernel,
-this was worked around by making the look-up of path components during
-path walking lock-free.
-
-
-Safe lock-free look-up of dcache hash table
-===========================================
-
-Dcache is a complex data structure with the hash table entries also
-linked together in other lists. In 2.4 kernel, dcache_lock protected
-all the lists. We applied RCU only on hash chain walking. The rest of
-the lists are still protected by dcache_lock.  Some of the important
-changes are :
-
-1. The deletion from hash chain is done using hlist_del_rcu() macro
-   which doesn't initialize next pointer of the deleted dentry and
-   this allows us to walk safely lock-free while a deletion is
-   happening.
-
-2. Insertion of a dentry into the hash table is done using
-   hlist_add_head_rcu() which take care of ordering the writes - the
-   writes to the dentry must be visible before the dentry is
-   inserted. This works in conjunction with hlist_for_each_rcu(),
-   which has since been replaced by hlist_for_each_entry_rcu(), while
-   walking the hash chain. The only requirement is that all
-   initialization to the dentry must be done before
-   hlist_add_head_rcu() since we don't have dcache_lock protection
-   while traversing the hash chain. This isn't different from the
-   existing code.
-
-3. The dentry looked up without holding dcache_lock by cannot be
-   returned for walking if it is unhashed. It then may have a NULL
-   d_inode or other bogosity since RCU doesn't protect the other
-   fields in the dentry. We therefore use a flag DCACHE_UNHASHED to
-   indicate unhashed dentries and use this in conjunction with a
-   per-dentry lock (d_lock). Once looked up without the dcache_lock,
-   we acquire the per-dentry lock (d_lock) and check if the dentry is
-   unhashed. If so, the look-up is failed. If not, the reference count
-   of the dentry is increased and the dentry is returned.
-
-4. Once a dentry is looked up, it must be ensured during the path walk
-   for that component it doesn't go away. In pre-2.5.10 code, this was
-   done holding a reference to the dentry. dcache_rcu does the same.
-   In some sense, dcache_rcu path walking looks like the pre-2.5.10
-   version.
-
-5. All dentry hash chain updates must take the dcache_lock as well as
-   the per-dentry lock in that order. dput() does this to ensure that
-   a dentry that has just been looked up in another CPU doesn't get
-   deleted before dget() can be done on it.
-
-6. There are several ways to do reference counting of RCU protected
-   objects. One such example is in ipv4 route cache where deferred
-   freeing (using call_rcu()) is done as soon as the reference count
-   goes to zero. This cannot be done in the case of dentries because
-   tearing down of dentries require blocking (dentry_iput()) which
-   isn't supported from RCU callbacks. Instead, tearing down of
-   dentries happen synchronously in dput(), but actual freeing happens
-   later when RCU grace period is over. This allows safe lock-free
-   walking of the hash chains, but a matched dentry may have been
-   partially torn down. The checking of DCACHE_UNHASHED flag with
-   d_lock held detects such dentries and prevents them from being
-   returned from look-up.
-
-
-Maintaining POSIX rename semantics
-==================================
-
-Since look-up of dentries is lock-free, it can race against a
-concurrent rename operation. For example, during rename of file A to
-B, look-up of either A or B must succeed.  So, if look-up of B happens
-after A has been removed from the hash chain but not added to the new
-hash chain, it may fail.  Also, a comparison while the name is being
-written concurrently by a rename may result in false positive matches
-violating rename semantics.  Issues related to race with rename are
-handled as described below :
-
-1. Look-up can be done in two ways - d_lookup() which is safe from
-   simultaneous renames and __d_lookup() which is not.  If
-   __d_lookup() fails, it must be followed up by a d_lookup() to
-   correctly determine whether a dentry is in the hash table or
-   not. d_lookup() protects look-ups using a sequence lock
-   (rename_lock).
-
-2. The name associated with a dentry (d_name) may be changed if a
-   rename is allowed to happen simultaneously. To avoid memcmp() in
-   __d_lookup() go out of bounds due to a rename and false positive
-   comparison, the name comparison is done while holding the
-   per-dentry lock. This prevents concurrent renames during this
-   operation.
-
-3. Hash table walking during look-up may move to a different bucket as
-   the current dentry is moved to a different bucket due to rename.
-   But we use hlists in dcache hash table and they are
-   null-terminated.  So, even if a dentry moves to a different bucket,
-   hash chain walk will terminate. [with a list_head list, it may not
-   since termination is when the list_head in the original bucket is
-   reached].  Since we redo the d_parent check and compare name while
-   holding d_lock, lock-free look-up will not race against d_move().
-
-4. There can be a theoretical race when a dentry keeps coming back to
-   original bucket due to double moves. Due to this look-up may
-   consider that it has never moved and can end up in a infinite loop.
-   But this is not any worse that theoretical livelocks we already
-   have in the kernel.
-
-
-Important guidelines for filesystem developers related to dcache_rcu
-====================================================================
-
-1. Existing dcache interfaces (pre-2.5.62) exported to filesystem
-   don't change. Only dcache internal implementation changes. However
-   filesystems *must not* delete from the dentry hash chains directly
-   using the list macros like allowed earlier. They must use dcache
-   APIs like d_drop() or __d_drop() depending on the situation.
-
-2. d_flags is now protected by a per-dentry lock (d_lock). All access
-   to d_flags must be protected by it.
-
-3. For a hashed dentry, checking of d_count needs to be protected by
-   d_lock.
-
-
-Papers and other documentation on dcache locking
-================================================
-
-1. Scaling dcache with RCU (http://linuxjournal.com/article.php?sid=7124).
-
-2. http://lse.sourceforge.net/locking/dcache/dcache.html
-
-
-

Documentation/filesystems/path-lookup.txt

View file

diff --git a/Documentation/filesystems/path-lookup.txt b/Documentation/filesystems/path-lookup.txt
new file mode 100644
index 0000000..eb59c8b
--- /dev/null
+++ b/Documentation/filesystems/path-lookup.txt
@@ -0,0 +1,382 @@
+Path walking and name lookup locking
+====================================
+
+Path resolution is the finding a dentry corresponding to a path name string, by
+performing a path walk. Typically, for every open(), stat() etc., the path name
+will be resolved. Paths are resolved by walking the namespace tree, starting
+with the first component of the pathname (eg. root or cwd) with a known dentry,
+then finding the child of that dentry, which is named the next component in the
+path string. Then repeating the lookup from the child dentry and finding its
+child with the next element, and so on.
+
+Since it is a frequent operation for workloads like multiuser environments and
+web servers, it is important to optimize this code.
+
+Path walking synchronisation history:
+Prior to 2.5.10, dcache_lock was acquired in d_lookup (dcache hash lookup) and
+thus in every component during path look-up. Since 2.5.10 onwards, fast-walk
+algorithm changed this by holding the dcache_lock at the beginning and walking
+as many cached path component dentries as possible. This significantly
+decreases the number of acquisition of dcache_lock. However it also increases
+the lock hold time significantly and affects performance in large SMP machines.
+Since 2.5.62 kernel, dcache has been using a new locking model that uses RCU to
+make dcache look-up lock-free.
+
+All the above algorithms required taking a lock and reference count on the
+dentry that was looked up, so that may be used as the basis for walking the
+next path element. This is inefficient and unscalable. It is inefficient
+because of the locks and atomic operations required for every dentry element
+slows things down. It is not scalable because many parallel applications that
+are path-walk intensive tend to do path lookups starting from a common dentry
+(usually, the root "/" or current working directory). So contention on these
+common path elements causes lock and cacheline queueing.
+
+Since 2.6.38, RCU is used to make a significant part of the entire path walk
+(including dcache look-up) completely "store-free" (so, no locks, atomics, or
+even stores into cachelines of common dentries). This is known as "rcu-walk"
+path walking.
+
+Path walking overview
+=====================
+
+A name string specifies a start (root directory, cwd, fd-relative) and a
+sequence of elements (directory entry names), which together refer to a path in
+the namespace. A path is represented as a (dentry, vfsmount) tuple. The name
+elements are sub-strings, seperated by '/'.
+
+Name lookups will want to find a particular path that a name string refers to
+(usually the final element, or parent of final element). This is done by taking
+the path given by the name's starting point (which we know in advance -- eg.
+current->fs->cwd or current->fs->root) as the first parent of the lookup. Then
+iteratively for each subsequent name element, look up the child of the current
+parent with the given name and if it is not the desired entry, make it the
+parent for the next lookup.
+
+A parent, of course, must be a directory, and we must have appropriate
+permissions on the parent inode to be able to walk into it.
+
+Turning the child into a parent for the next lookup requires more checks and
+procedures. Symlinks essentially substitute the symlink name for the target
+name in the name string, and require some recursive path walking.  Mount points
+must be followed into (thus changing the vfsmount that subsequent path elements
+refer to), switching from the mount point path to the root of the particular
+mounted vfsmount. These behaviours are variously modified depending on the
+exact path walking flags.
+
+Path walking then must, broadly, do several particular things:
+- find the start point of the walk;
+- perform permissions and validity checks on inodes;
+- perform dcache hash name lookups on (parent, name element) tuples;
+- traverse mount points;
+- traverse symlinks;
+- lookup and create missing parts of the path on demand.
+
+Safe store-free look-up of dcache hash table
+============================================
+
+Dcache name lookup
+------------------
+In order to lookup a dcache (parent, name) tuple, we take a hash on the tuple
+and use that to select a bucket in the dcache-hash table. The list of entries
+in that bucket is then walked, and we do a full comparison of each entry
+against our (parent, name) tuple.
+
+The hash lists are RCU protected, so list walking is not serialised with
+concurrent updates (insertion, deletion from the hash). This is a standard RCU
+list application with the exception of renames, which will be covered below.
+
+Parent and name members of a dentry, as well as its membership in the dcache
+hash, and its inode are protected by the per-dentry d_lock spinlock. A
+reference is taken on the dentry (while the fields are verified under d_lock),
+and this stabilises its d_inode pointer and actual inode. This gives a stable
+point to perform the next step of our path walk against.
+
+These members are also protected by d_seq seqlock, although this offers
+read-only protection and no durability of results, so care must be taken when
+using d_seq for synchronisation (see seqcount based lookups, below).
+
+Renames
+-------
+Back to the rename case. In usual RCU protected lists, the only operations that
+will happen to an object is insertion, and then eventually removal from the
+list. The object will not be reused until an RCU grace period is complete.
+This ensures the RCU list traversal primitives can run over the object without
+problems (see RCU documentation for how this works).
+
+However when a dentry is renamed, its hash value can change, requiring it to be
+moved to a new hash list. Allocating and inserting a new alias would be
+expensive and also problematic for directory dentries. Latency would be far to
+high to wait for a grace period after removing the dentry and before inserting
+it in the new hash bucket. So what is done is to insert the dentry into the
+new list immediately.
+
+However, when the dentry's list pointers are updated to point to objects in the
+new list before waiting for a grace period, this can result in a concurrent RCU
+lookup of the old list veering off into the new (incorrect) list and missing
+the remaining dentries on the list.
+
+There is no fundamental problem with walking down the wrong list, because the
+dentry comparisons will never match. However it is fatal to miss a matching
+dentry. So a seqlock is used to detect when a rename has occurred, and so the
+lookup can be retried.
+
+         1      2      3
+        +---+  +---+  +---+
+hlist-->| N-+->| N-+->| N-+->
+head <--+-P |<-+-P |<-+-P |
+        +---+  +---+  +---+
+
+Rename of dentry 2 may require it deleted from the above list, and inserted
+into a new list. Deleting 2 gives the following list.
+
+         1             3
+        +---+         +---+     (don't worry, the longer pointers do not
+hlist-->| N-+-------->| N-+->    impose a measurable performance overhead
+head <--+-P |<--------+-P |      on modern CPUs)
+        +---+         +---+
+          ^      2      ^
+          |    +---+    |
+          |    | N-+----+
+          +----+-P |
+               +---+
+
+This is a standard RCU-list deletion, which leaves the deleted object's
+pointers intact, so a concurrent list walker that is currently looking at
+object 2 will correctly continue to object 3 when it is time to traverse the
+next object.
+
+However, when inserting object 2 onto a new list, we end up with this:
+
+         1             3
+        +---+         +---+
+hlist-->| N-+-------->| N-+->
+head <--+-P |<--------+-P |
+        +---+         +---+
+                 2
+               +---+
+               | N-+---->
+          <----+-P |
+               +---+
+
+Because we didn't wait for a grace period, there may be a concurrent lookup
+still at 2. Now when it follows 2's 'next' pointer, it will walk off into
+another list without ever having checked object 3.
+
+A related, but distinctly different, issue is that of rename atomicity versus
+lookup operations. If a file is renamed from 'A' to 'B', a lookup must only
+find either 'A' or 'B'. So if a lookup of 'A' returns NULL, a subsequent lookup
+of 'B' must succeed (note the reverse is not true).
+
+Between deleting the dentry from the old hash list, and inserting it on the new
+hash list, a lookup may find neither 'A' nor 'B' matching the dentry. The same
+rename seqlock is also used to cover this race in much the same way, by
+retrying a negative lookup result if a rename was in progress.
+
+Seqcount based lookups
+----------------------
+In refcount based dcache lookups, d_lock is used to serialise access to
+the dentry, stabilising it while comparing its name and parent and then
+taking a reference count (the reference count then gives a stable place to
+start the next part of the path walk from).
+
+As explained above, we would like to do path walking without taking locks or
+reference counts on intermediate dentries along the path. To do this, a per
+dentry seqlock (d_seq) is used to take a "coherent snapshot" of what the dentry
+looks like (its name, parent, and inode). That snapshot is then used to start
+the next part of the path walk. When loading the coherent snapshot under d_seq,
+care must be taken to load the members up-front, and use those pointers rather
+than reloading from the dentry later on (otherwise we'd have interesting things
+like d_inode going NULL underneath us, if the name was unlinked).
+
+Also important is to avoid performing any destructive operations (pretty much:
+no non-atomic stores to shared data), and to recheck the seqcount when we are
+"done" with the operation. Retry or abort if the seqcount does not match.
+Avoiding destructive or changing operations means we can easily unwind from
+failure.
+
+What this means is that a caller, provided they are holding RCU lock to
+protect the dentry object from disappearing, can perform a seqcount based
+lookup which does not increment the refcount on the dentry or write to
+it in any way. This returned dentry can be used for subsequent operations,
+provided that d_seq is rechecked after that operation is complete.
+
+Inodes are also rcu freed, so the seqcount lookup dentry's inode may also be
+queried for permissions.
+
+With this two parts of the puzzle, we can do path lookups without taking
+locks or refcounts on dentry elements.
+
+RCU-walk path walking design
+============================
+
+Path walking code now has two distinct modes, ref-walk and rcu-walk. ref-walk
+is the traditional[*] way of performing dcache lookups using d_lock to
+serialise concurrent modifications to the dentry and take a reference count on
+it. ref-walk is simple and obvious, and may sleep, take locks, etc while path
+walking is operating on each dentry. rcu-walk uses seqcount based dentry
+lookups, and can perform lookup of intermediate elements without any stores to
+shared data in the dentry or inode. rcu-walk can not be applied to all cases,
+eg. if the filesystem must sleep or perform non trivial operations, rcu-walk
+must be switched to ref-walk mode.
+
+[*] RCU is still used for the dentry hash lookup in ref-walk, but not the full
+    path walk.
+
+Where ref-walk uses a stable, refcounted ``parent'' to walk the remaining
+path string, rcu-walk uses a d_seq protected snapshot. When looking up a
+child of this parent snapshot, we open d_seq critical section on the child
+before closing d_seq critical section on the parent. This gives an interlocking
+ladder of snapshots to walk down.
+
+
+     proc 101
+      /----------------\
+     / comm:    "vi"    \
+    /  fs.root: dentry0  \
+    \  fs.cwd:  dentry2  /
+     \                  /
+      \----------------/
+
+So when vi wants to open("/home/npiggin/test.c", O_RDWR), then it will
+start from current->fs->root, which is a pinned dentry. Alternatively,
+"./test.c" would start from cwd; both names refer to the same path in
+the context of proc101.
+
+     dentry 0
+    +---------------------+   rcu-walk begins here, we note d_seq, check the
+    | name:    "/"        |   inode's permission, and then look up the next
+    | inode:   10         |   path element which is "home"...
+    | children:"home", ...|
+    +---------------------+
+              |
+     dentry 1 V
+    +---------------------+   ... which brings us here. We find dentry1 via
+    | name:    "home"     |   hash lookup, then note d_seq and compare name
+    | inode:   678        |   string and parent pointer. When we have a match,
+    | children:"npiggin"  |   we now recheck the d_seq of dentry0. Then we
+    +---------------------+   check inode and look up the next element.
+              |
+     dentry2  V
+    +---------------------+   Note: if dentry0 is now modified, lookup is
+    | name:    "npiggin"  |   not necessarily invalid, so we need only keep a
+    | inode:   543        |   parent for d_seq verification, and grandparents
+    | children:"a.c", ... |   can be forgotten.
+    +---------------------+
+              |
+     dentry3  V
+    +---------------------+   At this point we have our destination dentry.
+    | name:    "a.c"      |   We now take its d_lock, verify d_seq of this
+    | inode:   14221      |   dentry. If that checks out, we can increment
+    | children:NULL       |   its refcount because we're holding d_lock.
+    +---------------------+
+
+Taking a refcount on a dentry from rcu-walk mode, by taking its d_lock,
+re-checking its d_seq, and then incrementing its refcount is called
+"dropping rcu" or dropping from rcu-walk into ref-walk mode.
+
+It is, in some sense, a bit of a house of cards. If the seqcount check of the
+parent snapshot fails, the house comes down, because we had closed the d_seq
+section on the grandparent, so we have nothing left to stand on. In that case,
+the path walk must be fully restarted (which we do in ref-walk mode, to avoid
+live locks). It is costly to have a full restart, but fortunately they are
+quite rare.
+
+When we reach a point where sleeping is required, or a filesystem callout
+requires ref-walk, then instead of restarting the walk, we attempt to drop rcu
+at the last known good dentry we have. Avoiding a full restart in ref-walk in
+these cases is fundamental for performance and scalability because blocking
+operations such as creates and unlinks are not uncommon.
+
+The detailed design for rcu-walk is like this:
+* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
+* Take the RCU lock for the entire path walk, starting with the acquiring
+  of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
+  not required for dentry persistence.
+* synchronize_rcu is called when unregistering a filesystem, so we can
+  access d_ops and i_ops during rcu-walk.
+* Similarly take the vfsmount lock for the entire path walk. So now mnt
+  refcounts are not required for persistence. Also we are free to perform mount
+  lookups, and to assume dentry mount points and mount roots are stable up and
+  down the path.
+* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
+  so we can load this tuple atomically, and also check whether any of its
+  members have changed.
+* Dentry lookups (based on parent, candidate string tuple) recheck the parent
+  sequence after the child is found in case anything changed in the parent
+  during the path walk.
+* inode is also RCU protected so we can load d_inode and use the inode for
+  limited things.
+* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
+* i_op can be loaded.
+* When the destination dentry is reached, drop rcu there (ie. take d_lock,
+  verify d_seq, increment refcount).
+* If seqlock verification fails anywhere along the path, do a full restart
+  of the path lookup in ref-walk mode. -ECHILD tends to be used (for want of
+  a better errno) to signal an rcu-walk failure.
+
+The cases where rcu-walk cannot continue are:
+* NULL dentry (ie. any uncached path element)
+* Following links
+
+It may be possible eventually to make following links rcu-walk aware.
+
+Uncached path elements will always require dropping to ref-walk mode, at the
+very least because i_mutex needs to be grabbed, and objects allocated.
+
+Final note:
+"store-free" path walking is not strictly store free. We take vfsmount lock
+and refcounts (both of which can be made per-cpu), and we also store to the
+stack (which is essentially CPU-local), and we also have to take locks and
+refcount on final dentry.
+
+The point is that shared data, where practically possible, is not locked
+or stored into. The result is massive improvements in performance and
+scalability of path resolution.
+
+
+Interesting statistics
+======================
+
+The following table gives rcu lookup statistics for a few simple workloads
+(2s12c24t Westmere, debian non-graphical system). Ungraceful are attempts to
+drop rcu that fail due to d_seq failure and requiring the entire path lookup
+again. Other cases are successful rcu-drops that are required before the final
+element, nodentry for missing dentry, revalidate for filesystem revalidate
+routine requiring rcu drop, permission for permission check requiring drop,
+and link for symlink traversal requiring drop.
+
+     rcu-lookups     restart  nodentry          link  revalidate  permission
+bootup     47121           0      4624          1010       10283        7852
+dbench  25386793           0   6778659(26.7%)     55         549        1156
+kbuild   2696672          10     64442(2.3%)  108764(4.0%)     1        1590
+git diff   39605           0        28             2           0         106
+vfstest 24185492        4945    708725(2.9%) 1076136(4.4%)     0        2651
+