| Message ID | 88cb6a4d7a074fd4c4c6b59076df766c7de54105.1646922313.git.bcodding@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | NFS: Trigger "ls -l" readdir heuristic sooner | expand |
On Fri, Mar 11, 2022 at 9:40 AM Benjamin Coddington <bcodding@redhat.com> wrote: > > .. Something like this does the trick in my testing, but yes will have an > impact on regular workloads: > > 8<------------------------------------------------------------------------ > > Since commit 1a34c8c9a49e ("NFS: Support larger readdir buffers") has > updated dtsize and recent improvements to the READDIRPLUS helper heuristic, > the heuristic may not trigger until many dentries are emitted to userspace, > which may cause many thousands of GETATTR calls for "ls -l" when the > directory's pagecache has already been populated. This typically manifests > as a much slower total runtime for a _second_ invocation of "ls -l" within > the directory attribute cache timeouts. > > Fix this by emitting only 17 entries for any first pass through the NFS > directory's ->iterate_shared(), which will allow userpace to prime the > counters for the heuristic. Here's for what it's worth. An experiment between linux to linux where the linux server had a "small" directory structure of 57274 directories, 5727390 files in total where each directory had ~100 files each. With this patch: date; time tree vol1 > tree.out && date; time tree vol1 > tree.out Wed Mar 16 12:21:30 EDT 2022 real 11m7.923s user 0m20.507s sys 0m39.683s Wed Mar 16 12:32:38 EDT 2022 real 40m1.751s user 0m23.477s sys 0m45.663s Without the patch: date; time tree vol1 > tree.out && date; time tree vol1 > tree.out Wed Mar 16 13:49:12 EDT 2022 real 10m52.909s user 0m21.342s sys 0m39.198s Wed Mar 16 14:00:05 EDT 2022 real 222m56.990s user 0m30.392s sys 2m25.202s > > Signed-off-by: Benjamin Coddington <bcodding@redhat.com> > --- > fs/nfs/dir.c | 9 +++++++-- > 1 file changed, 7 insertions(+), 2 deletions(-) > > diff --git a/fs/nfs/dir.c b/fs/nfs/dir.c > index 7e12102b29e7..dc5fc9ba2c49 100644 > --- a/fs/nfs/dir.c > +++ b/fs/nfs/dir.c > @@ -1060,6 +1060,8 @@ static int readdir_search_pagecache(struct nfs_readdir_descriptor *desc) > return res; > } > > +#define NFS_READDIR_CACHE_MISS_THRESHOLD (16UL) > + > /* > * Once we've found the start of the dirent within a page: fill 'er up... > */ > @@ -1069,6 +1071,7 @@ static void nfs_do_filldir(struct nfs_readdir_descriptor *desc, > struct file *file = desc->file; > struct nfs_cache_array *array; > unsigned int i; > + bool first_emit = !desc->dir_cookie; > > array = kmap(desc->page); > for (i = desc->cache_entry_index; i < array->size; i++) { > @@ -1092,6 +1095,10 @@ static void nfs_do_filldir(struct nfs_readdir_descriptor *desc, > desc->ctx->pos = desc->dir_cookie; > else > desc->ctx->pos++; > + if (first_emit && i > NFS_READDIR_CACHE_MISS_THRESHOLD + 1) { > + desc->eob = true; > + break; > + } > } > if (array->page_is_eof) > desc->eof = !desc->eob; > @@ -1173,8 +1180,6 @@ static int uncached_readdir(struct nfs_readdir_descriptor *desc) > return status; > } > > -#define NFS_READDIR_CACHE_MISS_THRESHOLD (16UL) > - > static bool nfs_readdir_handle_cache_misses(struct inode *inode, > struct nfs_readdir_descriptor *desc, > unsigned int cache_misses, > -- > 2.31.1 >
diff --git a/fs/nfs/dir.c b/fs/nfs/dir.c index 7e12102b29e7..dc5fc9ba2c49 100644 --- a/fs/nfs/dir.c +++ b/fs/nfs/dir.c @@ -1060,6 +1060,8 @@ static int readdir_search_pagecache(struct nfs_readdir_descriptor *desc) return res; } +#define NFS_READDIR_CACHE_MISS_THRESHOLD (16UL) + /* * Once we've found the start of the dirent within a page: fill 'er up... */ @@ -1069,6 +1071,7 @@ static void nfs_do_filldir(struct nfs_readdir_descriptor *desc, struct file *file = desc->file; struct nfs_cache_array *array; unsigned int i; + bool first_emit = !desc->dir_cookie; array = kmap(desc->page); for (i = desc->cache_entry_index; i < array->size; i++) { @@ -1092,6 +1095,10 @@ static void nfs_do_filldir(struct nfs_readdir_descriptor *desc, desc->ctx->pos = desc->dir_cookie; else desc->ctx->pos++; + if (first_emit && i > NFS_READDIR_CACHE_MISS_THRESHOLD + 1) { + desc->eob = true; + break; + } } if (array->page_is_eof) desc->eof = !desc->eob; @@ -1173,8 +1180,6 @@ static int uncached_readdir(struct nfs_readdir_descriptor *desc) return status; } -#define NFS_READDIR_CACHE_MISS_THRESHOLD (16UL) - static bool nfs_readdir_handle_cache_misses(struct inode *inode, struct nfs_readdir_descriptor *desc, unsigned int cache_misses,
.. Something like this does the trick in my testing, but yes will have an impact on regular workloads: 8<------------------------------------------------------------------------ Since commit 1a34c8c9a49e ("NFS: Support larger readdir buffers") has updated dtsize and recent improvements to the READDIRPLUS helper heuristic, the heuristic may not trigger until many dentries are emitted to userspace, which may cause many thousands of GETATTR calls for "ls -l" when the directory's pagecache has already been populated. This typically manifests as a much slower total runtime for a _second_ invocation of "ls -l" within the directory attribute cache timeouts. Fix this by emitting only 17 entries for any first pass through the NFS directory's ->iterate_shared(), which will allow userpace to prime the counters for the heuristic. Signed-off-by: Benjamin Coddington <bcodding@redhat.com> --- fs/nfs/dir.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)