| Message ID | 162575623717.2532.8517369487503961860.stgit@klimt.1015granger.net (mailing list archive) |
|---|---|
| Headers | show |
| Series | Bulk-release pages during NFSD read splice | expand |
On Fri, 09 Jul 2021, Chuck Lever wrote: > > In this version of the series, each nfsd thread never accrues more > than 16 pages. We can easily make that larger or smaller, but 16 > already reduces the rate of put_pages() calls to a minute fraction > of what it was, and does not consume much additional space in struct > svc_rqst. > > Comments welcome! Very nice. Does "1/16" really count as "minute"? Or did I miss something and it is actually a smaller fraction? Either way: excellent work. Reviewed-by: NeilBrown <neilb@suse.de> NeilBrown > > --- > > Chuck Lever (3): > NFSD: Clean up splice actor > SUNRPC: Add svc_rqst_replace_page() API > NFSD: Batch release pages during splice read > > > fs/nfsd/vfs.c | 20 +++++--------------- > include/linux/sunrpc/svc.h | 5 +++++ > net/sunrpc/svc.c | 29 +++++++++++++++++++++++++++++ > 3 files changed, 39 insertions(+), 15 deletions(-) > > -- > Chuck Lever > >
> On Jul 8, 2021, at 7:23 PM, NeilBrown <neilb@suse.de> wrote: > > On Fri, 09 Jul 2021, Chuck Lever wrote: >> >> In this version of the series, each nfsd thread never accrues more >> than 16 pages. We can easily make that larger or smaller, but 16 >> already reduces the rate of put_pages() calls to a minute fraction >> of what it was, and does not consume much additional space in struct >> svc_rqst. >> >> Comments welcome! > > Very nice. Does "1/16" really count as "minute"? Or did I miss > something and it is actually a smaller fraction? 6% is better than an order of magnitude fewer calls. I can drop the "minute". > Either way: excellent work. > > Reviewed-by: NeilBrown <neilb@suse.de> Thanks! > NeilBrown > >> >> --- >> >> Chuck Lever (3): >> NFSD: Clean up splice actor >> SUNRPC: Add svc_rqst_replace_page() API >> NFSD: Batch release pages during splice read >> >> >> fs/nfsd/vfs.c | 20 +++++--------------- >> include/linux/sunrpc/svc.h | 5 +++++ >> net/sunrpc/svc.c | 29 +++++++++++++++++++++++++++++ >> 3 files changed, 39 insertions(+), 15 deletions(-) >> >> -- >> Chuck Lever >> >> -- Chuck Lever
I'm using "v3" simply because the v2 series of NFSD page allocator work included the same bulk-release concept in a different form. v2 has now been merged (thanks, Mel!). However, the bulk-release part of that series was postponed. Consider v3 to be an RFC refresh. As with the page allocation side, I'm trying to reduce the average number of times NFSD invokes the page allocation and release APIs because they can be expensive, and because it is a resource that is shared amongst all nfsd threads and thus access to it is partially serialized. This small series tackles a code path that is frequently invoked when NFSD handles READ operations on local filesystems that support splice (i.e., most of the popular ones). The previous version of this proposal placed the unused pages on a local list and then re-used the pages directly in svc_alloc_arg() before invoking alloc_pages_bulk_array() to fill in any remaining missing rq_pages entries. This meant there would be the possibility of some workloads that caused accrual of pages without bounds, so the finished version of that logic would have to be complex and possibly involve a shrinker. In this version, I'm simply handing the pages back to the page allocator, so all that complexity vanishes. What makes it more efficient is that instead of calling put_page() for each page, the code collects the unused pages in a per-nfsd thread array, and returns them to the allocator using a bulk free API (release_pages) when the array is full. In this version of the series, each nfsd thread never accrues more than 16 pages. We can easily make that larger or smaller, but 16 already reduces the rate of put_pages() calls to a minute fraction of what it was, and does not consume much additional space in struct svc_rqst. Comments welcome! --- Chuck Lever (3): NFSD: Clean up splice actor SUNRPC: Add svc_rqst_replace_page() API NFSD: Batch release pages during splice read fs/nfsd/vfs.c | 20 +++++--------------- include/linux/sunrpc/svc.h | 5 +++++ net/sunrpc/svc.c | 29 +++++++++++++++++++++++++++++ 3 files changed, 39 insertions(+), 15 deletions(-) -- Chuck Lever