| Message ID | 20200430164356.15543-1-fdmanana@kernel.org (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | percpu: make pcpu_alloc() aware of current gfp context | expand |
On Thu, Apr 30, 2020 at 05:43:56PM +0100, fdmanana@kernel.org wrote: > From: Filipe Manana <fdmanana@suse.com> > > Since 5.7-rc1, on btrfs we have a percpu counter initialization for which > we always pass a GFP_KERNEL gfp_t argument (this happens since commit > 2992df73268f78 ("btrfs: Implement DREW lock")). That is safe in some > contextes but not on others where allowing fs reclaim could lead to a > deadlock because we are either holding some btrfs lock needed for a > transaction commit or holding a btrfs transaction handle open. Because > of that we surround the call to the function that initializes the percpu > counter with a NOFS context using memalloc_nofs_save() (this is done at > btrfs_init_fs_root()). > > However it turns out that this is not enough to prevent a possible > deadlock because percpu_alloc() determines if it is in an atomic context > by looking exclusively at the gfp flags passed to it (GFP_KERNEL in this > case) and it is not aware that a NOFS context is set. Because it thinks > it is in a non atomic context it locks the pcpu_alloc_mutex, which can > result in a btrfs deadlock when pcpu_balance_workfn() is running, has > acquired that mutex and is waiting for reclaim, while the btrfs task that > called percpu_counter_init() (and therefore percpu_alloc()) is holding > either the btrfs commit_root semaphore or a transaction handle (done at > fs/btrfs/backref.c:iterate_extent_inodes()), which prevents reclaim from > finishing as an attempt to commit the current btrfs transaction will > deadlock. ... > This could be fixed by making btrfs pass GFP_NOFS instead of GFP_KERNEL to > percpu_counter_init() in contextes where it is not reclaim safe, however > that type of approach is discouraged since memalloc_[nofs|noio]_save() > were introduced. Therefore this change makes pcpu_alloc() look up into > an existing nofs/noio context before deciding whether it is in an atomic > context or not. > > Signed-off-by: Filipe Manana <fdmanana@suse.com> Acked-by: Tejun Heo <tj@kernel.org> Thanks.
On Thu, 30 Apr 2020 17:43:56 +0100 fdmanana@kernel.org wrote: > From: Filipe Manana <fdmanana@suse.com> > > Since 5.7-rc1, on btrfs we have a percpu counter initialization for which > we always pass a GFP_KERNEL gfp_t argument (this happens since commit > 2992df73268f78 ("btrfs: Implement DREW lock")). That is safe in some > contextes but not on others where allowing fs reclaim could lead to a > deadlock because we are either holding some btrfs lock needed for a > transaction commit or holding a btrfs transaction handle open. Because > of that we surround the call to the function that initializes the percpu > counter with a NOFS context using memalloc_nofs_save() (this is done at > btrfs_init_fs_root()). > > However it turns out that this is not enough to prevent a possible > deadlock because percpu_alloc() determines if it is in an atomic context > by looking exclusively at the gfp flags passed to it (GFP_KERNEL in this > case) and it is not aware that a NOFS context is set. Because it thinks > it is in a non atomic context it locks the pcpu_alloc_mutex, which can > result in a btrfs deadlock when pcpu_balance_workfn() is running, has > acquired that mutex and is waiting for reclaim, while the btrfs task that > called percpu_counter_init() (and therefore percpu_alloc()) is holding > either the btrfs commit_root semaphore or a transaction handle (done at > fs/btrfs/backref.c:iterate_extent_inodes()), which prevents reclaim from > finishing as an attempt to commit the current btrfs transaction will > deadlock. > Patch looks good and seems sensible, thanks. But why did btrfs use memalloc_nofs_save()/restore() rather than s/GFP_KERNEL/GFP_NOFS/?
On Thu, Apr 30, 2020 at 02:40:18PM -0700, Andrew Morton wrote: > On Thu, 30 Apr 2020 17:43:56 +0100 fdmanana@kernel.org wrote: > > > From: Filipe Manana <fdmanana@suse.com> > > > > Since 5.7-rc1, on btrfs we have a percpu counter initialization for which > > we always pass a GFP_KERNEL gfp_t argument (this happens since commit > > 2992df73268f78 ("btrfs: Implement DREW lock")). That is safe in some > > contextes but not on others where allowing fs reclaim could lead to a > > deadlock because we are either holding some btrfs lock needed for a > > transaction commit or holding a btrfs transaction handle open. Because > > of that we surround the call to the function that initializes the percpu > > counter with a NOFS context using memalloc_nofs_save() (this is done at > > btrfs_init_fs_root()). > > > > However it turns out that this is not enough to prevent a possible > > deadlock because percpu_alloc() determines if it is in an atomic context > > by looking exclusively at the gfp flags passed to it (GFP_KERNEL in this > > case) and it is not aware that a NOFS context is set. Because it thinks > > it is in a non atomic context it locks the pcpu_alloc_mutex, which can > > result in a btrfs deadlock when pcpu_balance_workfn() is running, has > > acquired that mutex and is waiting for reclaim, while the btrfs task that > > called percpu_counter_init() (and therefore percpu_alloc()) is holding > > either the btrfs commit_root semaphore or a transaction handle (done at > > fs/btrfs/backref.c:iterate_extent_inodes()), which prevents reclaim from > > finishing as an attempt to commit the current btrfs transaction will > > deadlock. > > > > Patch looks good and seems sensible, thanks. > Acked-by: Dennis Zhou <dennis@kernel.org> > But why did btrfs use memalloc_nofs_save()/restore() rather than > s/GFP_KERNEL/GFP_NOFS/? I would also like to know. Thanks, Dennis
On Thu, Apr 30, 2020 at 11:23 PM Dennis Zhou <dennis@kernel.org> wrote: > > On Thu, Apr 30, 2020 at 02:40:18PM -0700, Andrew Morton wrote: > > On Thu, 30 Apr 2020 17:43:56 +0100 fdmanana@kernel.org wrote: > > > > > From: Filipe Manana <fdmanana@suse.com> > > > > > > Since 5.7-rc1, on btrfs we have a percpu counter initialization for which > > > we always pass a GFP_KERNEL gfp_t argument (this happens since commit > > > 2992df73268f78 ("btrfs: Implement DREW lock")). That is safe in some > > > contextes but not on others where allowing fs reclaim could lead to a > > > deadlock because we are either holding some btrfs lock needed for a > > > transaction commit or holding a btrfs transaction handle open. Because > > > of that we surround the call to the function that initializes the percpu > > > counter with a NOFS context using memalloc_nofs_save() (this is done at > > > btrfs_init_fs_root()). > > > > > > However it turns out that this is not enough to prevent a possible > > > deadlock because percpu_alloc() determines if it is in an atomic context > > > by looking exclusively at the gfp flags passed to it (GFP_KERNEL in this > > > case) and it is not aware that a NOFS context is set. Because it thinks > > > it is in a non atomic context it locks the pcpu_alloc_mutex, which can > > > result in a btrfs deadlock when pcpu_balance_workfn() is running, has > > > acquired that mutex and is waiting for reclaim, while the btrfs task that > > > called percpu_counter_init() (and therefore percpu_alloc()) is holding > > > either the btrfs commit_root semaphore or a transaction handle (done at > > > fs/btrfs/backref.c:iterate_extent_inodes()), which prevents reclaim from > > > finishing as an attempt to commit the current btrfs transaction will > > > deadlock. > > > > > > > Patch looks good and seems sensible, thanks. > > > > Acked-by: Dennis Zhou <dennis@kernel.org> > > > But why did btrfs use memalloc_nofs_save()/restore() rather than > > s/GFP_KERNEL/GFP_NOFS/? > > I would also like to know. For 2 reasons: 1) It's the preferred way to do it since memalloc_nofs_save()/restore() was added (according to Documentation/core-api/gfp_mask-from-fs-io.rst); 2) According to Documentation/core-api/gfp_mask-from-fs-io.rst, passing GFP_NOFS to __vmalloc() doesn't work, so one has to use the memalloc_nofs_save()/restore() API for that. And pcpu_alloc() calls helpers that end up calling __vmalloc() (through pcpu_mem_zalloc()). And that's it. Thanks. > > Thanks, > Dennis
On Thu, Apr 30, 2020 at 11:43:20PM +0100, Filipe Manana wrote: > On Thu, Apr 30, 2020 at 11:23 PM Dennis Zhou <dennis@kernel.org> wrote: > > > > On Thu, Apr 30, 2020 at 02:40:18PM -0700, Andrew Morton wrote: > > > On Thu, 30 Apr 2020 17:43:56 +0100 fdmanana@kernel.org wrote: > > > > > > > From: Filipe Manana <fdmanana@suse.com> > > > > > > > > Since 5.7-rc1, on btrfs we have a percpu counter initialization for which > > > > we always pass a GFP_KERNEL gfp_t argument (this happens since commit > > > > 2992df73268f78 ("btrfs: Implement DREW lock")). That is safe in some > > > > contextes but not on others where allowing fs reclaim could lead to a > > > > deadlock because we are either holding some btrfs lock needed for a > > > > transaction commit or holding a btrfs transaction handle open. Because > > > > of that we surround the call to the function that initializes the percpu > > > > counter with a NOFS context using memalloc_nofs_save() (this is done at > > > > btrfs_init_fs_root()). > > > > > > > > However it turns out that this is not enough to prevent a possible > > > > deadlock because percpu_alloc() determines if it is in an atomic context > > > > by looking exclusively at the gfp flags passed to it (GFP_KERNEL in this > > > > case) and it is not aware that a NOFS context is set. Because it thinks > > > > it is in a non atomic context it locks the pcpu_alloc_mutex, which can > > > > result in a btrfs deadlock when pcpu_balance_workfn() is running, has > > > > acquired that mutex and is waiting for reclaim, while the btrfs task that > > > > called percpu_counter_init() (and therefore percpu_alloc()) is holding > > > > either the btrfs commit_root semaphore or a transaction handle (done at > > > > fs/btrfs/backref.c:iterate_extent_inodes()), which prevents reclaim from > > > > finishing as an attempt to commit the current btrfs transaction will > > > > deadlock. > > > > > > > > > > Patch looks good and seems sensible, thanks. > > > > > > > Acked-by: Dennis Zhou <dennis@kernel.org> > > > > > But why did btrfs use memalloc_nofs_save()/restore() rather than > > > s/GFP_KERNEL/GFP_NOFS/? > > > > I would also like to know. > > For 2 reasons: > > 1) It's the preferred way to do it since > memalloc_nofs_save()/restore() was added (according to > Documentation/core-api/gfp_mask-from-fs-io.rst); > Thanks. I didn't realize it completely superceded GFP_NOFS. > 2) According to Documentation/core-api/gfp_mask-from-fs-io.rst, > passing GFP_NOFS to __vmalloc() doesn't work, so one has to use the > memalloc_nofs_save()/restore() API for that. And pcpu_alloc() calls > helpers that end up calling __vmalloc() (through pcpu_mem_zalloc()). > > And that's it. > I'm starting to remember a bit more. I guess it's not great how percpu manages GFP_ATOMIC as !GFP_KERNEL for the possible vmalloc() calls. At the time I believe the whitelist was the only way to deal with the recursive case. If I get a chance I'll look at the flags again and see if we can't do something better/ more aligned today. > > > > > Thanks, > > Dennis
diff --git a/mm/percpu.c b/mm/percpu.c index e9844086b236..f5258d5a9c16 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -80,6 +80,7 @@ #include <linux/workqueue.h> #include <linux/kmemleak.h> #include <linux/sched.h> +#include <linux/sched/mm.h> #include <asm/cacheflush.h> #include <asm/sections.h> @@ -1557,10 +1558,9 @@ static struct pcpu_chunk *pcpu_chunk_addr_search(void *addr) static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, gfp_t gfp) { - /* whitelisted flags that can be passed to the backing allocators */ - gfp_t pcpu_gfp = gfp & (GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN); - bool is_atomic = (gfp & GFP_KERNEL) != GFP_KERNEL; - bool do_warn = !(gfp & __GFP_NOWARN); + gfp_t pcpu_gfp; + bool is_atomic; + bool do_warn; static int warn_limit = 10; struct pcpu_chunk *chunk, *next; const char *err; @@ -1569,6 +1569,12 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, void __percpu *ptr; size_t bits, bit_align; + gfp = current_gfp_context(gfp); + /* whitelisted flags that can be passed to the backing allocators */ + pcpu_gfp = gfp & (GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN); + is_atomic = (gfp & GFP_KERNEL) != GFP_KERNEL; + do_warn = !(gfp & __GFP_NOWARN); + /* * There is now a minimum allocation size of PCPU_MIN_ALLOC_SIZE, * therefore alignment must be a minimum of that many bytes.