From patchwork Thu May 28 11:53:32 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Layton X-Patchwork-Id: 6497571 Return-Path: X-Original-To: patchwork-linux-nfs@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 45BC29F1CC for ; Thu, 28 May 2015 11:53:56 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 3E48520625 for ; Thu, 28 May 2015 11:53:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1026D20584 for ; Thu, 28 May 2015 11:53:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754199AbbE1Lxs (ORCPT ); Thu, 28 May 2015 07:53:48 -0400 Received: from mail-wi0-f174.google.com ([209.85.212.174]:33504 "EHLO mail-wi0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753904AbbE1Lxn (ORCPT ); Thu, 28 May 2015 07:53:43 -0400 Received: by wicmx19 with SMTP id mx19so121008450wic.0 for ; Thu, 28 May 2015 04:53:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-type; bh=IvZZ40wG19Tjg9zznCSqtbm/I/HrikWz9jcsvhiwCD0=; b=Nq1Wvk2621KKwGswhoav02Z+hAyvAPXA+hKRsxPLGsi30uFCyYVrser29UsB/fTfGJ drEjGUYu9l/hW6h8onr4hLMO6YXTZTICgwnPeWxEhCeynfmVTOumoKIr+9i1Xjp5BvCw E/nchAdcEk+AqTRSZbzWMBGL/QlDQyeFtgrxCOMKLNG9n37ERxJ2gVO7Td9SCpi5v36h MiMnHxn+FASEefY6FLeGLRBn64Z4bZVthOrcLeeSB8UM/VzTXEoxBLcADG4A5q+5J/kb jo3OZYzMlBtGAeaH+mHgzMvHuUekf695JHwWcxfYuxoQfftmnF85h9PUp2yeeZPmGLZt Tb2g== X-Gm-Message-State: ALoCoQkKKJ/PPlpt9SdMOZ+flEGr3/y9lxtx5jBp5MdAN5nCtuufWkOEprsXOo/1FalC2qGwr5NK X-Received: by 10.180.84.8 with SMTP id u8mr61038002wiy.39.1432814021608; Thu, 28 May 2015 04:53:41 -0700 (PDT) Received: from synchrony.poochiereds.net ([2606:a000:1105:8e:fa16:54ff:fe51:6224]) by mx.google.com with ESMTPSA id h1sm8213442wiz.6.2015.05.28.04.53.39 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 28 May 2015 04:53:41 -0700 (PDT) Date: Thu, 28 May 2015 07:53:32 -0400 From: Jeff Layton To: Jerome Marchand Cc: Mel Gorman , Jeff Layton , "'Linux-MM'" , "'linux-kernel'" , trond.myklebust@primarydata.com, linux-nfs@vger.kernel.org Subject: Re: swap: nfs: Sleeping function called from an rcu read section in nfs_swap_activate Message-ID: <20150528075332.6945fd09@synchrony.poochiereds.net> In-Reply-To: <5564732E.4090607@redhat.com> References: <5564732E.4090607@redhat.com> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.28; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD, T_TVD_MIME_EPI, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Tue, 26 May 2015 15:20:46 +0200 Jerome Marchand wrote: > > Commit dad2b015 added an rcu read lock around the call to xs_swapper() > in nfs_activate()/deactivate(), which can sleep, thus raising a bug at > each swapon and swapoff over NFS. > I'm not sure if this is related or not, but swapoff also triggers the > WARN_ON(sk->sk_forward_alloc) in sk_clear_memalloc(). > > [ 243.668067] =============================== > [ 243.668665] [ INFO: suspicious RCU usage. ] > [ 243.669293] 4.1.0-rc1-lock_stat-dbg-next-20150430+ #235 Not tainted > [ 243.670301] ------------------------------- > [ 243.670905] include/linux/rcupdate.h:570 Illegal context switch in RCU read-side critical section! > [ 243.672163] > other info that might help us debug this: > > [ 243.673025] > rcu_scheduler_active = 1, debug_locks = 0 > [ 243.673565] 2 locks held by swapon/1176: > [ 243.673893] #0: (&sb->s_type->i_mutex_key#17){+.+.+.}, at: [] SyS_swapon+0x2b0/0x1000 > [ 243.674758] #1: (rcu_read_lock){......}, at: [] nfs_swap_activate+0x5/0x180 [nfs] > [ 243.675591] > stack backtrace: > [ 243.675957] CPU: 0 PID: 1176 Comm: swapon Not tainted 4.1.0-rc1-lock_stat-dbg-next-20150430+ #235 > [ 243.676687] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 > [ 243.677179] 0000000000000000 00000000ef88d841 ffff88003327bcd8 ffffffff818861f0 > [ 243.677854] 0000000000000000 ffff880078e38000 ffff88003327bd08 ffffffff8110d237 > [ 243.678514] 0000000000000000 ffffffff81c650e4 0000000000000268 ffff880078e38000 > [ 243.679171] Call Trace: > [ 243.679383] [] dump_stack+0x4c/0x65 > [ 243.679811] [] lockdep_rcu_suspicious+0xe7/0x120 > [ 243.680348] [] ___might_sleep+0xaf/0x250 > [ 243.680815] [] __might_sleep+0x4d/0x90 > [ 243.681279] [] mutex_lock_nested+0x47/0x430 > [ 243.681762] [] static_key_slow_inc+0x7c/0xc0 > [ 243.682264] [] sk_set_memalloc+0x27/0x30 > [ 243.682736] [] xs_swapper+0x54/0x60 [sunrpc] > [ 243.683238] [] nfs_swap_activate+0x93/0x180 [nfs] > [ 243.683760] [] ? nfs_swap_activate+0x5/0x180 [nfs] > [ 243.684316] [] SyS_swapon+0xad4/0x1000 > [ 243.684766] [] ? syscall_return+0x16/0x59 > [ 243.685245] [] system_call_fastpath+0x12/0x76 > [ 243.685743] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:616 > [ 243.686439] in_atomic(): 1, irqs_disabled(): 0, pid: 1176, name: swapon > [ 243.687053] INFO: lockdep is turned off. > [ 243.687429] CPU: 0 PID: 1176 Comm: swapon Not tainted 4.1.0-rc1-lock_stat-dbg-next-20150430+ #235 > [ 243.688313] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 > [ 243.688845] 0000000000000000 00000000ef88d841 ffff88003327bd08 ffffffff818861f0 > [ 243.689570] 0000000000000000 ffff880078e38000 ffff88003327bd38 ffffffff810df29c > [ 243.690353] ffff880000000001 ffffffff81c650e4 0000000000000268 0000000000000000 > [ 243.691057] Call Trace: > [ 243.691315] [] dump_stack+0x4c/0x65 > [ 243.691785] [] ___might_sleep+0x18c/0x250 > [ 243.692306] [] __might_sleep+0x4d/0x90 > [ 243.692807] [] mutex_lock_nested+0x47/0x430 > [ 243.693346] [] static_key_slow_inc+0x7c/0xc0 > [ 243.693887] [] sk_set_memalloc+0x27/0x30 > [ 243.694416] [] xs_swapper+0x54/0x60 [sunrpc] > [ 243.694959] [] nfs_swap_activate+0x93/0x180 [nfs] > [ 243.695535] [] ? nfs_swap_activate+0x5/0x180 [nfs] > [ 243.696193] [] SyS_swapon+0xad4/0x1000 > [ 243.696699] [] ? syscall_return+0x16/0x59 > [ 243.697299] [] system_call_fastpath+0x12/0x76 > [ 243.702101] Adding 524284k swap on /mnt/swapfile512. Priority:-2 extents:1 across:524284k FS > [ 325.151350] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:616 > [ 325.152688] in_atomic(): 1, irqs_disabled(): 0, pid: 1199, name: swapoff > [ 325.153737] INFO: lockdep is turned off. > [ 325.154457] CPU: 1 PID: 1199 Comm: swapoff Not tainted 4.1.0-rc1-lock_stat-dbg-next-20150430+ #235 > [ 325.156204] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 > [ 325.157120] 0000000000000000 00000000a7682b83 ffff88007ac3fce8 ffffffff818861f0 > [ 325.158361] 0000000000000000 ffff880032434c00 ffff88007ac3fd18 ffffffff810df29c > [ 325.159592] 0000000000000000 ffffffff81c650e4 0000000000000268 0000000000000000 > [ 325.160798] Call Trace: > [ 325.161251] [] dump_stack+0x4c/0x65 > [ 325.162071] [] ___might_sleep+0x18c/0x250 > [ 325.163073] [] __might_sleep+0x4d/0x90 > [ 325.163934] [] mutex_lock_nested+0x47/0x430 > [ 325.164927] [] atomic_dec_and_mutex_lock+0x4f/0x70 > [ 325.166020] [] __static_key_slow_dec+0x27/0xc0 > [ 325.166942] [] static_key_slow_dec+0x26/0x50 > [ 325.167955] [] sk_clear_memalloc+0x2f/0x80 > [ 325.169075] [] xs_swapper+0x41/0x60 [sunrpc] > [ 325.170241] [] nfs_swap_deactivate+0x87/0x170 [nfs] > [ 325.171276] [] ? nfs_swap_deactivate+0x5/0x170 [nfs] > [ 325.172349] [] destroy_swap_extents+0x77/0x90 > [ 325.173754] [] SyS_swapoff+0x215/0x600 > [ 325.174726] [] ? trace_hardirqs_on_thunk+0x17/0x19 > [ 325.175971] [] system_call_fastpath+0x12/0x76 > [ 325.178052] ------------[ cut here ]------------ > [ 325.178892] WARNING: CPU: 1 PID: 1199 at net/core/sock.c:364 sk_clear_memalloc+0x51/0x80() > [ 325.180363] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iosf_mbi crct10dif_pclmul crc32_pclmul crc32c_intel ppdev ghash_clmulni_intel joydev nfsd parport_pc pcspkr virtio_console serio_raw virtio_balloon parport pvpanic i2c_piix4 acpi_cpufreq auth_rpcgss nfs_acl lockd grace sunrpc virtio_blk qxl virtio_net drm_kms_helper ttm drm virtio_pci virtio_ring virtio ata_generic pata_acpi floppy > [ 325.192279] CPU: 1 PID: 1199 Comm: swapoff Not tainted 4.1.0-rc1-lock_stat-dbg-next-20150430+ #235 > [ 325.193605] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 > [ 325.194491] 0000000000000000 00000000a7682b83 ffff88007ac3fdf8 ffffffff818861f0 > [ 325.195692] 0000000000000000 0000000000000000 ffff88007ac3fe38 ffffffff810af5ca > [ 325.196891] ffff88007ac3fe78 ffff88007b068000 ffff88007b484a00 ffff88007b484aa8 > [ 325.198119] Call Trace: > [ 325.198555] [] dump_stack+0x4c/0x65 > [ 325.199380] [] warn_slowpath_common+0x8a/0xc0 > [ 325.200601] [] warn_slowpath_null+0x1a/0x20 > [ 325.201536] [] sk_clear_memalloc+0x51/0x80 > [ 325.202468] [] xs_swapper+0x41/0x60 [sunrpc] > [ 325.203398] [] nfs_swap_deactivate+0x87/0x170 [nfs] > [ 325.204426] [] ? nfs_swap_deactivate+0x5/0x170 [nfs] > [ 325.205456] [] destroy_swap_extents+0x77/0x90 > [ 325.206406] [] SyS_swapoff+0x215/0x600 > [ 325.207287] [] ? trace_hardirqs_on_thunk+0x17/0x19 > [ 325.208300] [] system_call_fastpath+0x12/0x76 > [ 325.209248] ---[ end trace 13f1014b56e5e711 ]--- > Jerome, Can you test the attached patch and let me know if this fixes the problem? If so, then I'll send it on "officially" to Trond. Thanks, From 527e41205915d605f68c356a636dfc12b7545b7b Mon Sep 17 00:00:00 2001 From: Jeff Layton Date: Thu, 28 May 2015 07:39:39 -0400 Subject: [PATCH] nfs: don't hold rcu_read_lock over xs_swapper call Jerome reported seeing a WARN_ON pop up when enabling a swapfile on NFS. The xs_swapper call can sleep so we do not want to hold the rcu_read_lock when calling it. Fix this by taking a reference to the xprt in the critical section, calling xs_swapper outside it and then putting that reference. Reported-by: Jerome Marchand Cc: Mel Gorman Signed-off-by: Jeff Layton --- fs/nfs/file.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/fs/nfs/file.c b/fs/nfs/file.c index 8b8d83a526ce..3ee08460c4cf 100644 --- a/fs/nfs/file.c +++ b/fs/nfs/file.c @@ -561,12 +561,18 @@ static int nfs_swap_activate(struct swap_info_struct *sis, struct file *file, { int ret; struct rpc_clnt *clnt = NFS_CLIENT(file->f_mapping->host); + struct rpc_xprt *xprt; *span = sis->pages; rcu_read_lock(); - ret = xs_swapper(rcu_dereference(clnt->cl_xprt), 1); + xprt = xprt_get(rcu_dereference(clnt->cl_xprt)); rcu_read_unlock(); + if (!xprt) + return -EINVAL; + + ret = xs_swapper(xprt, 1); + xprt_put(xprt); return ret; } @@ -574,10 +580,16 @@ static int nfs_swap_activate(struct swap_info_struct *sis, struct file *file, static void nfs_swap_deactivate(struct file *file) { struct rpc_clnt *clnt = NFS_CLIENT(file->f_mapping->host); + struct rpc_xprt *xprt; rcu_read_lock(); - xs_swapper(rcu_dereference(clnt->cl_xprt), 0); + xprt = xprt_get(rcu_dereference(clnt->cl_xprt)); rcu_read_unlock(); + if (!xprt) + return; + + xs_swapper(xprt, 0); + xprt_put(xprt); } #endif -- 2.4.1