From patchwork Tue Jan 21 20:21:33 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Weston Andros Adamson X-Patchwork-Id: 3519501 Return-Path: X-Original-To: patchwork-linux-nfs@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 25C1E9F2E9 for ; Tue, 21 Jan 2014 20:21:41 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 1AD152011E for ; Tue, 21 Jan 2014 20:21:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1F7582010E for ; Tue, 21 Jan 2014 20:21:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752690AbaAUUVi (ORCPT ); Tue, 21 Jan 2014 15:21:38 -0500 Received: from mail-ie0-f175.google.com ([209.85.223.175]:53563 "EHLO mail-ie0-f175.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752684AbaAUUVh (ORCPT ); Tue, 21 Jan 2014 15:21:37 -0500 Received: by mail-ie0-f175.google.com with SMTP id ar20so5607045iec.6 for ; Tue, 21 Jan 2014 12:21:36 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=HTFD5ZEhEC199KPpvgGTIPuH089OamL2u6cB6RVqRCg=; b=YjMS1bpL9B1ChiQr1JVCuUCLRCTZH37x9D7azznXhSIoiIMkd/JA66bWgpBfYiohwb Kdp9CbzzYSxdfWd6u2B+UqY1JpIBogy32AA3iMGrYqdMUUJ9XijXxmtcLH9pSTpEX9pq MHg7QnrO0GKhuyjbD1L3hCKxb6WZZnw7vKk1s6gvcG449f0P5GDW3/OFmWYiKc4uSoJX 8qKZtfjDOjfgfdD//K2T7pmBQpar/VncejSeaH94Vk056JuKqR59nYWeSNP6DsI5aEAd Mh7YrrfyJHRhKTjNv1NW5adgwu/tw69KOKPy1Rb3r5RbYH6F5jkI95VwCCl5ixRMDCgJ /GGQ== X-Gm-Message-State: ALoCoQkBKzJHSiwBJTzUTtiYUG4kHOQGJioiIcCx9098d5wrop9yurzxru8JqSxJjtBmrWW5Nn33 X-Received: by 10.43.98.202 with SMTP id cp10mr20281525icc.28.1390335696760; Tue, 21 Jan 2014 12:21:36 -0800 (PST) Received: from gavrio.gateway.2wire.net (99-82-247-89.lightspeed.livnmi.sbcglobal.net. [99.82.247.89]) by mx.google.com with ESMTPSA id kb10sm14695389igb.6.2014.01.21.12.21.35 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 21 Jan 2014 12:21:36 -0800 (PST) From: Weston Andros Adamson To: trond.myklebust@primarydata.com Cc: linux-nfs@vger.kernel.org, Weston Andros Adamson Subject: [PATCH] pnfs: fix BUG in filelayout_recover_commit_reqs Date: Tue, 21 Jan 2014 15:21:33 -0500 Message-Id: <1390335693-12004-1-git-send-email-dros@primarydata.com> X-Mailer: git-send-email 1.8.3.4 (Apple Git-47) Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Spam-Status: No, score=-7.5 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP cond_resched_lock(cinfo->lock) is called everywhere else while holding the cinfo->lock spinlock. Not holding this lock while calling transfer_commit_list in filelayout_recover_commit_reqs causes the BUG below. It's true that we can't hold this lock while calling pnfs_put_lseg, because that might try to lock the inode lock - which might be the same lock as cinfo->lock. To reproduce, mount a 2 DS pynfs server and run an O_DIRECT command that crosses a stripe boundary and is not page aligned, such as: dd if=/dev/zero of=/mnt/f bs=17000 count=1 oflag=direct BUG: sleeping function called from invalid context at linux/fs/nfs/nfs4filelayout.c:1161 in_atomic(): 0, irqs_disabled(): 0, pid: 27, name: kworker/0:1 2 locks held by kworker/0:1/27: #0: (events){.+.+.+}, at: [] process_one_work+0x175/0x3a5 #1: ((&dreq->work)){+.+...}, at: [] process_one_work+0x175/0x3a5 CPU: 0 PID: 27 Comm: kworker/0:1 Not tainted 3.13.0-rc3-branch-dros_testing+ #21 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013 Workqueue: events nfs_direct_write_schedule_work [nfs] 0000000000000000 ffff88007a39bbb8 ffffffff81491256 ffff88007b87a130 ffff88007a39bbd8 ffffffff8105f103 ffff880079614000 ffff880079617d40 ffff88007a39bc20 ffffffffa011603e ffff880078988b98 0000000000000000 Call Trace: [] dump_stack+0x4d/0x66 [] __might_sleep+0x100/0x105 [] transfer_commit_list+0x94/0xf1 [nfs_layout_nfsv41_files] [] filelayout_recover_commit_reqs+0x3b/0x68 [nfs_layout_nfsv41_files] [] nfs_direct_write_reschedule+0x9f/0x1d6 [nfs] [] ? mark_lock+0x1df/0x224 [] ? trace_hardirqs_off_caller+0x37/0xa4 [] ? trace_hardirqs_off+0xd/0xf [] nfs_direct_write_schedule_work+0x9d/0xb7 [nfs] [] ? process_one_work+0x175/0x3a5 [] process_one_work+0x1f6/0x3a5 [] ? process_one_work+0x175/0x3a5 [] worker_thread+0x149/0x1f5 [] ? rescuer_thread+0x28d/0x28d [] kthread+0xd2/0xda [] ? __kthread_parkme+0x61/0x61 [] ret_from_fork+0x7c/0xb0 [] ? __kthread_parkme+0x61/0x61 Signed-off-by: Weston Andros Adamson --- I'm pretty sure this is the correct approach - it certainly fixes the BUG for me. fs/nfs/nfs4filelayout.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/fs/nfs/nfs4filelayout.c b/fs/nfs/nfs4filelayout.c index 0a93e79..03fd8be 100644 --- a/fs/nfs/nfs4filelayout.c +++ b/fs/nfs/nfs4filelayout.c @@ -1216,17 +1216,17 @@ static void filelayout_recover_commit_reqs(struct list_head *dst, struct pnfs_commit_bucket *b; int i; - /* NOTE cinfo->lock is NOT held, relying on fact that this is - * only called on single thread per dreq. - * Can't take the lock because need to do pnfs_put_lseg - */ + spin_lock(cinfo->lock); for (i = 0, b = cinfo->ds->buckets; i < cinfo->ds->nbuckets; i++, b++) { if (transfer_commit_list(&b->written, dst, cinfo, 0)) { + spin_unlock(cinfo->lock); pnfs_put_lseg(b->wlseg); b->wlseg = NULL; + spin_lock(cinfo->lock); } } cinfo->ds->nwritten = 0; + spin_unlock(cinfo->lock); } static unsigned int