From patchwork Tue Jul 29 08:47:02 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Junxiao Bi X-Patchwork-Id: 4639181 Return-Path: X-Original-To: patchwork-ocfs2-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 60EBB9F32F for ; Tue, 29 Jul 2014 08:51:38 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 7F6612015D for ; Tue, 29 Jul 2014 08:51:36 +0000 (UTC) Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7935920149 for ; Tue, 29 Jul 2014 08:51:35 +0000 (UTC) Received: from acsinet21.oracle.com (acsinet21.oracle.com [141.146.126.237]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id s6T8oj3n020967 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 29 Jul 2014 08:50:47 GMT Received: from oss.oracle.com (oss-external.oracle.com [137.254.96.51]) by acsinet21.oracle.com (8.14.4+Sun/8.14.4) with ESMTP id s6T8ofxR020774 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 29 Jul 2014 08:50:41 GMT Received: from localhost ([127.0.0.1] helo=oss.oracle.com) by oss.oracle.com with esmtp (Exim 4.63) (envelope-from ) id 1XC34G-0003WM-2D; Tue, 29 Jul 2014 01:47:32 -0700 Received: from ucsinet21.oracle.com ([156.151.31.93]) by oss.oracle.com with esmtp (Exim 4.63) (envelope-from ) id 1XC33g-0003Vg-AH for ocfs2-devel@oss.oracle.com; Tue, 29 Jul 2014 01:46:56 -0700 Received: from userz7021.oracle.com (userz7021.oracle.com [156.151.31.85]) by ucsinet21.oracle.com (8.14.4+Sun/8.14.4) with ESMTP id s6T8ktuf001626 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Tue, 29 Jul 2014 08:46:55 GMT Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by userz7021.oracle.com (8.14.4+Sun/8.14.4) with ESMTP id s6T8kt2x001621; Tue, 29 Jul 2014 08:46:55 GMT Received: from [10.182.39.153] (/10.182.39.153) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 29 Jul 2014 01:46:54 -0700 Message-ID: <53D75F86.5040109@oracle.com> Date: Tue, 29 Jul 2014 16:47:02 +0800 From: Junxiao Bi User-Agent: Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Andrew Morton References: <1406192280-6643-1-git-send-email-junxiao.bi@oracle.com> <20140728164012.e4c201a4b8c1512468909eaa@linux-foundation.org> In-Reply-To: <20140728164012.e4c201a4b8c1512468909eaa@linux-foundation.org> Cc: mfasheh@suse.com, joe.jin@oracle.com, ocfs2-devel@oss.oracle.com Subject: Re: [Ocfs2-devel] [PATCH] ocfs2: fix journal commit deadlock X-BeenThere: ocfs2-devel@oss.oracle.com X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: ocfs2-devel-bounces@oss.oracle.com Errors-To: ocfs2-devel-bounces@oss.oracle.com X-Source-IP: acsinet21.oracle.com [141.146.126.237] X-Spam-Status: No, score=-4.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Hi Andrew, On 07/29/2014 07:40 AM, Andrew Morton wrote: > On Thu, 24 Jul 2014 16:58:00 +0800 Junxiao Bi wrote: > >> For buffer write, page lock will be got in write_begin and released >> in write_end, in ocfs2_write_end_nolock(), before it unlock the page >> in ocfs2_free_write_ctxt(), it calls ocfs2_run_deallocs(), this will >> ask for the read lock of journal->j_trans_barrier. Holding page lock >> and ask for journal->j_trans_barrier breaks the locking order. >> >> This will cause a deadlock with journal commit threads, ocfs2cmt will >> get write lock of journal->j_trans_barrier first, then it wakes up >> kjournald2 to do the commit work, at last it waits until done. To >> commit journal, kjournald2 needs flushing data first, it needs get >> the cache page lock. >> >> Since some ocfs2 cluster locks are holding by write process, this >> deadlock may hung the whole cluster. >> >> unlock pages before ocfs2_run_deallocs() can fix the locking order, >> also put unlock before ocfs2_commit_trans() to make page lock is >> unlocked before j_trans_barrier to preserve unlocking order. > This conflicts with > http://ozlabs.org/~akpm/mmots/broken-out/ocfs2-call-ocfs2_journal_access_di-before-ocfs2_journal_dirty-in-ocfs2_write_end_nolock.patch > in ways which I am not confident in resolving. Could you please redo > the patch against linux-next and retest? Resolved. See the following fix. @@ -915,7 +915,11 @@ static void ocfs2_free_write_ctxt(struct ocfs2_write_ctxt *wc) page_cache_release(wc->w_target_page); } ocfs2_unlock_and_free_pages(wc->w_pages, wc->w_num_pages); +} +static void ocfs2_free_write_ctxt(struct ocfs2_write_ctxt *wc) +{ + ocfs2_unlock_pages(wc); brelse(wc->w_di_bh); kfree(wc); } @@ -2040,11 +2044,19 @@ out_write_size: ocfs2_journal_dirty(handle, wc->w_di_bh); out: + /* unlock pages before dealloc since it needs acquiring j_trans_barrier + * lock, or it will cause a deadlock since journal commit threads holds + * this lock and will ask for the page lock when flushing the data. + * put it here to preserve the unlock order. + */ + ocfs2_unlock_pages(wc); + ocfs2_commit_trans(osb, handle); ocfs2_run_deallocs(osb, &wc->w_dealloc); - ocfs2_free_write_ctxt(wc); + brelse(wc->w_di_bh); + kfree(wc); return copied; } Thanks, Junxiao. > diff --git a/fs/ocfs2/aops.c b/fs/ocfs2/aops.c index a06b237..2e63621 100644 --- a/fs/ocfs2/aops.c +++ b/fs/ocfs2/aops.c @@ -894,7 +894,7 @@ void ocfs2_unlock_and_free_pages(struct page **pages, int num_pages) } } -static void ocfs2_free_write_ctxt(struct ocfs2_write_ctxt *wc) +static void ocfs2_unlock_pages(struct ocfs2_write_ctxt *wc) { int i;