From patchwork Thu Jun 19 00:08:33 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 4379971 Return-Path: X-Original-To: patchwork-linux-btrfs@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 742419F3DF for ; Thu, 19 Jun 2014 00:08:43 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 7A03C20382 for ; Thu, 19 Jun 2014 00:08:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id CE3F720380 for ; Thu, 19 Jun 2014 00:08:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755787AbaFSAIh (ORCPT ); Wed, 18 Jun 2014 20:08:37 -0400 Received: from g2t2354.austin.hp.com ([15.217.128.53]:36351 "EHLO g2t2354.austin.hp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753772AbaFSAIh (ORCPT ); Wed, 18 Jun 2014 20:08:37 -0400 Received: from g2t2360.austin.hp.com (g2t2360.austin.hp.com [16.197.8.247]) by g2t2354.austin.hp.com (Postfix) with ESMTP id DC53E11E; Thu, 19 Jun 2014 00:08:35 +0000 (UTC) Received: from [10.152.32.90] (ospra1.fc.hp.com [16.79.38.118]) by g2t2360.austin.hp.com (Postfix) with ESMTP id 7966E55; Thu, 19 Jun 2014 00:08:34 +0000 (UTC) Message-ID: <53A22A01.7080505@hp.com> Date: Wed, 18 Jun 2014 20:08:33 -0400 From: Waiman Long User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20130109 Thunderbird/10.0.12 MIME-Version: 1.0 To: Marc Dionne CC: Chris Mason , Josef Bacik , linux-btrfs@vger.kernel.org, t-itoh@jp.fujitsu.com Subject: Re: Lockups with btrfs on 3.16-rc1 - bisected References: <53A20FFF.3010807@hp.com> <53A2125B.3050701@fb.com> <53A21702.8090109@hp.com> <53A21C78.1040809@fb.com> <53A21E84.2050103@hp.com> <53A22064.7080400@fb.com> <53A2212E.7090907@hp.com> <53A2268F.3080807@fb.com> In-Reply-To: Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On 06/18/2014 08:03 PM, Marc Dionne wrote: > On Wed, Jun 18, 2014 at 7:53 PM, Chris Mason wrote: >> On 06/18/2014 07:30 PM, Waiman Long wrote: >>> On 06/18/2014 07:27 PM, Chris Mason wrote: >>>> On 06/18/2014 07:19 PM, Waiman Long wrote: >>>>> On 06/18/2014 07:10 PM, Josef Bacik wrote: >>>>>> On 06/18/2014 03:47 PM, Waiman Long wrote: >>>>>>> On 06/18/2014 06:27 PM, Josef Bacik wrote: >>>>>>>> On 06/18/2014 03:17 PM, Waiman Long wrote: >>>>>>>>> On 06/18/2014 04:57 PM, Marc Dionne wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I've been seeing very reproducible soft lockups with 3.16-rc1 >>>>>>>>>> similar >>>>>>>>>> to what is reported here: >>>>>>>>>> https://urldefense.proofpoint.com/v1/url?u=http://marc.info/?l%3Dlinux-btrfs%26m%3D140290088532203%26w%3D2&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=cKCbChRKsMpTX8ybrSkonQ%3D%3D%0A&m=aoagvtZMwVb16gh1HApZZL00I7eP50GurBpuEo3l%2B5g%3D%0A&s=c62558feb60a480bbb52802093de8c97b5e1f23d4100265b6120c8065bd99565 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> , along with the >>>>>>>>>> occasional hard lockup, making it impossible to complete a parallel >>>>>>>>>> build on a btrfs filesystem for the package I work on. This was >>>>>>>>>> working fine just a few days before rc1. >>>>>>>>>> >>>>>>>>>> Bisecting brought me to the following commit: >>>>>>>>>> >>>>>>>>>> commit bd01ec1a13f9a327950c8e3080096446c7804753 >>>>>>>>>> Author: Waiman Long >>>>>>>>>> Date: Mon Feb 3 13:18:57 2014 +0100 >>>>>>>>>> >>>>>>>>>> x86, locking/rwlocks: Enable qrwlocks on x86 >>>>>>>>>> >>>>>>>>>> And sure enough if I revert that commit on top of current mainline, >>>>>>>>>> I'm unable to reproduce the soft lockups and hangs. >>>>>>>>>> >>>>>>>>>> Marc >>>>>>>>> The queue rwlock is fair. As a result, recursive read_lock is not >>>>>>>>> allowed unless the task is in an interrupt context. Doing recursive >>>>>>>>> read_lock will hang the process when a write_lock happens >>>>>>>>> somewhere in >>>>>>>>> between. Are recursive read_lock being done in the btrfs code? >>>>>>>>> >>>>>>>> We walk down a tree and read lock each node as we walk down, is that >>>>>>>> what you mean? Or do you mean read_lock multiple times on the same >>>>>>>> lock in the same process, cause we definitely don't do that. Thanks, >>>>>>>> >>>>>>>> Josef >>>>>>> I meant recursively read_lock the same lock in a process. >>>>>> I take it back, we do actually do this in some cases. Thanks, >>>>>> >>>>>> Josef >>>>> This is what I thought when I looked at the looking code in btrfs. The >>>>> unlock code doesn't clear the lock_owner pid, this may cause the >>>>> lock_nested to be set incorrectly. >>>>> >>>>> Anyway, are you going to do something about it? >>>> Thanks for reporting this, we shouldn't be actually taking the lock >>>> recursively. Could you please try with lockdep enabled? If the problem >>>> goes away with lockdep on, I think I know what's causing it. Otherwise, >>>> lockdep should clue us in. >>>> >>>> -chris >>> I am not sure if lockdep will report recursive read_lock as this is >>> possible in the past. If not, we certainly need to add that capability >>> to it. >>> >>> One more thing, I saw comment in btrfs tree locking code about taking a >>> read lock after taking a write (partial?) lock. That is not possible >>> with even with the old rwlock code. >> With lockdep on, the clear_path_blocking function you're hitting >> softlockups in is different. Futjitsu hit a similar problem during >> quota rescans, and it goes away with lockdep on. I'm trying to nail >> down where we went wrong, but please try lockdep on. >> >> -chris > With lockdep on I'm unable to reproduce the lockups, and there are no > lockdep warnings. > > Marc Enabling lockdep may change the lock timing that make it hard to reproduce the problem. Anyway, could you try to apply the following patch to see if it shows any warning? -Longman * We're holding the nest_lock, which serializes this lock's @@ -1852,8 +1862,10 @@ check_prev_add(struct task_struct *curr, struct held_lock * write-lock never takes any other locks, then the reads are * equivalent to a NOP. */ +#ifndef CONFIG_QUEUE_RWLOCK if (next->read == 2 || prev->read == 2) return 1; +#endif /* * Is the -> dependency already present? * --- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c index d24e433..b6c9f2e 100644 --- a/kernel/locking/lockdep.c +++ b/kernel/locking/lockdep.c @@ -1766,12 +1766,22 @@ check_deadlock(struct task_struct *curr, struct held_loc if (hlock_class(prev) != hlock_class(next)) continue; +#ifdef CONFIG_QUEUE_RWLOCK + /* + * Queue rwlock only allows read-after-read recursion of the + * same lock class when the latter read is in an interrupt + * context. + */ + if ((read == 2) && prev->read && in_interrupt()) + return 2; +#else /* * Allow read-after-read recursion of the same * lock class (i.e. read_lock(lock)+read_lock(lock)): */ if ((read == 2) && prev->read) return 2; +#endif /*