From patchwork Mon Jan 2 21:11:36 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 9494151 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 7C6B9606A6 for ; Mon, 2 Jan 2017 21:14:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6D0FF1FF35 for ; Mon, 2 Jan 2017 21:14:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 614FB2684F; Mon, 2 Jan 2017 21:14:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DC96B1FF35 for ; Mon, 2 Jan 2017 21:14:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751536AbdABVMD (ORCPT ); Mon, 2 Jan 2017 16:12:03 -0500 Received: from gum.cmpxchg.org ([85.214.110.215]:35716 "EHLO gum.cmpxchg.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755602AbdABVLu (ORCPT ); Mon, 2 Jan 2017 16:11:50 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cmpxchg.org ; s=x; h=In-Reply-To:Content-Type:MIME-Version:References:Message-ID:Subject: Cc:To:From:Date:Sender:Reply-To:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=dvdADJ16UaV3aVcKcIBtG9E64K/4brJAw6vJWTwG8bg=; b=QfS/jfcC45J6dZ/UDbfN+kkALU sHcaaoC7SyDSidybdPvLWfGbIxXVI9s/tt1r+Z7J2wXb8X3WYrdlYGOQvc0VOEOUX+AYoVQOp0ZFc yS9q4klG6DZ3M9STZxHPDTukTBP9Yf1usQBkreqlWSYaQyCcNBpqxDJbkbXGdyNIT6zk=; Date: Mon, 2 Jan 2017 16:11:36 -0500 From: Johannes Weiner To: Hugh Dickins Cc: Linus Torvalds , Dave Chinner , Chris Leech , Linux Kernel Mailing List , Lee Duncan , open-iscsi@googlegroups.com, Linux SCSI List , linux-block@vger.kernel.org, Christoph Hellwig , Jan Kara , Andrea Arcangeli Subject: Re: [4.10, panic, regression] iscsi: null pointer deref at iscsi_tcp_segment_done+0x20d/0x2e0 Message-ID: <20170102211136.GA3189@cmpxchg.org> References: <20161214222953.GI4326@dastard> <20161216185906.t2wmrr6wqjdsrduw@straylight.hirudinean.org> <20161221221638.GD4758@dastard> <20161222001303.nvrtm22szn3hgxar@straylight.hirudinean.org> <20161222051322.GF4758@dastard> <20161223073241.GA13584@cmpxchg.org> <20161223083329.GA13952@cmpxchg.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20161223083329.GA13952@cmpxchg.org> User-Agent: Mutt/1.7.1 (2016-10-04) Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Fri, Dec 23, 2016 at 03:33:29AM -0500, Johannes Weiner wrote: > On Fri, Dec 23, 2016 at 02:32:41AM -0500, Johannes Weiner wrote: > > On Thu, Dec 22, 2016 at 12:22:27PM -0800, Hugh Dickins wrote: > > > On Wed, 21 Dec 2016, Linus Torvalds wrote: > > > > On Wed, Dec 21, 2016 at 9:13 PM, Dave Chinner wrote: > > > > > I unmounted the fs, mkfs'd it again, ran the > > > > > workload again and about a minute in this fired: > > > > > > > > > > [628867.607417] ------------[ cut here ]------------ > > > > > [628867.608603] WARNING: CPU: 2 PID: 16925 at mm/workingset.c:461 shadow_lru_isolate+0x171/0x220 > > > > > > > > Well, part of the changes during the merge window were the shadow > > > > entry tracking changes that came in through Andrew's tree. Adding > > > > Johannes Weiner to the participants. > > > > > > > > > Now, this workload does not touch the page cache at all - it's > > > > > entirely an XFS metadata workload, so it should not really be > > > > > affecting the working set code. > > > > > > > > Well, I suspect that anything that creates memory pressure will end up > > > > triggering the working set code, so .. > > > > > > > > That said, obviously memory corruption could be involved and result in > > > > random issues too, but I wouldn't really expect that in this code. > > > > > > > > It would probably be really useful to get more data points - is the > > > > problem reliably in this area, or is it going to be random and all > > > > over the place. > > > > > > Data point: kswapd got WARNING on mm/workingset.c:457 in shadow_lru_isolate, > > > soon followed by NULL pointer deref in list_lru_isolate, one time when > > > I tried out Sunday's git tree. Not seen since, I haven't had time to > > > investigate, just set it aside as something to worry about if it happens > > > again. But it looks like shadow_lru_isolate() has issues beyond Dave's > > > case (I've no XFS and no iscsi), suspect unrelated to his other problems. > > > > This seems consistent with what Dave observed: we encounter regular > > pages in radix tree nodes on the shadow LRU that should only contain > > nodes full of exceptional shadow entries. It could be an issue in the > > new slot replacement code and the node tracking callback. > > Both encounters seem to indicate use-after-free. Dave's node didn't > warn about an unexpected node->count / node->exceptional state, but > had entries that were inconsistent with that. Hugh got the counter > warning but crashed on a list_head that's not NULLed in a live node. > > workingset_update_node() should be called on page cache radix tree > leaf nodes that go empty. I must be missing an update_node callback > where a leaf node gets freed somewhere. Sorry for dropping silent on this. I'm traveling over the holidays with sporadic access to my emails and no access to real equipment. The times I managed to sneak away to look at the code didn't turn up anything useful yet. Andrea encountered the warning as well and I gave him a debugging patch (attached below), but he hasn't been able to reproduce this condition. I've personally never seen the warning trigger, even though the patches have been running on my main development machine for quite a while now. Albeit against an older base; I've updated to Linus's master branch now in case it's an interaction with other new code. If anybody manages to reproduce this, that would be helpful. Any extra eyes on this would be much appreciated too until I'm back at my desk. Thanks --- To unsubscribe from this list: send the line "unsubscribe linux-block" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/lib/radix-tree.c b/lib/radix-tree.c index 6f382e07de77..0783af1c0ebb 100644 --- a/lib/radix-tree.c +++ b/lib/radix-tree.c @@ -640,6 +640,8 @@ static inline void radix_tree_shrink(struct radix_tree_root *root, update_node(node, private); } + WARN_ON_ONCE(!list_empty(&node->private_list)); + radix_tree_node_free(node); } } @@ -666,6 +668,8 @@ static void delete_node(struct radix_tree_root *root, root->rnode = NULL; } + WARN_ON_ONCE(!list_empty(&node->private_list)); + radix_tree_node_free(node); node = parent; @@ -767,6 +771,7 @@ static void radix_tree_free_nodes(struct radix_tree_node *node) struct radix_tree_node *old = child; offset = child->offset + 1; child = child->parent; + WARN_ON_ONCE(!list_empty(&node->private_list)); radix_tree_node_free(old); if (old == entry_to_node(node)) return;