From patchwork Mon Jul 18 06:02:15 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Chinner X-Patchwork-Id: 9234083 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 6B2436075D for ; Mon, 18 Jul 2016 06:02:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5DF4C2094D for ; Mon, 18 Jul 2016 06:02:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 52976228C8; Mon, 18 Jul 2016 06:02:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0DDB4212F2 for ; Mon, 18 Jul 2016 06:02:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751811AbcGRGCX (ORCPT ); Mon, 18 Jul 2016 02:02:23 -0400 Received: from ipmail07.adl2.internode.on.net ([150.101.137.131]:16600 "EHLO ipmail07.adl2.internode.on.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751802AbcGRGCV (ORCPT ); Mon, 18 Jul 2016 02:02:21 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2ASDwA+cIxXEHvHLHlcgz+BUoZvnRIBAQEBAQEGjFWKEYYUAgIBAQKBK00BAQEBAQEHAQEBAQEBAj5AhFwBAQUnExwjEAgDFQMJJQ8FJQMHGhOIL78YAQEBAQEFAQEBAQEBIR6FRIUVhAgUAYNPgi8FhgpMkk6FKoZXglSBdYgIhUSGX4k/gl8cgV4qMoV2AQ4XgR4BAQE Received: from ppp121-44-199-123.lns20.syd7.internode.on.net (HELO dastard) ([121.44.199.123]) by ipmail07.adl2.internode.on.net with ESMTP; 18 Jul 2016 15:32:16 +0930 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1bP1d9-0004F4-I0; Mon, 18 Jul 2016 16:02:15 +1000 Date: Mon, 18 Jul 2016 16:02:15 +1000 From: Dave Chinner To: Calvin Owens Cc: linux-block@vger.kernel.org, kernel-team@fb.com, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, xfs@oss.sgi.com Subject: Re: [BUG] Slab corruption during XFS writeback under memory pressure Message-ID: <20160718060215.GB16044@dastard> References: <28f77d74-5ab4-d913-2921-df90da53f393@fb.com> <20160717000003.GW1922@dastard> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20160717000003.GW1922@dastard> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Sun, Jul 17, 2016 at 10:00:03AM +1000, Dave Chinner wrote: > On Fri, Jul 15, 2016 at 05:18:02PM -0700, Calvin Owens wrote: > > Hello all, > > > > I've found a nasty source of slab corruption. Based on seeing similar symptoms > > on boxes at Facebook, I suspect it's been around since at least 3.10. > > > > It only reproduces under memory pressure so far as I can tell: the issue seems > > to be that XFS reclaims pages from buffers that are still in use by > > scsi/block. I'm not sure which side the bug lies on, but I've only observed it > > with XFS. [....] > But this indicates that the page is under writeback at this point, > so that tends to indicate that the above freeing was incorrect. > > Hmmm - it's clear we've got direct reclaim involved here, and the > suspicion of a dirty page that has had it's bufferheads cleared. > Are there any other warnings in the log from XFS prior to kasan > throwing the error? Can you try the patch below? -Dave. Tested-by: Calvin Owens diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c index 80714eb..0cfb944 100644 --- a/fs/xfs/xfs_aops.c +++ b/fs/xfs/xfs_aops.c @@ -87,6 +87,12 @@ xfs_find_bdev_for_inode( * We're now finished for good with this page. Update the page state via the * associated buffer_heads, paying attention to the start and end offsets that * we need to process on the page. + * + * Landmine Warning: bh->b_end_io() will call end_page_writeback() on the last + * buffer in the IO. Once it does this, it is unsafe to access the bufferhead or + * the page at all, as we may be racing with memory reclaim and it can free both + * the bufferhead chain and the page as it will see the page as clean and + * unused. */ static void xfs_finish_page_writeback( @@ -95,8 +101,9 @@ xfs_finish_page_writeback( int error) { unsigned int end = bvec->bv_offset + bvec->bv_len - 1; - struct buffer_head *head, *bh; + struct buffer_head *head, *bh, *next; unsigned int off = 0; + unsigned int bsize; ASSERT(bvec->bv_offset < PAGE_SIZE); ASSERT((bvec->bv_offset & ((1 << inode->i_blkbits) - 1)) == 0); @@ -105,15 +112,17 @@ xfs_finish_page_writeback( bh = head = page_buffers(bvec->bv_page); + bsize = bh->b_size; do { + next = bh->b_this_page; if (off < bvec->bv_offset) goto next_bh; if (off > end) break; bh->b_end_io(bh, !error); next_bh: - off += bh->b_size; - } while ((bh = bh->b_this_page) != head); + off += bsize; + } while ((bh = next) != head); } /*