From patchwork Fri Jul 20 10:36:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tetsuo Handa X-Patchwork-Id: 10536431 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 6AD7D6029B for ; Fri, 20 Jul 2018 10:37:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 567D928D09 for ; Fri, 20 Jul 2018 10:37:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 46B5928D50; Fri, 20 Jul 2018 10:37:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 93B7928D09 for ; Fri, 20 Jul 2018 10:37:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6CC5D6B0006; Fri, 20 Jul 2018 06:37:06 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 67C5C6B0007; Fri, 20 Jul 2018 06:37:06 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 56BB56B000A; Fri, 20 Jul 2018 06:37:06 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-it0-f71.google.com (mail-it0-f71.google.com [209.85.214.71]) by kanga.kvack.org (Postfix) with ESMTP id 2D34A6B0006 for ; Fri, 20 Jul 2018 06:37:06 -0400 (EDT) Received: by mail-it0-f71.google.com with SMTP id g6-v6so8445829iti.7 for ; Fri, 20 Jul 2018 03:37:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:to:cc :references:from:message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=VZb10+t9AIPsGJDlmrqk6h20AbU+MAtwVO5HfaATsqg=; b=MVMadzQksLzWSfcGkeGxqxwOSvEYXMpYSqfqTZyY6F1q99W5QWyHSn+imsgbgx/Iox R1fdhcWD9VJKbQEhIdl3o9A33Hm2Ab01QNrGBpI0pQrXDv+ntgIen3UeZbm06jdczbNB rIHD6sdDay+8odcLjOMRIXbi4qqoeWycmvnqWhvjXTmyxVgYhB9VLseJIYbuH2xxieH4 XpiPJkleckntAG5FOOzpXM8HCr1V0urwon5PrwJdbo7wj42XuPa70cPenf0VkeFQqEs/ Zf1NKaLTcJN09eMMqqNCrunSCHQmcMSrwxWhlMOa+RvGBXImIRZP98K11NkaTEteyWVm /zXg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of penguin-kernel@i-love.sakura.ne.jp designates 202.181.97.72 as permitted sender) smtp.mailfrom=penguin-kernel@i-love.sakura.ne.jp X-Gm-Message-State: AOUpUlGaousc+N24JKyf4KXSiVn6EplRyOCXARo4y7NKZaNfsE81VvbA f1NKG8PzdXNfKAG+FPDuuI21Sto8+oYjLV7/Hb/UV1CiHogCFbPfwvDOSC4REUlBEk94s9eIFqQ WmoZS22CZsQNTBqtiyPEJl1yNhfXZGDXYn8KE+sbQHnPoGOGHpaEWFOWbBL95v4OMnQ== X-Received: by 2002:a24:e506:: with SMTP id g6-v6mr1411593iti.117.1532083025957; Fri, 20 Jul 2018 03:37:05 -0700 (PDT) X-Google-Smtp-Source: AAOMgpeN0ioOyGcU7BJwyviwYM59tzLLDv75dYI3Qj9bHksm5sJZ8DDtEA2S/DSKqM2KUBm3Y9Cr X-Received: by 2002:a24:e506:: with SMTP id g6-v6mr1411560iti.117.1532083025125; Fri, 20 Jul 2018 03:37:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1532083025; cv=none; d=google.com; s=arc-20160816; b=Wbho3HhZB/SfQEjKrZqkXUio7H0LG1CAvo3uyMvsiwYuzSdVroLQV+d5MJgHwCoVqq B5VhHgdJ6fY3EIs2tZ2A/MTCq9PwuhHvZVLU1+F1R4Rss+ayMZbNsjMDabC22LLtuv5S b9izfpTT/jPs1P/c/g80fRShdKgiC1q/vZwjcHGs5fxuSFr5iPqIMm3mqcpG9gxW6Ky9 XxVS9MX8Tsp4zVTA1JXlJJDSwZ+eQupyhGWx0nlrKcQAuJoPtgiVPLgmEEqe2TNKkcuS WZ2RggQonAOwhgZ1MgrhNCRSG99juT3BOeestyWyrH8xl7NSbZb3WzVb1slcpQDL9PZ6 HUJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=VZb10+t9AIPsGJDlmrqk6h20AbU+MAtwVO5HfaATsqg=; b=UjvvExWBCKVcvNGwgAv06WKdWtfrdz6DLFks6Bp5actckgYoACIBYceKvIzNgRMkzu RP7j7x+9GV4piWGC24zGits7uzyReKi/evcMoMmvO0/jUPnC6HB6lG4vw5l/rRavPAR9 MpOtU/+CuFevQpXlrQq7bBRHu0C3m8TKGmy0joq5crKgr68EpIZQQeJUtX+LMZooMn61 PpzV4tmcPKFrgaBX6FErjlCfmImKZrUkIJO7+pF7ebdW1FOApzSwK0OaNg4DkAH4Tofc 6kUVnOcei9lyHKhcx2UqB/DoOuzUHsI/exs1CHoruB8JrERG6eIPaIsFTcrXacbv3aZQ ZFIw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of penguin-kernel@i-love.sakura.ne.jp designates 202.181.97.72 as permitted sender) smtp.mailfrom=penguin-kernel@i-love.sakura.ne.jp Received: from www262.sakura.ne.jp (www262.sakura.ne.jp. [202.181.97.72]) by mx.google.com with ESMTPS id m125-v6si1219200iof.217.2018.07.20.03.37.04 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 20 Jul 2018 03:37:04 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of penguin-kernel@i-love.sakura.ne.jp designates 202.181.97.72 as permitted sender) client-ip=202.181.97.72; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of penguin-kernel@i-love.sakura.ne.jp designates 202.181.97.72 as permitted sender) smtp.mailfrom=penguin-kernel@i-love.sakura.ne.jp Received: from fsav404.sakura.ne.jp (fsav404.sakura.ne.jp [133.242.250.103]) by www262.sakura.ne.jp (8.15.2/8.15.2) with ESMTP id w6KAaMw4027435; Fri, 20 Jul 2018 19:36:22 +0900 (JST) (envelope-from penguin-kernel@I-love.SAKURA.ne.jp) Received: from www262.sakura.ne.jp (202.181.97.72) by fsav404.sakura.ne.jp (F-Secure/fsigk_smtp/530/fsav404.sakura.ne.jp); Fri, 20 Jul 2018 19:36:22 +0900 (JST) X-Virus-Status: clean(F-Secure/fsigk_smtp/530/fsav404.sakura.ne.jp) Received: from [192.168.1.8] (softbank126074194044.bbtec.net [126.74.194.44]) (authenticated bits=0) by www262.sakura.ne.jp (8.15.2/8.15.2) with ESMTPSA id w6KAaLZW027413 (version=TLSv1.2 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 20 Jul 2018 19:36:22 +0900 (JST) (envelope-from penguin-kernel@I-love.SAKURA.ne.jp) Subject: Re: INFO: task hung in generic_file_write_iter To: akpm@linux-foundation.org, Alexander Viro Cc: syzbot , linux-mm@kvack.org, mgorman@techsingularity.net, Michal Hocko , ak@linux.intel.com, jack@suse.cz, jlayton@redhat.com, linux-kernel@vger.kernel.org, mawilcox@microsoft.com, syzkaller-bugs@googlegroups.com, tim.c.chen@linux.intel.com, linux-fsdevel References: <0000000000009ce88d05714242a8@google.com> <4b349bff-8ad4-6410-250d-593b13d8d496@I-love.SAKURA.ne.jp> From: Tetsuo Handa Message-ID: <9b9fcdda-c347-53ee-fdbb-8a7d11cf430e@I-love.SAKURA.ne.jp> Date: Fri, 20 Jul 2018 19:36:23 +0900 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <4b349bff-8ad4-6410-250d-593b13d8d496@I-love.SAKURA.ne.jp> Content-Language: en-US X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP On 2018/07/18 19:28, Tetsuo Handa wrote: > There are many reports which are stalling inside __getblk_gfp(). Currently 18 reports out of 65 "INFO: task hung in " reports. INFO: task hung in aead_recvmsg INFO: task hung in inode_sleep_on_writeback INFO: task hung in __writeback_inodes_sb_nr INFO: task hung in __blkdev_get (2) INFO: task hung in lookup_slow INFO: task hung in iterate_supers INFO: task hung in flush_work INFO: task hung in vfs_setxattr INFO: task hung in lock_mount INFO: task hung in __get_super INFO: task hung in do_unlinkat INFO: task hung in fat_fallocate INFO: task hung in generic_file_write_iter INFO: task hung in d_alloc_parallel INFO: task hung in __fdget_pos (2) INFO: task hung in path_openat INFO: task hung in do_truncate INFO: task hung in filename_create > And there is horrible comment for __getblk_gfp(): > > /* > * __getblk_gfp() will locate (and, if necessary, create) the buffer_head > * which corresponds to the passed block_device, block and size. The > * returned buffer has its reference count incremented. > * > * __getblk_gfp() will lock up the machine if grow_dev_page's > * try_to_free_buffers() attempt is failing. FIXME, perhaps? > */ > > This report is stalling after mount() completed and process used remap_file_pages(). > I think that we might need to use debug printk(). But I don't know what to examine. > Andrew, can you pick up this debug printk() patch? I guess we can get the result within one week. From 8f55e00b21fefffbc6abd9085ac503c52a302464 Mon Sep 17 00:00:00 2001 From: Tetsuo Handa Date: Fri, 20 Jul 2018 19:29:06 +0900 Subject: [PATCH] fs/buffer.c: add debug print for __getblk_gfp() stall problem Among syzbot's unresolved hung task reports, 18 out of 65 reports contain __getblk_gfp() line in the backtrace. Since there is a comment block that says that __getblk_gfp() will lock up the machine if try_to_free_buffers() attempt from grow_dev_page() is failing, let's start from checking whether syzbot is hitting that case. This change will be removed after the bug is fixed. Signed-off-by: Tetsuo Handa Cc: Dmitry Vyukov Signed-off-by: Tetsuo Handa Signed-off-by: Tetsuo Handa --- fs/buffer.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++-- include/linux/sched.h | 7 +++++++ lib/Kconfig.debug | 6 ++++++ 3 files changed, 61 insertions(+), 2 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index be31e28..ebf78ab 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -955,10 +955,20 @@ static sector_t blkdev_max_block(struct block_device *bdev, unsigned int size) end_block = init_page_buffers(page, bdev, (sector_t)index << sizebits, size); +#ifdef CONFIG_DEBUG_AID_FOR_SYZBOT + current->getblk_executed |= 0x01; +#endif goto done; } - if (!try_to_free_buffers(page)) + if (!try_to_free_buffers(page)) { +#ifdef CONFIG_DEBUG_AID_FOR_SYZBOT + current->getblk_executed |= 0x02; +#endif goto failed; + } +#ifdef CONFIG_DEBUG_AID_FOR_SYZBOT + current->getblk_executed |= 0x04; +#endif } /* @@ -978,6 +988,9 @@ static sector_t blkdev_max_block(struct block_device *bdev, unsigned int size) spin_unlock(&inode->i_mapping->private_lock); done: ret = (block < end_block) ? 1 : -ENXIO; +#ifdef CONFIG_DEBUG_AID_FOR_SYZBOT + current->getblk_executed |= 0x08; +#endif failed: unlock_page(page); put_page(page); @@ -1033,6 +1046,12 @@ static sector_t blkdev_max_block(struct block_device *bdev, unsigned int size) return NULL; } +#ifdef CONFIG_DEBUG_AID_FOR_SYZBOT + current->getblk_stamp = jiffies; + current->getblk_executed = 0; + current->getblk_bh_count = 0; + current->getblk_bh_state = 0; +#endif for (;;) { struct buffer_head *bh; int ret; @@ -1044,6 +1063,18 @@ static sector_t blkdev_max_block(struct block_device *bdev, unsigned int size) ret = grow_buffers(bdev, block, size, gfp); if (ret < 0) return NULL; + +#ifdef CONFIG_DEBUG_AID_FOR_SYZBOT + if (!time_after(jiffies, current->getblk_stamp + 3 * HZ)) + continue; + printk(KERN_ERR "%s(%u): getblk(): executed=%x bh_count=%d bh_state=%lx\n", + current->comm, current->pid, current->getblk_executed, + current->getblk_bh_count, current->getblk_bh_state); + current->getblk_executed = 0; + current->getblk_bh_count = 0; + current->getblk_bh_state = 0; + current->getblk_stamp = jiffies; +#endif } } @@ -3216,6 +3247,11 @@ int sync_dirty_buffer(struct buffer_head *bh) */ static inline int buffer_busy(struct buffer_head *bh) { +#ifdef CONFIG_DEBUG_AID_FOR_SYZBOT + current->getblk_executed |= 0x80; + current->getblk_bh_count = atomic_read(&bh->b_count); + current->getblk_bh_state = bh->b_state; +#endif return atomic_read(&bh->b_count) | (bh->b_state & ((1 << BH_Dirty) | (1 << BH_Lock))); } @@ -3254,11 +3290,18 @@ int try_to_free_buffers(struct page *page) int ret = 0; BUG_ON(!PageLocked(page)); - if (PageWriteback(page)) + if (PageWriteback(page)) { +#ifdef CONFIG_DEBUG_AID_FOR_SYZBOT + current->getblk_executed |= 0x10; +#endif return 0; + } if (mapping == NULL) { /* can this still happen? */ ret = drop_buffers(page, &buffers_to_free); +#ifdef CONFIG_DEBUG_AID_FOR_SYZBOT + current->getblk_executed |= 0x20; +#endif goto out; } @@ -3282,6 +3325,9 @@ int try_to_free_buffers(struct page *page) if (ret) cancel_dirty_page(page); spin_unlock(&mapping->private_lock); +#ifdef CONFIG_DEBUG_AID_FOR_SYZBOT + current->getblk_executed |= 0x40; +#endif out: if (buffers_to_free) { struct buffer_head *bh = buffers_to_free; diff --git a/include/linux/sched.h b/include/linux/sched.h index c4e0aba..95b143e 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1207,6 +1207,13 @@ struct task_struct { unsigned long prev_lowest_stack; #endif +#ifdef CONFIG_DEBUG_AID_FOR_SYZBOT + unsigned long getblk_stamp; + unsigned int getblk_executed; + unsigned int getblk_bh_count; + unsigned long getblk_bh_state; +#endif + /* * New fields for task_struct should be added above here, so that * they are included in the randomized portion of task_struct. diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index c731ff9..0747ce7 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2058,3 +2058,9 @@ config IO_STRICT_DEVMEM if the driver using a given range cannot be disabled. If in doubt, say Y. + +config DEBUG_AID_FOR_SYZBOT + bool "Additional debug code for syzbot" + default n + help + This option is intended for testing by syzbot.