From patchwork Sat Apr 25 17:31:05 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Filipe Manana X-Patchwork-Id: 6275021 Return-Path: X-Original-To: patchwork-linux-btrfs@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 880FBBF4A6 for ; Sat, 25 Apr 2015 17:31:33 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 6B09F2024F for ; Sat, 25 Apr 2015 17:31:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 19E452024D for ; Sat, 25 Apr 2015 17:31:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S966638AbbDYRb1 (ORCPT ); Sat, 25 Apr 2015 13:31:27 -0400 Received: from victor.provo.novell.com ([137.65.250.26]:39521 "EHLO prv3-mh.provo.novell.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S964961AbbDYRb1 (ORCPT ); Sat, 25 Apr 2015 13:31:27 -0400 Received: from debian3.lan (prv-ext-foundry1int.gns.novell.com [137.65.251.240]) by prv3-mh.provo.novell.com with ESMTP (NOT encrypted); Sat, 25 Apr 2015 11:31:12 -0600 From: Filipe Manana To: linux-btrfs@vger.kernel.org Cc: clm@fb.com, Filipe Manana Subject: [PATCH] Btrfs: fix deadlock when starting writeback of bg caches Date: Sat, 25 Apr 2015 18:31:05 +0100 Message-Id: <1429983065-22174-1-git-send-email-fdmanana@suse.com> X-Mailer: git-send-email 2.1.3 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP While starting the writes of the dirty block group caches, if we don't find a block group item in the extent tree we were leaving without releasing our path, running delayed references and then looping again to process any new dirty block groups. However this second iteration of the loop could cause a deadlock because it tries to lock some other extent tree node/leaf which another task already locked and it's blocked because it's waiting for a lock on some node/leaf that is in our path that was not released before. We could also deadlock when running the delayed references - as we could end up trying to lock the same nodes/leafs that we have in our local path (with a different lock type). Got into such case when running xfstests: [20892.242791] ------------[ cut here ]------------ [20892.243776] WARNING: CPU: 0 PID: 13299 at fs/btrfs/super.c:260 __btrfs_abort_transaction+0x52/0x114 [btrfs]() [20892.245874] BTRFS: Transaction aborted (error -2) (...) [20892.269378] Call Trace: [20892.269915] [] dump_stack+0x4f/0x7b [20892.271097] [] ? console_unlock+0x361/0x3ad [20892.272173] [] warn_slowpath_common+0xa1/0xbb [20892.273386] [] ? __btrfs_abort_transaction+0x52/0x114 [btrfs] [20892.274857] [] warn_slowpath_fmt+0x46/0x48 [20892.275851] [] __btrfs_abort_transaction+0x52/0x114 [btrfs] [20892.277341] [] write_one_cache_group+0x68/0xaf [btrfs] [20892.278628] [] btrfs_start_dirty_block_groups+0x18d/0x29b [btrfs] [20892.280191] [] btrfs_commit_transaction+0x130/0x9c9 [btrfs] (...) [20892.291316] ---[ end trace 597f77e664245373 ]--- [20892.293955] BTRFS: error (device sdg) in write_one_cache_group:3184: errno=-2 No such entry [20892.297390] BTRFS info (device sdg): forced readonly [20892.298222] ------------[ cut here ]------------ [20892.299190] WARNING: CPU: 0 PID: 13299 at fs/btrfs/ctree.c:2683 btrfs_search_slot+0x7e/0x7d2 [btrfs]() (...) [20892.326253] Call Trace: [20892.326904] [] dump_stack+0x4f/0x7b [20892.329503] [] ? console_unlock+0x361/0x3ad [20892.330815] [] warn_slowpath_common+0xa1/0xbb [20892.332556] [] ? btrfs_search_slot+0x7e/0x7d2 [btrfs] [20892.333955] [] warn_slowpath_null+0x1a/0x1c [20892.335562] [] btrfs_search_slot+0x7e/0x7d2 [btrfs] [20892.336849] [] ? arch_local_irq_save+0x9/0xc [20892.338222] [] ? cache_save_setup+0x43/0x2a5 [btrfs] [20892.339823] [] ? cache_save_setup+0x57/0x2a5 [btrfs] [20892.341275] [] ? _raw_spin_unlock+0x32/0x46 [20892.342810] [] write_one_cache_group+0x3f/0xaf [btrfs] [20892.344184] [] btrfs_start_dirty_block_groups+0x18d/0x29b [btrfs] [20892.347162] [] btrfs_commit_transaction+0x130/0x9c9 [btrfs] (...) [20892.361015] ---[ end trace 597f77e664245374 ]--- [21120.688097] INFO: task kworker/u8:17:29854 blocked for more than 120 seconds. [21120.689881] Tainted: G W 4.0.0-rc5-btrfs-next-9+ #2 [21120.691384] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. (...) [21120.703696] Call Trace: [21120.704310] [] schedule+0x74/0x83 [21120.705490] [] btrfs_tree_lock+0xd7/0x236 [btrfs] [21120.706757] [] ? signal_pending_state+0x31/0x31 [21120.708156] [] lock_extent_buffer_for_io+0x3e/0x194 [btrfs] [21120.709892] [] ? btree_write_cache_pages+0x273/0x385 [btrfs] [21120.711605] [] btree_write_cache_pages+0x32f/0x385 [btrfs] [21120.723440] [] btree_writepages+0x23/0x5c [btrfs] [21120.724943] [] do_writepages+0x23/0x2c [21120.726008] [] __writeback_single_inode+0x73/0x2fa [21120.727230] [] ? writeback_sb_inodes+0xe5/0x38b [21120.728526] [] ? writeback_sb_inodes+0x196/0x38b [21120.729701] [] writeback_sb_inodes+0x205/0x38b (...) [21120.747853] INFO: task btrfs:13282 blocked for more than 120 seconds. [21120.749459] Tainted: G W 4.0.0-rc5-btrfs-next-9+ #2 [21120.751137] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. (...) [21120.768457] Call Trace: [21120.769039] [] schedule+0x74/0x83 [21120.770107] [] btrfs_commit_transaction+0x315/0x9c9 [btrfs] [21120.771558] [] ? signal_pending_state+0x31/0x31 [21120.773659] [] prepare_to_relocate+0xcb/0xd2 [btrfs] [21120.776257] [] relocate_block_group+0x44/0x4a9 [btrfs] [21120.777755] [] ? btrfs_relocate_block_group+0x161/0x288 [btrfs] [21120.779459] [] btrfs_relocate_block_group+0x169/0x288 [btrfs] [21120.781153] [] btrfs_relocate_chunk.isra.29+0x3e/0xa7 [btrfs] [21120.783918] [] btrfs_balance+0xaa4/0xc52 [btrfs] [21120.785436] [] ? cpu_cache_get.isra.39+0xe/0x1f [21120.786434] [] btrfs_ioctl_balance+0x23f/0x2b0 [btrfs] (...) [21120.889251] INFO: task fsstress:13288 blocked for more than 120 seconds. [21120.890526] Tainted: G W 4.0.0-rc5-btrfs-next-9+ #2 [21120.891773] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. (...) [21120.899960] Call Trace: [21120.900743] [] schedule+0x74/0x83 [21120.903004] [] btrfs_tree_lock+0xd7/0x236 [btrfs] [21120.904383] [] ? signal_pending_state+0x31/0x31 [21120.905608] [] btrfs_search_slot+0x766/0x7d2 [btrfs] [21120.906812] [] ? virt_to_head_page+0x9/0x2c [21120.907874] [] ? cache_alloc_debugcheck_after.isra.42+0x16c/0x1cb [21120.909551] [] btrfs_insert_empty_items+0x5d/0xa8 [btrfs] [21120.910914] [] btrfs_insert_item+0x5a/0xa5 [btrfs] [21120.912181] [] ? btrfs_create_pending_block_groups+0x96/0x130 [btrfs] [21120.913784] [] btrfs_create_pending_block_groups+0xaf/0x130 [btrfs] [21120.915374] [] __btrfs_end_transaction+0x84/0x366 [btrfs] [21120.916735] [] btrfs_end_transaction+0x10/0x12 [btrfs] [21120.917996] [] btrfs_check_data_free_space+0x11f/0x27c [btrfs] [21120.919478] [] btrfs_delalloc_reserve_space+0x1e/0x51 [btrfs] [21120.921226] [] btrfs_truncate_page+0x85/0x2c4 [btrfs] [21120.923121] [] btrfs_cont_expand+0x41/0x3ef [btrfs] [21120.924449] [] ? btrfs_file_write_iter+0x19a/0x431 [btrfs] [21120.926602] [] ? arch_local_irq_save+0x9/0xc [21120.927769] [] ? btrfs_file_write_iter+0x19a/0x431 [btrfs] [21120.929324] [] ? btrfs_file_write_iter+0x1a9/0x431 [btrfs] [21120.930723] [] btrfs_file_write_iter+0x1e2/0x431 [btrfs] [21120.931897] [] ? get_parent_ip+0xe/0x3e [21120.934446] [] new_sync_write+0x7c/0xa0 [21120.935528] [] vfs_write+0xb2/0x117 (...) Fixes: 1bbc621ef284 ("Btrfs: allow block group cache writeout outside critical section in commit") Signed-off-by: Filipe Manana --- fs/btrfs/extent-tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index d5f3ef0..0ec8e22 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-tree.c @@ -3178,8 +3178,8 @@ static int write_one_cache_group(struct btrfs_trans_handle *trans, bi = btrfs_item_ptr_offset(leaf, path->slots[0]); write_extent_buffer(leaf, &cache->item, bi, sizeof(cache->item)); btrfs_mark_buffer_dirty(leaf); - btrfs_release_path(path); fail: + btrfs_release_path(path); if (ret) btrfs_abort_transaction(trans, root, ret); return ret;