From patchwork Fri Mar 3 03:01:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13158302 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19DFFC678D4 for ; Fri, 3 Mar 2023 03:02:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD4FE6B0075; Thu, 2 Mar 2023 22:02:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A5D656B0078; Thu, 2 Mar 2023 22:02:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8FDDE6B007B; Thu, 2 Mar 2023 22:02:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 818816B0075 for ; Thu, 2 Mar 2023 22:02:28 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 566ADA0632 for ; Fri, 3 Mar 2023 03:02:28 +0000 (UTC) X-FDA: 80526088776.12.508EE19 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by imf19.hostedemail.com (Postfix) with ESMTP id 1F6731A0007 for ; Fri, 3 Mar 2023 03:02:25 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=j7jBgMsD; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf19.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677812546; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=P7UN7dIaegGLl1PNUKXygcXYp0tGSGNvhXvzdmlN3Hs=; b=rs14cmkJa/KbopBasAN6DlPjmGqprlJf6sLzBPSg09owbqhrnFurFrArMotSZgqebGW/UY gfW7QAtjEIWiEQjjHuhMlyYac412QpuLax6yJuHCMfBkqF3ftC64F4Quk7vTQK2S93Wy1u 0C8QjwpOI+3WshJVCdXwt0K5NpJ1Rj8= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=j7jBgMsD; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf19.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677812546; a=rsa-sha256; cv=none; b=3d7hkPI5nYWu4sb8PNivn4UF9c/RlLYav+garTdBvxZjvLK+lIwnVjvyD7YtxX0IvliXgO 5AKJYTIlePkBlL3N+rrx1DFi01NYZhhL76jftN7J7lwgmk0qo0iL1Rv1K7iVd7/n3xH8yL 28fzxhYykWaoG7SMuQsDVtDb4VFmjXI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677812546; x=1709348546; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dadG4erEiBTCMMwG4DB9rDq0meqMywzEolW1Kh5p2sI=; b=j7jBgMsDfU2F/WxeHalKwT8np/AIiZHTyYHyEcQHP/mNWNmCGesWjk3b JoUDzCUVqJVfuYoWDrAjvTAwiq1YKEs198yOSUVobqYi/5b3ZpF5gtvAA /XJRq6QqyhzyL5wIGZ95Mcnbr+ERv0wsLQ6NqHOaUk/FZcnKcDcq8oAob x9g3oDT2pajXKcWogr8zeTQtwcW03VV92zswtVveCayKg4LQu2K9WbHb0 0PZ60I2KoZgvB8Ubx8gUyCT1hFS8adDaujMqP5M+TeOLGzke1lc2mG3mr PxjBJwGDvNFDhuJQR3gfHNNE1hclFpysaLiUYANtWH4vWFBWNelcXB7/V g==; X-IronPort-AV: E=McAfee;i="6500,9779,10637"; a="421207099" X-IronPort-AV: E=Sophos;i="5.98,229,1673942400"; d="scan'208";a="421207099" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Mar 2023 19:02:25 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10637"; a="668497684" X-IronPort-AV: E=Sophos;i="5.98,229,1673942400"; d="scan'208";a="668497684" Received: from xinyumao-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.255.31.231]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Mar 2023 19:02:21 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Hugh Dickins , "Xu, Pengfei" , Jan Kara , Baolin Wang , Christoph Hellwig , Stefan Roesch , Tejun Heo , Xin Hao , Zi Yan , Yang Shi , Matthew Wilcox , Mike Kravetz Subject: [PATCH -V2 1/3] migrate_pages: fix deadlock in batched migration Date: Fri, 3 Mar 2023 11:01:53 +0800 Message-Id: <20230303030155.160983-2-ying.huang@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230303030155.160983-1-ying.huang@intel.com> References: <20230303030155.160983-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1F6731A0007 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 379za5tkfu1nfnig388tygksjshgbxz3 X-HE-Tag: 1677812545-124349 X-HE-Meta: U2FsdGVkX18iAlJT1JgCGdYHBJnszSYoPLJ3hRZsq+CUVv/QW98gTzlfbcUYpBX6vfQGIV0V+R2X6IWJi06+XFh+yLUJGXTyuHLcvfVgrtB9uqFzQZ2qu2I+97oj96vczjgtCmrGKeZDerrfPCZj2jLczjayXM7cWd9LYqW0nHbrPuNYS4Cbr3mSBAGsUpHYWJRFSA/aurIWXmleT4LcmqBm3AtAIzMG4zea7ZLflPT1AT1kaErYLQKHAjIxE9w8Oi5vJQNS6WFbMEZ6uZDk8UF4agUHJJeI9OeBdaqd2448DL+2eXZ2+PUaNmFJERLIZkHrw5u20MnC5MC6GgFOB8SuLq1BgUrFPvDMWuMSWUau180g/ZmlNk3BKN4hD/7WhOtHzelwXlC5k5AIrLEULdL1RVSpLOnuTNbHddRu5oRWtTSqsqk/xY7fSfRIwpFjdhcpgQEWg31BvgWKit57LHUZHHurJxv+qbpwFR+kjNAwiHT0gKzOwM6sISAe1VRwIGSiEaCHrOo6fNuPc1Z+LSzfYwPGUxu5UHjqxh3PSwM2A+k2AwIPBwRcsfcokXb2zOUeRREWiaj3wEby8u9iYzFMonDqsG8LlQkvweaAlHiqvV7dycQSKjd+nIabf4uxEw2SBEWu6JaZraj6t5EnCgX5ENSPPvCwHjrRoM2c0hGJWPG1KmKYsdzpO2qRsiZVnMq6KV8WftB3CsqDxov4ZN4SdQoGHa5VyWk7QhpfRieYESQLX7t+Lf3u8rOr+2RHJunnbAC41F7Db/qBwgE6fb3L6oVZFjVYIJBiCdSo3uqJHRiiUT1NEc1MQ8Gb6d1wu7LAeWN3i4HYVKYXT+JkG7FTaXDEWkdWJHzZ8+jYR8IoC/h0JF+4vw4zhliLzPStc+zF2GkSb7/5SeitvyfnSp8FeH2+8wt4XRN742Rv3H/VHEt/7g8N/wb99e3DOQ2YpDICk1pmXd3VyycsBUH J5okN6r+ LgsjKRRS2c4z8O6cA+SZhZ1vAp5BuGaRyGuoYgfRTPQLWVr6Pym9ZKal+tF7w59QudQ73cJVeMAjKhdxAPNbMhJy7hyoJXvYy3RPwHUSEP/zlBYU00XzrRveZwxPek286ZD+MmtdDybX7NjKdq2vu2mIAg90rUfdk07eqr35vB8wjdt7vaI+vmFV5L4pqdI6KOOLsZyiWbAjfNzw35KjQo7/qsHt59WiWpElU+rK/OnlBDlVjfsIhGiVDK0yVz0ffbMeC+drPprilSnUQHikb6106C4ypw6BuGlcsCdu/1FEzxdsZJJK+6sfYJcml9cQI7jym/td10dnPxlmqcvtPt4Ter2Foi0na30IlQfIzliTf4Fxcm4cvyv9LKqE3qOk9TqRrlwQ89OPE9JxJ14nLHMGmoklw2FJJdzB8Z0nWQas8tC6dY5D2Ywid+9+XFSqe8r6mj4NAfFIAWJA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Two deadlock bugs were reported for the migrate_pages() batching series. Thanks Hugh and Pengfei! For example, in the following deadlock trace snippet, INFO: task kworker/u4:0:9 blocked for more than 147 seconds. Not tainted 6.2.0-rc4-kvm+ #1314 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:kworker/u4:0 state:D stack:0 pid:9 ppid:2 flags:0x00004000 Workqueue: loop4 loop_rootcg_workfn Call Trace: __schedule+0x43b/0xd00 schedule+0x6a/0xf0 io_schedule+0x4a/0x80 folio_wait_bit_common+0x1b5/0x4e0 ? __pfx_wake_page_function+0x10/0x10 __filemap_get_folio+0x73d/0x770 shmem_get_folio_gfp+0x1fd/0xc80 shmem_write_begin+0x91/0x220 generic_perform_write+0x10e/0x2e0 __generic_file_write_iter+0x17e/0x290 ? generic_write_checks+0x12b/0x1a0 generic_file_write_iter+0x97/0x180 ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 do_iter_readv_writev+0x13c/0x210 ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 do_iter_write+0xf6/0x330 vfs_iter_write+0x46/0x70 loop_process_work+0x723/0xfe0 loop_rootcg_workfn+0x28/0x40 process_one_work+0x3cc/0x8d0 worker_thread+0x66/0x630 ? __pfx_worker_thread+0x10/0x10 kthread+0x153/0x190 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x29/0x50 INFO: task repro:1023 blocked for more than 147 seconds. Not tainted 6.2.0-rc4-kvm+ #1314 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:repro state:D stack:0 pid:1023 ppid:360 flags:0x00004004 Call Trace: __schedule+0x43b/0xd00 schedule+0x6a/0xf0 io_schedule+0x4a/0x80 folio_wait_bit_common+0x1b5/0x4e0 ? compaction_alloc+0x77/0x1150 ? __pfx_wake_page_function+0x10/0x10 folio_wait_bit+0x30/0x40 folio_wait_writeback+0x2e/0x1e0 migrate_pages_batch+0x555/0x1ac0 ? __pfx_compaction_alloc+0x10/0x10 ? __pfx_compaction_free+0x10/0x10 ? __this_cpu_preempt_check+0x17/0x20 ? lock_is_held_type+0xe6/0x140 migrate_pages+0x100e/0x1180 ? __pfx_compaction_free+0x10/0x10 ? __pfx_compaction_alloc+0x10/0x10 compact_zone+0xe10/0x1b50 ? lock_is_held_type+0xe6/0x140 ? check_preemption_disabled+0x80/0xf0 compact_node+0xa3/0x100 ? __sanitizer_cov_trace_const_cmp8+0x1c/0x30 ? _find_first_bit+0x7b/0x90 sysctl_compaction_handler+0x5d/0xb0 proc_sys_call_handler+0x29d/0x420 proc_sys_write+0x2b/0x40 vfs_write+0x3a3/0x780 ksys_write+0xb7/0x180 __x64_sys_write+0x26/0x30 do_syscall_64+0x3b/0x90 entry_SYSCALL_64_after_hwframe+0x72/0xdc RIP: 0033:0x7f3a2471f59d RSP: 002b:00007ffe567f7288 EFLAGS: 00000217 ORIG_RAX: 0000000000000001 RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f3a2471f59d RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000005 RBP: 00007ffe567f72a0 R08: 0000000000000010 R09: 0000000000000010 R10: 0000000000000010 R11: 0000000000000217 R12: 00000000004012e0 R13: 00007ffe567f73e0 R14: 0000000000000000 R15: 0000000000000000 The page migration task has held the lock of the shmem folio A, and is waiting the writeback of the folio B of the file system on the loop block device to complete. While the loop worker task which writes back the folio B is waiting to lock the shmem folio A, because the folio A backs the folio B in the loop device. Thus deadlock is triggered. In general, if we have locked some other folios except the one we are migrating, it's not safe to wait synchronously, for example, to wait the writeback to complete or wait to lock the buffer head. To fix the deadlock, in this patch, we avoid to batch the page migration except for MIGRATE_ASYNC mode. In MIGRATE_ASYNC mode, synchronous waiting is avoided. The fix can be improved further. We will do that as soon as possible. Link: https://lore.kernel.org/linux-mm/87a6c8c-c5c1-67dc-1e32-eb30831d6e3d@google.com/ Link: https://lore.kernel.org/linux-mm/874jrg7kke.fsf@yhuang6-desk2.ccr.corp.intel.com/ Link: https://lore.kernel.org/linux-mm/20230227110614.dngdub2j3exr6dfp@quack3/ Fixes: 5dfab109d519 ("migrate_pages: batch _unmap and _move") Signed-off-by: "Huang, Ying" Reported-by: Hugh Dickins Reported-by: "Xu, Pengfei" Cc: Jan Kara Cc: Baolin Wang Cc: Christoph Hellwig Cc: Stefan Roesch Cc: Tejun Heo Cc: Xin Hao Cc: Zi Yan Cc: Yang Shi Cc: Matthew Wilcox Cc: Mike Kravetz --- mm/migrate.c | 69 ++++++++++++++++++++-------------------------------- 1 file changed, 26 insertions(+), 43 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 04b9b3eeecef..7d751d6a2c15 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1115,7 +1115,7 @@ static void migrate_folio_done(struct folio *src, /* Obtain the lock on page, remove all ptes. */ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, unsigned long private, struct folio *src, - struct folio **dstp, int force, bool avoid_force_lock, + struct folio **dstp, int force, enum migrate_mode mode, enum migrate_reason reason, struct list_head *ret) { @@ -1166,17 +1166,6 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page if (current->flags & PF_MEMALLOC) goto out; - /* - * We have locked some folios and are going to wait to lock - * this folio. To avoid a potential deadlock, let's bail - * out and not do that. The locked folios will be moved and - * unlocked, then we can wait to lock this folio. - */ - if (avoid_force_lock) { - rc = -EDEADLOCK; - goto out; - } - folio_lock(src); } locked = true; @@ -1256,7 +1245,7 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page /* Establish migration ptes */ VM_BUG_ON_FOLIO(folio_test_anon(src) && !folio_test_ksm(src) && !anon_vma, src); - try_to_migrate(src, TTU_BATCH_FLUSH); + try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0); page_was_mapped = 1; } @@ -1270,7 +1259,7 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page * A folio that has not been unmapped will be restored to * right list unless we want to retry. */ - if (rc == -EAGAIN || rc == -EDEADLOCK) + if (rc == -EAGAIN) ret = NULL; migrate_folio_undo_src(src, page_was_mapped, anon_vma, locked, ret); @@ -1621,6 +1610,11 @@ static int migrate_hugetlbs(struct list_head *from, new_page_t get_new_page, /* * migrate_pages_batch() first unmaps folios in the from list as many as * possible, then move the unmapped folios. + * + * We only batch migration if mode == MIGRATE_ASYNC to avoid to wait a + * lock or bit when we have locked more than one folio. Which may cause + * deadlock (e.g., for loop device). So, if mode != MIGRATE_ASYNC, the + * length of the from list must be <= 1. */ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, free_page_t put_new_page, unsigned long private, @@ -1643,11 +1637,11 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, LIST_HEAD(dst_folios); bool nosplit = (reason == MR_NUMA_MISPLACED); bool no_split_folio_counting = false; - bool avoid_force_lock; + VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC && + !list_empty(from) && !list_is_singular(from)); retry: rc_saved = 0; - avoid_force_lock = false; retry = 1; for (pass = 0; pass < NR_MAX_MIGRATE_PAGES_RETRY && (retry || large_retry); @@ -1692,15 +1686,14 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, } rc = migrate_folio_unmap(get_new_page, put_new_page, private, - folio, &dst, pass > 2, avoid_force_lock, - mode, reason, ret_folios); + folio, &dst, pass > 2, mode, + reason, ret_folios); /* * The rules are: * Success: folio will be freed * Unmap: folio will be put on unmap_folios list, * dst folio put on dst_folios list * -EAGAIN: stay on the from list - * -EDEADLOCK: stay on the from list * -ENOMEM: stay on the from list * Other errno: put on ret_folios list */ @@ -1752,14 +1745,6 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, goto out; else goto move; - case -EDEADLOCK: - /* - * The folio cannot be locked for potential deadlock. - * Go move (and unlock) all locked folios. Then we can - * try again. - */ - rc_saved = rc; - goto move; case -EAGAIN: if (is_large) { large_retry++; @@ -1774,11 +1759,6 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, stats->nr_thp_succeeded += is_thp; break; case MIGRATEPAGE_UNMAP: - /* - * We have locked some folios, don't force lock - * to avoid deadlock. - */ - avoid_force_lock = true; list_move_tail(&folio->lru, &unmap_folios); list_add_tail(&dst->lru, &dst_folios); break; @@ -1903,17 +1883,15 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, */ list_splice_init(from, ret_folios); list_splice_init(&split_folios, from); + /* + * Force async mode to avoid to wait lock or bit when we have + * locked more than one folios. + */ + mode = MIGRATE_ASYNC; no_split_folio_counting = true; goto retry; } - /* - * We have unlocked all locked folios, so we can force lock now, let's - * try again. - */ - if (rc == -EDEADLOCK) - goto retry; - return rc; } @@ -1948,7 +1926,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, enum migrate_mode mode, int reason, unsigned int *ret_succeeded) { int rc, rc_gather; - int nr_pages; + int nr_pages, batch; struct folio *folio, *folio2; LIST_HEAD(folios); LIST_HEAD(ret_folios); @@ -1962,6 +1940,11 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, mode, reason, &stats, &ret_folios); if (rc_gather < 0) goto out; + + if (mode == MIGRATE_ASYNC) + batch = NR_MAX_BATCHED_MIGRATION; + else + batch = 1; again: nr_pages = 0; list_for_each_entry_safe(folio, folio2, from, lru) { @@ -1972,11 +1955,11 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, } nr_pages += folio_nr_pages(folio); - if (nr_pages > NR_MAX_BATCHED_MIGRATION) + if (nr_pages >= batch) break; } - if (nr_pages > NR_MAX_BATCHED_MIGRATION) - list_cut_before(&folios, from, &folio->lru); + if (nr_pages >= batch) + list_cut_before(&folios, from, &folio2->lru); else list_splice_init(from, &folios); rc = migrate_pages_batch(&folios, get_new_page, put_new_page, private,