From patchwork Thu Mar 7 03:19:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13585013 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 489D3C54E41 for ; Thu, 7 Mar 2024 03:20:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D50E76B00F3; Wed, 6 Mar 2024 22:20:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D00AC6B00F5; Wed, 6 Mar 2024 22:20:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA1926B00F6; Wed, 6 Mar 2024 22:20:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A80FC6B00F3 for ; Wed, 6 Mar 2024 22:20:25 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E69501C1274 for ; Thu, 7 Mar 2024 03:20:24 +0000 (UTC) X-FDA: 81868789968.06.B3670BA Received: from mail-oi1-f173.google.com (mail-oi1-f173.google.com [209.85.167.173]) by imf30.hostedemail.com (Postfix) with ESMTP id 69F6A8000D for ; Thu, 7 Mar 2024 03:20:22 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=GO3JJTMI; spf=pass (imf30.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.167.173 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709781622; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=c1jvkih2U4k9BuhBY0/Fz/4FIDzcIUMMLDQD8hNmvUE=; b=PDqg/pJXm4n9GjQHa7+V0NsSZtrTnTGqHJn0cBGqD8vP/RApNOf/FQGLyuIJbE7jdAX9Fb LtrhN6YpQIg26kjI+nyOG4CSyvvjgueq+qwrAmeZPPjGr+/C9WMfmS31YRWln7kTxNv2V9 njbIA+G6PY3hHxXcTKkeh4sHeS8+jZo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709781622; a=rsa-sha256; cv=none; b=dW4dGC0Wsdd8aTaa3HyS4iDreSxuHa5njhHWJBox2AUgvlrHBjVSdgoWAOFxi+9tOeVqwV 0JhgbsnI8DyIDR2CCc++yD1Jc83PE49R+fgNnPctZmrN8FY0OaOV8VK4vi12EeZu5El2FQ x8tHn/JKLhkCQ5jXAq2Si+JW+oBE9ec= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=GO3JJTMI; spf=pass (imf30.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.167.173 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oi1-f173.google.com with SMTP id 5614622812f47-3c1e992f060so166971b6e.0 for ; Wed, 06 Mar 2024 19:20:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709781621; x=1710386421; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=c1jvkih2U4k9BuhBY0/Fz/4FIDzcIUMMLDQD8hNmvUE=; b=GO3JJTMIm35XID8VskSoEzx/vSXZhKIUJsvkOqG9JVN5UjWWNISM/1gO93xHv5tLuu lv/fKTXDeDyRhq87+2tKqMKwxVp9ny4TmoEgXTSzc7TfUucEQFPCozBLL8NtYWdcwAXs 9tYSGWVLy31vZVqz87Jt3nFToRoOlHGi/SgHZpGx10xQ7yec6XjYKWFJgvJYyLCH6SsV 8bWZe0gEygv3W6nETM6jEVY/Ea9fjVu7biIxfwkQ3hYcTHiypgJd2t+JWpDyJUfVfmqw 7ku0B3/6iBHYm9322Ukri+2h+4R36FdcYp1L+3BXE9y2NMyZMTQ9QkWfsnm7Y8kxOWyH 8jLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709781621; x=1710386421; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=c1jvkih2U4k9BuhBY0/Fz/4FIDzcIUMMLDQD8hNmvUE=; b=MNh82Ifofs1A1lBhlpUYO10x8X+X95PMCgq9o7RJbj7/o2w480Cp6gTy6JYDQ7wyLi j+QSb7LuMgh7erdH+EqVrds//UTQvRY0kw2CfywIha38gg8XID5mtdmt/PXeQfbSAmwT ZokV84FhlIxwh2m9H0QZiR0G+Weu0gVph7TBSM2TN/XuHRNBmZgiztV7ZdpNBi8jVaJv hJPr0bwAv8tmJ83LNpxX3b6qJ0YE6+xAbHFbsHVjkq9lATneGe+XGlnnT2R671jSizYq 72x+arlSBhp5+le1ev3KNN2PaeFdUHgPoFSHDLe5oCsbVxjubdSBlDSbIpSOcdKfDYyF Db5g== X-Gm-Message-State: AOJu0YzE9C907VhpDugNaw4teEnB/e7vBOA1g1T7teYFR1SN/9pP0BmD J/hRBpGUQLH1L6T2S+kFZvY4Vsc1LrKQQC+0YX+SU1wl7OQimFaN X-Google-Smtp-Source: AGHT+IFh9KHzJ7ETy6KC4crTv8rbNLfsKcuL6AV0XMw0BLC93LPGYxe2i39LiW/y5IXaIDyezzsdYg== X-Received: by 2002:a05:6808:2016:b0:3c1:ec24:e4b5 with SMTP id q22-20020a056808201600b003c1ec24e4b5mr8621103oiw.8.1709781621424; Wed, 06 Mar 2024 19:20:21 -0800 (PST) Received: from localhost.localdomain ([39.144.104.165]) by smtp.gmail.com with ESMTPSA id y15-20020aa7854f000000b006e5a99942c6sm10368678pfn.88.2024.03.06.19.20.17 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Mar 2024 19:20:20 -0800 (PST) From: Yafang Shao To: akpm@linux-foundation.org, yuzhao@google.com Cc: linux-mm@kvack.org, Yafang Shao , stable@vger.kernel.org Subject: [PATCH] mm: mglru: Fix soft lockup attributed to scanning folios Date: Thu, 7 Mar 2024 11:19:52 +0800 Message-Id: <20240307031952.2123-1-laoar.shao@gmail.com> X-Mailer: git-send-email 2.30.1 (Apple Git-130) MIME-Version: 1.0 X-Stat-Signature: qoqjonb3h79z8rgk11mrqcpbmpczzfqc X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 69F6A8000D X-Rspam-User: X-HE-Tag: 1709781622-174290 X-HE-Meta: U2FsdGVkX18Unv1TTNJtkbX1Lm00h4aOnXJSHbgqG25eZY2BCHafZttyU4yn9FdXEt/kV+klbUW8gdgm8Mh8o4mPafJNHyN2SittEg7GEr37yomsSP/by29ZyK+CmwJKxPYoMVlBldNiouVwauCgTQQPg+er5MvjarX8LJFX95IFAl35MdKrsVZnub1kDn3+nNhZb3RJdWSv7YpnXrElln8j5aYFXueLu+HsPCbLaG+wBOSQM/Y267J2p97/TvmXA1NCNpD0AP0KbNYyTRGmWV3YM3wk0CwJyi2PtPVDProIvCK4F4Ga3w42cnWUVQkdOBphd2TZBxU5ZcxGZztgeda5+KNgHuSpcUtlvuv3lY8UeFxSb61Y5GVdoov0+TtPV2hRNM0MflNRViZ47CodiJY1uvCAY+wxfzzPrUt6WQvTYtzB7qec0aN4utswHkdgx16M+5YxRqbrfxpnKpRSYpo+G3/0iMRdHlaQg6G7ZYaLnVApUqntE2NG9oBTIDcIL7PdDCFCABqymM7kufXJWmgKd1sRdWO1iCifn8wmz1PH8XHVLtMaBTBHjNtZECAoz3KU5M+54+207z7+Cr6QtF2JY7zNvBiJ57opfMyGhdkhQskklMunWClSwkrVNt2n1zpm4tYUihyNH959d1QUnQ7nEY4R5IdMxSb+0jT7DoMhjKmg6463c0yUoYxFpdZoVXfLyv5lt1QQJ0rTpKD10PJf6P5hcG938/ORy7RWOCVtC5EVkOh2FcWgt/QzNhMuFB40nF+AsQkQPoDMLJGc5cHnDBo9PKkJQF5MCdcl/z2LEWThECcPh8hpakI/KLKqfFnnrwdIk8mwh/gYJOPD4+QPTMt+7Zyi/odO90Gt+4giy8m1jvTVLmaW2Qo2bvkN1WCvIo4+ZvoZu04v5bpT98JJpKezth3qJ1iHUuEKC09TJUE+jbK/EDeWpCd5amUQxZ3EwLwp+pBX2G907/Z qrSji5Hj GVlmwq8xdhpF524tbkPxWIbs+vdB30a+TdaRVqNAsOrjFqjObJxJbeZUPdzrb52lWRKD20JOUaNHEDiU1BWwl70hkrp5cSu+zocMF9OkRgltUez4j8LW01eH/7A0W8D/0ySoSTtaes3QkNbIncjmlREWKs4daqnQ/CpSKAvRTLZQsOocw9eZ9R7yPzVgYEnYR6uN4C0zQXZ3oVtIQ8Jw+L3II1Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: After we enabled mglru on our 384C1536GB production servers, we encountered frequent soft lockups attributed to scanning folios. The soft lockup as follows, [Sat Feb 24 02:29:42 2024] watchdog: BUG: soft lockup - CPU#215 stuck for 111s! [kworker/215:0:2200100] [Sat Feb 24 02:29:42 2024] Call Trace: [Sat Feb 24 02:29:42 2024] [Sat Feb 24 02:29:42 2024] ? show_regs.cold+0x1a/0x1f [Sat Feb 24 02:29:42 2024] ? watchdog_timer_fn+0x1c4/0x220 [Sat Feb 24 02:29:42 2024] ? softlockup_fn+0x30/0x30 [Sat Feb 24 02:29:42 2024] ? __hrtimer_run_queues+0xa2/0x2b0 [Sat Feb 24 02:29:42 2024] ? hrtimer_interrupt+0x109/0x220 [Sat Feb 24 02:29:42 2024] ? __sysvec_apic_timer_interrupt+0x5e/0x110 [Sat Feb 24 02:29:42 2024] ? sysvec_apic_timer_interrupt+0x7b/0x90 [Sat Feb 24 02:29:42 2024] [Sat Feb 24 02:29:42 2024] [Sat Feb 24 02:29:42 2024] ? asm_sysvec_apic_timer_interrupt+0x1b/0x20 [Sat Feb 24 02:29:42 2024] ? folio_end_writeback+0x73/0xa0 [Sat Feb 24 02:29:42 2024] ? folio_rotate_reclaimable+0x8c/0x90 [Sat Feb 24 02:29:42 2024] ? folio_rotate_reclaimable+0x57/0x90 [Sat Feb 24 02:29:42 2024] ? folio_rotate_reclaimable+0x8c/0x90 [Sat Feb 24 02:29:42 2024] folio_end_writeback+0x73/0xa0 [Sat Feb 24 02:29:42 2024] iomap_finish_ioend+0x1d4/0x420 [Sat Feb 24 02:29:42 2024] iomap_finish_ioends+0x5e/0xe0 [Sat Feb 24 02:29:42 2024] xfs_end_ioend+0x65/0x150 [xfs] [Sat Feb 24 02:29:42 2024] xfs_end_io+0xbc/0xf0 [xfs] [Sat Feb 24 02:29:42 2024] process_one_work+0x1ec/0x3c0 [Sat Feb 24 02:29:42 2024] worker_thread+0x4d/0x390 [Sat Feb 24 02:29:42 2024] ? process_one_work+0x3c0/0x3c0 [Sat Feb 24 02:29:42 2024] kthread+0xee/0x120 [Sat Feb 24 02:29:42 2024] ? kthread_complete_and_exit+0x20/0x20 [Sat Feb 24 02:29:42 2024] ret_from_fork+0x1f/0x30 [Sat Feb 24 02:29:42 2024] From our analysis of the vmcore generated by the soft lockup, the thread was waiting for the spinlock lruvec->lru_lock: PID: 2200100 TASK: ffff9a221d8b4000 CPU: 215 COMMAND: "kworker/215:0" #0 [fffffe000319ae20] crash_nmi_callback at ffffffff8e055419 #1 [fffffe000319ae58] nmi_handle at ffffffff8e0253c0 #2 [fffffe000319aea0] default_do_nmi at ffffffff8eae5985 #3 [fffffe000319aec8] exc_nmi at ffffffff8eae5b78 #4 [fffffe000319aef0] end_repeat_nmi at ffffffff8ec015f0 [exception RIP: queued_spin_lock_slowpath+59] RIP: ffffffff8eaf9b8b RSP: ffffb58b01d4fc20 RFLAGS: 00000002 RAX: 0000000000000001 RBX: ffffb58b01d4fc90 RCX: 0000000000000000 RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff99d2b6ff9050 RBP: ffffb58b01d4fc40 R8: 0000000000035b21 R9: 0000000000000040 R10: 0000000000035b00 R11: 0000000000000001 R12: ffff99d2b6ff9050 R13: 0000000000000046 R14: ffffffff8e28bd30 R15: 0000000000000000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- --- #5 [ffffb58b01d4fc20] queued_spin_lock_slowpath at ffffffff8eaf9b8b #6 [ffffb58b01d4fc48] _raw_spin_lock_irqsave at ffffffff8eaf9b11 #7 [ffffb58b01d4fc68] folio_lruvec_lock_irqsave at ffffffff8e337a82 #8 [ffffb58b01d4fc88] folio_batch_move_lru at ffffffff8e28dbcf #9 [ffffb58b01d4fcd0] folio_batch_add_and_move at ffffffff8e28dce7 #10 [ffffb58b01d4fce0] folio_rotate_reclaimable at ffffffff8e28eee7 #11 [ffffb58b01d4fcf8] folio_end_writeback at ffffffff8e27bfb3 #12 [ffffb58b01d4fd10] iomap_finish_ioend at ffffffff8e3d9d04 #13 [ffffb58b01d4fd98] iomap_finish_ioends at ffffffff8e3d9fae #14 [ffffb58b01d4fde0] xfs_end_ioend at ffffffffc0fae835 [xfs] #15 [ffffb58b01d4fe20] xfs_end_io at ffffffffc0fae9dc [xfs] #16 [ffffb58b01d4fe60] process_one_work at ffffffff8e0ae08c #17 [ffffb58b01d4feb0] worker_thread at ffffffff8e0ae2ad #18 [ffffb58b01d4ff10] kthread at ffffffff8e0b671e #19 [ffffb58b01d4ff50] ret_from_fork at ffffffff8e002dcf While the spinlock (RDI: ffff99d2b6ff9050) was held by a task which was scanning folios: PID: 2400713 TASK: ffff996be1d14000 CPU: 50 COMMAND: "chitu_main" --- --- #5 [ffffb58b14ef76e8] __mod_zone_page_state at ffffffff8e2a9c36 #6 [ffffb58b14ef76f0] folio_inc_gen at ffffffff8e2990bd #7 [ffffb58b14ef7740] sort_folio at ffffffff8e29afbb #8 [ffffb58b14ef7748] sysvec_apic_timer_interrupt at ffffffff8eae79f0 #9 [ffffb58b14ef77b0] scan_folios at ffffffff8e29b49b #10 [ffffb58b14ef7878] evict_folios at ffffffff8e29bb53 #11 [ffffb58b14ef7968] lru_gen_shrink_lruvec at ffffffff8e29cb57 #12 [ffffb58b14ef7a28] shrink_lruvec at ffffffff8e29e135 #13 [ffffb58b14ef7af0] shrink_node at ffffffff8e29e78c #14 [ffffb58b14ef7b88] do_try_to_free_pages at ffffffff8e29ec08 #15 [ffffb58b14ef7bf8] try_to_free_mem_cgroup_pages at ffffffff8e2a17a6 #16 [ffffb58b14ef7ca8] try_charge_memcg at ffffffff8e338879 #17 [ffffb58b14ef7d48] charge_memcg at ffffffff8e3394f8 #18 [ffffb58b14ef7d70] __mem_cgroup_charge at ffffffff8e33aded #19 [ffffb58b14ef7d98] do_anonymous_page at ffffffff8e2c6523 #20 [ffffb58b14ef7dd8] __handle_mm_fault at ffffffff8e2cc27d #21 [ffffb58b14ef7e78] handle_mm_fault at ffffffff8e2cc3ba #22 [ffffb58b14ef7eb8] do_user_addr_fault at ffffffff8e073a99 #23 [ffffb58b14ef7f20] exc_page_fault at ffffffff8eae82f7 #24 [ffffb58b14ef7f50] asm_exc_page_fault at ffffffff8ec00bb7 There were a total of 22 tasks waiting for this spinlock (RDI: ffff99d2b6ff9050): crash> foreach RU bt | grep -B 8 queued_spin_lock_slowpath | grep "RDI: ffff99d2b6ff9050" | wc -l 22 Additionally, two other threads were also engaged in scanning folios, one with 19 waiters and the other with 15 waiters. To address this issue under heavy reclaim conditions, we introduced a hotfix version of the fix, incorporating cond_resched() in scan_folios(). Following the application of this hotfix to our servers, the soft lockup issue ceased. Signed-off-by: Yafang Shao Cc: Yu Zhao Cc: stable@vger.kernel.org # 6.1+ --- mm/vmscan.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/vmscan.c b/mm/vmscan.c index 4f9c854ce6cc..8f2877285b9a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4367,6 +4367,10 @@ static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, if (!--remaining || max(isolated, skipped_zone) >= MIN_LRU_BATCH) break; + + spin_unlock_irq(&lruvec->lru_lock); + cond_resched(); + spin_lock_irq(&lruvec->lru_lock); } if (skipped_zone) {