From patchwork Tue Feb 11 08:18:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chen Ridong X-Patchwork-Id: 13969513 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18A25C021A1 for ; Tue, 11 Feb 2025 08:29:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8BDA3280004; Tue, 11 Feb 2025 03:29:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 86DD7280002; Tue, 11 Feb 2025 03:29:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7361A280004; Tue, 11 Feb 2025 03:29:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 54C8D280002 for ; Tue, 11 Feb 2025 03:29:26 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 30B8B16125C for ; Tue, 11 Feb 2025 08:29:25 +0000 (UTC) X-FDA: 83106989490.04.76946D8 Received: from dggsgout11.his.huawei.com (dggsgout11.his.huawei.com [45.249.212.51]) by imf23.hostedemail.com (Postfix) with ESMTP id 3E6F6140004 for ; Tue, 11 Feb 2025 08:29:20 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=none; spf=pass (imf23.hostedemail.com: domain of chenridong@huaweicloud.com designates 45.249.212.51 as permitted sender) smtp.mailfrom=chenridong@huaweicloud.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739262563; a=rsa-sha256; cv=none; b=ElhfhY2/a/oOsHZ5fA7L+3/1EXZBuSb8JgPHAizWPDCFGNWQfDNF+kvesdVY9m1uryt6Gc FJWfctfI/xpNzykNyN2Mwl08iD4ufS6DCEnweegIAi173sAdgxCv/MN1/nLi7r3VzRv/pP 7rTZt4+fPQRR5znP/5dh2hvCSHuHSJo= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=none; spf=pass (imf23.hostedemail.com: domain of chenridong@huaweicloud.com designates 45.249.212.51 as permitted sender) smtp.mailfrom=chenridong@huaweicloud.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739262563; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references; bh=QfSyCSZNb+npW3k16ZFsCPCNy/Aw+gB7+jrlNmgysM8=; b=IdZZm6TQFJ7oOfbhF37GPH6YVqSUi8+ezw4Gp5WZeCY4mNCUtf2/VxNMTWVeMKFxAklfYg LWXl7rggNzSrYuRCUbMyBefpOLV3A7n6NjUjpGED9L9Bgo8G9oRoAVRLW80+FK4/toPA/C /RfYGmmhLY4dpfBm13NaMerBCw4VTjo= Received: from mail.maildlp.com (unknown [172.19.93.142]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4YsZNL4GQ3z4f3lCf for ; Tue, 11 Feb 2025 16:28:50 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.252]) by mail.maildlp.com (Postfix) with ESMTP id 46EE41A0E60 for ; Tue, 11 Feb 2025 16:29:13 +0800 (CST) Received: from hulk-vt.huawei.com (unknown [10.67.174.121]) by APP3 (Coremail) with SMTP id _Ch0CgAHp8FICqtnfiPrDQ--.42730S2; Tue, 11 Feb 2025 16:29:13 +0800 (CST) From: Chen Ridong To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org Cc: cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chenridong@huawei.com, wangweiyang2@huawei.com Subject: [PATCH] memcg: avoid dead loop when setting memory.max Date: Tue, 11 Feb 2025 08:18:19 +0000 Message-Id: <20250211081819.33307-1-chenridong@huaweicloud.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-CM-TRANSID: _Ch0CgAHp8FICqtnfiPrDQ--.42730S2 X-Coremail-Antispam: 1UD129KBjvJXoWxJw1fZr4DXw15ZF4ftw1kXwb_yoW5Cw1DpF 13Jw1UGF48Jr1kXr4UJF1fWr15Ga1kCFy7JrW7ur1rA3ZxG3WUJ34rK3yUXryDXr1rZr1a vF1DJw4xtw4DJaUanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUv2b4IE77IF4wAFF20E14v26r4j6ryUM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rwA2F7IY1VAKz4 vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_Ar0_tr1l84ACjcxK6xIIjxv20xvEc7Cj xVAFwI0_Gr1j6F4UJwA2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwVC2z280aVCY1x 0267AKxVW0oVCq3wAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG 6I80ewAv7VC0I7IYx2IY67AKxVWUGVWUXwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFV Cjc4AY6r1j6r4UM4x0Y48IcxkI7VAKI48JM4IIrI8v6xkF7I0E8cxan2IY04v7MxkF7I0E n4kS14v26r1q6r43MxAIw28IcxkI7VAKI48JMxC20s026xCaFVCjc4AY6r1j6r4UMI8I3I 0E5I8CrVAFwI0_Jr0_Jr4lx2IqxVCjr7xvwVAFwI0_JrI_JrWlx4CE17CEb7AF67AKxVWU tVW8ZwCIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v26r1j6r1xMIIF0xvE2Ix0cI8IcV CY1x0267AKxVW8JVWxJwCI42IY6xAIw20EY4v20xvaj40_Jr0_JF4lIxAIcVC2z280aVAF wI0_Jr0_Gr1lIxAIcVC2z280aVCY1x0267AKxVW8JVW8JrUvcSsGvfC2KfnxnUUI43ZEXa 7IUbmii3UUUUU== X-CM-SenderInfo: hfkh02xlgr0w46kxt4xhlfz01xgou0bp/ X-Stat-Signature: 4j4ucdjse9cd3t67piqnu1mibz9t6o1b X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 3E6F6140004 X-Rspam-User: X-HE-Tag: 1739262560-703703 X-HE-Meta: U2FsdGVkX19CgX+rlFuJ3kOMGRgXANf1C6LHsn1D1NcOwLul2IQ+i6R9WgK8yz3jVxI/KuH94IFXTap9LLfBYH4Eqq72YfHZrir8a9hHEhZeXiJcw3i0w4MSGxtiTKNvc8n8BoFBM3XY1AnM92pcA0fme9lzVuyGFt9A4j7O0h/fc17AolB+hScXXB8rG/qpPMKBfkdV0R8cI9GPvdRsL+EEhWiTgo1dgYC2lXdrrCbK29xJgOvlK8QbS/zYwqW0WpHWCb+Bg/ZovFL+w8j1pCriylyJvbWsOwlI8t/4/6aq1TNpC9+aPO1HZXqdpw+MhnttG41A8o5F9/ckOZV5HFGTSpwmOX1P6COrfZdl4tb0myWCSQvHQI+Egc4JbkHwfCSrT1RX9IetpNXgD6VRyvUfMyvcG1CI8dZK7eSIg+USZn6WhO4CjF69Tm+sDGc4xxMMpx45M5KUNqEIPNY0Yh0Bhqng/YzRFxzIVaYoPJpVr7FuPFmtDn4LFHs09VtV//mXBpkyC8fwhaDTMEWUMaEBr1sfoIt9D+jKo7/8YyQFCHdC2QvxODVE/4FRNQJ/H2gFGTABgWB6POJtlBQKb232sG3iT2MpGS5ZHo8ovKxPu0S4XJ1xqy6VZNFtNE553uB5n9BUzjuXhFETLmuExLspI7QRBMVSoKVXu5u8JjbVAczMs/UFcoIs4EUP7RCgOLUVTnnIzqb0iQyeYNw4tKA/sjjMFe2UTvqo9l540mtW4DbpD5EVcMGDeJ12pgdifCRPV5FB/POwzU8Vc7BGrMcprJtqcipKGvGJMfgpy5T5md3zTuXKmJO0d4vB00k/2o9+WkXKzAWp9Pkz2rPaBUeFq57A/f4VTdeucuK6Gw9wb8gJKnXWJXjp2SooarO+l3LLPLTeCUBDhl578olP0WiPBu4F1uTh3Q9ryd+jc27NoazMC3gyJ8//TXyLY2lQ2t56Nekq6jgZW3wKAgq Nfhb4m0n jeSrilLc7qIUJWBjTtXLWAVPUyTy99xWNVLijzMLCS5WbtQ8tlsMqT6788HFKn8SUVUdFAkeyysAd2rQ202hO5PoE0E5JKALkKEiGOzVnGyoW/9g0E/QX4zKQ6l/mSjrmxBVE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chen Ridong A softlockup issue was found with stress test: watchdog: BUG: soft lockup - CPU#27 stuck for 26s! [migration/27:181] CPU: 27 UID: 0 PID: 181 Comm: migration/27 6.14.0-rc2-next-20250210 #1 Stopper: multi_cpu_stop <- stop_machine_from_inactive_cpu RIP: 0010:stop_machine_yield+0x2/0x10 RSP: 0000:ff4a0dcecd19be48 EFLAGS: 00000246 RAX: ffffffff89c0108f RBX: ff4a0dcec03afe44 RCX: 0000000000000000 RDX: ff1cdaaf6eba5808 RSI: 0000000000000282 RDI: ff1cda80c1775a40 RBP: 0000000000000001 R08: 00000011620096c6 R09: 7fffffffffffffff R10: 0000000000000001 R11: 0000000000000100 R12: ff1cda80c1775a40 R13: 0000000000000000 R14: 0000000000000001 R15: ff4a0dcec03afe20 FS: 0000000000000000(0000) GS:ff1cdaaf6eb80000(0000) CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 00000025e2c2a001 CR4: 0000000000773ef0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 PKRU: 55555554 Call Trace: multi_cpu_stop+0x8f/0x100 cpu_stopper_thread+0x90/0x140 smpboot_thread_fn+0xad/0x150 kthread+0xc2/0x100 ret_from_fork+0x2d/0x50 The stress test involves CPU hotplug operations and memory control group (memcg) operations. The scenario can be described as follows: echo xx > memory.max cache_ap_online oom_reaper (CPU23) (CPU50) xx < usage stop_machine_from_inactive_cpu for(;;) // all active cpus trigger OOM queue_stop_cpus_work // waiting oom_reaper multi_cpu_stop(migration/xx) // sync all active cpus ack // waiting cpu23 ack // CPU50 loops in multi_cpu_stop waiting cpu50 Detailed explanation: 1. When the usage is larger than xx, an OOM may be triggered. If the process does not handle with ths kill signal immediately, it will loop in the memory_max_write. 2. When cache_ap_online is triggered, the multi_cpu_stop is queued to the active cpus. Within the multi_cpu_stop function, it attempts to synchronize the CPU states. However, the CPU23 didn't acknowledge because it is stuck in a loop within the for(;;). 3. The oom_reaper process is blocked because CPU50 is in a loop, waiting for CPU23 to acknowledge the synchronization request. 4. Finally, it formed cyclic dependency and lead to softlockup and dead loop. To fix this issue, add cond_resched() in the memory_max_write, so that it will not block migration task. Fixes: b6e6edcfa405 ("mm: memcontrol: reclaim and OOM kill when shrinking memory.max below usage") Signed-off-by: Chen Ridong Acked-by: Michal Hocko --- mm/memcontrol.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 8d21c1a44220..16f3bdbd37d8 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4213,6 +4213,7 @@ static ssize_t memory_max_write(struct kernfs_open_file *of, memcg_memory_event(memcg, MEMCG_OOM); if (!mem_cgroup_out_of_memory(memcg, GFP_KERNEL, 0)) break; + cond_resched(); } memcg_wb_domain_size_changed(memcg);