From patchwork Tue Dec 17 12:18:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chen Ridong X-Patchwork-Id: 13911698 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83C6BE7717F for ; Tue, 17 Dec 2024 12:28:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F352A6B00C0; Tue, 17 Dec 2024 07:28:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EE5456B00C2; Tue, 17 Dec 2024 07:28:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD4ED6B00C3; Tue, 17 Dec 2024 07:28:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C222F6B00C0 for ; Tue, 17 Dec 2024 07:28:39 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4DF4680703 for ; Tue, 17 Dec 2024 12:28:39 +0000 (UTC) X-FDA: 82904379222.02.4458287 Received: from dggsgout12.his.huawei.com (dggsgout12.his.huawei.com [45.249.212.56]) by imf23.hostedemail.com (Postfix) with ESMTP id 5CA74140007 for ; Tue, 17 Dec 2024 12:28:13 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf23.hostedemail.com: domain of chenridong@huaweicloud.com designates 45.249.212.56 as permitted sender) smtp.mailfrom=chenridong@huaweicloud.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734438489; a=rsa-sha256; cv=none; b=5mqHIaLdX1zP/1FJ4DTZkW2nLfZlvCMg1EziU7FPXUZe50ie2SP4x3Jnj1UE33MbmyX+9s v0rjY4DJYlJYd11yeTNovKEVbyfyAsU0wusTwN1ICZoSezedKTON2fPnwbn6GxjEHSqDCz Ue4+XOnUzLtOwWbYHOgg9/3hyZ3AO5w= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf23.hostedemail.com: domain of chenridong@huaweicloud.com designates 45.249.212.56 as permitted sender) smtp.mailfrom=chenridong@huaweicloud.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734438489; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references; bh=P3ZeZeVaBm1DbgfJdGwG4k1VkFZtdP6Ub5hHRYUp+8U=; b=8da1Yl1m8Erry8J2Z5yKmdIOPerjEQKubO8dVTzk8ckN+1QBAEr6UWP0bfuHoAY1ZBm0JK EwhoUYr2bdI3LOXnln4K9GEnfhCSe5HMzTni+0JShdh1nA2+fYVnaJEuAOsbYpVPe2Y3G2 8G8+JPyAPM+LQPjXNVc41y7u0+JUSZA= Received: from mail.maildlp.com (unknown [172.19.163.235]) by dggsgout12.his.huawei.com (SkyGuard) with ESMTP id 4YCGLK0Yh1z4f3jsf for ; Tue, 17 Dec 2024 20:28:09 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.128]) by mail.maildlp.com (Postfix) with ESMTP id 8888C1A058E for ; Tue, 17 Dec 2024 20:28:28 +0800 (CST) Received: from hulk-vt.huawei.com (unknown [10.67.174.121]) by APP4 (Coremail) with SMTP id gCh0CgAX8oVebmFn4SNwEw--.64435S2; Tue, 17 Dec 2024 20:28:28 +0800 (CST) From: Chen Ridong To: akpm@linux-foundation.org, mhocko@kernel.org, hannes@cmpxchg.org, yosryahmed@google.com, roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, davidf@vimeo.com, vbabka@suse.cz, handai.szj@taobao.com, rientjes@google.com, kamezawa.hiroyu@jp.fujitsu.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, chenridong@huawei.com, wangweiyang2@huawei.com Subject: [PATCH v1] memcg: fix soft lockup in the OOM process Date: Tue, 17 Dec 2024 12:18:28 +0000 Message-Id: <20241217121828.3219752-1-chenridong@huaweicloud.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-CM-TRANSID: gCh0CgAX8oVebmFn4SNwEw--.64435S2 X-Coremail-Antispam: 1UD129KBjvJXoW7Cr1fGF43CFWUXrW3Gr43Awb_yoW8KF45p3 ZrJFWUGw4rJr15Xr1xAr1Fgr15Ja17Aa1UXrsFvr1rJF17G3WUAry7JrWUtr1kGr4fZFy3 tFn0qw48tr1DZaUanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUvFb4IE77IF4wAFF20E14v26ryj6rWUM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28lY4IEw2IIxxk0rwA2F7IY1VAKz4 vEj48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_Xr0_Ar1l84ACjcxK6xIIjxv20xvEc7Cj xVAFwI0_Cr0_Gr1UM28EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I8E87Iv6xkF7I 0E14v26rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI64kE6c02F40E x7xfMcIj6xIIjxv20xvE14v26r106r15McIj6I8E87Iv67AKxVWUJVW8JwAm72CE4IkC6x 0Yz7v_Jr0_Gr1lF7xvr2IYc2Ij64vIr41lFIxGxcIEc7CjxVA2Y2ka0xkIwI1lc7CjxVAa w2AFwI0_GFv_Wryl42xK82IYc2Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxV Aqx4xG67AKxVWUJVWUGwC20s026x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r4a 6rW5MIIYrxkI7VAKI48JMIIF0xvE2Ix0cI8IcVAFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY6x kF7I0E14v26r4j6F4UMIIF0xvE42xK8VAvwI8IcIk0rVWUJVWUCwCI42IY6I8E87Iv67AK xVWUJVW8JwCI42IY6I8E87Iv6xkF7I0E14v26r4j6r4UJbIYCTnIWIevJa73UjIFyTuYvj xUxo7KDUUUU X-CM-SenderInfo: hfkh02xlgr0w46kxt4xhlfz01xgou0bp/ X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 5CA74140007 X-Stat-Signature: 7c3r4k3hdpxrmnsafcq3npx9eoot9swi X-Rspam-User: X-HE-Tag: 1734438493-753318 X-HE-Meta: U2FsdGVkX19uOWhwlpqtjdQLK8snfy/KydLSAHlzDZ7YgNTLap0MW/XVME7t0YEWMsNtLHtt0XFVaCL9RP24d/O6u2rGrpmPvbsEMrfUM+9dzCt2lVR0WJeFOD9zXCPaBYjg4aMaldyt3qdBEPOGr72XwtAEiIRg1qWESvA1SlYXQNupaT4S3+P3U9tUSJVinhPNEvboI0izcql75TuQP40+/r2Axf8lCkrrcJzQwZL9x3J8xW73khhiUahG8JkzRp0eWzELdPZS3J0QggGLVOT8Pf5XlQTO0Ji59RDOsQY+etYSKiXdxoRFPW1sspnlsfNP7gV9FPt8yyPo44q0gjj5oV5/FabtaVwzk4KOq5jWqpHNMHX+zgP5B+23WoEwn8FLbGUTdVGXikeCjYePx+m1Av9F9rw1SfpW63CbpuNnXrZtjG1kFFqa8EAOPblzkOpng07gD3oF1aZiO6NJ6iADLWO68pKNpeRE518wlECA8FTzhuKOWPtQj13KJTcr9BkFEBEXiUHl7bNLSPfbV9faMmHgt7TwTNUAC900P/R201bMhvuzcVI6fueV5cLJNPiQ5aHAdt8/fqvbv5tTOMgzz0/TXk8CLUBR4gBWDjmnPvS/vUKBHLFNQ/j+BRM7gra3a2RDQ2vgoFqx9R72AYCwrxl5YdYItnCsa+SqIZOhEqYZKCDew6BsqTdmQsLsZ20MOT17K9/6IUE/98gIN/Wg5pynGVbzRIeLsh8NZvIo2O/dvUm8OrfWkYS7arj8entpgXPr/dqLa0fIvfU1QMjYfxnVXIymND7Bu144ZC3Rq46rtWLgqgpRp4ahr2mT3H6Zak4Ci6f3wXWVrVaWr02wi6Ny8T1Ndowx97anJzeeGm6crqmDTL9cJh1CTebw+rI4y4QApJWJQz5tvUYHSVA8S6jQpnjzDjMlsg5cIJrvkFKGpHKYOiUgVSo43jlWZghJ4mvN5v0zYZ9aecn Rl4m4jP+ auEv+P6n3VShVWiRJa4HUWljgSvRC9nONgImMcJxh5YEFmlHRoN57WMOuI7+PL1nVkOkczLLKod42lSdCyYdvBkfQAQ6S8az5Ppz1d2DX6yPbCEtop2dn3Lvne4dnsxUHyiOb X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chen Ridong A soft lockup issue was found in the product with. About 56,000 tasks were in the OOM cgroup, it was traversing them when the soft lockup was triggered. watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [VM Thread:1503066] CPU: 2 PID: 1503066 Comm: VM Thread Kdump: loaded Tainted: G Hardware name: Huawei Cloud OpenStack Nova, BIOS RIP: 0010:console_unlock+0x343/0x540 RSP: 0000:ffffb751447db9a0 EFLAGS: 00000247 ORIG_RAX: ffffffffffffff13 RAX: 0000000000000001 RBX: 0000000000000000 RCX: 00000000ffffffff RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000247 RBP: ffffffffafc71f90 R08: 0000000000000000 R09: 0000000000000040 R10: 0000000000000080 R11: 0000000000000000 R12: ffffffffafc74bd0 R13: ffffffffaf60a220 R14: 0000000000000247 R15: 0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f2fe6ad91f0 CR3: 00000004b2076003 CR4: 0000000000360ee0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: vprintk_emit+0x193/0x280 printk+0x52/0x6e dump_task+0x114/0x130 mem_cgroup_scan_tasks+0x76/0x100 dump_header+0x1fe/0x210 oom_kill_process+0xd1/0x100 out_of_memory+0x125/0x570 mem_cgroup_out_of_memory+0xb5/0xd0 try_charge+0x720/0x770 mem_cgroup_try_charge+0x86/0x180 mem_cgroup_try_charge_delay+0x1c/0x40 do_anonymous_page+0xb5/0x390 handle_mm_fault+0xc4/0x1f0 This is because thousands of processes are in the OOM cgroup, it takes a long time to traverse all of them. As a result, this lead to soft lockup in the OOM process. To fix this issue, add 'cond_resched' in the 'dump_task' function. Fixes: 9cbb78bb3143 ("mm, memcg: introduce own oom handler to iterate only over its own threads") Signed-off-by: Chen Ridong --- mm/oom_kill.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/oom_kill.c b/mm/oom_kill.c index 1c485beb0b93..14260381cccc 100644 --- a/mm/oom_kill.c +++ b/mm/oom_kill.c @@ -390,6 +390,7 @@ static int dump_task(struct task_struct *p, void *arg) if (!is_memcg_oom(oc) && !oom_cpuset_eligible(p, oc)) return 0; + cond_resched(); task = find_lock_task_mm(p); if (!task) { /*