From patchwork Tue Sep 8 14:24:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Chunxin Zang X-Patchwork-Id: 11763419 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AB05859D for ; Tue, 8 Sep 2020 14:28:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5DE0222228 for ; Tue, 8 Sep 2020 14:28:54 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=bytedance-com.20150623.gappssmtp.com header.i=@bytedance-com.20150623.gappssmtp.com header.b="TcMeF22U" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5DE0222228 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 44D496B0055; Tue, 8 Sep 2020 10:28:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3FF166B005A; Tue, 8 Sep 2020 10:28:53 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2ECF48E0001; Tue, 8 Sep 2020 10:28:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0246.hostedemail.com [216.40.44.246]) by kanga.kvack.org (Postfix) with ESMTP id 0D97A6B0055 for ; Tue, 8 Sep 2020 10:28:53 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id C5C7A824805A for ; Tue, 8 Sep 2020 14:28:52 +0000 (UTC) X-FDA: 77240125704.17.man13_0406b6a270d5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id 9BE8D18027F98 for ; Tue, 8 Sep 2020 14:25:11 +0000 (UTC) X-Spam-Summary: 1,0,0,4081e56d05740917,d41d8cd98f00b204,zangchunxin@bytedance.com,,RULES_HIT:41:152:355:379:541:800:960:966:973:988:989:1260:1277:1311:1313:1314:1345:1437:1515:1516:1518:1535:1543:1593:1594:1711:1730:1747:1777:1792:1801:2196:2199:2393:2559:2562:2693:3138:3139:3140:3141:3142:3355:3865:3866:3870:3871:4117:4321:4385:4605:5007:6261:6653:7576:7903:7904:8784:9149:9592:10004:10400:11026:11658:11914:12043:12220:12296:12297:12517:12519:12555:12679:12740:12895:12986:13161:13229:13894:14096:14097:14181:14394:14659:14721:21080:21444:21451:21627:21789:21795:21939:21990:30051:30054:30075,0,RBL:209.85.215.194:@bytedance.com:.lbl8.mailshell.net-62.2.0.100 66.100.201.201;04yroa8sk4g6bupkb38bndr4x8ehnypmfpe86r5a93yoo68cwn1aqnigwcgjtt4.errtiz9xk838t5r6qpd6ud6z41wy71dqf7c3bh4ce8rpeeu4z4mzmyexnaudqxn.h-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:no ne X-HE-Tag: man13_0406b6a270d5 X-Filterd-Recvd-Size: 6364 Received: from mail-pg1-f194.google.com (mail-pg1-f194.google.com [209.85.215.194]) by imf23.hostedemail.com (Postfix) with ESMTP for ; Tue, 8 Sep 2020 14:25:10 +0000 (UTC) Received: by mail-pg1-f194.google.com with SMTP id m8so4861057pgi.3 for ; Tue, 08 Sep 2020 07:25:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=qbyfb7IENIIZnzycuSlXwlchWqptmqZvtFIJsiTST2Q=; b=TcMeF22UNZIEg9d0gzz98TwFeIxyqfPvLagxomEInN8Jz1RMp7VDdtXA6EehKzjSG8 /5SlRkTXBSFEzI7pP7AjUsnj7qEH/1FwaqMYorliQ5tbjGHarxMvO8K2yyl3kx+Odxn2 3p+pVF9186L5mOvCvvk4+cuQxqF1uHjnnMNSMl/4X7ONNFqdgPB5cc4Rdoy/RJMqBX7L bEIWU1IN9uRC1oJ6e3XY9xep4JvDEaHn65KJdAK/LLiIaIQAFmKTXm5LoKEfec4Q8zPo Af1hYqV97gBqQ2vTs0XQpUYzWpFqhNlJegb6cSioxGUw8S/cvolkKosVgz5eZAOnnSmN NcTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=qbyfb7IENIIZnzycuSlXwlchWqptmqZvtFIJsiTST2Q=; b=srnT08vT431akJaXG9V+039st1da71AWGn5ipEWDMavS+hqCa1aiRclltY5kH0jdZO vWBn/hByT4EnuR5VsA1CYsEn0+VJ6SnhpEMh3NV3Qv0qxjlKXsfBDCFrub6FyCuAtsbC MnyJzLU6qyDv1OOeYc4yYe5myz6rHvZx8/7CsbBaA3UaoNCz1k1LQoBMdMA/v7x9xH/w 5vg3YSQUzYbF4GzK17zeZTplcI6JkeoCFYEo6vZvBcUkmtpH4sqZSUbm/yjrZCERE00L sIX8XFspEEdKbI8YHIQYeih4DAMfo9Bd0fN04TIWRbedan499eZD+SEGysWLoCYllgL/ X2hw== X-Gm-Message-State: AOAM532gsqtUSeBa56gcVuv6bEAqT/4b2lOddTIwB52P8UNACdTa73uB PMtqlinba5ykLBAUsRTuhaGJmw== X-Google-Smtp-Source: ABdhPJxgxtlU8ucp+kFiZL0rAJPS9+qutAhqRqo5Pl2vMcjnCNUKCP13XLhDLlCxt6b4VxJLHDqNIQ== X-Received: by 2002:aa7:8e85:: with SMTP id a5mr23961248pfr.96.1599575109593; Tue, 08 Sep 2020 07:25:09 -0700 (PDT) Received: from localhost.localdomain ([103.136.220.67]) by smtp.gmail.com with ESMTPSA id v26sm6120436pgo.83.2020.09.08.07.25.06 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 Sep 2020 07:25:08 -0700 (PDT) From: zangchunxin@bytedance.com To: akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chunxin Zang , Muchun Song Subject: [PATCH] mm/vmscan: fix infinite loop in drop_slab_node Date: Tue, 8 Sep 2020 22:24:56 +0800 Message-Id: <20200908142456.89626-1-zangchunxin@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) MIME-Version: 1.0 X-Rspamd-Queue-Id: 9BE8D18027F98 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Chunxin Zang On our server, there are about 10k memcg in one machine. They use memory very frequently. When I tigger drop caches，the process will infinite loop in drop_slab_node. There are two reasons: 1.We have too many memcgs, even though one object freed in one memcg, the sum of object is bigger than 10. 2.We spend a lot of time in traverse memcg once. So, the memcg who traversed at the first have been freed many objects. Traverse memcg next time, the freed count bigger than 10 again. We can get the following info through 'ps': root:~# ps -aux | grep drop root 357956 ... R Aug25 21119854:55 echo 3 > /proc/sys/vm/drop_caches root 1771385 ... R Aug16 21146421:17 echo 3 > /proc/sys/vm/drop_caches root 1986319 ... R 18:56 117:27 echo 3 > /proc/sys/vm/drop_caches root 2002148 ... R Aug24 5720:39 echo 3 > /proc/sys/vm/drop_caches root 2564666 ... R 18:59 113:58 echo 3 > /proc/sys/vm/drop_caches root 2639347 ... R Sep03 2383:39 echo 3 > /proc/sys/vm/drop_caches root 3904747 ... R 03:35 993:31 echo 3 > /proc/sys/vm/drop_caches root 4016780 ... R Aug21 7882:18 echo 3 > /proc/sys/vm/drop_caches Use bpftrace follow 'freed' value in drop_slab_node: root:~# bpftrace -e 'kprobe:drop_slab_node+70 {@ret=hist(reg("bp")); }' Attaching 1 probe... ^B^C @ret: [64, 128) 1 | | [128, 256) 28 | | [256, 512) 107 |@ | [512, 1K) 298 |@@@ | [1K, 2K) 613 |@@@@@@@ | [2K, 4K) 4435 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [4K, 8K) 442 |@@@@@ | [8K, 16K) 299 |@@@ | [16K, 32K) 100 |@ | [32K, 64K) 139 |@ | [64K, 128K) 56 | | [128K, 256K) 26 | | [256K, 512K) 2 | | In one drop caches action, only traverse memcg once maybe is better. If user need more memory, they can do drop caches again. Signed-off-by: Chunxin Zang Signed-off-by: Muchun Song --- mm/vmscan.c | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index b6d84326bdf2..9d8ee2ae5824 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -699,17 +699,12 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, void drop_slab_node(int nid) { - unsigned long freed; + struct mem_cgroup *memcg = NULL; + memcg = mem_cgroup_iter(NULL, NULL, NULL); do { - struct mem_cgroup *memcg = NULL; - - freed = 0; - memcg = mem_cgroup_iter(NULL, NULL, NULL); - do { - freed += shrink_slab(GFP_KERNEL, nid, memcg, 0); - } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); - } while (freed > 10); + shrink_slab(GFP_KERNEL, nid, memcg, 0); + } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); } void drop_slab(void)