From patchwork Sat Jul 11 00:58:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 11657595 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9266413B1 for ; Sat, 11 Jul 2020 00:59:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 692C42080D for ; Sat, 11 Jul 2020 00:59:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 692C42080D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AC0FD8D0003; Fri, 10 Jul 2020 20:59:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 90ECC8D0006; Fri, 10 Jul 2020 20:59:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6C3DE8D0003; Fri, 10 Jul 2020 20:59:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0251.hostedemail.com [216.40.44.251]) by kanga.kvack.org (Postfix) with ESMTP id 51D5F8D0005 for ; Fri, 10 Jul 2020 20:59:07 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 1997F180AD801 for ; Sat, 11 Jul 2020 00:59:07 +0000 (UTC) X-FDA: 77023985934.21.grape87_580147c26ed2 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin21.hostedemail.com (Postfix) with ESMTP id DFFE4180442C0 for ; Sat, 11 Jul 2020 00:59:06 +0000 (UTC) X-Spam-Summary: 1,0,0,b84ed0cf38355acb,d41d8cd98f00b204,alex.shi@linux.alibaba.com,,RULES_HIT:2:41:69:152:355:379:541:967:968:973:988:989:1260:1261:1277:1311:1313:1314:1345:1381:1437:1515:1516:1518:1535:1593:1594:1605:1606:1730:1747:1777:1792:1801:2198:2199:2393:2525:2553:2559:2563:2682:2685:2693:2859:2933:2937:2939:2942:2945:2947:2951:2954:3022:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4037:4117:4250:4362:4470:4605:5007:6119:6261:6737:7903:7974:8957:9010:9025:9040:10004:11026:11657:11658:11914:12043:12048:12050:12296:12297:12438:12895:13146:13161:13229:13230:13845:13894:14096:14097:14394:14659:14915:21060:21080:21451:21627:21740:30012:30054:30055:30056:30070:30090,0,RBL:115.124.30.42:@linux.alibaba.com:.lbl8.mailshell.net-64.201.201.201 62.20.2.100;04yrnobfmt4cyj6e33u1z57wk7ue7ocwtthr6zh4nm3tyu58oa17a5b17srcbe9.t97xxwbqknpjd8t5xmbiw3byjrzbxj3oqw17umpkz38b3n61o7pz14kp5ii3e8p.y-lbl8.mailshell.net-223.238. 255.100, X-HE-Tag: grape87_580147c26ed2 X-Filterd-Recvd-Size: 6837 Received: from out30-42.freemail.mail.aliyun.com (out30-42.freemail.mail.aliyun.com [115.124.30.42]) by imf09.hostedemail.com (Postfix) with ESMTP for ; Sat, 11 Jul 2020 00:59:05 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R221e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01f04427;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=17;SR=0;TI=SMTPD_---0U2KGAZr_1594429139; Received: from alexshi-test.localdomain(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0U2KGAZr_1594429139) by smtp.aliyun-inc.com(127.0.0.1); Sat, 11 Jul 2020 08:58:59 +0800 From: Alex Shi To: akpm@linux-foundation.org, mgorman@techsingularity.net, tj@kernel.org, hughd@google.com, khlebnikov@yandex-team.ru, daniel.m.jordan@oracle.com, yang.shi@linux.alibaba.com, willy@infradead.org, hannes@cmpxchg.org, lkp@intel.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, shakeelb@google.com, iamjoonsoo.kim@lge.com, richard.weiyang@gmail.com, kirill@shutemov.name Subject: [PATCH v16 00/22] per memcg lru_lock Date: Sat, 11 Jul 2020 08:58:34 +0800 Message-Id: <1594429136-20002-1-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 MIME-Version: 1.0 X-Rspamd-Queue-Id: DFFE4180442C0 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The new version which bases on v5.8-rc4. Add 2 more patchs: 'mm/thp: remove code path which never got into' 'mm/thp: add tail pages into lru anyway in split_huge_page()' and modified 'mm/mlock: reorder isolation sequence during munlock' Current lru_lock is one for each of node, pgdat->lru_lock, that guard for lru lists, but now we had moved the lru lists into memcg for long time. Still using per node lru_lock is clearly unscalable, pages on each of memcgs have to compete each others for a whole lru_lock. This patchset try to use per lruvec/memcg lru_lock to repleace per node lru lock to guard lru lists, make it scalable for memcgs and get performance gain. Currently lru_lock still guards both lru list and page's lru bit, that's ok. but if we want to use specific lruvec lock on the page, we need to pin down the page's lruvec/memcg during locking. Just taking lruvec lock first may be undermined by the page's memcg charge/migration. To fix this problem, we could take out the page's lru bit clear and use it as pin down action to block the memcg changes. That's the reason for new atomic func TestClearPageLRU. So now isolating a page need both actions: TestClearPageLRU and hold the lru_lock. The typical usage of this is isolate_migratepages_block() in compaction.c we have to take lru bit before lru lock, that serialized the page isolation in memcg page charge/migration which will change page's lruvec and new lru_lock in it. The above solution suggested by Johannes Weiner, and based on his new memcg charge path, then have this patchset. (Hugh Dickins tested and contributed much code from compaction fix to general code polish, thanks a lot!). The patchset includes 3 parts: 1, some code cleanup and minimum optimization as a preparation. 2, use TestCleanPageLRU as page isolation's precondition 3, replace per node lru_lock with per memcg per node lru_lock Following Daniel Jordan's suggestion, I have run 208 'dd' with on 104 containers on a 2s * 26cores * HT box with a modefied case: https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-lru-file-readtwice With this patchset, the readtwice performance increased about 80% in concurrent containers. Thanks Hugh Dickins and Konstantin Khlebnikov, they both brought this idea 8 years ago, and others who give comments as well: Daniel Jordan, Mel Gorman, Shakeel Butt, Matthew Wilcox etc. Thanks for Testing support from Intel 0day and Rong Chen, Fengguang Wu, and Yun Wang. Hugh Dickins also shared his kbuild-swap case. Thanks! Alex Shi (20): mm/vmscan: remove unnecessary lruvec adding mm/page_idle: no unlikely double check for idle page counting mm/compaction: correct the comments of compact_defer_shift mm/compaction: rename compact_deferred as compact_should_defer mm/thp: move lru_add_page_tail func to huge_memory.c mm/thp: clean up lru_add_page_tail mm/thp: remove code path which never got into mm/thp: narrow lru locking mm/memcg: add debug checking in lock_page_memcg mm/swap: fold vm event PGROTATED into pagevec_move_tail_fn mm/lru: move lru_lock holding in func lru_note_cost_page mm/lru: move lock into lru_note_cost mm/lru: introduce TestClearPageLRU mm/thp: add tail pages into lru anyway in split_huge_page() mm/compaction: do page isolation first in compaction mm/mlock: reorder isolation sequence during munlock mm/swap: serialize memcg changes during pagevec_lru_move_fn mm/lru: replace pgdat lru_lock with lruvec lock mm/lru: introduce the relock_page_lruvec function mm/pgdat: remove pgdat lru_lock Hugh Dickins (2): mm/vmscan: use relock for move_pages_to_lru mm/lru: revise the comments of lru_lock Documentation/admin-guide/cgroup-v1/memcg_test.rst | 15 +- Documentation/admin-guide/cgroup-v1/memory.rst | 21 +-- Documentation/trace/events-kmem.rst | 2 +- Documentation/vm/unevictable-lru.rst | 22 +-- include/linux/compaction.h | 4 +- include/linux/memcontrol.h | 98 +++++++++++ include/linux/mm_types.h | 2 +- include/linux/mmzone.h | 6 +- include/linux/page-flags.h | 1 + include/linux/swap.h | 4 +- include/trace/events/compaction.h | 2 +- mm/compaction.c | 113 ++++++++---- mm/filemap.c | 4 +- mm/huge_memory.c | 47 +++-- mm/memcontrol.c | 71 +++++++- mm/memory.c | 3 - mm/mlock.c | 93 +++++----- mm/mmzone.c | 1 + mm/page_alloc.c | 1 - mm/page_idle.c | 8 - mm/rmap.c | 4 +- mm/swap.c | 189 ++++++++------------- mm/swap_state.c | 2 - mm/vmscan.c | 174 ++++++++++--------- mm/workingset.c | 2 - 25 files changed, 524 insertions(+), 365 deletions(-) Signed-off-by: Hugh Dickins