From patchwork Mon Aug 24 12:54:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 11733157 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9265A13B1 for ; Mon, 24 Aug 2020 12:56:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 694EF20838 for ; Mon, 24 Aug 2020 12:56:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 694EF20838 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 204808D0015; Mon, 24 Aug 2020 08:55:37 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 081348D0017; Mon, 24 Aug 2020 08:55:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CB1EB8D0018; Mon, 24 Aug 2020 08:55:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0067.hostedemail.com [216.40.44.67]) by kanga.kvack.org (Postfix) with ESMTP id 9C1748D0015 for ; Mon, 24 Aug 2020 08:55:36 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 4FA7A181AEF1E for ; Mon, 24 Aug 2020 12:55:36 +0000 (UTC) X-FDA: 77185458672.23.ray28_220201427053 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin23.hostedemail.com (Postfix) with ESMTP id 1D57037606 for ; Mon, 24 Aug 2020 12:55:36 +0000 (UTC) X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,alex.shi@linux.alibaba.com,,RULES_HIT:30012:30054:30055:30056:30070:30090,0,RBL:47.88.44.36:@linux.alibaba.com:.lbl8.mailshell.net-62.18.0.100 64.10.201.10;04yf73ufgoeors7oh7rjurkzhegktoczwrjxa4e1c55mwo1po97hgnu7y7zz74s.p4n7c73ac1ngmaz3ek594stn5qtnsjwohbxgpd53a1nyixczr6z8j11n69efn9b.e-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: ray28_220201427053 X-Filterd-Recvd-Size: 7461 Received: from out4436.biz.mail.alibaba.com (out4436.biz.mail.alibaba.com [47.88.44.36]) by imf30.hostedemail.com (Postfix) with ESMTP for ; Mon, 24 Aug 2020 12:55:34 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R191e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e07484;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=21;SR=0;TI=SMTPD_---0U6k9-bl_1598273712; Received: from aliy80.localdomain(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0U6k9-bl_1598273712) by smtp.aliyun-inc.com(127.0.0.1); Mon, 24 Aug 2020 20:55:13 +0800 From: Alex Shi To: akpm@linux-foundation.org, mgorman@techsingularity.net, tj@kernel.org, hughd@google.com, khlebnikov@yandex-team.ru, daniel.m.jordan@oracle.com, willy@infradead.org, hannes@cmpxchg.org, lkp@intel.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, shakeelb@google.com, iamjoonsoo.kim@lge.com, richard.weiyang@gmail.com, kirill@shutemov.name, alexander.duyck@gmail.com, rong.a.chen@intel.com, mhocko@suse.com, vdavydov.dev@gmail.com, shy828301@gmail.com Subject: [PATCH v18 00/32] per memcg lru_lock Date: Mon, 24 Aug 2020 20:54:33 +0800 Message-Id: <1598273705-69124-1-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 MIME-Version: 1.0 X-Rspamd-Queue-Id: 1D57037606 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The new version which bases on v5.9-rc2. The first 6 patches was picked into linux-mm, and add patch 25-32 that do some further post optimization. The patchset includes 4 parts: 1, some code cleanup and minimum optimization as a preparation. patch 1-15. 2, use TestCleanPageLRU as page isolation's precondition. patch 16-19 3, replace per node lru_lock with per memcg per node lru_lock. patch 20 4, some post optimization. patch 21-32 Current lru_lock is one for each of node, pgdat->lru_lock, that guard for lru lists, but now we had moved the lru lists into memcg for long time. Still using per node lru_lock is clearly unscalable, pages on each of memcgs have to compete each others for a whole lru_lock. This patchset try to use per lruvec/memcg lru_lock to repleace per node lru lock to guard lru lists, make it scalable for memcgs and get performance gain. Currently lru_lock still guards both lru list and page's lru bit, that's ok. but if we want to use specific lruvec lock on the page, we need to pin down the page's lruvec/memcg during locking. Just taking lruvec lock first may be undermined by the page's memcg charge/migration. To fix this problem, we could take out the page's lru bit clear and use it as pin down action to block the memcg changes. That's the reason for new atomic func TestClearPageLRU. So now isolating a page need both actions: TestClearPageLRU and hold the lru_lock. The typical usage of this is isolate_migratepages_block() in compaction.c we have to take lru bit before lru lock, that serialized the page isolation in memcg page charge/migration which will change page's lruvec and new lru_lock in it. The above solution suggested by Johannes Weiner, and based on his new memcg charge path, then have this patchset. (Hugh Dickins tested and contributed much code from compaction fix to general code polish, thanks a lot!). Following Daniel Jordan's suggestion, I have run 208 'dd' with on 104 containers on a 2s * 26cores * HT box with a modefied case: https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-lru-file-readtwice With this patchset, the readtwice performance increased about 80% in concurrent containers. Thanks Hugh Dickins and Konstantin Khlebnikov, they both brought this idea 8 years ago, and others who give comments as well: Daniel Jordan, Mel Gorman, Shakeel Butt, Matthew Wilcox etc. Thanks for Testing support from Intel 0day and Rong Chen, Fengguang Wu, and Yun Wang. Hugh Dickins also shared his kbuild-swap case. Thanks! Alex Shi (23): mm/memcg: warning on !memcg after readahead page charged mm/memcg: bail out early from swap accounting when memcg is disabled mm/thp: move lru_add_page_tail func to huge_memory.c mm/thp: clean up lru_add_page_tail mm/thp: remove code path which never got into mm/thp: narrow lru locking mm/swap.c: stop deactivate_file_page if page not on lru mm/vmscan: remove unnecessary lruvec adding mm/page_idle: no unlikely double check for idle page counting mm/compaction: rename compact_deferred as compact_should_defer mm/memcg: add debug checking in lock_page_memcg mm/swap.c: fold vm event PGROTATED into pagevec_move_tail_fn mm/lru: move lru_lock holding in func lru_note_cost_page mm/lru: move lock into lru_note_cost mm/lru: introduce TestClearPageLRU mm/compaction: do page isolation first in compaction mm/thp: add tail pages into lru anyway in split_huge_page() mm/swap.c: serialize memcg changes in pagevec_lru_move_fn mm/lru: replace pgdat lru_lock with lruvec lock mm/pgdat: remove pgdat lru_lock mm/mlock: remove lru_lock on TestClearPageMlocked in munlock_vma_page mm/mlock: remove __munlock_isolate_lru_page mm/swap.c: optimizing __pagevec_lru_add lru_lock Alexander Duyck (6): mm/lru: introduce the relock_page_lruvec function mm/compaction: Drop locked from isolate_migratepages_block mm: Identify compound pages sooner in isolate_migratepages_block mm: Drop use of test_and_set_skip in favor of just setting skip mm: Add explicit page decrement in exception path for isolate_lru_pages mm: Split release_pages work into 3 passes Hugh Dickins (3): mm/memcg: optimize mem_cgroup_page_lruvec mm/vmscan: use relock for move_pages_to_lru mm/lru: revise the comments of lru_lock Documentation/admin-guide/cgroup-v1/memcg_test.rst | 15 +- Documentation/admin-guide/cgroup-v1/memory.rst | 21 +- Documentation/trace/events-kmem.rst | 2 +- Documentation/vm/unevictable-lru.rst | 22 +- include/linux/compaction.h | 4 +- include/linux/memcontrol.h | 110 ++++++++ include/linux/mm_types.h | 2 +- include/linux/mmdebug.h | 13 + include/linux/mmzone.h | 6 +- include/linux/page-flags.h | 1 + include/linux/swap.h | 4 +- include/trace/events/compaction.h | 2 +- mm/compaction.c | 166 +++++------ mm/filemap.c | 4 +- mm/huge_memory.c | 48 +++- mm/memcontrol.c | 92 +++++- mm/mlock.c | 76 ++--- mm/mmzone.c | 1 + mm/page_alloc.c | 1 - mm/page_idle.c | 8 - mm/rmap.c | 4 +- mm/swap.c | 307 +++++++++++---------- mm/vmscan.c | 178 ++++++------ mm/workingset.c | 2 - 24 files changed, 646 insertions(+), 443 deletions(-) Acked-by: Hugh Dickins Acked-by: Hugh Dickins Acked-by: Hugh Dickins Acked-by: Hugh Dickins Acked-by: Hugh Dickins Signed-off-by: Wei Yang Acked-by: Hugh Dickins Acked-by: Hugh Dickins Acked-by: Hugh Dickins Acked-by: Hugh Dickins Signed-off-by: Hugh Dickins