From patchwork Mon Apr 17 07:56:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13213419 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A14EEC77B72 for ; Mon, 17 Apr 2023 07:55:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 40D7D900002; Mon, 17 Apr 2023 03:55:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3BDF98E0001; Mon, 17 Apr 2023 03:55:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 285CA900002; Mon, 17 Apr 2023 03:55:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 197D28E0001 for ; Mon, 17 Apr 2023 03:55:44 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id DAC7B80426 for ; Mon, 17 Apr 2023 07:55:43 +0000 (UTC) X-FDA: 80690123766.07.EBAA053 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf23.hostedemail.com (Postfix) with ESMTP id BD39C14000A for ; Mon, 17 Apr 2023 07:55:41 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=F2lP4krB; spf=pass (imf23.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681718142; a=rsa-sha256; cv=none; b=TwgqWW2Cac8vDXmvlIUBxm7aoWkuc066KBQ2IPLSoBPDZRek8AVn8SrPLWFsgdm2nwOPKO zVzXbNCdH5f715ynlUezmnr4o3SsYEXOZtse6ZEqJXKoPUAc4/7uMbnqBuX0t8SySJKCQV Ka7p9rwdRvODO9fnvMym8Joyber7j7k= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=F2lP4krB; spf=pass (imf23.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681718142; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YplWFnCAw5REiCL+91O+oQDL0q3EFb5pySxQGdi9SXo=; b=QHYrgu2fm57K7ZXMmivTQs45q+Pxi+c3as3huCo7PFrc+U2UfzhcqoP0w1cfLJqEm62T2q r2tIBZrJ0vl4XCVF07TxmcvN/ut4yVpwD0qLsBmMEIr/INO/81sIRzj7z2YafPVuKnMpRd UpNcr5fu1jz+xHBv0wNYNi53YMDOKhU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1681718141; x=1713254141; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=2aOPxNaioygwyhundx6GQZJnjyHxRbJrogJB60G/n8k=; b=F2lP4krBqMbpiozk+szBVE/KAXiuF9+8toakv2Hg9B1FTqPoswtjsSrF +6ijcl7wh7f9H1cuB6RHGloEdRSxJgfEkJ55wXwlAMakKxm49mQOYmPFd Hbr4hfApXX9o8n7cGm4Quo0ZZM5yEMFqWclco1PXrw+WYErq7TGTjdLXa Wj69s7LfiGRxfz12FNW46k1Z2N/ueU0vieFt22PTVOT9bQqrCl/p9+Scb qASXDpLXQkOAuYHdzFXaPObPnqPue6JdeqzvVjyY8E1ubfa7msdr4DjSQ NFibzJReZ/eHHA2bi26cbo+wAflGiH7dUHnBiz2UTqwCtCYEJWLP0A+I0 g==; X-IronPort-AV: E=McAfee;i="6600,9927,10682"; a="324453865" X-IronPort-AV: E=Sophos;i="5.99,203,1677571200"; d="scan'208";a="324453865" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Apr 2023 00:55:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10682"; a="723167476" X-IronPort-AV: E=Sophos;i="5.99,203,1677571200"; d="scan'208";a="723167476" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by orsmga001.jf.intel.com with ESMTP; 17 Apr 2023 00:55:37 -0700 From: Yin Fengwei To: linux-mm@kvack.org, akpm@linux-foundation.org, willy@infradead.org, yuzhao@google.com, ryan.roberts@arm.com Cc: fengwei.yin@intel.com Subject: [PATCH 2/2] lru: allow large batched add large folio to lru list Date: Mon, 17 Apr 2023 15:56:43 +0800 Message-Id: <20230417075643.3287513-3-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230417075643.3287513-1-fengwei.yin@intel.com> References: <20230417075643.3287513-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: BD39C14000A X-Rspamd-Server: rspam01 X-Stat-Signature: wf8j39q9e3bsemahc8fnpt4ry9i9x8rs X-HE-Tag: 1681718141-994292 X-HE-Meta: U2FsdGVkX18U1u7x/JHu0uyOyg9cZhf31cAvHlVFQRox4lQut73DtKOAzR67sNATLk9a61HCtKfO27oI7+1WKCuWSIH5YH2yRdpZgRHxHV7kqO83N0XjsWGhp99j9NjQOKkAAPGiYxw/iIR8e8X8sYL/i+oVtMmoB5pKwc9jPxcW0lxAJ6CXNHNRL0epqD040RE+0hVJPdRjhnEzG8KKjzKEDE9jkom0+AWpnGBf+I1xVQ0xV5ZLfmbPjiZqunbEGyt27lYhSmx9aoa6fJC6d9JaMCJLobe7hHbu+rBZRoLeXa3/zAStiQn7Hqk2ZsqsClqKo1ARzMiUUMhYUJgEBF1eZdepbdjgwxSViI/lr5ChAnHpWbEEir40TukG9HChdXEN2cZmhMa8urZ9B4332jbEgz+ULNvCskZQGebC627rpW6Q3hbrXv/mMXgM/EyO8O54xMKnnPLqA7vta621q0LF7yOLeMIfwm7qo2RJNDpM4X0IpsO4NOoVl5NW+yDzj5kB3TQif0AtsR2ao5xoFfXC21UCjVHG/kLp5KAqDksF2SyRXI4DvJorimemmzWB2hvjj5dN7VeOO+lqXyD2UqXXmgLlAQwUxHGXv/8brHjb/QOMJdnCs/Ard3hyXr1Q8PRFag/+BvqTdTttTZCWlxbryLBxXxJ4U/mCqUA4kUaqjbMsqtyB47T7w4zwX7lP3+jdbj5PHlKxZYgOniqwyJBNAyPp9sXNY6QPKBho4+a/8jDg4iA3s41VaSX1GOkejuEAw3QL8v8M686h0+eG1hQ+chEqz7KdMfC2/hyn28GkVxnWlYsn8rT3zjwr8zK1sir5sjZMgHN0i+NbytLVf6aCWEYksCe6DlI3DpPEA3wv7Pru6UDo5WteXkgZgR8tX6QBeMn1mFnUH6akhupTExHF2SGsQULGjfCD5NSRMbBgIjsUyxd0g7x4i5ikznqi32IWqki1kf0JGzJDAx7 pyC4SSLf AOmnGp89auaFZqIxnyjNUofPTnlVqSfoYqzJiKxNKP13UDxk2isCclLkw3hDQ9o/6jr0vNxl9xQSAk4PS1zQ5nPAo6PTdEj5lAN8RhQbUdqzXUUpp7jmBrijmtu6T3vOqikPF24aVKhkxVmgVV3xiTH3nSHoBEqHUM+tdxotGIvVMTns= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, large folio is not batched added to lru list. Which cause high lru lock contention after enable large folio for anonymous mappping. Running page_fault1 of will-it-scale + order 2 folio with 96 processes on Ice Lake 48C/96T, the lru lock contention could be around 65%: - 65.38% 0.17% page_fault1_pro [kernel.kallsyms] [k] folio_lruvec_lock_irqsave - 65.21% folio_lruvec_lock_irqsave With this patch, the lru lock contention dropped to 45% with same testing: - 44.93% 0.17% page_fault1_pro [kernel.kallsyms] [k] folio_lruvec_lock_irqsave + 44.75% folio_lruvec_lock_irqsave Signed-off-by: Yin Fengwei --- include/linux/pagevec.h | 19 +++++++++++++++++-- mm/swap.c | 3 +-- 2 files changed, 18 insertions(+), 4 deletions(-) diff --git a/include/linux/pagevec.h b/include/linux/pagevec.h index f582f7213ea52..d719f7ad5a567 100644 --- a/include/linux/pagevec.h +++ b/include/linux/pagevec.h @@ -10,6 +10,7 @@ #define _LINUX_PAGEVEC_H #include +#include /* 15 pointers + header align the pagevec structure to a power of two */ #define PAGEVEC_SIZE 15 @@ -22,6 +23,7 @@ struct address_space; struct pagevec { unsigned char nr; bool percpu_pvec_drained; + unsigned short pages_nr; struct page *pages[PAGEVEC_SIZE]; }; @@ -30,6 +32,7 @@ void __pagevec_release(struct pagevec *pvec); static inline void pagevec_init(struct pagevec *pvec) { pvec->nr = 0; + pvec->pages_nr = 0; pvec->percpu_pvec_drained = false; } @@ -54,7 +57,12 @@ static inline unsigned pagevec_space(struct pagevec *pvec) static inline unsigned pagevec_add(struct pagevec *pvec, struct page *page) { pvec->pages[pvec->nr++] = page; - return pagevec_space(pvec); + pvec->pages_nr += compound_nr(page); + + if (pvec->pages_nr > PAGEVEC_SIZE) + return 0; + else + return pagevec_space(pvec); } static inline void pagevec_release(struct pagevec *pvec) @@ -75,6 +83,7 @@ static inline void pagevec_release(struct pagevec *pvec) struct folio_batch { unsigned char nr; bool percpu_pvec_drained; + unsigned short pages_nr; struct folio *folios[PAGEVEC_SIZE]; }; @@ -92,6 +101,7 @@ static_assert(offsetof(struct pagevec, pages) == static inline void folio_batch_init(struct folio_batch *fbatch) { fbatch->nr = 0; + fbatch->pages_nr = 0; fbatch->percpu_pvec_drained = false; } @@ -124,7 +134,12 @@ static inline unsigned folio_batch_add(struct folio_batch *fbatch, struct folio *folio) { fbatch->folios[fbatch->nr++] = folio; - return fbatch_space(fbatch); + fbatch->pages_nr += folio_nr_pages(folio); + + if (fbatch->pages_nr > PAGEVEC_SIZE) + return 0; + else + return fbatch_space(fbatch); } static inline void folio_batch_release(struct folio_batch *fbatch) diff --git a/mm/swap.c b/mm/swap.c index 423199ee8478c..59e3f1e3701c3 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -228,8 +228,7 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) static void folio_batch_add_and_move(struct folio_batch *fbatch, struct folio *folio, move_fn_t move_fn) { - if (folio_batch_add(fbatch, folio) && !folio_test_large(folio) && - !lru_cache_disabled()) + if (folio_batch_add(fbatch, folio) && !lru_cache_disabled()) return; folio_batch_move_lru(fbatch, move_fn); }