From patchwork Fri Dec 2 22:35:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 13063302 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10678C4321E for ; Fri, 2 Dec 2022 22:37:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A7746B0071; Fri, 2 Dec 2022 17:37:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 657C46B0072; Fri, 2 Dec 2022 17:37:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 51F206B0073; Fri, 2 Dec 2022 17:37:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3F72B6B0071 for ; Fri, 2 Dec 2022 17:37:16 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E2108C0385 for ; Fri, 2 Dec 2022 22:37:15 +0000 (UTC) X-FDA: 80198828430.19.4AC6CFE Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf20.hostedemail.com (Postfix) with ESMTP id 947F11C0016 for ; Fri, 2 Dec 2022 22:37:14 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=LHn2nRg2; spf=pass (imf20.hostedemail.com: domain of 3GX6KYwsKCGIALMASRYMINAGOOGLE.COMLINUX-MMKVACK.ORG@flex--almasrymina.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3GX6KYwsKCGIALMASRYMINAGOOGLE.COMLINUX-MMKVACK.ORG@flex--almasrymina.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670020634; a=rsa-sha256; cv=none; b=iIkgOp8/CzZZXkNIVfTt3V76FXctIhRROEu4r5LKlFldxaMznBRWWncw9E14FhPD2nkBC6 ZxNuEda5/zUjBa3bB3rN+9ZzRtgTkhJvuHrVlQJDH+aXOQUKWISHV2KZhHP0QRK+o72gBA 2ffng173aU86wO05q7+pRrE8Awgge4g= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=LHn2nRg2; spf=pass (imf20.hostedemail.com: domain of 3GX6KYwsKCGIALMASRYMINAGOOGLE.COMLINUX-MMKVACK.ORG@flex--almasrymina.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3GX6KYwsKCGIALMASRYMINAGOOGLE.COMLINUX-MMKVACK.ORG@flex--almasrymina.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670020634; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=+RO437GEZ+WElFZehPW6Ok02BB1y3djf+6Wg2yQ+h9s=; b=uxziiNbnNsPDlJk+q7oZOlJcr7+9ycKxCBNbs8cbkGb188fbxS4EILQ+It3aOo64o5tBvl 1ckbvo8u1YuIbYMj2+VtF1+o8DcAoPtqxBdjlrlzYuMxYpANClS49Dqu0fae9dwT/hnXpG NiEqN1da3B5+NrL2QMDRTpDCFO4mrZs= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-3b102317668so64187677b3.23 for ; Fri, 02 Dec 2022 14:37:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=+RO437GEZ+WElFZehPW6Ok02BB1y3djf+6Wg2yQ+h9s=; b=LHn2nRg2oFSaB0dJIO0yooCpv3dQ9QXMByKCwdl+egE/WOxwP4FVQGWDYz6Y9tkPE5 Jgp0CK7S4yTl0+kb59ClT1OaReoxyzdt+FA3VldMFUb3CnSkvhWbbGUhX/Xy35QjivbK TO20TuKzlLUlQOLWrz0krvOhwUBh7koeYBCRszO+0ZfI78AbvD3Oqx3YxjNGYNKY5M87 9/Vr//ig9WyTbABAr+ryFE+y1QKpod6J16FyKzolUGaMMhBukvF/UQe6iw3yqO5cm5hD Dlel/jBqm1zRF7mqaUmelu3Yt6iL0EO5IjYDyTYElc9xM5l/JhwKJVbiEHqf4k/4b4Gn ursQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=+RO437GEZ+WElFZehPW6Ok02BB1y3djf+6Wg2yQ+h9s=; b=zOta8kBQIOvN50/A6/RuardpqRKbGm4r+zmJ9GEQffXOSferFTuKAPSKV4ILQVL/D0 dF6K5XjUV/To+tFQpDCK6R607rg8YQI+3HkjwvSSlO+nYECxZ1P5J6j3dt+GZKKyXnt+ /GWdRHTjMOzQtfCrgOu4Mg0l7P0Rt97dSRqGtDAaYTmlanb9odzyK+cZxn+CtJb5RBf+ 69U0lMnz2sP3PaeBdxkuF4xomtEGI35yr+59PzgkLjwthaQ4stqJeZ+w5Uh55msFAMFK oH5C50iAyhgVy4fjSz9gtQE4+TkraBGKHrAmVjgY4c61L8g3vvpghChCIqdZBHDj6H9o LkXw== X-Gm-Message-State: ANoB5pm4mzg7Vos/RgD5E03XofA2jRyJeZHrwKIZ5+w0gycqrOM3DEXl c+C+kD2lBWWqlpJTbr5V6WMq0q1CVJw1k8A4JA== X-Google-Smtp-Source: AA0mqf4PIdv0aQLEqmeEeWbJELak73oDPHmxGVKwPDJayj60lAtluQD5SfzgXdsUavXKcKM8KW79xWNR2mzBs49Dfg== X-Received: from almasrymina.svl.corp.google.com ([2620:15c:2d4:203:e655:31e2:2ad4:2421]) (user=almasrymina job=sendgmr) by 2002:a81:fe09:0:b0:3b0:b9aa:46eb with SMTP id j9-20020a81fe09000000b003b0b9aa46ebmr44197742ywn.150.1670020633813; Fri, 02 Dec 2022 14:37:13 -0800 (PST) Date: Fri, 2 Dec 2022 14:35:31 -0800 Mime-Version: 1.0 X-Mailer: git-send-email 2.39.0.rc0.267.gcb52ba06e7-goog Message-ID: <20221202223533.1785418-1-almasrymina@google.com> Subject: [PATCH v3] mm: Add nodes= arg to memory.reclaim From: Mina Almasry To: Tejun Heo , Zefan Li , Johannes Weiner , Jonathan Corbet , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton Cc: Huang Ying , Yang Shi , Yosry Ahmed , weixugc@google.com, fvdl@google.com, Mina Almasry , Michal Hocko , bagasdotme@gmail.com, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspam-User: X-Spamd-Result: default: False [-1.10 / 9.00]; BAYES_HAM(-6.00)[100.00%]; SORBS_IRL_BL(3.00)[209.85.128.202:from]; MID_CONTAINS_FROM(1.00)[]; MV_CASE(0.50)[]; FORGED_SENDER(0.30)[almasrymina@google.com,3GX6KYwsKCGIALMASRYMINAGOOGLE.COMLINUX-MMKVACK.ORG@flex--almasrymina.bounces.google.com]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; BAD_REP_POLICIES(0.10)[]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_DKIM_ALLOW(0.00)[google.com:s=20210112]; RCVD_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; R_SPF_ALLOW(0.00)[+ip4:209.85.128.0/17]; RCPT_COUNT_TWELVE(0.00)[21]; DKIM_TRACE(0.00)[google.com:+]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; DMARC_POLICY_ALLOW(0.00)[google.com,reject]; FROM_NEQ_ENVFROM(0.00)[almasrymina@google.com,3GX6KYwsKCGIALMASRYMINAGOOGLE.COMLINUX-MMKVACK.ORG@flex--almasrymina.bounces.google.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 947F11C0016 X-Rspamd-Server: rspam01 X-Stat-Signature: pd4sgxgwxob383x4645wkngtkhtpnpox X-HE-Tag: 1670020634-28544 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The nodes= arg instructs the kernel to only scan the given nodes for proactive reclaim. For example use cases, consider a 2 tier memory system: nodes 0,1 -> top tier nodes 2,3 -> second tier $ echo "1m nodes=0" > memory.reclaim This instructs the kernel to attempt to reclaim 1m memory from node 0. Since node 0 is a top tier node, demotion will be attempted first. This is useful to direct proactive reclaim to specific nodes that are under pressure. $ echo "1m nodes=2,3" > memory.reclaim This instructs the kernel to attempt to reclaim 1m memory in the second tier, since this tier of memory has no demotion targets the memory will be reclaimed. $ echo "1m nodes=0,1" > memory.reclaim Instructs the kernel to reclaim memory from the top tier nodes, which can be desirable according to the userspace policy if there is pressure on the top tiers. Since these nodes have demotion targets, the kernel will attempt demotion first. Since commit 3f1509c57b1b ("Revert "mm/vmscan: never demote for memcg reclaim""), the proactive reclaim interface memory.reclaim does both reclaim and demotion. Reclaim and demotion incur different latency costs to the jobs in the cgroup. Demoted memory would still be addressable by the userspace at a higher latency, but reclaimed memory would need to incur a pagefault. The 'nodes' arg is useful to allow the userspace to control demotion and reclaim independently according to its policy: if the memory.reclaim is called on a node with demotion targets, it will attempt demotion first; if it is called on a node without demotion targets, it will only attempt reclaim. Acked-by: Michal Hocko Signed-off-by: Mina Almasry Acked-by: Shakeel Butt Acked-by: Muchun Song --- v3: - Dropped RFC tag from subject. - Added Michal's Ack - Applied some of Bagas's comment suggestions. - Converted try_to_fre_mem_cgorup_pages() to take nodemask_t* instead of nodemask_t as Shakeel and Muchun suggested. Cc: bagasdotme@gmail.com Thanks for the comments and reviews. --- Documentation/admin-guide/cgroup-v2.rst | 15 +++--- include/linux/swap.h | 3 +- mm/memcontrol.c | 67 ++++++++++++++++++++----- mm/vmscan.c | 4 +- 4 files changed, 68 insertions(+), 21 deletions(-) -- 2.39.0.rc0.267.gcb52ba06e7-goog diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index 74cec76be9f2..c8ae7c897f14 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -1245,17 +1245,13 @@ PAGE_SIZE multiple when read back. This is a simple interface to trigger memory reclaim in the target cgroup. - This file accepts a single key, the number of bytes to reclaim. - No nested keys are currently supported. + This file accepts a string which contains the number of bytes to + reclaim. Example:: echo "1G" > memory.reclaim - The interface can be later extended with nested keys to - configure the reclaim behavior. For example, specify the - type of memory to reclaim from (anon, file, ..). - Please note that the kernel can over or under reclaim from the target cgroup. If less bytes are reclaimed than the specified amount, -EAGAIN is returned. @@ -1267,6 +1263,13 @@ PAGE_SIZE multiple when read back. This means that the networking layer will not adapt based on reclaim induced by memory.reclaim. + This file also allows the user to specify the nodes to reclaim from, + via the 'nodes=' key, for example:: + + echo "1G nodes=0,1" > memory.reclaim + + The above instructs the kernel to reclaim memory from nodes 0,1. + memory.peak A read-only single value file which exists on non-root cgroups. diff --git a/include/linux/swap.h b/include/linux/swap.h index 0ceed49516ad..2787b84eaf12 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -418,7 +418,8 @@ extern unsigned long try_to_free_pages(struct zonelist *zonelist, int order, extern unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, unsigned long nr_pages, gfp_t gfp_mask, - unsigned int reclaim_options); + unsigned int reclaim_options, + nodemask_t *nodemask); extern unsigned long mem_cgroup_shrink_node(struct mem_cgroup *mem, gfp_t gfp_mask, bool noswap, pg_data_t *pgdat, diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 23750cec0036..0f02f47a87e4 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -63,6 +63,7 @@ #include #include #include +#include #include "internal.h" #include #include @@ -2392,7 +2393,8 @@ static unsigned long reclaim_high(struct mem_cgroup *memcg, psi_memstall_enter(&pflags); nr_reclaimed += try_to_free_mem_cgroup_pages(memcg, nr_pages, gfp_mask, - MEMCG_RECLAIM_MAY_SWAP); + MEMCG_RECLAIM_MAY_SWAP, + NULL); psi_memstall_leave(&pflags); } while ((memcg = parent_mem_cgroup(memcg)) && !mem_cgroup_is_root(memcg)); @@ -2683,7 +2685,8 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, psi_memstall_enter(&pflags); nr_reclaimed = try_to_free_mem_cgroup_pages(mem_over_limit, nr_pages, - gfp_mask, reclaim_options); + gfp_mask, reclaim_options, + NULL); psi_memstall_leave(&pflags); if (mem_cgroup_margin(mem_over_limit) >= nr_pages) @@ -3503,7 +3506,8 @@ static int mem_cgroup_resize_max(struct mem_cgroup *memcg, } if (!try_to_free_mem_cgroup_pages(memcg, 1, GFP_KERNEL, - memsw ? 0 : MEMCG_RECLAIM_MAY_SWAP)) { + memsw ? 0 : MEMCG_RECLAIM_MAY_SWAP, + NULL)) { ret = -EBUSY; break; } @@ -3614,7 +3618,8 @@ static int mem_cgroup_force_empty(struct mem_cgroup *memcg) return -EINTR; if (!try_to_free_mem_cgroup_pages(memcg, 1, GFP_KERNEL, - MEMCG_RECLAIM_MAY_SWAP)) + MEMCG_RECLAIM_MAY_SWAP, + NULL)) nr_retries--; } @@ -6407,7 +6412,8 @@ static ssize_t memory_high_write(struct kernfs_open_file *of, } reclaimed = try_to_free_mem_cgroup_pages(memcg, nr_pages - high, - GFP_KERNEL, MEMCG_RECLAIM_MAY_SWAP); + GFP_KERNEL, MEMCG_RECLAIM_MAY_SWAP, + NULL); if (!reclaimed && !nr_retries--) break; @@ -6456,7 +6462,8 @@ static ssize_t memory_max_write(struct kernfs_open_file *of, if (nr_reclaims) { if (!try_to_free_mem_cgroup_pages(memcg, nr_pages - max, - GFP_KERNEL, MEMCG_RECLAIM_MAY_SWAP)) + GFP_KERNEL, MEMCG_RECLAIM_MAY_SWAP, + NULL)) nr_reclaims--; continue; } @@ -6579,21 +6586,54 @@ static ssize_t memory_oom_group_write(struct kernfs_open_file *of, return nbytes; } +enum { + MEMORY_RECLAIM_NODES = 0, + MEMORY_RECLAIM_NULL, +}; + +static const match_table_t if_tokens = { + { MEMORY_RECLAIM_NODES, "nodes=%s" }, + { MEMORY_RECLAIM_NULL, NULL }, +}; + static ssize_t memory_reclaim(struct kernfs_open_file *of, char *buf, size_t nbytes, loff_t off) { struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); unsigned int nr_retries = MAX_RECLAIM_RETRIES; unsigned long nr_to_reclaim, nr_reclaimed = 0; - unsigned int reclaim_options; - int err; + unsigned int reclaim_options = MEMCG_RECLAIM_MAY_SWAP | + MEMCG_RECLAIM_PROACTIVE; + char *old_buf, *start; + substring_t args[MAX_OPT_ARGS]; + int token; + char value[256]; + nodemask_t nodemask = NODE_MASK_ALL; buf = strstrip(buf); - err = page_counter_memparse(buf, "", &nr_to_reclaim); - if (err) - return err; - reclaim_options = MEMCG_RECLAIM_MAY_SWAP | MEMCG_RECLAIM_PROACTIVE; + old_buf = buf; + nr_to_reclaim = memparse(buf, &buf) / PAGE_SIZE; + if (buf == old_buf) + return -EINVAL; + + buf = strstrip(buf); + + while ((start = strsep(&buf, " ")) != NULL) { + if (!strlen(start)) + continue; + token = match_token(start, if_tokens, args); + match_strlcpy(value, args, sizeof(value)); + switch (token) { + case MEMORY_RECLAIM_NODES: + if (nodelist_parse(value, nodemask) < 0) + return -EINVAL; + break; + default: + return -EINVAL; + } + } + while (nr_reclaimed < nr_to_reclaim) { unsigned long reclaimed; @@ -6610,7 +6650,8 @@ static ssize_t memory_reclaim(struct kernfs_open_file *of, char *buf, reclaimed = try_to_free_mem_cgroup_pages(memcg, nr_to_reclaim - nr_reclaimed, - GFP_KERNEL, reclaim_options); + GFP_KERNEL, reclaim_options, + &nodemask); if (!reclaimed && !nr_retries--) return -EAGAIN; diff --git a/mm/vmscan.c b/mm/vmscan.c index 7b8e8e43806b..62b0c9b46bd2 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -6735,7 +6735,8 @@ unsigned long mem_cgroup_shrink_node(struct mem_cgroup *memcg, unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, unsigned long nr_pages, gfp_t gfp_mask, - unsigned int reclaim_options) + unsigned int reclaim_options, + nodemask_t *nodemask) { unsigned long nr_reclaimed; unsigned int noreclaim_flag; @@ -6750,6 +6751,7 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, .may_unmap = 1, .may_swap = !!(reclaim_options & MEMCG_RECLAIM_MAY_SWAP), .proactive = !!(reclaim_options & MEMCG_RECLAIM_PROACTIVE), + .nodemask = nodemask, }; /* * Traverse the ZONELIST_FALLBACK zonelist of the current node to put