From patchwork Fri Nov 29 02:52:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wang <00107082@163.com> X-Patchwork-Id: 13895576 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C997BE7716C for ; Thu, 5 Dec 2024 15:29:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 079B36B011D; Thu, 5 Dec 2024 10:19:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 384AA6B00E0; Thu, 5 Dec 2024 10:19:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B49516B00C8; Thu, 5 Dec 2024 10:19:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3D5976B0083 for ; Thu, 28 Nov 2024 21:52:53 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D6AA440AC4 for ; Fri, 29 Nov 2024 02:52:52 +0000 (UTC) X-FDA: 82837609848.19.9FB3BB6 Received: from m16.mail.163.com (m16.mail.163.com [220.197.31.2]) by imf20.hostedemail.com (Postfix) with ESMTP id EA6E01C0007 for ; Fri, 29 Nov 2024 02:52:31 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=163.com header.s=s110527 header.b=C8de6NLX; spf=pass (imf20.hostedemail.com: domain of 00107082@163.com designates 220.197.31.2 as permitted sender) smtp.mailfrom=00107082@163.com; dmarc=pass (policy=none) header.from=163.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732848766; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iAZhN5QA1fVtJxok5Hv4iANgsbTu1WmImQVILA5WLKo=; b=BlK9thE1QaJf2NsryTE65A7snaSHWsm/d/HO5wX6u+baQpM6Q1ksrOQg0JvsuRfrjVZwnF T5jMyDm90qxc6bRWDgjUDwttgcAn7CCjpCnEWdwauM39NN+atlitgrPMR8mKGekaWdKJVF Vd96iOIguDuyRThlty6b6dLbLLeaU/E= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732848766; a=rsa-sha256; cv=none; b=mieJcIEIHZnCJ7/YS6ZocdDkk6S+cJumes8ECf0btEqAck7JFZUIPrZEPw6deIoLM8F3HH 1mgisdsyYk5aMmCNC4yZLDc2z8oScae243uL09AF7ZoGguFbf4rWNroKPP9nUKN9o2jhkV +dkZaRB1mva3SPmtx8E9PTrpfEOLMp0= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=163.com header.s=s110527 header.b=C8de6NLX; spf=pass (imf20.hostedemail.com: domain of 00107082@163.com designates 220.197.31.2 as permitted sender) smtp.mailfrom=00107082@163.com; dmarc=pass (policy=none) header.from=163.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:Subject:Date:Message-Id:MIME-Version; bh=iAZhN 5QA1fVtJxok5Hv4iANgsbTu1WmImQVILA5WLKo=; b=C8de6NLXQRLSmrnogsPMv NGFCsRFI+x4CccNUOEqADPh3JWFjym3e+z3L8wSOcmtXYjWad836fneN+7kAaukl Tzl8DdblXOoqxxZEr/Sfg5xVOeztPoQkFTzFQslbb1RIJ2kBvlX162Q5ZVEs27EL V8jMf5ijwWFzrOGdl5IKhQ= Received: from localhost.localdomain (unknown [111.35.188.140]) by gzga-smtp-mtada-g0-2 (Coremail) with SMTP id _____wDXjT5eLElnqS_yKQ--.30366S4; Fri, 29 Nov 2024 10:52:27 +0800 (CST) From: David Wang <00107082@163.com> To: surenb@google.com, yuzhao@google.com, kent.overstreet@linux.dev, akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, David Wang <00107082@163.com> Subject: [PATCH v2] mm/codetag: swap tags when migrate pages Date: Fri, 29 Nov 2024 10:52:13 +0800 Message-Id: <20241129025213.34836-1-00107082@163.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241128102619.707071-1-00107082@163.com> References: <20241128102619.707071-1-00107082@163.com> MIME-Version: 1.0 X-CM-TRANSID: _____wDXjT5eLElnqS_yKQ--.30366S4 X-Coremail-Antispam: 1Uf129KBjvJXoWxAFWkZrW3JFyxtF43Cw1Dtrb_yoWrZrWkpF W0g3W7Ga98JFy7WrWv9r1qvr13X3yxt3yxAFWaka4fu3W3tw4vkw1Dt34jqF45ZFyUJFna vr4vqF1Fg3WUJa7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x0pE8nYUUUUUU= X-Originating-IP: [111.35.188.140] X-CM-SenderInfo: qqqrilqqysqiywtou0bp/1tbiMximqmdJKQiGQgAAss X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: EA6E01C0007 X-Stat-Signature: jfm7s8zi5pqzskaguohb9wsjms8b661z X-Rspam-User: X-HE-Tag: 1732848751-961402 X-HE-Meta: U2FsdGVkX18RIP28Z5V5Jq7oTkS89NaIw4MUZqPsOBFHukyJ6hzA6DZPx2qFqo7AkaQ9xziVQm61F/pndc/XewT9hTBsSyDWox3cnpgEjTFHFWkWfpkgnBLE4KVavNv4cMZ25RxKU1prKedHkFUMelkzdndgDFW+g93Yhgs7Qd/VAg1FpWrHYdsKLaDpXYlwzM7YmsedZturcB561oEJ5oFS2sgv5Mhx6OfeflKK4g2nWzyoGv0wfFps1UzC9dbfYvCaRFv+uXi5VmLjHxlem2ulJvMe/qdR3tW2qm/Wjl5A+9AujSipzX5iYDAvgcExOHp6tFE/hyPRCGq0nVealuDRG6/sXao2PnfrU+OBbZU6ujWni36XASPCtbxVqcl6u9HT5T3hjdyN+PyKc5cMztl+aDIS7TVmaiiTOK8HYRqotg+R2b3EWNP1lE2bTIO2M2bbL7De9d2lBDLC1MUgc/04B8eYASFgXievya+XFfbgsLqniDiB+cCO/yj2P1y6WiKSX2acxHegh8tPA6dWXTP580VeC3obrdPkUtGQoL9vzC10XrUZ/g4ua9dr7ayj/UrGsppn4eieS26CotGw3cEAo/mjlGXczLZ86aF9NeHqGAzBZ5nV2VVifh0Ic0+JnQPVG8Xh+r+f9SfMiY6tsVO4S076x/2j9MStUjcwXdJsXSil9jblbMFbZl1dnFRTR/TjI1MPqJ5hHmuhpU9FevRCC+IuFkA92R9mOqlVIIK8TPCXQ7KX7c8xOrkyNavHk4XcCXOHmhqy2cIBlQEoPDge4MxXtcWnCXvNdtr6qU4ENYjn2sx76MlbCXDIuArP5oA5ODYDp6jLV9+SduVjtTEHbrV9woq0zMH8J4eaupCpw1bxXj9QGSj6iE4xR4UgzFv47WsJYPgugtys0zQRQM2buwVHoEPclySc3RwBE91O62RSI6J+gOFIw4Mb8L3K/sp1OAQlfDoDBZ5kNgS fmct70O9 AgO+e3Xt73U5Rh5b0Faxnq84CHGDJw3vxSQrPEKxwGFRhAQcR8RDR/nJlr3R0RhnbC+NQoTo2O8XacGSs7XVHTSuL8gikzUbj9wSoZPeUcft5hr6OIf3gBrFMtbtUsED7zkQZgyrRexx2ImRtCTbx9Jf/GSPrn7jNJSLANS/Z+bSEPdKXuVUe1MGvxRYGylvicDvgQTKDs0btSNqi29iXTE0IpR/312LmePc4fTQ7O7vhARhVn+OBJg6bCSGHUvaFMLyLl+aiaRHDQTm/xe3qWIhQeJ8IoUYuwfpVZJvYy76e9WwAGaXT0qeNbMDI8kox30EZt1mY9XKvTuPyJLwjQANZrKavd0eAE1xTLYcNT+RdqIpu4vhW/x0FK3QcEgZOVWEn0JKQjg90gH3/O1zy4qrNGdbdbceABDWvl3TUxDaINi3ibJ0E0RimIkAZgksHo29/6nmT3IKWZjp/1mJWsTqPss9A2vNeia2Jzq5dD85JB0pfz15IyO92zZMd3rWUj8TqttbZkWdiaTM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Current solution to adjust codetag references during page migration is done in 3 steps: 1. sets the codetag reference of the old page as empty (not pointing to any codetag); 2. subtracts counters of the new page to compensate for its own allocation; 3. sets codetag reference of the new page to point to the codetag of the old page. This does not work if CONFIG_MEM_ALLOC_PROFILING_DEBUG=n because set_codetag_empty() becomes NOOP. Instead, let's simply swap codetag references so that the new page is referencing the old codetag and the old page is referencing the new codetag. This way accounting stays valid and the logic makes more sense. Fixes: e0a955bf7f61 ("mm/codetag: add pgalloc_tag_copy()") Signed-off-by: David Wang <00107082@163.com> Closes: https://lore.kernel.org/lkml/20241124074318.399027-1-00107082@163.com/ Acked-by: Suren Baghdasaryan Suggested-by: Suren Baghdasaryan --- include/linux/pgalloc_tag.h | 4 ++-- lib/alloc_tag.c | 36 ++++++++++++++++++++++-------------- mm/migrate.c | 2 +- 3 files changed, 25 insertions(+), 17 deletions(-) diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 0e43ab653ab6..3469c4b20105 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -231,7 +231,7 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) } void pgalloc_tag_split(struct folio *folio, int old_order, int new_order); -void pgalloc_tag_copy(struct folio *new, struct folio *old); +void pgalloc_tag_swap(struct folio *new, struct folio *old); void __init alloc_tag_sec_init(void); @@ -245,7 +245,7 @@ static inline struct alloc_tag *pgalloc_tag_get(struct page *page) { return NULL static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) {} static inline void alloc_tag_sec_init(void) {} static inline void pgalloc_tag_split(struct folio *folio, int old_order, int new_order) {} -static inline void pgalloc_tag_copy(struct folio *new, struct folio *old) {} +static inline void pgalloc_tag_swap(struct folio *new, struct folio *old) {} #endif /* CONFIG_MEM_ALLOC_PROFILING */ diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 2414a7ee7ec7..35f7560a309a 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -189,26 +189,34 @@ void pgalloc_tag_split(struct folio *folio, int old_order, int new_order) } } -void pgalloc_tag_copy(struct folio *new, struct folio *old) +void pgalloc_tag_swap(struct folio *new, struct folio *old) { - union pgtag_ref_handle handle; - union codetag_ref ref; - struct alloc_tag *tag; + union pgtag_ref_handle handle_old, handle_new; + union codetag_ref ref_old, ref_new; + struct alloc_tag *tag_old, *tag_new; - tag = pgalloc_tag_get(&old->page); - if (!tag) + tag_old = pgalloc_tag_get(&old->page); + if (!tag_old) + return; + tag_new = pgalloc_tag_get(&new->page); + if (!tag_new) return; - if (!get_page_tag_ref(&new->page, &ref, &handle)) + if (!get_page_tag_ref(&old->page, &ref_old, &handle_old)) return; + if (!get_page_tag_ref(&new->page, &ref_new, &handle_new)) { + put_page_tag_ref(handle_old); + return; + } + + /* swap tags */ + __alloc_tag_ref_set(&ref_old, tag_new); + update_page_tag_ref(handle_old, &ref_old); + __alloc_tag_ref_set(&ref_new, tag_old); + update_page_tag_ref(handle_new, &ref_new); - /* Clear the old ref to the original allocation tag. */ - clear_page_tag_ref(&old->page); - /* Decrement the counters of the tag on get_new_folio. */ - alloc_tag_sub(&ref, folio_size(new)); - __alloc_tag_ref_set(&ref, tag); - update_page_tag_ref(handle, &ref); - put_page_tag_ref(handle); + put_page_tag_ref(handle_old); + put_page_tag_ref(handle_new); } static void shutdown_mem_profiling(bool remove_file) diff --git a/mm/migrate.c b/mm/migrate.c index 2ce6b4b814df..cc68583c86f9 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -745,7 +745,7 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio) folio_set_readahead(newfolio); folio_copy_owner(newfolio, folio); - pgalloc_tag_copy(newfolio, folio); + pgalloc_tag_swap(newfolio, folio); mem_cgroup_migrate(folio, newfolio); }