From patchwork Fri Apr 11 08:13:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dev Jain X-Patchwork-Id: 14047815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82F55C369A8 for ; Fri, 11 Apr 2025 08:13:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5BC1C280186; Fri, 11 Apr 2025 04:13:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 56AE428017D; Fri, 11 Apr 2025 04:13:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 45878280186; Fri, 11 Apr 2025 04:13:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 29C3728017D for ; Fri, 11 Apr 2025 04:13:13 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 3243BC2162 for ; Fri, 11 Apr 2025 08:13:14 +0000 (UTC) X-FDA: 83321047908.29.10CB349 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf27.hostedemail.com (Postfix) with ESMTP id 682A740009 for ; Fri, 11 Apr 2025 08:13:12 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf27.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744359192; a=rsa-sha256; cv=none; b=hCgEKe+EptGiGdrkk8oHv2xdqZVem+r1rr9oy4ChbpfBHGAqemrum+xa43lNvVZ89SsZmY 4M/yceEKO3bSe48GeFkZe9s8Ol9AVddIZz0h/mFTXL1K0RKz4n6O21+QAMOND0yS9Z877F SBP7b4o53268YkY7RyhuYMv1R1Z4fXM= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf27.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744359192; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references; bh=pAeLAEv1hlQkSH8CFyArnpSyV0PUAGPte5Vn4vAGRns=; b=qqj1Z3m1JcvPTNuLK+q3O/q1wABF2O43HQd1pnv6IvaYXU9FupegnhVw9PeyYzdhNr5y3T VnHC44rjq4I6b+RluF0/xqT4HjvpLdTAGLtD8nHlhkghrdCGnJ8MHbHQEAKbGOaAphABmd 9kEg5MztyQuv8haV7T2/x/rMrf/McIQ= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1214E106F; Fri, 11 Apr 2025 01:13:11 -0700 (PDT) Received: from K4MQJ0H1H2.arm.com (unknown [10.163.73.102]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 068CB3F694; Fri, 11 Apr 2025 01:13:05 -0700 (PDT) From: Dev Jain To: akpm@linux-foundation.org Cc: ryan.roberts@arm.com, david@redhat.com, willy@infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, hughd@google.com, vishal.moola@gmail.com, yang@os.amperecomputing.com, ziy@nvidia.com, Dev Jain Subject: [PATCH] mempolicy: Optimize queue_folios_pte_range by PTE batching Date: Fri, 11 Apr 2025 13:43:01 +0530 Message-Id: <20250411081301.8533-1-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) MIME-Version: 1.0 X-Rspamd-Queue-Id: 682A740009 X-Stat-Signature: h39t4o5k8grf6n1eupm4zmhj9a5sbjhm X-Rspam-User: X-Rspamd-Server: rspam06 X-HE-Tag: 1744359192-952795 X-HE-Meta: U2FsdGVkX1/oYQQCMsUCe+o2TbxP+ZU/xkMGdEIeziIp184PTqFJBf1d3sdTXVhq4cTwKDt/gOrkUZnkXr5nF1IBAbsgL0Towt+yUp+NrzMmnFkYNZTYG+udnKLEb733L/UcAC0dvoa+JznmxN25SkrwVLGB7K3EAR/p9wet4Hcwjaj8tAtUFfy3yq+WbmMQXW78yVTOitQ3AFpTcNC557aznaPHcB5NOOOE5Y5lL3/I711LNcnZvqoa9IdyzJclw5fZEpOcPUkKpQJxTWrjMXvYDTat3Jap1bJA9vpR2BcYbT6I2R/UKomu3ezyWsn7jbJdyQrzwMH8cPyTX5rpEXmhRVe/6F8Ifq9ZUAKhRnSFop59mZi5kuv55KfwBTBENWc4YIQBUTO4IT2hJGfQY/3GELBVdBujpil5/FKbKO7QC8hTC2qBDWifuLKDv+Stjw/XIY/XGgERWBH3aRpeSyeYquavHYOg7kiV6qUuhvPQhWI8o029MHPr47vSYVRCIAGNtdCr2PYjmjCo8YLMj9sLyE+/UvilTzryufVv0/FvAH/BAYo7vo5VsGIV5gS8dsrlGa7DM/dl6W2qULFNXyRl+i8slg4yld8SWh0mSIkQ/PbpL39H4RkYQH/+LqCBbKSrH9c0HF7lkvcK0xrnNyYYDAHhEJCZ3SgUohzpxCdXvltVHqlaOI9IZY9/hMticuQGJmSHWFwKlzIMcU3WnQ7HBteU80uSQyl/QQMVUCdr3g6wiIMsR+VPREg3HXihslehQrK+rgIoNBZTCm2ZblbPBPPjoiUeTQgN/kTfUnbPkVxLzMNPi0h4PSlWHswvigKEbX5o0NmJeeJw8fhtj/LvmufZtBWi8KfWNyj+xW8Ce+Qr/YiGyA6j+BrRvDp1pgt2XbOiZtf++82aZjWCJ7dfdsSl2zd8hS4Z6XS+/yNoNxamotVso7BbBzVZjkkhYuN5k6RQZAInLbG7mp7 Ci4N6c6Q 4Xo0cnKy/yvGzOBVhVD5bbgE185hj0XiJ/fM0lJ7Cuh91McfJ7iiBmpchWdK/xjwpCFAZd5TnLqPra/xs/YehTLUOPG1YOsTlt4GO2DJsNglwsB4oOM5aZjyqcLUVqlCMrjRFLTSrJXIkto9LPJJtb39U5aLo7m907rgeGVQy0wYQovYgLUNr8YF51XD81aoLGrUlLbdy8jcNKIWG5fXruA4CkzUr8nE09imQOucJGym/vmM6BpbibSCqX4OgbV+4ImrSvjeqyXxUBj27eaQJ4+TcZw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: After the check for queue_folio_required(), the code only cares about the folio in the for loop, i.e the PTEs are redundant. Therefore, optimize this loop by skipping over a PTE batch mapping the same folio. Signed-off-by: Dev Jain --- Unfortunately I have only build tested this since my test environment is broken. mm/mempolicy.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index b28a1e6ae096..b019524da8a2 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -573,6 +573,9 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, pte_t *pte, *mapped_pte; pte_t ptent; spinlock_t *ptl; + int max_nr; + const fpb_t fpb_flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY; + int nr = 1; ptl = pmd_trans_huge_lock(pmd, vma); if (ptl) { @@ -586,7 +589,8 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, walk->action = ACTION_AGAIN; return 0; } - for (; addr != end; pte++, addr += PAGE_SIZE) { + for (; addr != end; pte += nr, addr += nr * PAGE_SIZE) { + nr = 1; ptent = ptep_get(pte); if (pte_none(ptent)) continue; @@ -607,6 +611,11 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, if (!queue_folio_required(folio, qp)) continue; if (folio_test_large(folio)) { + max_nr = (end - addr) >> PAGE_SHIFT; + if (max_nr != 1) + nr = folio_pte_batch(folio, addr, pte, ptent, + max_nr, fpb_flags, + NULL, NULL, NULL); /* * A large folio can only be isolated from LRU once, * but may be mapped by many PTEs (and Copy-On-Write may @@ -633,6 +642,7 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, qp->nr_failed++; if (strictly_unmovable(flags)) break; + qp->nr_failed += nr - 1; } } pte_unmap_unlock(mapped_pte, ptl);