From patchwork Thu Mar 21 22:07:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13599410 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92D89CD11DC for ; Thu, 21 Mar 2024 22:08:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B1FE6B009D; Thu, 21 Mar 2024 18:08:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 13D186B009E; Thu, 21 Mar 2024 18:08:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA7EB6B009F; Thu, 21 Mar 2024 18:08:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D2B6B6B009D for ; Thu, 21 Mar 2024 18:08:22 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9A3B7C048F for ; Thu, 21 Mar 2024 22:08:22 +0000 (UTC) X-FDA: 81922435644.28.0CD2B56 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf01.hostedemail.com (Postfix) with ESMTP id 803C540010 for ; Thu, 21 Mar 2024 22:08:20 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Gq0YrZaZ; spf=pass (imf01.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711058900; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dI3BkO2nCtQEnBDt3PLoa+NL/Gig0J3ZsiByK6YiGEU=; b=npzWGjXQqCHUMrbyOPIIcY5ddBDMMhTlA75BoWRdlmeERWgEyXKN3ctjkFeiPn7Px7olNJ sKDvj8s99FoChA2oQMyYMLaHSMrfjYqCWMRnCc2oN1nFsR7vQWX1RU2Hc5L6R1me3IhI+5 kj1dtIX2qvMHDTWlp989/9G0EYROL1Q= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711058900; a=rsa-sha256; cv=none; b=HgUFm+EHn6x65tpD3ucHXPBu9A3v+ff6nO9i/hOpojnwXcf0OK82XRFJOa4bRN9OJQp8Un uayC1VCr06riV1xnR6ZFYQiS+zs9UNQTVBXAQ3exszMTqaVo0aOyWxwaF2E1JQFg9YxgtZ Nu7pW3T2Q+Vja9Rd43wshS/HPvwHl/o= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Gq0YrZaZ; spf=pass (imf01.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058899; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dI3BkO2nCtQEnBDt3PLoa+NL/Gig0J3ZsiByK6YiGEU=; b=Gq0YrZaZJDba3/oiLHQ5k0QmoQm24cQ+2uRUbyM7sQ1AQDcS7Dy2L/3pkR7wVjlqpxGD3z mcAbI1PWYLUrjrMR4xJKymVdTk2i//qVxoi6PSNXD+zK4MGzMyJGLIS+lin8FeDhv0Hmcl 3dxdDIBhC+tRORZawc/LBLVRrZXzBG8= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-588-Ym7j8uxwPQSKPiE1AwZfuA-1; Thu, 21 Mar 2024 18:08:18 -0400 X-MC-Unique: Ym7j8uxwPQSKPiE1AwZfuA-1 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-78a0e8a60f8so22227885a.0 for ; Thu, 21 Mar 2024 15:08:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058897; x=1711663697; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dI3BkO2nCtQEnBDt3PLoa+NL/Gig0J3ZsiByK6YiGEU=; b=GVMBSlbSIJTU5EbCGCzA/6FC739WyTM7iffblImKGFKqj4jF5XrTvnjIjwhNP9DquF wBjhYyixKgMr+ouADDTVqsYvLn0TOPSlNBjJ3pOKWCTBp6aJzTUytbGoNcJghmY3Ulm+ vRmc3m6QSLRLCi8i03Jqwbn96tjLnNxwHyUZZ2xb/Xj+poHm07IMopp5NIJ1SjMoTDPE MJbWcbDPgfCaY+IkfrYMamEMb/cPI13Fxryq2ecgGsW8QEtPN8g9zg6PoYFViV4MEMf4 BWKYy+t/hTnWDG7YmXRj73DkYW5bUdYFcYsahZfnPaNeofTk2dpOsG3yaW95rCpY9L67 Lkhw== X-Gm-Message-State: AOJu0YyTK/9SYMdS43GBu/V3rmo95k1iQdN2heePLs8v2tudsOKHuKYQ bd9Iv7hhKeOdLF7cnfzBfbzbw9fmWbx4iB6zlLjlU1msejbrb5YULsyf8G/qWntxyVbVZSadyIN vSwvHuuluLBFsnoyKXo/85szSM7H6VM0vvr9Izqw3nEcclwM7zRdeYAEjKs/z4s9q+Cpf37B4Zv xtVMByhVo7pmcssWNlPh6muImAkr5zNA== X-Received: by 2002:a05:6214:4489:b0:68f:e779:70f2 with SMTP id on9-20020a056214448900b0068fe77970f2mr443561qvb.3.1711058897286; Thu, 21 Mar 2024 15:08:17 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHGuIUGmtGDilXkO+Txm1IkzpifWGQBAYBHvb3QP196cSPWL36y2snARbq2rVPrLWqoQw6FQA== X-Received: by 2002:a05:6214:4489:b0:68f:e779:70f2 with SMTP id on9-20020a056214448900b0068fe77970f2mr443515qvb.3.1711058896774; Thu, 21 Mar 2024 15:08:16 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:16 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Christophe Leroy , Matthew Wilcox , Rik van Riel , Lorenzo Stoakes , Axel Rasmussen , peterx@redhat.com, Yang Shi , John Hubbard , linux-arm-kernel@lists.infradead.org, "Kirill A . Shutemov" , Andrew Jones , Vlastimil Babka , Mike Rapoport , Andrew Morton , Muchun Song , Christoph Hellwig , linux-riscv@lists.infradead.org, James Houghton , David Hildenbrand , Jason Gunthorpe , Andrea Arcangeli , "Aneesh Kumar K . V" , Mike Kravetz Subject: [PATCH v3 06/12] mm/gup: Refactor record_subpages() to find 1st small page Date: Thu, 21 Mar 2024 18:07:56 -0400 Message-ID: <20240321220802.679544-7-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 803C540010 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 39ehau4kxu7nmjczfh8nfwt77ojsq1em X-HE-Tag: 1711058900-207467 X-HE-Meta: U2FsdGVkX19UdH039RSD684lfw4/zkoZxSSsm6FuonffrkS/ImjUnS4YTnBQXktpel37RRaOxK356XtnxwOyNUKQmlPM15+4wQYBrdf8E1zWqxeWwux0O/z6D2XCjmY3G96RlJmQ5buuDfsopP9GLqmE0DMwYqsromnhdcdapooHCz1hlfquGFvW4WQn7nYTgKiaAOQfRV44J8233Q709yNk3e9Tt+UOmWoFI3dmLnQ04CD7iJm6sarhMhD0IY3cVWpe26WOOQw/IHPcV2A906qiKwJkseS+IojVyDp436egjc0d6l+SdcTg0BeIaHehLumMGbKDuTMNmKja8KzFPwjGxirFIm8p3AmGWzRvP4OmAo1SfWMpKTFFZlQrpOrhylbVOZVL6MnnQ2FCzsFheBd9HIW39G1KNfwTXwNa+yzjknX53T4/gUXxAXd02DNMvw2OIjokH3MkN+vOPf0bkDqMNMhANyi5rB9M7MEp8agYEx0qyRJVY1Yr1tq1tgD7leXUlwqgvHwvY+FHMSIw2qwIZTu1k5JmBjDHbiSe3WSHfN4mYmHBNaToeb19DPcbl41RtyXztzFyXC1yhNd7VHqe4GcVdNszDovpC7K80s8xC6DjXsukYKkOcbPVc1soz49L0SLV9aa0qrW7mhiysWfBnigAKuDlmsUH+dqTag0oZe5wXxxPADERKT/rtwSW6CE7ASYBSNSdXO+nYMUO0OsW5Z1kfPDjJj4CSUxRl9Kd8mUjZF550RVr3+9ChWeZ1meFfr8+CsipxC72b54gR++CY0I4w2HDyqfVtgtpczesB4OF1vGrMwzj5DKcXhtaAQ5Xu05ollOf1LSNt4CL1TnQeXeTvXasHJVHVHX7/QRQUg1PX1yBRlxGivI8dfcf7QYD0t8ccvEynOAzU8PuuG9mmbAJg9677i9Wqu0lttRCIGiklOwn/7QkljfOx/ScS8O2TnUMa5IKex6T5iN osETQJsE HUrj0jND/xb69qXoGEeHB6Gg6s9rEz5bXYQFsHARGcIeb2dFkw55KdQVwnmTkQmod9yoHl0o5wcFR9+dBI6ClDrQ4gHa1UpFrXKCTV5uPjLo1rnR43rkqWqZyo9fNQDk9j4FJhlHfMXph3PMCYBgY/qPAF0h6jwcsR7mgTphkSjntcGFkvGrKDwsDpf2mcU2X3UYdUIBBOkrwNGs5KiK6mloP7NwrcNe+EgmRm2Eo6krsixJVV2UqB+L4u/FjNfOBpB9XFttLSDb1hfDBl0dSYR22zF14CIuUAC3LWHve8SKt4jqRzAOgOWjX5LAcXctHgtNgfFpInLmenpH5+6eNbu2LvkwTYOoe3DXRJlAgEOmHrBSmLYnlyXUk3Q4Yw+q+WDIVv8W5TZgZ2kXu/j4mwchv2DmIBdiu+eFYl+08ENg1ocJ//lrOmN0fky6AZVWuQCOdxK6txA7OMQo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peter Xu All the fast-gup functions take a tail page to operate, always need to do page mask calculations before feeding that into record_subpages(). Merge that logic into record_subpages(), so that it will do the nth_page() calculation. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- mm/gup.c | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 9127ec5515ac..f3ae8f6ce8a4 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2778,13 +2778,16 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, } #endif -static int record_subpages(struct page *page, unsigned long addr, - unsigned long end, struct page **pages) +static int record_subpages(struct page *page, unsigned long sz, + unsigned long addr, unsigned long end, + struct page **pages) { + struct page *start_page; int nr; + start_page = nth_page(page, (addr & (sz - 1)) >> PAGE_SHIFT); for (nr = 0; addr != end; nr++, addr += PAGE_SIZE) - pages[nr] = nth_page(page, nr); + pages[nr] = nth_page(start_page, nr); return nr; } @@ -2819,8 +2822,8 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, /* hugepages are never "special" */ VM_BUG_ON(!pfn_valid(pte_pfn(pte))); - page = nth_page(pte_page(pte), (addr & (sz - 1)) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pte_page(pte); + refs = record_subpages(page, sz, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2893,8 +2896,8 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, pages, nr); } - page = nth_page(pmd_page(orig), (addr & ~PMD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pmd_page(orig); + refs = record_subpages(page, PMD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2937,8 +2940,8 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, pages, nr); } - page = nth_page(pud_page(orig), (addr & ~PUD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pud_page(orig); + refs = record_subpages(page, PUD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2977,8 +2980,8 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, BUILD_BUG_ON(pgd_devmap(orig)); - page = nth_page(pgd_page(orig), (addr & ~PGDIR_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pgd_page(orig); + refs = record_subpages(page, PGDIR_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio)