From patchwork Tue Dec 19 07:55:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13497943 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0C2DC46CA2 for ; Tue, 19 Dec 2023 07:57:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A11D6B0072; Tue, 19 Dec 2023 02:57:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 62A806B007B; Tue, 19 Dec 2023 02:57:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4CB7D6B007D; Tue, 19 Dec 2023 02:57:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 383BE6B0072 for ; Tue, 19 Dec 2023 02:57:25 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 1423180888 for ; Tue, 19 Dec 2023 07:57:25 +0000 (UTC) X-FDA: 81582812850.15.F891CC2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf16.hostedemail.com (Postfix) with ESMTP id 5C0E2180006 for ; Tue, 19 Dec 2023 07:57:23 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SyDs1X7f; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf16.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702972643; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4pxuQ+zeUrZD43W0p30SCM4P3ajCE4Euxhqhb6PE8fI=; b=H4PCmNdk7J2HD+yNK3px9tq9IdD81yDVyNedwESRL9ox7TAXC571v2ZfvgtspzLl6eTbyW ophGjleFa26v1Cc/W/08Rbn7oK8jheKRdZ30+i/M9fLCv2puX20Uen7OIE6xIySfqOu6gU bTXsvZCrUWPpAk5E1ShJjfuPp3BoA1k= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SyDs1X7f; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf16.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702972643; a=rsa-sha256; cv=none; b=ZoZNOuVVAhL1VvD16GiifDDwKXDjpvMgiVE3zQzZfNHBafbcO6TVhBXWMp2emjjnpwfpYZ y5fCDB5XcAa1fWgPCdmK0jOxIq5VHL9IpOXpW2EEVf8B2hgp+hDQ6Zj10UzNFutZ/gHPpZ K3m/Ib2y+lJ9IpXbaYe61O9pp3fanBg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702972642; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4pxuQ+zeUrZD43W0p30SCM4P3ajCE4Euxhqhb6PE8fI=; b=SyDs1X7fCzcJCaDo1qG3sb0Y1RCZgbsFRW9xDymmwRPhJiJgRh3knwcM+Y9NOcR83XDiWI 7Xn0XgTV/dPk3RXRhGCWokxAapZMEiROHZe2cjWr672wEaeUJusV2QrCs0MC2UtF59wDWB h1sprC6SnqFJ4swoJtytquBaWcfKiEA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-308-O_LLgdEzO2yAjxB50bIomw-1; Tue, 19 Dec 2023 02:57:17 -0500 X-MC-Unique: O_LLgdEzO2yAjxB50bIomw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 94A54837221; Tue, 19 Dec 2023 07:57:16 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.117]) by smtp.corp.redhat.com (Postfix) with ESMTP id B40302026D66; Tue, 19 Dec 2023 07:57:05 +0000 (UTC) From: peterx@redhat.com To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Christophe Leroy , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Mike Kravetz , Mike Rapoport , Christoph Hellwig , John Hubbard , Andrew Jones , linux-arm-kernel@lists.infradead.org, Michael Ellerman , "Kirill A . Shutemov" , linuxppc-dev@lists.ozlabs.org, Rik van Riel , linux-riscv@lists.infradead.org, Yang Shi , James Houghton , "Aneesh Kumar K . V" , Andrew Morton , Jason Gunthorpe , Andrea Arcangeli , peterx@redhat.com, Axel Rasmussen Subject: [PATCH 07/13] mm/gup: Refactor record_subpages() to find 1st small page Date: Tue, 19 Dec 2023 15:55:32 +0800 Message-ID: <20231219075538.414708-8-peterx@redhat.com> In-Reply-To: <20231219075538.414708-1-peterx@redhat.com> References: <20231219075538.414708-1-peterx@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspamd-Queue-Id: 5C0E2180006 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: knbwns4cpjue79keyojg9thf8hcjpzay X-HE-Tag: 1702972643-389740 X-HE-Meta: U2FsdGVkX1/hEhCxYAgZMm8tce7blzhBZ9xhTQBZM5rpiiB+zDV3nA34hc4b+NkBYvA5fbv5Dj/WJxNvLydKIMNHbRcq1IrdY/oNuJ6GDU2clQbLRGL4DzULYiIds+Tj6H/5Vzo0zVHMjvmkzqWnXfQuur0M+OZ7OMJO45pVgXOevsn5iSslFmi5sDJ3xzV22zgGF7rFOj0dMExoI+7DnzF68iB+AHwqBLehQk4WSYjUs40yGTCbjA6eGQC5tVi2K/FGqaP6vbTokCyWcYA10ao+0yOhVlsYM0GZtwI0vvBJlmRKy8fmEbP3/YMrrHRByyJv4eX69WOSaq6MAfzLtjLzpyRt1p1Ru1hRimHAvmFSCkmkrWlsRd4VY7L6j0Zrw6jCxyX4qnWvTyqvF498/A5wiBcREEWeTcWOI0noB4SwWY5rmic9LDCnAUYyg5MJ/FYFwGXdgBd//0kU8ah9xYS8JZaYP1wT7c41q8w76fNujaxA58F6a+PcDxeHA6Cw1dU6vpFm15xi4Fjfj3Uf9aDIRSZJcC2dxGnwvXsnbIen1aYZ4AxxmVJte/tyF+j4adEpesQW/MGDbETPbxZWZ1GRRyapSdMipHGjjLUAe//zYMiSesIHq7r7rih4lycdBs1vkKcC3FPscGbI98AkOfL1w9Cke14/Tp/TvnNCRzpz8zs7XVAPeVpGL19QU0ddfVYD9VRG8MmaKYIXWTvQ005hgpGOlVmx2cKZpyTMC0X2HMN2D8CQH9hqqrTOLQHUJTCCs5RPM7evkTc0nZv8R4+j3fD2oC9iFU8G99zm719xjgHo6BRknxWcW8AzMalckQ2gBPXliN558uBaIZ9yZJ/XRMsuviqOoPTVP0N1UXmxopBaQ6PqBr64JN7NUINEJwFYVtoRj5IgHmBLmSlkA/sRqD75MbFHitINrFLfnHPqyzKvAvOt+i3fPlFoU3YEK0tCRWUscx5gqoqJ+lA O9oZN5tn QM/djwTRQv3zlntyUo5OvCAbtN6hCK1mwDHZfYE1G9NeiCf8p8w0cjYp1uLPA0YViSfX+6wlmNCxG8r+ghWABsUSCnvPU2+cq4uTLG25nGi+KvgK+5vN7SKRbIpy+i+HzUemCVIb6DeNRuw8im44REekbvpOIV4giFkdtX58+7uXZP2NvWoDH1luh8wmz3kkdBNZ+tNyEMr/GUyrZglsGET3rRQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peter Xu All the fast-gup functions take a tail page to operate, always need to do page mask calculations before feeding that into record_subpages(). Merge that logic into record_subpages(), so that it will do the nth_page() calculation. Signed-off-by: Peter Xu --- mm/gup.c | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index bb5b7134f10b..82d28d517d0d 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2767,13 +2767,16 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, } #endif -static int record_subpages(struct page *page, unsigned long addr, - unsigned long end, struct page **pages) +static int record_subpages(struct page *page, unsigned long sz, + unsigned long addr, unsigned long end, + struct page **pages) { + struct page *start_page; int nr; + start_page = nth_page(page, (addr & (sz - 1)) >> PAGE_SHIFT); for (nr = 0; addr != end; nr++, addr += PAGE_SIZE) - pages[nr] = nth_page(page, nr); + pages[nr] = nth_page(start_page, nr); return nr; } @@ -2808,8 +2811,8 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, /* hugepages are never "special" */ VM_BUG_ON(!pfn_valid(pte_pfn(pte))); - page = nth_page(pte_page(pte), (addr & (sz - 1)) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pte_page(pte); + refs = record_subpages(page, sz, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2882,8 +2885,8 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, pages, nr); } - page = nth_page(pmd_page(orig), (addr & ~PMD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pmd_page(orig); + refs = record_subpages(page, PMD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2926,8 +2929,8 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, pages, nr); } - page = nth_page(pud_page(orig), (addr & ~PUD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pud_page(orig); + refs = record_subpages(page, PUD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2966,8 +2969,8 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, BUILD_BUG_ON(pgd_devmap(orig)); - page = nth_page(pgd_page(orig), (addr & ~PGDIR_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pgd_page(orig); + refs = record_subpages(page, PGDIR_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio)