From patchwork Thu Nov 16 01:29:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13457485 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0754C072A2 for ; Thu, 16 Nov 2023 01:29:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 998FB6B03F0; Wed, 15 Nov 2023 20:29:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9215A6B03F1; Wed, 15 Nov 2023 20:29:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 778036B03F2; Wed, 15 Nov 2023 20:29:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 622B46B03F0 for ; Wed, 15 Nov 2023 20:29:33 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 25F04C01A6 for ; Thu, 16 Nov 2023 01:29:33 +0000 (UTC) X-FDA: 81462085026.06.642552D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 1A18A14000F for ; Thu, 16 Nov 2023 01:29:28 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AaZXdQAj; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700098171; a=rsa-sha256; cv=none; b=WlAhYWB3NDAM6T4eGR7FyStPpQdhDoUBTbJwL46d0dkBQElVSzAYlDLB9z8vAi61EjyGAD QQoQfdPU1BCHmsLocJTv7eKzaPCUC076g+ouVhM9YuMAoWVddcaHfP6tlXjlV7BNSiz8/8 ShwwFOfOHPExWfnUhzW5Tw3mzN/+l14= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AaZXdQAj; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700098171; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/m4Nmkq0cr3qM8eU/2c8RM1XZmbD+yMVfcEDigP2nYM=; b=dchGU9j0wwI46E4UX2SxCM5pXWDeuEiYVFjJADAoPPGzonc0Di7uKnu9PxjnJdocN5wXCn +FXsIidWyvPx9cP3QSrH4qeBkrLLHMHjZ24HQRPK/dKBjo67Nm819HmNOgwAUlljWapWdc 3f63QGZE6+Bf9gXxL1sz4E9CWM62KyU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098167; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/m4Nmkq0cr3qM8eU/2c8RM1XZmbD+yMVfcEDigP2nYM=; b=AaZXdQAjUiZsWmiDJsnE2gWSIvItsTcUG0/jF35lGa55OU8f6z0ukCzb1K4ECWvQ9o3NRo nRZQfgVhl5Bu7PAr/U0zfNUkeV6+U0hnDqCMAwi4OokNOfEaa1uIbLy1cO9cFxRXWTtSND OvWJhy9Ld8btIKfVNugE+jQa81ZuM04= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-330-egE_FXqrPhW-05S524YTPQ-1; Wed, 15 Nov 2023 20:29:24 -0500 X-MC-Unique: egE_FXqrPhW-05S524YTPQ-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-420c7034fddso915871cf.0 for ; Wed, 15 Nov 2023 17:29:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098163; x=1700702963; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/m4Nmkq0cr3qM8eU/2c8RM1XZmbD+yMVfcEDigP2nYM=; b=qOWbLe66429WIejW0KaK8Y0UTCKln3vBQDaM8XgICRUgbJI6f7o7Y4ktnVDbzMsUYb 43wo7x2dzlQH6PlncWdi7/0bXRWzLq9Bf+9t9DQKBAE5LKcW1urxbMAwpd8OAcD6EAbn vr89oQVAd7BIF+b7aHaGWKBClNLIselTY9DssoIJyoD4lborj8ODnNphsiu9sI8hIJEj NDTbjoUYP46CKMIEAyzf89SFGmld9yHNp4Xy/VJz9YNVhULVGBC+v209d4K6TbEY/YHx TK2EAv4JS1Yw4YPopSKk0UH7ySgOCYw3+oidRR27XIKqifytkhCo96M8UfxInAwaWHvM RECA== X-Gm-Message-State: AOJu0Yww7pzg1+5u8eFYDZWB9U6RSc7s9o8zLH7x+nEWvzIAPe1617iv +MXmHKZBV4OAW8nyrNVeGzljTUOCuZhXTuy2HpHheXghcP/56GI277lqzhngqD3eH1zyXm2yROM 202tvtQS0M2M= X-Received: by 2002:ac8:1496:0:b0:421:9fe5:6dc1 with SMTP id l22-20020ac81496000000b004219fe56dc1mr6484666qtj.3.1700098162895; Wed, 15 Nov 2023 17:29:22 -0800 (PST) X-Google-Smtp-Source: AGHT+IFEPgMVSzfIxtQlfIF+wS0OHZ+NUQWxFmPrijPw2JerykvyjqCin/P2xZqUrm2jMmgcDcZfrw== X-Received: by 2002:ac8:1496:0:b0:421:9fe5:6dc1 with SMTP id l22-20020ac81496000000b004219fe56dc1mr6484654qtj.3.1700098162561; Wed, 15 Nov 2023 17:29:22 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:22 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 07/12] mm/gup: Refactor record_subpages() to find 1st small page Date: Wed, 15 Nov 2023 20:29:03 -0500 Message-ID: <20231116012908.392077-8-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 1A18A14000F X-Stat-Signature: srnwxk3w9uzjci3ii96kkoye9hxeohcs X-HE-Tag: 1700098168-898148 X-HE-Meta: U2FsdGVkX19uA7riHIn5atXgS1sVbh9gi4pIV2WC4WjeHX5ABLXEAHgH39XtGweP/kIQ59v5SPa/RqBpUrfJCEKlDYqtGst4EWyH8raoOHGNHV3mqEa8hyxGpiWyxQgS7pi6uN4Tm7g2+IbEvI3sM1bNcKKgm0PPaPdOdWCvmMdwBTzx+A9gGqLr1K76cbHuzGYiDo6SGYAFo+YSQ4RQzcxDpFUUJgFIYLsWvOZ8SFtdrDfI1VrtePwH3rmYy66Qcu/dt25K2yiaJhsuaiIIhPQ1YD2a0Jtzm+0f6GwfDqDF/6x9xaXZOL8CZ9jvU21rRv/ssXm78q9lOV2E7Lvo/rQsJx/XEfveFgNtH+q20Lt475HjAqrccIjrXUwWPbOPFbITKY3jaMcvmApBkxBRcNhoZszO7ptStfbxkZJmu4xLbvDg9jXqPEvqADP5IG+ULTqjXL2mnnHlCVu77ji55aMqxwZBEaP+XGU8Mm4qLxCbfcbIoPJ9MNhryvmrAVQR+MYm/hb8uPS4DzJ4E3gkuwW4izqk20cfk8c7LFwB/FZeYRMgvMdljPAw4rhUGKx07w7SufxleN/RSg/NWFx+qMNEVaDfZTtYzYITn3J5iwEwkjUMKrtbSSjQydcS8FxI5TAnN2dByXvU7RF6sS85kNaZSE3mnGgb50v6HMtc6q3kVqDEdXLLtpfqNRHXpyL6Duzxb7OOUZJoXuNzxn2S8PlxGDLgPvkDXVhT6JdAv53ne0m2WNpee/lxKzzZEe5rxOfJEkdYtvBD/qqxQeIoTmoeqlmVgSoEvDXRu/tA7/QvU5lrZh9Cff0/Ygl18+PbeyOmR1Mc+FaYt6B8UuJBPCrJ9UaCCRQwxcC+ZuqmTG6ggNMtpHobakZbGFYAFsTEj/gSOckveVhW8wQQE/mr3csPZ52Btizm5JSFVkUR1dZipVgl/ooBUSUXdOEGZZbdrVlzyYxLUZtD19kYQua LfutFIZW ivPKg6xzO+0kD4OZ/R/dSb4fC0EiiP23zz4JKsv/ZFxN0QjY13auB2hoCnR6KerX0jnb/xvWI3w+Mk7Ul8hancEpOkhWx+V5As+KLbSDnIE6WDpMwY9uc4ChZJQXjj5fvN0EYzdS4t7iWYclsf14Q/NwrxOCq+zWf2dzP5nJ86+19lqcRCq7TQ2Eb46+UZSUth2QZX0TU0y/qAPc6KbYR4xJz6/SzOFy2bR5rZminuyQ25XWRahjUEbKrdWSmyqTiXkaN8oROj05jixNp8QvK2YMOtvEWbvJqToCivgsfpNcBLyJueh8ztIezJKDp7j1F+/bI4gx5oWJT1F7WTwXz6k5H2vFOaVCHQ3XF0mIAdkkWOd0qnUtbjYC8pgijgUs6GOT0wmnZs3jwTIB0hHF+zl8rgFOKbGmcTCBIfRU9ofjaOAczogCl0lsl9A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All the fast-gup functions take a tail page to operate, always need to do page mask calculations before feeding that into record_subpages(). Merge that logic into record_subpages(), so that we always take a head page, and leave the rest calculation to record_subpages(). Signed-off-by: Peter Xu --- mm/gup.c | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 424d45e1afb3..69dae51f3eb1 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2763,11 +2763,14 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, } #endif -static int record_subpages(struct page *page, unsigned long addr, - unsigned long end, struct page **pages) +static int record_subpages(struct page *head, unsigned long sz, + unsigned long addr, unsigned long end, + struct page **pages) { + struct page *page; int nr; + page = nth_page(head, (addr & (sz - 1)) >> PAGE_SHIFT); for (nr = 0; addr != end; nr++, addr += PAGE_SIZE) pages[nr] = nth_page(page, nr); @@ -2804,8 +2807,8 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, /* hugepages are never "special" */ VM_BUG_ON(!pfn_valid(pte_pfn(pte))); - page = nth_page(pte_page(pte), (addr & (sz - 1)) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pte_page(pte); + refs = record_subpages(page, sz, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2870,8 +2873,8 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, pages, nr); } - page = nth_page(pmd_page(orig), (addr & ~PMD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pmd_page(orig); + refs = record_subpages(page, PMD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2914,8 +2917,8 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, pages, nr); } - page = nth_page(pud_page(orig), (addr & ~PUD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pud_page(orig); + refs = record_subpages(page, PUD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2954,8 +2957,8 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, BUILD_BUG_ON(pgd_devmap(orig)); - page = nth_page(pgd_page(orig), (addr & ~PGDIR_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pgd_page(orig); + refs = record_subpages(page, PGDIR_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio)