From patchwork Wed Mar 27 15:23:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13606791 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCC6DC47DD9 for ; Wed, 27 Mar 2024 15:24:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 737296B00A1; Wed, 27 Mar 2024 11:24:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6BD676B00A3; Wed, 27 Mar 2024 11:24:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4EA106B00A4; Wed, 27 Mar 2024 11:24:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2BC9B6B00A1 for ; Wed, 27 Mar 2024 11:24:02 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E562E80A6B for ; Wed, 27 Mar 2024 15:24:01 +0000 (UTC) X-FDA: 81943189482.16.AB3DF6A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 00413140020 for ; Wed, 27 Mar 2024 15:23:59 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BSGJl9wx; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711553040; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ldvmB2LhfsIRddCCGOUltuspZLyBYJ6Ab85tSc3WzTs=; b=tS49UKCYQRUkiW+CougM+/wzc3Fex+S5iyL668F1lJ8OKMJbKPMd185QU/h7f6A81oq9PN IzuLLeJyNH23ZY+Oewuo7VJril/QjwmeQ1XQqjZxszbl60B/i6T2p7FVdbXqjmM9EysP+0 GzZ5C0yzbYEw1xJoOERHJnfU6NTrs84= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BSGJl9wx; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711553040; a=rsa-sha256; cv=none; b=VtkgEyeQw/K5JC/GxJbT3o53MvyHeEVYXsET0QZ/SjBGJLVp+8A3m3csTro9n78nO3qEEH WZJaue7aUcaLbm/VZZp8NSg2U28ATMyx6V4olmDxAmZI1/lq5F1gSxKztXLRQHsvrtVwLc C75xp53GQ8WwvPRunI5VTZ4bPnECP44= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711553039; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ldvmB2LhfsIRddCCGOUltuspZLyBYJ6Ab85tSc3WzTs=; b=BSGJl9wxcjjAjeAVZsrZtKz/inf8oY4eNgmD7SzvW9j8nX1r6qONCLUACsEcD9j+p6vZx5 bJf3lXQ1oePSHc+cLgmQW+EBgjyHATvgXsiQO2kQvT0cwBCVyiWVGXTarhhLhQ+bpSKDBA hZWmAnR3ci3QXQuYPOW46iUQhgBcBsM= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-266-ltTAhUhMP7u23L-pAkafqw-1; Wed, 27 Mar 2024 11:23:58 -0400 X-MC-Unique: ltTAhUhMP7u23L-pAkafqw-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-690ab2b0de1so15530976d6.0 for ; Wed, 27 Mar 2024 08:23:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711553037; x=1712157837; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ldvmB2LhfsIRddCCGOUltuspZLyBYJ6Ab85tSc3WzTs=; b=EgYNK9HcVmJk4Ov8ZuimIn/qsTAuA9NB3cjMMikYmic+3N6ujMAAgssjgxe7lmC1cM nghVN3tWJN+ykbwZMcOrYlgeOkWAW+56kOFhVT+gjAmazJuwrVbwvJQZOHS7Pxs3Rwkh PI4zMuEmDzrKrEbK7mKTKn0mF/hxXtPNVgrRjF8hr8J0LN5q8L92pCZqtsjA5pjIjwsi MjQ/L/u7gqtMZ8N2bsdda5mhHOSi3j5Z46x3Vq5xK5roeU0i9WnxRqyo1r05G7D2RRnw eTUHaLBB9UzPE8u+64dq1072lsuvDLt3sgxVMcdi8XAQHKEGV9gMbitf4C/adgMPJim1 sZ7w== X-Gm-Message-State: AOJu0YydNZnvsS+ySriug4w4p6XyRe8iW5cVU9aNmjsD0UJYmGxqGAAu iDoK8PRJ7hVm5KZTGfj0XJODz0Rjh88ZFOu7FQ5wYeMQqIl2LBiGpO1z2KdZqrE6fZChwFcm85k kXKlCO4MB8JXNr4MTklH8Uv0bdC7hpQjVF4lio6uP72Vhj4U2J6fXE6ukLJQFCLKhZt4BD0pFc4 Y4AZzZlsELTPkNL0niaCt6VCw9QWCOcA== X-Received: by 2002:a05:6214:3d8c:b0:696:6f95:4421 with SMTP id om12-20020a0562143d8c00b006966f954421mr14386007qvb.1.1711553037086; Wed, 27 Mar 2024 08:23:57 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEnSeTu/B6EMntVMOCEyhcTmSAZJByvhuxAj5BPZo0z5woD7z6vaVN6Kd8/ZBw0a5bsOP9d6g== X-Received: by 2002:a05:6214:3d8c:b0:696:6f95:4421 with SMTP id om12-20020a0562143d8c00b006966f954421mr14385952qvb.1.1711553036462; Wed, 27 Mar 2024 08:23:56 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id hu4-20020a056214234400b00690dd47a41csm6412639qvb.86.2024.03.27.08.23.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Mar 2024 08:23:56 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Yang Shi , "Kirill A . Shutemov" , Mike Kravetz , John Hubbard , Michael Ellerman , peterx@redhat.com, Andrew Jones , Muchun Song , linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Andrew Morton , Christoph Hellwig , Lorenzo Stoakes , Matthew Wilcox , Rik van Riel , linux-arm-kernel@lists.infradead.org, Andrea Arcangeli , David Hildenbrand , "Aneesh Kumar K . V" , Vlastimil Babka , James Houghton , Jason Gunthorpe , Mike Rapoport , Axel Rasmussen Subject: [PATCH v4 10/13] mm/gup: Handle huge pud for follow_pud_mask() Date: Wed, 27 Mar 2024 11:23:29 -0400 Message-ID: <20240327152332.950956-11-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240327152332.950956-1-peterx@redhat.com> References: <20240327152332.950956-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 00413140020 X-Rspam-User: X-Stat-Signature: aih5dknr3tca6bti9m6ctd78cuj1fxbj X-Rspamd-Server: rspam01 X-HE-Tag: 1711553039-323136 X-HE-Meta: U2FsdGVkX18ErmHLJw0pB4lUVycyfkZ7AvA8Ln4CfrGcquOrFkjksU9cHkPlmegdqON6FLnMnf/wtZzC+n5TcE9nR/EjKn+TOivBcvFZfVdJFUXwxOY3qjRnAjbMI6y/B011zHO5mKLTv7ghthAWr3LoE71J+qCnqvxe4zomv/dXyHMmeKvvXAJwOpLJJkxzy2x/+DkpyZitcEeF3QDm/7+O3V/vOLb14+kuAhO1VS10/Gz/Z4BuK1x/nk5QKKbQ30FJGqy8Z7MQoNghxeD3czIki53A6WS31DSdKqoWWuDJsdhEr9DeXrBVkuyJv0GOClaTc/8pbsJRnTY6ax2twRgTTVX5lO0TmUoXahQ3sqIKbDHdkoPmqrRCtk6DP8r4PHhQfYJVl6V8ZbElApjoUUHsJS0p4CdoxbqZQmrDGc4IiziELmBPc+nejhIVxEEO4bShRJv+uCxND8V1CutPVpVrfDYP5qfR/LvmiIdBqXFszdDSF+kIniZvKw9TgBuOSnAcjLkQcteusOsFiwd2UCCul31/s7SzZ9FrHPfPr5dMvIW82r4QR8XiG7lbo0cMWaczt1HzD7yu5FINy+AFH1osycFCKUM8XJdjrDAcKNV+ZAlK+ilITksES0hkH5YEsvfCQ+pTArGeFySfZtxoaRrqkAnRaE3N2Np/HJ8nbrqmOORhpKhfuGbYcLVrcD4NnDhjgW6KUdBFIIyiqlDXYnVtFZSFA1RMtbR+WfZX/HLCL/KZYaxVRi6FKKRDzctTSC1hz4UMedT/+/EHR9/xvcVxv3CkRfRNGVPooIIa03yQG5a6Mh5LDNlkx30cprFfSMiGoCAPpJCXImPoH16I/tx5SSqOH+/JjmUZM1RG9FSwSZxcrdTEIhpMsY3s/GiuTDfy0fznTXX+mY+foujlmqeTkvkc1a46NfwVlgwSQEM42PgMWSxLe5ysGYqYqpSNun3OVGqkP1WJ8mRzJ8T GmAKKaNe oaPPRq7Vm4av65M+XJMRxOGyZYdB66KMsSPqdN2xU9yGxjjAYzYSrCxDGbCeGBItI/Oabe9wzyQgxr/qL6sp8vIzSOSDhk7WVLTixzU1ps2iUXDlMht2kOEygj2mR6ZLSfoNY1zJ6OXb/2Io= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peter Xu Teach follow_pud_mask() to be able to handle normal PUD pages like hugetlb. Rename follow_devmap_pud() to follow_huge_pud() so that it can process either huge devmap or hugetlb. Move it out of TRANSPARENT_HUGEPAGE_PUD and and huge_memory.c (which relies on CONFIG_THP). Switch to pud_leaf() to detect both cases in the slow gup. In the new follow_huge_pud(), taking care of possible CoR for hugetlb if necessary. touch_pud() needs to be moved out of huge_memory.c to be accessable from gup.c even if !THP. Since at it, optimize the non-present check by adding a pud_present() early check before taking the pgtable lock, failing the follow_page() early if PUD is not present: that is required by both devmap or hugetlb. Use pud_huge() to also cover the pud_devmap() case. One more trivial thing to mention is, introduce "pud_t pud" in the code paths along the way, so the code doesn't dereference *pudp multiple time. Not only because that looks less straightforward, but also because if the dereference really happened, it's not clear whether there can be race to see different *pudp values when it's being modified at the same time. Setting ctx->page_mask properly for a PUD entry. As a side effect, this patch should also be able to optimize devmap GUP on PUD to be able to jump over the whole PUD range, but not yet verified. Hugetlb already can do so prior to this patch. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- include/linux/huge_mm.h | 8 ----- mm/gup.c | 70 +++++++++++++++++++++++++++++++++++++++-- mm/huge_memory.c | 47 ++------------------------- mm/internal.h | 2 ++ 4 files changed, 71 insertions(+), 56 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index d3bb25c39482..3f36511bdc02 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -351,8 +351,6 @@ static inline bool folio_test_pmd_mappable(struct folio *folio) struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, int flags, struct dev_pagemap **pgmap); -struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, - pud_t *pud, int flags, struct dev_pagemap **pgmap); vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf); @@ -507,12 +505,6 @@ static inline struct page *follow_devmap_pmd(struct vm_area_struct *vma, return NULL; } -static inline struct page *follow_devmap_pud(struct vm_area_struct *vma, - unsigned long addr, pud_t *pud, int flags, struct dev_pagemap **pgmap) -{ - return NULL; -} - static inline bool thp_migration_supported(void) { return false; diff --git a/mm/gup.c b/mm/gup.c index 26b8cca24077..1e5d42211bb4 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -525,6 +525,70 @@ static struct page *no_page_table(struct vm_area_struct *vma, return NULL; } +#ifdef CONFIG_PGTABLE_HAS_HUGE_LEAVES +static struct page *follow_huge_pud(struct vm_area_struct *vma, + unsigned long addr, pud_t *pudp, + int flags, struct follow_page_context *ctx) +{ + struct mm_struct *mm = vma->vm_mm; + struct page *page; + pud_t pud = *pudp; + unsigned long pfn = pud_pfn(pud); + int ret; + + assert_spin_locked(pud_lockptr(mm, pudp)); + + if ((flags & FOLL_WRITE) && !pud_write(pud)) + return NULL; + + if (!pud_present(pud)) + return NULL; + + pfn += (addr & ~PUD_MASK) >> PAGE_SHIFT; + + if (IS_ENABLED(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) && + pud_devmap(pud)) { + /* + * device mapped pages can only be returned if the caller + * will manage the page reference count. + * + * At least one of FOLL_GET | FOLL_PIN must be set, so + * assert that here: + */ + if (!(flags & (FOLL_GET | FOLL_PIN))) + return ERR_PTR(-EEXIST); + + if (flags & FOLL_TOUCH) + touch_pud(vma, addr, pudp, flags & FOLL_WRITE); + + ctx->pgmap = get_dev_pagemap(pfn, ctx->pgmap); + if (!ctx->pgmap) + return ERR_PTR(-EFAULT); + } + + page = pfn_to_page(pfn); + + if (!pud_devmap(pud) && !pud_write(pud) && + gup_must_unshare(vma, flags, page)) + return ERR_PTR(-EMLINK); + + ret = try_grab_page(page, flags); + if (ret) + page = ERR_PTR(ret); + else + ctx->page_mask = HPAGE_PUD_NR - 1; + + return page; +} +#else /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */ +static struct page *follow_huge_pud(struct vm_area_struct *vma, + unsigned long addr, pud_t *pudp, + int flags, struct follow_page_context *ctx) +{ + return NULL; +} +#endif /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */ + static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address, pte_t *pte, unsigned int flags) { @@ -760,11 +824,11 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, pudp = pud_offset(p4dp, address); pud = READ_ONCE(*pudp); - if (pud_none(pud)) + if (!pud_present(pud)) return no_page_table(vma, flags, address); - if (pud_devmap(pud)) { + if (pud_leaf(pud)) { ptl = pud_lock(mm, pudp); - page = follow_devmap_pud(vma, address, pudp, flags, &ctx->pgmap); + page = follow_huge_pud(vma, address, pudp, flags, ctx); spin_unlock(ptl); if (page) return page; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index bc6fa82d9815..2979198d7b71 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1377,8 +1377,8 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, } #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD -static void touch_pud(struct vm_area_struct *vma, unsigned long addr, - pud_t *pud, bool write) +void touch_pud(struct vm_area_struct *vma, unsigned long addr, + pud_t *pud, bool write) { pud_t _pud; @@ -1390,49 +1390,6 @@ static void touch_pud(struct vm_area_struct *vma, unsigned long addr, update_mmu_cache_pud(vma, addr, pud); } -struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, - pud_t *pud, int flags, struct dev_pagemap **pgmap) -{ - unsigned long pfn = pud_pfn(*pud); - struct mm_struct *mm = vma->vm_mm; - struct page *page; - int ret; - - assert_spin_locked(pud_lockptr(mm, pud)); - - if (flags & FOLL_WRITE && !pud_write(*pud)) - return NULL; - - if (pud_present(*pud) && pud_devmap(*pud)) - /* pass */; - else - return NULL; - - if (flags & FOLL_TOUCH) - touch_pud(vma, addr, pud, flags & FOLL_WRITE); - - /* - * device mapped pages can only be returned if the - * caller will manage the page reference count. - * - * At least one of FOLL_GET | FOLL_PIN must be set, so assert that here: - */ - if (!(flags & (FOLL_GET | FOLL_PIN))) - return ERR_PTR(-EEXIST); - - pfn += (addr & ~PUD_MASK) >> PAGE_SHIFT; - *pgmap = get_dev_pagemap(pfn, *pgmap); - if (!*pgmap) - return ERR_PTR(-EFAULT); - page = pfn_to_page(pfn); - - ret = try_grab_page(page, flags); - if (ret) - page = ERR_PTR(ret); - - return page; -} - int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, pud_t *dst_pud, pud_t *src_pud, unsigned long addr, struct vm_area_struct *vma) diff --git a/mm/internal.h b/mm/internal.h index 6c8d3844b6a3..eee8c82740b5 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1111,6 +1111,8 @@ int __must_check try_grab_page(struct page *page, unsigned int flags); /* * mm/huge_memory.c */ +void touch_pud(struct vm_area_struct *vma, unsigned long addr, + pud_t *pud, bool write); struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, unsigned int flags);