From patchwork Sat Feb 18 00:27:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13145377 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13148C05027 for ; Sat, 18 Feb 2023 00:29:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 803F9280007; Fri, 17 Feb 2023 19:28:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7DB57280002; Fri, 17 Feb 2023 19:28:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6A310280007; Fri, 17 Feb 2023 19:28:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5D02A280002 for ; Fri, 17 Feb 2023 19:28:57 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2EA0CA019B for ; Sat, 18 Feb 2023 00:28:57 +0000 (UTC) X-FDA: 80478527514.06.06E8998 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf23.hostedemail.com (Postfix) with ESMTP id 727F2140009 for ; Sat, 18 Feb 2023 00:28:55 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=V1UHwis9; spf=pass (imf23.hostedemail.com: domain of 3xhvwYwoKCOMOYMTZLMYTSLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3xhvwYwoKCOMOYMTZLMYTSLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676680135; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Pz2nYroaVRrGSVy94qGG4nZ9E30MSqR5I0rmAzuTvq0=; b=t3FTpAlD4m7nwRSVdS8fPbRzKfVJdAACiLh86wQSdCKsr8E04W5DWm0qwoxugi6l2RKfi4 oL+C86i/YXXBmV+5RdhPL+7XZztxYo3FC1hfWYenWJ4Mw7VFZ7WXxGtrgVwoDqksbZgCOm MQW9qX4LjaFbpnmjLvBzqJmMrFL2LWA= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=V1UHwis9; spf=pass (imf23.hostedemail.com: domain of 3xhvwYwoKCOMOYMTZLMYTSLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3xhvwYwoKCOMOYMTZLMYTSLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676680135; a=rsa-sha256; cv=none; b=H6EPd+w0GC5lxjkbnFF5AMSGpDNnd8YiLTgRGK0SD6HlPWH0WfVW3XzB73Qo1tGk7CLybh NIwByt+maSqGThFgj1CuCwQBJsOxLcnoS07VKjVPzGfgyBCbuaj3AsiTcNFezZGLOXWmqB mXkPVlCaTh278xSoR7l88NSH2PVi88Y= Received: by mail-yb1-f202.google.com with SMTP id 75-20020a250b4e000000b0090f2c84a6a4so1997283ybl.13 for ; Fri, 17 Feb 2023 16:28:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Pz2nYroaVRrGSVy94qGG4nZ9E30MSqR5I0rmAzuTvq0=; b=V1UHwis97dacsHy0l1b3lzUp4aEffElgkQqFRGAP4R9Hw6zzh9pL3pU8TQ6LeCkOOO LAnGR0E6fLjGxKpr6E0yb8yofQJ3Yplq8n0cLT/bshgkpsM/hMNaBPY6WVg4rCJqcfel PmhHCvaQjRLpj3PZA4l3c17dZhuSlGVknfSXIUsszWjHRSH2g8HHkboEveCGaLYpHpYI huKFnLPcITV45hT7FdZX9JtRZdP0XFlN1WbgnsZXqSux2N7F1a5KMCJ3UlJJdxC6K6rW 3mwyAccoa5IIH5iiNCO6YwhAkDP3JU0L27dI4gD0cC8hs7RdC32Vnk5NUwnl613BgcAj bg0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Pz2nYroaVRrGSVy94qGG4nZ9E30MSqR5I0rmAzuTvq0=; b=ULwLdJAPgoxJSHeq/wMFHEB7w7PZrfDUaIK+CsZxOrzD5nbbzO05OpX4txK3nzWMG1 gC8xPuk1H1bpp9Y+eCSrzNmWppZ178kvZGvMKRoIazyiypftV7LRZAqi2Rq5IuUJBagd lkX7ydw23a2Kui4N84FOuw7qocv9Vkl23B0VNV5nAIPTxk/koP09rsY8wbiFUh8HGk5g WlQ+Kl66h9NWF+5HyrQT8PJgT/MxJDw5f+xWTNVETvkHbDYxwsfAPwuMko0qGhD/gzC7 TOgyM6qytlCFbpl4hoZd3CvBMel5uMZzGTZ7BHcxLhwLmsfHHxfvyUpH+otFjBGThHXH 6/6g== X-Gm-Message-State: AO0yUKWn8c2vW1LgaBDXy8Q0bdJazmXZs93VJjdwYRnESeLb1bcQyfrM U4hOB5hZLb7jdipIWJ6Y9V+Z5rlduM/dRJbx X-Google-Smtp-Source: AK7set+zcld/vdofwpUCcvIXOcKa++WCtmqUm8NMGDjM4RjhBKN3Re9Z99sY+8b3ruIVEFKeHpX84TBdzsYueQO8 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6902:10c:b0:997:c919:4484 with SMTP id o12-20020a056902010c00b00997c9194484mr28393ybh.6.1676680134689; Fri, 17 Feb 2023 16:28:54 -0800 (PST) Date: Sat, 18 Feb 2023 00:27:44 +0000 In-Reply-To: <20230218002819.1486479-1-jthoughton@google.com> Mime-Version: 1.0 References: <20230218002819.1486479-1-jthoughton@google.com> X-Mailer: git-send-email 2.39.2.637.g21b0678d19-goog Message-ID: <20230218002819.1486479-12-jthoughton@google.com> Subject: [PATCH v2 11/46] hugetlb: add hugetlb_pte to track HugeTLB page table entries From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu , Andrew Morton Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , "Zach O'Keefe" , Manish Mishra , Naoya Horiguchi , "Dr . David Alan Gilbert" , "Matthew Wilcox (Oracle)" , Vlastimil Babka , Baolin Wang , Miaohe Lin , Yang Shi , Frank van der Linden , Jiaqi Yan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 727F2140009 X-Stat-Signature: dua91cuzwk6kbw4akt1sqkny5ee8fobe X-HE-Tag: 1676680135-920856 X-HE-Meta: U2FsdGVkX19y3u47WO0cQ3uI/004dTyXMgnWePvoemDGK29N5T90whP2nwP+ngvJE+jNElTv8P78aYh/XzGovgoN79EOA7TC4LpSx9xNR806WC7LSbAcasdOAbdGo78MU3Yk/mrCjZzX/buNe3JKt7Ya0XBEbn7cMKjiNOUE/BN5i97Zt9z13YbniO1wNkNzTLocYNq+/H7Os13idIPXKWahwGLoK3AI0+2pzeTYn+qU0EubtHSORIP9BLhWa5hoS3eIjNvWF2ITxCxPjNi86fP9z3E2AJOJ4RKunNqhHjRjaqiacinS+YwZRLyASiT5Bo9+DlulZSMkGv2ZxchomgIKxTuocMQKriMCRY+jDp6CK3vGJItDhnfYS2X01sW+WPEDpISQnWvvOGNayh2Sc20I5Tb/kQP7vX1l4peVUJekcbqO29xFir93YRoPV0vaDSbg4LB4bentdH7YCu1Mj7wPMT4PCHkg/M/RInap0UYohO2PLU0YRZEfL1Jzs5BtG468ZpQNX4bPlpNmrfsw5cZ6+6PzobOxerIPoHNypSzmi4sfjCZaxYWUtYKOvD3PS95gj7rRz31ancc6LrzoY6/BVet6JnXUnseeIfV7Yis1e2CodhUJqZsJcn2EOtVwt57llbDuaZcuthmIJYMwlfndyR5VprIF+sFW/FZJKe9rdcUALV606/SGENekJfqshm1IX1u8d1TcIJTzmB3rfvzBiWvaR/i0ZWNMPet/kDmn5vUJGHXFfJsR8acUFsUua57xmm4JmSHijB/cdfCOLxDiXGBVMxCU5uZM2UQ+X+Z/dwlrdDCYzXU1z9PhblT43of1pTAFwIj9JjPcNUxgADKz9kNjKi1dzi7b1zkG+NATT0j58+ABg02TOwudoGtZKiTK15lwQZT6tR86FvWsTvhcY4igfsisbs02n6ubvumLvrNhZy0V88UucWvySW/I9hdGD763re9iG4N0LG0 ftBggRxK dz+AVnab2PoOAuUJJmeWNww4nhDbsx/MSXfxy6rFiIJUKwKlfGTI0gi25v/JMwgUXhoreJWgiMkfK0x2/xxMYlTekEKpH3lXT50cs/WHJskELixydQlHh9tCO1n2IOvO4M571YMnVxEASkFuv4+rKqbA4nzK+wo06gAa1sm31o62nIuPiytyfNpSkYHeDuuvPAr23fmUp74fDBj4qaChjgqPdQylmRNSVjxjQTVhH/mFrSoiQOAQFX/zN5KLgxyLUGVZScfjDrSzzSlygJ0zhIKiVklvdfOIURSjL63Twr3YD9ln2l23pm+zsqfdK7p2veOXP7Lp8uVXV9xwPBojuuf5vV8trDCxEaX0Pqm9+E0/VMP/xFROFmTMwJGBYge5RnUJV6l1VhoubO2oS5o7EQG1Mf0f8Uuf2LctCtH+jDOSu5eF2C5nUeP/RNAXu8K7YBFxzMTfapqczC+qFgzxuLVQhwhEqmfcRG0F+4iWvaBw0kqxWz3e0fxIptUlF82rl3GOdC/sh44pzUNAhEfNch5A0+8O5RxcRlqtkAaTvxYxln56GkgrOx1hhY681npcspBJeUxHpf42T/e5OM+Q76NTMFbCv/UJfzCOAWFpXm/nGSgY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: After high-granularity mapping, page table entries for HugeTLB pages can be of any size/type. (For example, we can have a 1G page mapped with a mix of PMDs and PTEs.) This struct is to help keep track of a HugeTLB PTE after we have done a page table walk. Without this, we'd have to pass around the "size" of the PTE everywhere. We effectively did this before; it could be fetched from the hstate, which we pass around pretty much everywhere. hugetlb_pte_present_leaf is included here as a helper function that will be used frequently later on. Signed-off-by: James Houghton Reviewed-by: Mina Almasry Acked-by: Mike Kravetz diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index a1ceb9417f01..eeacadf3272b 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -26,6 +26,25 @@ typedef struct { unsigned long pd; } hugepd_t; #define __hugepd(x) ((hugepd_t) { (x) }) #endif +enum hugetlb_level { + HUGETLB_LEVEL_PTE = 1, + /* + * We always include PMD, PUD, and P4D in this enum definition so that, + * when logged as an integer, we can easily tell which level it is. + */ + HUGETLB_LEVEL_PMD, + HUGETLB_LEVEL_PUD, + HUGETLB_LEVEL_P4D, + HUGETLB_LEVEL_PGD, +}; + +struct hugetlb_pte { + pte_t *ptep; + unsigned int shift; + enum hugetlb_level level; + spinlock_t *ptl; +}; + #ifdef CONFIG_HUGETLB_PAGE #include @@ -39,6 +58,20 @@ typedef struct { unsigned long pd; } hugepd_t; */ #define __NR_USED_SUBPAGE 3 +static inline +unsigned long hugetlb_pte_size(const struct hugetlb_pte *hpte) +{ + return 1UL << hpte->shift; +} + +static inline +unsigned long hugetlb_pte_mask(const struct hugetlb_pte *hpte) +{ + return ~(hugetlb_pte_size(hpte) - 1); +} + +bool hugetlb_pte_present_leaf(const struct hugetlb_pte *hpte, pte_t pte); + struct hugepage_subpool { spinlock_t lock; long count; @@ -1234,6 +1267,45 @@ static inline spinlock_t *huge_pte_lock(struct hstate *h, return ptl; } +static inline +spinlock_t *hugetlb_pte_lockptr(struct hugetlb_pte *hpte) +{ + return hpte->ptl; +} + +static inline +spinlock_t *hugetlb_pte_lock(struct hugetlb_pte *hpte) +{ + spinlock_t *ptl = hugetlb_pte_lockptr(hpte); + + spin_lock(ptl); + return ptl; +} + +static inline +void __hugetlb_pte_init(struct hugetlb_pte *hpte, pte_t *ptep, + unsigned int shift, enum hugetlb_level level, + spinlock_t *ptl) +{ + /* + * If 'shift' indicates that this PTE is contiguous, then @ptep must + * be the first pte of the contiguous bunch. + */ + hpte->ptl = ptl; + hpte->ptep = ptep; + hpte->shift = shift; + hpte->level = level; +} + +static inline +void hugetlb_pte_init(struct mm_struct *mm, struct hugetlb_pte *hpte, + pte_t *ptep, unsigned int shift, + enum hugetlb_level level) +{ + __hugetlb_pte_init(hpte, ptep, shift, level, + huge_pte_lockptr(shift, mm, ptep)); +} + #if defined(CONFIG_HUGETLB_PAGE) && defined(CONFIG_CMA) extern void __init hugetlb_cma_reserve(int order); #else diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 5ca9eae0ac42..6c74adff43b6 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1269,6 +1269,35 @@ static bool vma_has_reserves(struct vm_area_struct *vma, long chg) return false; } +bool hugetlb_pte_present_leaf(const struct hugetlb_pte *hpte, pte_t pte) +{ + pgd_t pgd; + p4d_t p4d; + pud_t pud; + pmd_t pmd; + + switch (hpte->level) { + case HUGETLB_LEVEL_PGD: + pgd = __pgd(pte_val(pte)); + return pgd_present(pgd) && pgd_leaf(pgd); + case HUGETLB_LEVEL_P4D: + p4d = __p4d(pte_val(pte)); + return p4d_present(p4d) && p4d_leaf(p4d); + case HUGETLB_LEVEL_PUD: + pud = __pud(pte_val(pte)); + return pud_present(pud) && pud_leaf(pud); + case HUGETLB_LEVEL_PMD: + pmd = __pmd(pte_val(pte)); + return pmd_present(pmd) && pmd_leaf(pmd); + case HUGETLB_LEVEL_PTE: + return pte_present(pte); + default: + WARN_ON_ONCE(1); + return false; + } +} + + static void enqueue_hugetlb_folio(struct hstate *h, struct folio *folio) { int nid = folio_nid(folio);