From patchwork Fri Jun 24 17:36:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894926 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6D1EC43334 for ; Fri, 24 Jun 2022 17:37:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 390028E023F; Fri, 24 Jun 2022 13:37:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F1338E020E; Fri, 24 Jun 2022 13:37:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F5EC8E023F; Fri, 24 Jun 2022 13:37:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id EF9E08E020E for ; Fri, 24 Jun 2022 13:37:12 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id C56B660830 for ; Fri, 24 Jun 2022 17:37:12 +0000 (UTC) X-FDA: 79613835504.26.9CB1415 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf15.hostedemail.com (Postfix) with ESMTP id 5EF87A0022 for ; Fri, 24 Jun 2022 17:37:12 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id s6-20020a255e06000000b00669b21c51b5so2716064ybb.21 for ; Fri, 24 Jun 2022 10:37:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=0YfnJKuQrbw7pcq3GiMHpIZjYaYv7/J4Psozl7GAbkY=; b=rhedzQCKx2wNhVDSGuAJpH9VZYMHroyUpR1McnELZRf5xZDgeN7uNFXQFUQY36oI+V sQBZjS3MbDOeENElKNgbkCSeJdkA4KvbvhCfXewwUZV8VRhUz4h9mZPxYQQ152Wr/FN/ 5BdmdPLWcel0PhshC6zVgGbHZJRq7Q7vjb3TkLcOA4+olBP02aOCGQ9Oozydj+ubNW0a x/51FkRIRk4fHQ5AYf3+Jpxz4mZgUNsIXL+ynW22YI6IcrDX3OU3vcAglWcWj6mxHBI5 z3zP8e+fcp3Sh89Okw6YqgR52njYR4st6QIktXU0C0bgCozw42NZ4PuV/5cZ6iJXZUXt qYDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=0YfnJKuQrbw7pcq3GiMHpIZjYaYv7/J4Psozl7GAbkY=; b=ms/G3CvBGACYk0EQYR7WA2F2A/qW8OWhbv+H/AqVZ22Eb61mOxeWizbHsMsnNxs/yh wEax5QkUuxbU4M90doZhEZrjGl3Et0guaG6TVjK8J6AAik3y88qDmkiYQNTvy8lbMxf9 QqvhPOkYbV2n3y1M+BeH91v0rHxwDVqYnnXdmqGtKV+x4VrXTg5/OkohmjRq+mOyuHUX XF2H+UfIf28c7t4DDZBaA4pM4dofwML8vMa5/1WFoarl3rY0F5bx0xq/KUZfPejaAOuc Jzojov5VvcKmY97Kf53kqmaRsFqiu5qpjpA/ENrcfqIXRfiKEdD2sQIYeKwu3eO7zR1m +HAg== X-Gm-Message-State: AJIora9I20RUD9RZilvV7K2clbnuBvcLFcIFGtlKzE4ueQrj8+g6TeA7 sm/djI+YHW7SbwbJfsIo/GCDCabFdlgxS1Xt X-Google-Smtp-Source: AGRyM1s/D7w5lW8rHIOJ8tPXZ//y7BlXl1scm2CvIFSsZAI4asuL+1307Qq/vgDdbLiEG20sjasM/mVp1mTCdeSz X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a0d:f4c5:0:b0:317:7f89:f547 with SMTP id d188-20020a0df4c5000000b003177f89f547mr18475ywf.334.1656092231687; Fri, 24 Jun 2022 10:37:11 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:31 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-2-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 01/26] hugetlb: make hstate accessor functions const From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092232; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0YfnJKuQrbw7pcq3GiMHpIZjYaYv7/J4Psozl7GAbkY=; b=g9iPEINwbvfbrwqTWMIxJxrwd96amg+r6f/kdFXnlBVEvH+kodEXVzNiA7AWe8cDTeQkxn KssjMZfK98GvRIMuqRR74jdxvRLIfk+tR4BG+P7P9J5kUqTHqDqG4zKsZUz9YhBnxeMJt1 7BXB197ujrHDtSoXX6XhtMWx+/13Rbs= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=rhedzQCK; spf=pass (imf15.hostedemail.com: domain of 3R_a1YgoKCCgNXLSYKLXSRKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3R_a1YgoKCCgNXLSYKLXSRKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092232; a=rsa-sha256; cv=none; b=XEQdeA+RmAPmRW7RzVQoFD6kAAW4tJ+hxqtJHgWJ0A4c/jL1X29FaO/yBUgAMC2+aRJXry 3PUdR/8M8ZitrMgGRYqOzx2cNqsa4IcMMR0G4JQoJtO+orYYJm6Xm02KjKP+AoqPNgpCs9 nWJvhDU84vd4Y/PBM/BosBzMgNZugZU= Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=rhedzQCK; spf=pass (imf15.hostedemail.com: domain of 3R_a1YgoKCCgNXLSYKLXSRKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3R_a1YgoKCCgNXLSYKLXSRKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam06 X-Stat-Signature: rhmgixquu1rmmgkqjsmuuqmkktxhcqro X-Rspamd-Queue-Id: 5EF87A0022 X-HE-Tag: 1656092232-977833 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is just a const-correctness change so that the new hugetlb_pte changes can be const-correct too. Acked-by: David Rientjes Signed-off-by: James Houghton Reviewed-By: Mina Almasry --- include/linux/hugetlb.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index e4cff27d1198..498a4ae3d462 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -715,7 +715,7 @@ static inline struct hstate *hstate_vma(struct vm_area_struct *vma) return hstate_file(vma->vm_file); } -static inline unsigned long huge_page_size(struct hstate *h) +static inline unsigned long huge_page_size(const struct hstate *h) { return (unsigned long)PAGE_SIZE << h->order; } @@ -729,27 +729,27 @@ static inline unsigned long huge_page_mask(struct hstate *h) return h->mask; } -static inline unsigned int huge_page_order(struct hstate *h) +static inline unsigned int huge_page_order(const struct hstate *h) { return h->order; } -static inline unsigned huge_page_shift(struct hstate *h) +static inline unsigned huge_page_shift(const struct hstate *h) { return h->order + PAGE_SHIFT; } -static inline bool hstate_is_gigantic(struct hstate *h) +static inline bool hstate_is_gigantic(const struct hstate *h) { return huge_page_order(h) >= MAX_ORDER; } -static inline unsigned int pages_per_huge_page(struct hstate *h) +static inline unsigned int pages_per_huge_page(const struct hstate *h) { return 1 << h->order; } -static inline unsigned int blocks_per_huge_page(struct hstate *h) +static inline unsigned int blocks_per_huge_page(const struct hstate *h) { return huge_page_size(h) / 512; } From patchwork Fri Jun 24 17:36:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894927 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43948CCA473 for ; Fri, 24 Jun 2022 17:37:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BA6C08E0240; Fri, 24 Jun 2022 13:37:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B2F1D8E020E; Fri, 24 Jun 2022 13:37:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 90DF58E0240; Fri, 24 Jun 2022 13:37:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 76B058E020E for ; Fri, 24 Jun 2022 13:37:14 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 337C360830 for ; Fri, 24 Jun 2022 17:37:14 +0000 (UTC) X-FDA: 79613835588.17.5739C30 Received: from mail-ua1-f74.google.com (mail-ua1-f74.google.com [209.85.222.74]) by imf02.hostedemail.com (Postfix) with ESMTP id BF3B88002E for ; Fri, 24 Jun 2022 17:37:13 +0000 (UTC) Received: by mail-ua1-f74.google.com with SMTP id y15-20020ab0638f000000b00368a2d9b075so1012400uao.13 for ; Fri, 24 Jun 2022 10:37:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=cBo7lIiy2jlsdOG5qT1oBCHlWVvIiK17uVsMNAgjZEA=; b=NJEF0grG6xcnUMzJ7Xx8DncxrM3Dg/CCXuZcQoPGVS8RMCRW7OVDsyTMq2msiIkgR3 QD3rc43cPJSTJK/tuSyrg1IgeA7F7F0rpMnzeXl2raeM1+E+9KIbYlUZ6X/l+oQ1o3X8 FetdG0D5zD5JUcSef1agw+Hok5ZWwdiJaZLaL/pl4egaQHqe4C6GaU0IcgckBRNz/QIi XvX1VQlmkldTFcvliG4NEQiL7t+Leo3XWSFsPzela+hWWv2mTwnOC5NDGzw04gi1JEMx Hu59PBmYCm43RJHjH32Pik1VWkbEEt2CtqEMLAHbbH1gUokwIOPzG/lqTGqFU3e6iia/ o9Gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=cBo7lIiy2jlsdOG5qT1oBCHlWVvIiK17uVsMNAgjZEA=; b=o+d7ocXjihxG/nWga05noR8oASN53ts7cvQItCwbUvVGCyFVQ+KdfCjvxcrPyFgkO4 lG+E9blwNDF14OWYQbDbM+HfNoGm34OqXogTdfRyQg6M8oqXxn/BBcqexnUiRC9V3iFW lbq06TqKX0gPcwfXwcva0ywozBpCtdcpIZN42wJnYX9NdxgaUjC24zXy+RoU/RFN2TlP aPruVz4HLozl82cRQRToKYWwObKRiM7wR/62ENsQqaP+9Wmfb94ej19KHheM5pgkeTKT w6wt/1/AaCaocKrhStrLx2JpDvpm7Tb2MaS0UYuX86TvwcVxLyF7fiayqITP7NrskQDp n05w== X-Gm-Message-State: AJIora+ywPuRiyWEMmqDHVYSEnqQMba0eq9rHpY+uT7QzKx6m+co4TXB OMQspiIH5oZuvz0pqniIWMlSYc/+KO146U0s X-Google-Smtp-Source: AGRyM1uKLxV44YzBDP/q42+n2ejxUvrdr7EuCEClIr+ZyRlO8QyvjxzdjM1m0/iYLEcLM71c8MSRgqLgbnEZ/vLa X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6122:25b:b0:36c:5f1a:d94b with SMTP id t27-20020a056122025b00b0036c5f1ad94bmr11908478vko.31.1656092233083; Fri, 24 Jun 2022 10:37:13 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:32 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-3-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 02/26] hugetlb: sort hstates in hugetlb_init_hstates From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=NJEF0grG; spf=pass (imf02.hostedemail.com: domain of 3Sfa1YgoKCCoPZNUaMNZUTMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--jthoughton.bounces.google.com designates 209.85.222.74 as permitted sender) smtp.mailfrom=3Sfa1YgoKCCoPZNUaMNZUTMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092233; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cBo7lIiy2jlsdOG5qT1oBCHlWVvIiK17uVsMNAgjZEA=; b=Fu6E7okx6HfXouFZEOLfPOLWCS/T1/PJJFsmfVYgQKrJKlfXOmpVhsNqYb8TFTUFnXSQIA NryeuODQG2pywP9w/sS+v3jjTA4UE+vMx4wtIKDMrbnlxjoqgxK4B4CxswO3VP2T+HcDGs QZloblJ7+2z5Q7ecv7aqKrCMFKZe3Bk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092233; a=rsa-sha256; cv=none; b=QZlWY12xJlFK7uyeF76u9GCYmymreGgkK55FrQbjlfkANOTAT4vMrgyMzcPV60uss8x087 jnKfyQ6IK+H4L4MrhMM/WBoyqmICs6wG019rnVgEVzxtY3IoIqxhVDJcajiT5ON7P4e3uD cxs/P9IGpyLQ+p9UP8Xn312sHSvwO2Y= X-Stat-Signature: xurecfy9a5gw9tqpgcg54dzdyoyzjddr X-Rspamd-Queue-Id: BF3B88002E X-Rspam-User: Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=NJEF0grG; spf=pass (imf02.hostedemail.com: domain of 3Sfa1YgoKCCoPZNUaMNZUTMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--jthoughton.bounces.google.com designates 209.85.222.74 as permitted sender) smtp.mailfrom=3Sfa1YgoKCCoPZNUaMNZUTMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam02 X-HE-Tag: 1656092233-893495 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When using HugeTLB high-granularity mapping, we need to go through the supported hugepage sizes in decreasing order so that we pick the largest size that works. Consider the case where we're faulting in a 1G hugepage for the first time: we want hugetlb_fault/hugetlb_no_page to map it with a PUD. By going through the sizes in decreasing order, we will find that PUD_SIZE works before finding out that PMD_SIZE or PAGE_SIZE work too. Signed-off-by: James Houghton Reviewed-by: Mina Almasry --- mm/hugetlb.c | 40 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 37 insertions(+), 3 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a57e1be41401..5df838d86f32 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -33,6 +33,7 @@ #include #include #include +#include #include #include @@ -48,6 +49,10 @@ int hugetlb_max_hstate __read_mostly; unsigned int default_hstate_idx; +/* + * After hugetlb_init_hstates is called, hstates will be sorted from largest + * to smallest. + */ struct hstate hstates[HUGE_MAX_HSTATE]; #ifdef CONFIG_CMA @@ -3144,14 +3149,43 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) kfree(node_alloc_noretry); } +static int compare_hstates_decreasing(const void *a, const void *b) +{ + const int shift_a = huge_page_shift((const struct hstate *)a); + const int shift_b = huge_page_shift((const struct hstate *)b); + + if (shift_a < shift_b) + return 1; + if (shift_a > shift_b) + return -1; + return 0; +} + +static void sort_hstates(void) +{ + unsigned long default_hstate_sz = huge_page_size(&default_hstate); + + /* Sort from largest to smallest. */ + sort(hstates, hugetlb_max_hstate, sizeof(*hstates), + compare_hstates_decreasing, NULL); + + /* + * We may have changed the location of the default hstate, so we need to + * update it. + */ + default_hstate_idx = hstate_index(size_to_hstate(default_hstate_sz)); +} + static void __init hugetlb_init_hstates(void) { struct hstate *h, *h2; - for_each_hstate(h) { - if (minimum_order > huge_page_order(h)) - minimum_order = huge_page_order(h); + sort_hstates(); + /* The last hstate is now the smallest. */ + minimum_order = huge_page_order(&hstates[hugetlb_max_hstate - 1]); + + for_each_hstate(h) { /* oversize hugepages were init'ed in early boot */ if (!hstate_is_gigantic(h)) hugetlb_hstate_alloc_pages(h); From patchwork Fri Jun 24 17:36:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894928 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05823C433EF for ; Fri, 24 Jun 2022 17:37:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B311E8E0241; Fri, 24 Jun 2022 13:37:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A94198E020E; Fri, 24 Jun 2022 13:37:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7AF7E8E0241; Fri, 24 Jun 2022 13:37:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6403E8E020E for ; Fri, 24 Jun 2022 13:37:15 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 3C7B760BA6 for ; Fri, 24 Jun 2022 17:37:15 +0000 (UTC) X-FDA: 79613835630.16.4525EED Received: from mail-vk1-f201.google.com (mail-vk1-f201.google.com [209.85.221.201]) by imf18.hostedemail.com (Postfix) with ESMTP id CF53A1C0096 for ; Fri, 24 Jun 2022 17:37:14 +0000 (UTC) Received: by mail-vk1-f201.google.com with SMTP id l19-20020ac5c353000000b0036c1d249b61so910934vkk.12 for ; Fri, 24 Jun 2022 10:37:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=t5KTYrM6ct0JWom4isvAtP154HT/fkrm8rvSO1JL/WI=; b=M7hxU1278b0Lh3tABv6L9l9qZ4Alxa0YOqUCt2BED4X975N8JVezOpIosUzb1pRlq1 SX94qvz3QX67rX7o3GZ8S0leLb7FFHhNK9SdbX2Rb4HmWbI4dwX/Y22ct/UwPlrwOYgo BEGkS4HWHIzpNKTgxCJfh6kCkdYmxrPG3Nl2ni6y2MM4kjfXPF2tIOcXuallnel4T3/F /EtqJ7IGlIBOBHTK/RDkilRjqZ1ZMgNfsJXOMiaLWaQMYD1+JF22Fefkl8MXbu5xw4Co KYFTD830ASsGDKWn3Fb3Id7J7wa0mQKCv+6AwRNyDerpkb98fQtJVbUhtnQtE3T/QsCS 4F8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=t5KTYrM6ct0JWom4isvAtP154HT/fkrm8rvSO1JL/WI=; b=DnOeHtWsiwA7seiax8mHTYvAfwluTw+sLOmo+0NaYwTlrvKIpN1stqLccW36exdbSv fy9/r5XJAWP0Q8br8HPwZHdECk601pY5+xfJwKdAyIqF6i19BsmCqhLGUCuPC55Kwar2 eiKBpqied6B/PJ/J6aoOJ4PBpaHgX98vxtG9YU1FdA4QuHeymVPmIT4YtqkYuyFSNmMv vMtfQ9mNT6ChDY0lNhQel9KVXWJrJo1VMjrPoYXsC72gIlGQJwDgPeCZLGfC015+FM9v Es4TofQEEdLEIi9JUtGpy5C15M7FPHwBSRwgmyFlCJOkIc+wHw345vKptuwLMhSHyFwS b5eA== X-Gm-Message-State: AJIora+02pkYwLoc67+8PluoRrmhnhAydKLYGLe1EkJbRsBqbRRoVesX spQrG8PqOZPql2jvKNNpEhlmqMNp0sagNQ6o X-Google-Smtp-Source: AGRyM1s3Rhw53YqM2sURrEy49E22RRF/1tHTOyTtY+rHKd8zIb/K3DKzbqQUo9FnKORpPjSO3ZnTjnoOojVZvhpT X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6102:3ec8:b0:335:d67e:7535 with SMTP id n8-20020a0561023ec800b00335d67e7535mr71521vsv.47.1656092234176; Fri, 24 Jun 2022 10:37:14 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:33 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-4-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 03/26] hugetlb: add make_huge_pte_with_shift From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092234; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t5KTYrM6ct0JWom4isvAtP154HT/fkrm8rvSO1JL/WI=; b=aLcRqw9xrzXaHBmEnpgehTcgvnvO+AwhtQfxP2PYPN44cgl0KSe4110gmYNyawonhARv8q R/eTkIE8PBfsZPkG9mtlY7ZiNZR+r0bUUhx4qZScsnfXSmw2mhICGaof5LDD/Rbt8UV0Rw gPVCF1uNOQt2DMs84vTk49kn79RBxHQ= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=M7hxU127; spf=pass (imf18.hostedemail.com: domain of 3Sva1YgoKCCsQaOVbNOaVUNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--jthoughton.bounces.google.com designates 209.85.221.201 as permitted sender) smtp.mailfrom=3Sva1YgoKCCsQaOVbNOaVUNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092234; a=rsa-sha256; cv=none; b=vqArc7n1766+Drgu0Cv8f2MN3FnJNoSxE3QsAj1/R7/iJ46gdOGbjgZMT3luQIRY++Xu/5 p+/NffMXL4qhSIo29tfQgUdOrZDCmmJsu+XsNYFGzXWmYkq8kAGqrAt5EJzyopTEecE132 U1CnfWP3Ydq11LzwD2JH3Q85eMkc6Ps= Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=M7hxU127; spf=pass (imf18.hostedemail.com: domain of 3Sva1YgoKCCsQaOVbNOaVUNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--jthoughton.bounces.google.com designates 209.85.221.201 as permitted sender) smtp.mailfrom=3Sva1YgoKCCsQaOVbNOaVUNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam06 X-Stat-Signature: ebwgpjmdu79o4rz1ge4rk8se8g6bmkc6 X-Rspamd-Queue-Id: CF53A1C0096 X-HE-Tag: 1656092234-509218 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This allows us to make huge PTEs at shifts other than the hstate shift, which will be necessary for high-granularity mappings. Signed-off-by: James Houghton reviewed-by: manish.mishra@nutanix.com --- mm/hugetlb.c | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 5df838d86f32..0eec34edf3b2 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4686,23 +4686,30 @@ const struct vm_operations_struct hugetlb_vm_ops = { .pagesize = hugetlb_vm_op_pagesize, }; +static pte_t make_huge_pte_with_shift(struct vm_area_struct *vma, + struct page *page, int writable, + int shift) +{ + bool huge = shift > PAGE_SHIFT; + pte_t entry = huge ? mk_huge_pte(page, vma->vm_page_prot) + : mk_pte(page, vma->vm_page_prot); + + if (writable) + entry = huge ? huge_pte_mkwrite(entry) : pte_mkwrite(entry); + else + entry = huge ? huge_pte_wrprotect(entry) : pte_wrprotect(entry); + pte_mkyoung(entry); + if (huge) + entry = arch_make_huge_pte(entry, shift, vma->vm_flags); + return entry; +} + static pte_t make_huge_pte(struct vm_area_struct *vma, struct page *page, - int writable) + int writable) { - pte_t entry; unsigned int shift = huge_page_shift(hstate_vma(vma)); - if (writable) { - entry = huge_pte_mkwrite(huge_pte_mkdirty(mk_huge_pte(page, - vma->vm_page_prot))); - } else { - entry = huge_pte_wrprotect(mk_huge_pte(page, - vma->vm_page_prot)); - } - entry = pte_mkyoung(entry); - entry = arch_make_huge_pte(entry, shift, vma->vm_flags); - - return entry; + return make_huge_pte_with_shift(vma, page, writable, shift); } static void set_huge_ptep_writable(struct vm_area_struct *vma, From patchwork Fri Jun 24 17:36:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894929 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD2ACC43334 for ; Fri, 24 Jun 2022 17:37:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2CE028E0242; Fri, 24 Jun 2022 13:37:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 255F18E020E; Fri, 24 Jun 2022 13:37:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A8F18E0242; Fri, 24 Jun 2022 13:37:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D1C2D8E020E for ; Fri, 24 Jun 2022 13:37:16 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 9F20B35630 for ; Fri, 24 Jun 2022 17:37:16 +0000 (UTC) X-FDA: 79613835672.09.16A3880 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf17.hostedemail.com (Postfix) with ESMTP id 2BBD740023 for ; Fri, 24 Jun 2022 17:37:16 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id a83-20020a251a56000000b0066c354a19ccso1666131yba.6 for ; Fri, 24 Jun 2022 10:37:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Prs/g0eAXOv/ZKQzs92554GHZM/jn5SpuUFUCkOXcEo=; b=Px4lWScALFzFiAT+UIjCnXt+kA7DjkP32e6aEOW4CwgMESdfSnOQVe0TMA2O89nVFQ 2FSb4AFEEui36dp7cEO2cCeeW25R3wvs2P0Pw293UvGXo1bFkXyTw3c22GRwbp6ZeroB BKYDs88dfDTkiN+2kUji3gPFSrrfiflHbvNAwBPQkx1aAAnlNFHrCyf3UirmhSLZm9oL nyAorTU4OhiyKwV0sm0qey3GF9yE03pGEq+AkLOcUeZfuKYpmDNlnep1AHlKc5CHBoWs lchQ3wZR304ZcaHQH9yJaAqziuHYgjAFwrM+TJW+0jvW+Bqrs/0OMU3uJ876q5Jq/AM4 BUoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Prs/g0eAXOv/ZKQzs92554GHZM/jn5SpuUFUCkOXcEo=; b=qgymrKMrDHcKP3xFF41121ZeaI48nhlAd2YozyMvbdUqTKXi9LuVseGsu8Ug3iZM3J J+trOtCDYcaRjMOuibxFgWuVtL0/2wz7IWBrsEN9MeMLF4jBERsw2EVFTWw/Qzg6VDLn 1twsxx92IgG+GSekvMAbiLyuiT3nvHLUyYUUZ5S2Qo8frqQ+2pbFZNS5Uh0QCH4lyLkJ 1wcrKNs5S+7Ac6ehU42D4zuvXagpbJ7kwYs6nHT9XkyDI5O9Cbd5maFSpct7VA8t3TFg MNMZOnDgJGCSq0k08ucrKuOI1bx7zEQZjKVFTNjJLCMGgXBwNLSc6pNPTFRI1qi/GqZ0 yTaw== X-Gm-Message-State: AJIora/Y+zvgNZq1bStjVL+PgiOIkypuGbztuQAr6Pz8gKsChIqiWwJI lTuTvAyIjI+blN/ufzuUVwtHjCKU+H9LTjne X-Google-Smtp-Source: AGRyM1vNqp1siTq4+vM6xRRCFB8ViZUf9ItF5XtUfUff2x9P2l161GxyqwN1706DB9irdsmvY90a1AH0gr7PPFum X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a25:b19a:0:b0:668:a921:310d with SMTP id h26-20020a25b19a000000b00668a921310dmr271313ybj.341.1656092235523; Fri, 24 Jun 2022 10:37:15 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:34 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-5-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 04/26] hugetlb: make huge_pte_lockptr take an explicit shift argument. From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092236; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Prs/g0eAXOv/ZKQzs92554GHZM/jn5SpuUFUCkOXcEo=; b=796c5QE199jdPZ1FDCsuhhpeApg1+nOXEzWLg4wsIRIUIcvcY7hFR0of85s027gf6rgQPP No6fUjLJYmHigjV3cP8OXPozQ/4UCviEtXhSAV2n5lyqdi0Bj9m102++oDBC+z6VBJGiGy BkfN6VHxVcOJSWQjLBfUTpMT2bYJWtU= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Px4lWScA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3S_a1YgoKCCwRbPWcOPbWVOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3S_a1YgoKCCwRbPWcOPbWVOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092236; a=rsa-sha256; cv=none; b=XLWeYnl6WgGsQwvfFvhs2cVehoQRMg0ajE1WxXVpDeHoOrTaYPX/HccaeU2S1/z5+++C5v 1Yk23+Kzl8ACauo0gWTsApCQDMRkl7i7JPHw09LkVcFouc0AE1/j3n8jQ3AiU3Lcl3osxt 5+TJhcWHobPNkmFxxB1H7UO/jmvPgug= X-Rspamd-Queue-Id: 2BBD740023 Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Px4lWScA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3S_a1YgoKCCwRbPWcOPbWVOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3S_a1YgoKCCwRbPWcOPbWVOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--jthoughton.bounces.google.com X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 373nyokkfacnn8qhi8x99iwuuthzay4m X-HE-Tag: 1656092235-506515 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is needed to handle PTL locking with high-granularity mapping. We won't always be using the PMD-level PTL even if we're using the 2M hugepage hstate. It's possible that we're dealing with 4K PTEs, in which case, we need to lock the PTL for the 4K PTE. Signed-off-by: James Houghton --- arch/powerpc/mm/pgtable.c | 3 ++- include/linux/hugetlb.h | 19 ++++++++++++++----- mm/hugetlb.c | 9 +++++---- mm/migrate.c | 3 ++- mm/page_vma_mapped.c | 3 ++- 5 files changed, 25 insertions(+), 12 deletions(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index e6166b71d36d..663d591a8f08 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -261,7 +261,8 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, psize = hstate_get_psize(h); #ifdef CONFIG_DEBUG_VM - assert_spin_locked(huge_pte_lockptr(h, vma->vm_mm, ptep)); + assert_spin_locked(huge_pte_lockptr(huge_page_shift(h), + vma->vm_mm, ptep)); #endif #else diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 498a4ae3d462..5fe1db46d8c9 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -868,12 +868,11 @@ static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask) return modified_mask; } -static inline spinlock_t *huge_pte_lockptr(struct hstate *h, +static inline spinlock_t *huge_pte_lockptr(unsigned int shift, struct mm_struct *mm, pte_t *pte) { - if (huge_page_size(h) == PMD_SIZE) + if (shift == PMD_SHIFT) return pmd_lockptr(mm, (pmd_t *) pte); - VM_BUG_ON(huge_page_size(h) == PAGE_SIZE); return &mm->page_table_lock; } @@ -1076,7 +1075,7 @@ static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask) return 0; } -static inline spinlock_t *huge_pte_lockptr(struct hstate *h, +static inline spinlock_t *huge_pte_lockptr(unsigned int shift, struct mm_struct *mm, pte_t *pte) { return &mm->page_table_lock; @@ -1116,7 +1115,17 @@ static inline spinlock_t *huge_pte_lock(struct hstate *h, { spinlock_t *ptl; - ptl = huge_pte_lockptr(h, mm, pte); + ptl = huge_pte_lockptr(huge_page_shift(h), mm, pte); + spin_lock(ptl); + return ptl; +} + +static inline spinlock_t *huge_pte_lock_shift(unsigned int shift, + struct mm_struct *mm, pte_t *pte) +{ + spinlock_t *ptl; + + ptl = huge_pte_lockptr(shift, mm, pte); spin_lock(ptl); return ptl; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 0eec34edf3b2..d6d0d4c03def 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4817,7 +4817,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, continue; dst_ptl = huge_pte_lock(h, dst, dst_pte); - src_ptl = huge_pte_lockptr(h, src, src_pte); + src_ptl = huge_pte_lockptr(huge_page_shift(h), src, src_pte); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); entry = huge_ptep_get(src_pte); dst_entry = huge_ptep_get(dst_pte); @@ -4894,7 +4894,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, /* Install the new huge page if src pte stable */ dst_ptl = huge_pte_lock(h, dst, dst_pte); - src_ptl = huge_pte_lockptr(h, src, src_pte); + src_ptl = huge_pte_lockptr(huge_page_shift(h), + src, src_pte); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); entry = huge_ptep_get(src_pte); if (!pte_same(src_pte_old, entry)) { @@ -4948,7 +4949,7 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr, pte_t pte; dst_ptl = huge_pte_lock(h, mm, dst_pte); - src_ptl = huge_pte_lockptr(h, mm, src_pte); + src_ptl = huge_pte_lockptr(huge_page_shift(h), mm, src_pte); /* * We don't have to worry about the ordering of src and dst ptlocks @@ -6024,7 +6025,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, page_in_pagecache = true; } - ptl = huge_pte_lockptr(h, dst_mm, dst_pte); + ptl = huge_pte_lockptr(huge_page_shift(h), dst_mm, dst_pte); spin_lock(ptl); /* diff --git a/mm/migrate.c b/mm/migrate.c index e51588e95f57..a8a960992373 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -318,7 +318,8 @@ void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, void migration_entry_wait_huge(struct vm_area_struct *vma, struct mm_struct *mm, pte_t *pte) { - spinlock_t *ptl = huge_pte_lockptr(hstate_vma(vma), mm, pte); + spinlock_t *ptl = huge_pte_lockptr(huge_page_shift(hstate_vma(vma)), + mm, pte); __migration_entry_wait(mm, pte, ptl); } diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index c10f839fc410..8921dd4e41b1 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -174,7 +174,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (!pvmw->pte) return false; - pvmw->ptl = huge_pte_lockptr(hstate, mm, pvmw->pte); + pvmw->ptl = huge_pte_lockptr(huge_page_shift(hstate), + mm, pvmw->pte); spin_lock(pvmw->ptl); if (!check_pte(pvmw)) return not_found(pvmw); From patchwork Fri Jun 24 17:36:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894931 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84F7BC433EF for ; Fri, 24 Jun 2022 17:37:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D9D68E0245; Fri, 24 Jun 2022 13:37:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7122C8E0244; Fri, 24 Jun 2022 13:37:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 452378E0245; Fri, 24 Jun 2022 13:37:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 0EEDD8E020E for ; Fri, 24 Jun 2022 13:37:20 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 91D91120A48 for ; Fri, 24 Jun 2022 17:37:17 +0000 (UTC) X-FDA: 79613835714.11.4780C54 Received: from mail-vk1-f202.google.com (mail-vk1-f202.google.com [209.85.221.202]) by imf03.hostedemail.com (Postfix) with ESMTP id 4E5282001C for ; Fri, 24 Jun 2022 17:37:17 +0000 (UTC) Received: by mail-vk1-f202.google.com with SMTP id t191-20020a1f78c8000000b0036c28bfe148so923181vkc.6 for ; Fri, 24 Jun 2022 10:37:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=a/I3oVqLRkRj5HBns0MWW3Ig4smTJMVIYbM6WBPe2Ok=; b=Oi4b3uHib9QeDIaeFbYIYjHKGKKHh9PJftREg1F7x7dC3MdxzlFlkF4nFN2oUkEmJv ksO18/0H6IRArOgx9eqn0gBPvHv2o9/dMee0FdF4JnH+jRgLOM2Nmsz6Dpy3izb1wBXI nx/vriW2gKgSHU0JVyGwH1vUXSXmgF4EHLnj5axRDix5D/a9yC/E9keoj2kNivTofhb4 42LtC0DSUfNjTGTr1svMT6wtFWhev3hUy0WltrufYZ9eblHJrOi+VTLQFp5FE3ILODHz B23csyok2qwTFatLcA+FGCv8SsDhrPMlNnuTVoNybqfkwHtiNOSZnpNZ1wSXRcUNFGXT kHFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=a/I3oVqLRkRj5HBns0MWW3Ig4smTJMVIYbM6WBPe2Ok=; b=4AhT3/iCBEcD3+eiaz08LOyy5PefHLa21sP1+YoeIsTG3/FOmLPU/YpwUrUud5iul7 CkLqvcLUQZvOZFTLRLDbFuUo0VOwHh5/w1+fQKJTphGLiX/xT2I/Jg8QZu1h0HE6EoLe 8JQn4G4BEdxm4y3WdYw7fDkjK7wKdRyFgTctCHYAzbT5zaKSxxauIZ7UHc4GiGe6q8Qs amA7+7zIdPr3bL1d+Bvi4LXOLCdEW6n6ll8qU0aGQjYfzyjmkFxHmDZ26pqQoKz0yvD3 6ZYDP0CFIs3ou3lkwH7q04r0ca3VXeCpvs6CjzSRb1H94XmgOTMib/L99db7m86zHFTI /xLw== X-Gm-Message-State: AJIora9I/ow4+plmNP/IvDxwibydd7AvFr1IBqqub6bUOCNw0bxJvQTF O/AA9Z/hHP8VAnD38I3LQHDhHhq94IQiNt0N X-Google-Smtp-Source: AGRyM1ulcuwzT9arHdsQPyjRKEdIRoxwLhneCD7SBt/oCPOlZSq/mYJ1bqnOaa+5zikkIexlm8WeaXwx6yxitdUI X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a1f:2657:0:b0:36c:d88e:88cc with SMTP id m84-20020a1f2657000000b0036cd88e88ccmr256693vkm.36.1656092236628; Fri, 24 Jun 2022 10:37:16 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:35 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-6-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 05/26] hugetlb: add CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Oi4b3uHi; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3TPa1YgoKCC0ScQXdPQcXWPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--jthoughton.bounces.google.com designates 209.85.221.202 as permitted sender) smtp.mailfrom=3TPa1YgoKCC0ScQXdPQcXWPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092237; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=a/I3oVqLRkRj5HBns0MWW3Ig4smTJMVIYbM6WBPe2Ok=; b=2Cdwt7sDa6Ry0p0+BfZzhfPHwOZ1iB45xrnmj/RXzECIXhNepFmCno/vHoqgsPq83wNnkH wq5yCl9BPZ0vgRxPKc5kB+MytcrDdfUDX7JG+kbQ5ZKFD5eiwUtbarcbXQFCyJHakZ7ECM djCYrIPhqfd49XW37xb0KOTkTvwrstA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092237; a=rsa-sha256; cv=none; b=FEbgkbv/ZfXGrz4JiylUiEKPQKemLZyey2+0FLqNarkbPR+5Zb9I6BnACpGIeAC5KB6frG JSBeldfG1IQcJKLkT2pxVovng7UhB8hlp+XWUSxwNEJfgmiJqQarpXH8W3vvR7RyYra7gL Omx8Yqp+GFDoFaqPkfQC4t4XT7IRWRU= X-Stat-Signature: 6apoknntit4xxg31iio3i793znp5rqoq X-Rspamd-Queue-Id: 4E5282001C X-Rspam-User: Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Oi4b3uHi; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3TPa1YgoKCC0ScQXdPQcXWPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--jthoughton.bounces.google.com designates 209.85.221.202 as permitted sender) smtp.mailfrom=3TPa1YgoKCC0ScQXdPQcXWPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--jthoughton.bounces.google.com X-Rspamd-Server: rspam12 X-HE-Tag: 1656092237-887624 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This adds the Kconfig to enable or disable high-granularity mapping. It is enabled by default for architectures that use ARCH_WANT_GENERAL_HUGETLB. There is also an arch-specific config ARCH_HAS_SPECIAL_HUGETLB_HGM which controls whether or not the architecture has been updated to support HGM if it doesn't use general HugeTLB. Signed-off-by: James Houghton reviewed-by:manish.mishra@nutanix.com Reviewed-by: Mina Almasry --- fs/Kconfig | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/fs/Kconfig b/fs/Kconfig index 5976eb33535f..d76c7d812656 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -268,6 +268,13 @@ config HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON to enable optimizing vmemmap pages of HugeTLB by default. It can then be disabled on the command line via hugetlb_free_vmemmap=off. +config ARCH_HAS_SPECIAL_HUGETLB_HGM + bool + +config HUGETLB_HIGH_GRANULARITY_MAPPING + def_bool ARCH_WANT_GENERAL_HUGETLB || ARCH_HAS_SPECIAL_HUGETLB_HGM + depends on HUGETLB_PAGE + config MEMFD_CREATE def_bool TMPFS || HUGETLBFS From patchwork Fri Jun 24 17:36:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894930 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3FF02CCA473 for ; Fri, 24 Jun 2022 17:37:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5DB758E0243; Fri, 24 Jun 2022 13:37:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4ED2E8E020E; Fri, 24 Jun 2022 13:37:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2CAA28E0243; Fri, 24 Jun 2022 13:37:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 104768E020E for ; Fri, 24 Jun 2022 13:37:19 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E1E2660A1A for ; Fri, 24 Jun 2022 17:37:18 +0000 (UTC) X-FDA: 79613835756.18.93FAE17 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf23.hostedemail.com (Postfix) with ESMTP id 90593140002 for ; Fri, 24 Jun 2022 17:37:18 +0000 (UTC) Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-317765eb7ccso26892217b3.13 for ; Fri, 24 Jun 2022 10:37:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=/rA4FWSpYAR9L2sjJS2B0eCc4aGZeNFQ2hS7zEPXx88=; b=mjmAAKf/rdfFJdvjC9OXk41WB0xb48wdXBGxSuL0EQl9SkDoXcqxrY9GXlvPWLaEkg 8Hx1fw01wcTtT9K/GLcq3rroLEsW0zKCq/TUIz18NlMwdRcR1pA7Iaz2vn8ZlKAXcim1 wOaCTh2gfUTYSHpFrxRcxF2GTYKnhiAIuJO4LJxX4Vd8vg8DEaO1anDPe1vI6jmOnZyo xym6RDo4VAwtwvCcWa490l4s9zNK4x/xstjaksrlIYlIG61i/GAqmjnAH04jRi+sdL4m P8snVFmVGcbUsQWry1OYKwwBVkaqJWg10bFh2y6ecVXque/dyOxRMd1yJMz25o75f8Kd 6OwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=/rA4FWSpYAR9L2sjJS2B0eCc4aGZeNFQ2hS7zEPXx88=; b=jL96engpouxkPJpTM0jn2QhwaZqrL5LRjhOC3ENmO3mChfIpn6bvh+4PDQ4tQaRjPT qHJbnwFt0XVm3vLXwaOWl1snrgYzAdHat3NfvbYy3Dz8+M53hOWW+59KQvajRgCuEt37 PkWbCNRpIXlLZOaN4r9DecruL7ig4LKCdUVhdSQY3EYFxOJfdBXaepPv0kcuUoO0SpyK ksU017xKyPd27Yl/asVNzi/da0JTQsYnWDrf+tWvZqconva61R4JpGEV3BiXmSdzrq87 OvTGbdmHYX2qDBY71/Nsm1EyEW6FBBW8xH4NyBrnSJ5b1p8xDgBSsTUmFBB6fui2P/T4 H/UA== X-Gm-Message-State: AJIora8C9AqSjdrXCUvc44NmFc6Yf+Y1eloUyOKRCvjGCpyruosPq5qT vpKvZnANvw81sfMY7CfCCczjVCiPquhJvxL3 X-Google-Smtp-Source: AGRyM1t9tv0yjearXIr71+3GNqpFRDSr2edCHZwVgwQ6f0v6TT0Pfet2HMTd37UlieYMZq6c6zvogGIBBnnftS40 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a25:b806:0:b0:663:d35d:8b8a with SMTP id v6-20020a25b806000000b00663d35d8b8amr282022ybj.69.1656092237926; Fri, 24 Jun 2022 10:37:17 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:36 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-7-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 06/26] mm: make free_p?d_range functions public From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092238; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/rA4FWSpYAR9L2sjJS2B0eCc4aGZeNFQ2hS7zEPXx88=; b=4yXBwtw5O9tK/D76DZr9cnxKy4BtbfmQ3G7CGk7rtuQagI7nXQ5kChF6RIN3GWYEK8cfLQ vb68M0mBLd6u2MSbYDgrZ0XLo9hzxrlDVnF4I+5Sw1EqgGw1lj2gEIYdiRNfUGESWSGf/I 1HI7CfBHatd5JSZ+H7u1Y4COhjxus84= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="mjmAAKf/"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3Tfa1YgoKCC4TdRYeQRdYXQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--jthoughton.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Tfa1YgoKCC4TdRYeQRdYXQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092238; a=rsa-sha256; cv=none; b=qnGes6+cDZCFEqTj3uUXVqe72tvRV0S9ipasCCpEg7Ff+GoS9jDVsN+UPyTk2gDbY0NZXY URZFi7XGe+DD6mVX/obhTny5YBHIJSmP9OWf3egqzoMUOcfADVf7C6l2EFcjDz2Qu31jxu 9uvtrcKspKZVQYk52jz0+KrAPfSp4J4= X-Rspamd-Queue-Id: 90593140002 Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="mjmAAKf/"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3Tfa1YgoKCC4TdRYeQRdYXQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--jthoughton.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Tfa1YgoKCC4TdRYeQRdYXQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--jthoughton.bounces.google.com X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: kgrdjf6go8gr3cjaccxthr1hsbp1a3xg X-HE-Tag: 1656092238-259331 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This makes them usable for HugeTLB page table freeing operations. After HugeTLB high-granularity mapping, the page table for a HugeTLB VMA can get more complex, and these functions handle freeing page tables generally. Signed-off-by: James Houghton reviewed-by: manish.mishra@nutanix.com --- include/linux/mm.h | 7 +++++++ mm/memory.c | 8 ++++---- 2 files changed, 11 insertions(+), 4 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index bc8f326be0ce..07f5da512147 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1847,6 +1847,13 @@ void unmap_vmas(struct mmu_gather *tlb, struct vm_area_struct *start_vma, struct mmu_notifier_range; +void free_pte_range(struct mmu_gather *tlb, pmd_t *pmd, unsigned long addr); +void free_pmd_range(struct mmu_gather *tlb, pud_t *pud, unsigned long addr, + unsigned long end, unsigned long floor, unsigned long ceiling); +void free_pud_range(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr, + unsigned long end, unsigned long floor, unsigned long ceiling); +void free_p4d_range(struct mmu_gather *tlb, pgd_t *pgd, unsigned long addr, + unsigned long end, unsigned long floor, unsigned long ceiling); void free_pgd_range(struct mmu_gather *tlb, unsigned long addr, unsigned long end, unsigned long floor, unsigned long ceiling); int diff --git a/mm/memory.c b/mm/memory.c index 7a089145cad4..bb3b9b5b94fb 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -227,7 +227,7 @@ static void check_sync_rss_stat(struct task_struct *task) * Note: this doesn't free the actual pages themselves. That * has been handled earlier when unmapping all the memory regions. */ -static void free_pte_range(struct mmu_gather *tlb, pmd_t *pmd, +void free_pte_range(struct mmu_gather *tlb, pmd_t *pmd, unsigned long addr) { pgtable_t token = pmd_pgtable(*pmd); @@ -236,7 +236,7 @@ static void free_pte_range(struct mmu_gather *tlb, pmd_t *pmd, mm_dec_nr_ptes(tlb->mm); } -static inline void free_pmd_range(struct mmu_gather *tlb, pud_t *pud, +inline void free_pmd_range(struct mmu_gather *tlb, pud_t *pud, unsigned long addr, unsigned long end, unsigned long floor, unsigned long ceiling) { @@ -270,7 +270,7 @@ static inline void free_pmd_range(struct mmu_gather *tlb, pud_t *pud, mm_dec_nr_pmds(tlb->mm); } -static inline void free_pud_range(struct mmu_gather *tlb, p4d_t *p4d, +inline void free_pud_range(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr, unsigned long end, unsigned long floor, unsigned long ceiling) { @@ -304,7 +304,7 @@ static inline void free_pud_range(struct mmu_gather *tlb, p4d_t *p4d, mm_dec_nr_puds(tlb->mm); } -static inline void free_p4d_range(struct mmu_gather *tlb, pgd_t *pgd, +inline void free_p4d_range(struct mmu_gather *tlb, pgd_t *pgd, unsigned long addr, unsigned long end, unsigned long floor, unsigned long ceiling) { From patchwork Fri Jun 24 17:36:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894932 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33315C433EF for ; Fri, 24 Jun 2022 17:37:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B2C958E020E; Fri, 24 Jun 2022 13:37:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 98C818E0246; Fri, 24 Jun 2022 13:37:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 519998E020E; Fri, 24 Jun 2022 13:37:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1FF898E0244 for ; Fri, 24 Jun 2022 13:37:20 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id F1D92333E9 for ; Fri, 24 Jun 2022 17:37:19 +0000 (UTC) X-FDA: 79613835798.08.528CB4E Received: from mail-vk1-f201.google.com (mail-vk1-f201.google.com [209.85.221.201]) by imf18.hostedemail.com (Postfix) with ESMTP id 8131B1C00A0 for ; Fri, 24 Jun 2022 17:37:19 +0000 (UTC) Received: by mail-vk1-f201.google.com with SMTP id l19-20020ac5c353000000b0036c1d249b61so910934vkk.12 for ; Fri, 24 Jun 2022 10:37:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=C3fH1oRby2iG8QHlX2xdR+iuje/PS4n79lvPkGthMbg=; b=ZvJyRZYi5Luh9HoaMNclQG/FEjMLfzSsnyQ/9mxS9ijpI3JJW36iQDpaHsY87LXRBZ MYe/3q2xvO35L2IDpu2v2DM/tTOw5jERdNYlWr+w9FKaAmGF7Qw169EIGH/QBP5/M5Hu EnvP0MM3PVu+Qq3VOyCfhXtU1PqbPDjXRp9bYa+nmG69Ejf6C4LVUubpOccWnz+8x8al Ud1kt4H0mzAyAWMnP0xMUwaKTGQ7OAkmev0EEidQ3pcSQbK/8Wk2CIruV0DC5jWgMmae phVsDW4MQ5+a3GUwp4m0a+E3VZr2h8GwiRLlTkbUBbEknSpuJ1gbOzRLqoe5ZpouQ3g/ 4rOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=C3fH1oRby2iG8QHlX2xdR+iuje/PS4n79lvPkGthMbg=; b=nktw7VVFcaRAY9dpaDDwxP9uBsN3TUEAoFd9UQbAs1NAs4V4U2YOKm1wfY9AM+xF3e n2KENcKxqf/b4Fys4fFbthacq6FRx5BO7JjW+Wvl4Tc11SLKrAfIUYJdN5ZR5WKQRkzd Rho8CPPaRdvA2pByqycpM1sFiVxWoVhqC03sh0aLCCoRot0xdd/Jn/sNnxHp6WuDkUYr jCRnWU34BjK86Mrl1/0YzwXOL3o2haM0tnl2g/VSzT3nelG2G6AzSizzY9s94tf283yg PD9IZ+/Xj2jyKiJ8oLzEM1uvzxRfFsszPXxONkaXYlYxnIGoCVP1nf16zjA3OqkesSmk 8PHw== X-Gm-Message-State: AJIora/vuZP7zx3fBIUNXIGbkAMzaawFCM6A2JacrXOgzp/gYC9PndiX YV6d4nGBGmH4iehgzJgU7tqbK2WSl5OsuYze X-Google-Smtp-Source: AGRyM1sN569V6cAWRy0BJnUUQMejK8sKMmpW5w3o0qztJIemJ7kY/kSH0xH0s1Ckgj1aJGX5wLB5oOn168goMMsQ X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6122:506:b0:36c:3d23:38e7 with SMTP id x6-20020a056122050600b0036c3d2338e7mr9197vko.26.1656092239059; Fri, 24 Jun 2022 10:37:19 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:37 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-8-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 07/26] hugetlb: add hugetlb_pte to track HugeTLB page table entries From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092239; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=C3fH1oRby2iG8QHlX2xdR+iuje/PS4n79lvPkGthMbg=; b=juMaFKRksOlUhfu3aRZKabJuRj715lqjY42MbqsqEunIAIZDEugf5PQ2FNpEEFEjW69JKi VB5+O+Iqj4e5qEb4nkjdqMUX6kiYfcdePUiYOo3L3YPWeVj7N6C4+L/6rdXNMl2+3ra/DK jg8nWMz6kloowo7u68KCJNb345duyEA= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ZvJyRZYi; spf=pass (imf18.hostedemail.com: domain of 3T_a1YgoKCDAVfTagSTfaZSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--jthoughton.bounces.google.com designates 209.85.221.201 as permitted sender) smtp.mailfrom=3T_a1YgoKCDAVfTagSTfaZSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092239; a=rsa-sha256; cv=none; b=vBlfAefnYLE9LG9pAB5kJn045rx5d+OFRy66Y07HDQBGb8NhTWx509CQ9+yuYR++hFDWci sBwpTrggqCaHTGhT70uzzSDtm5L/3tko3krw0QJEMGvch00ye1DKdnUG8iqZdUhW57FfCR uZZFtCsEeOR+vKTOc4Ay2V4OEmBeonc= Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ZvJyRZYi; spf=pass (imf18.hostedemail.com: domain of 3T_a1YgoKCDAVfTagSTfaZSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--jthoughton.bounces.google.com designates 209.85.221.201 as permitted sender) smtp.mailfrom=3T_a1YgoKCDAVfTagSTfaZSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam06 X-Stat-Signature: ma1nuzacft4yu93rgz7n7ap555fd1taz X-Rspamd-Queue-Id: 8131B1C00A0 X-HE-Tag: 1656092239-470411 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: After high-granularity mapping, page table entries for HugeTLB pages can be of any size/type. (For example, we can have a 1G page mapped with a mix of PMDs and PTEs.) This struct is to help keep track of a HugeTLB PTE after we have done a page table walk. Without this, we'd have to pass around the "size" of the PTE everywhere. We effectively did this before; it could be fetched from the hstate, which we pass around pretty much everywhere. This commit includes definitions for some basic helper functions that are used later. These helper functions wrap existing PTE inspection/modification functions, where the correct version is picked depending on if the HugeTLB PTE is actually "huge" or not. (Previously, all HugeTLB PTEs were "huge"). For example, hugetlb_ptep_get wraps huge_ptep_get and ptep_get, where ptep_get is used when the HugeTLB PTE is PAGE_SIZE, and huge_ptep_get is used in all other cases. Signed-off-by: James Houghton --- include/linux/hugetlb.h | 84 +++++++++++++++++++++++++++++++++++++++++ mm/hugetlb.c | 57 ++++++++++++++++++++++++++++ 2 files changed, 141 insertions(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 5fe1db46d8c9..1d4ec9dfdebf 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -46,6 +46,68 @@ enum { __NR_USED_SUBPAGE, }; +struct hugetlb_pte { + pte_t *ptep; + unsigned int shift; +}; + +static inline +void hugetlb_pte_init(struct hugetlb_pte *hpte) +{ + hpte->ptep = NULL; +} + +static inline +void hugetlb_pte_populate(struct hugetlb_pte *hpte, pte_t *ptep, + unsigned int shift) +{ + BUG_ON(!ptep); + hpte->ptep = ptep; + hpte->shift = shift; +} + +static inline +unsigned long hugetlb_pte_size(const struct hugetlb_pte *hpte) +{ + BUG_ON(!hpte->ptep); + return 1UL << hpte->shift; +} + +static inline +unsigned long hugetlb_pte_mask(const struct hugetlb_pte *hpte) +{ + BUG_ON(!hpte->ptep); + return ~(hugetlb_pte_size(hpte) - 1); +} + +static inline +unsigned int hugetlb_pte_shift(const struct hugetlb_pte *hpte) +{ + BUG_ON(!hpte->ptep); + return hpte->shift; +} + +static inline +bool hugetlb_pte_huge(const struct hugetlb_pte *hpte) +{ + return !IS_ENABLED(CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING) || + hugetlb_pte_shift(hpte) > PAGE_SHIFT; +} + +static inline +void hugetlb_pte_copy(struct hugetlb_pte *dest, const struct hugetlb_pte *src) +{ + dest->ptep = src->ptep; + dest->shift = src->shift; +} + +bool hugetlb_pte_present_leaf(const struct hugetlb_pte *hpte); +bool hugetlb_pte_none(const struct hugetlb_pte *hpte); +bool hugetlb_pte_none_mostly(const struct hugetlb_pte *hpte); +pte_t hugetlb_ptep_get(const struct hugetlb_pte *hpte); +void hugetlb_pte_clear(struct mm_struct *mm, const struct hugetlb_pte *hpte, + unsigned long address); + struct hugepage_subpool { spinlock_t lock; long count; @@ -1130,6 +1192,28 @@ static inline spinlock_t *huge_pte_lock_shift(unsigned int shift, return ptl; } +static inline +spinlock_t *hugetlb_pte_lockptr(struct mm_struct *mm, struct hugetlb_pte *hpte) +{ + + BUG_ON(!hpte->ptep); + // Only use huge_pte_lockptr if we are at leaf-level. Otherwise use + // the regular page table lock. + if (hugetlb_pte_none(hpte) || hugetlb_pte_present_leaf(hpte)) + return huge_pte_lockptr(hugetlb_pte_shift(hpte), + mm, hpte->ptep); + return &mm->page_table_lock; +} + +static inline +spinlock_t *hugetlb_pte_lock(struct mm_struct *mm, struct hugetlb_pte *hpte) +{ + spinlock_t *ptl = hugetlb_pte_lockptr(mm, hpte); + + spin_lock(ptl); + return ptl; +} + #if defined(CONFIG_HUGETLB_PAGE) && defined(CONFIG_CMA) extern void __init hugetlb_cma_reserve(int order); extern void __init hugetlb_cma_check(void); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d6d0d4c03def..1a1434e29740 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1120,6 +1120,63 @@ static bool vma_has_reserves(struct vm_area_struct *vma, long chg) return false; } +bool hugetlb_pte_present_leaf(const struct hugetlb_pte *hpte) +{ + pgd_t pgd; + p4d_t p4d; + pud_t pud; + pmd_t pmd; + + BUG_ON(!hpte->ptep); + if (hugetlb_pte_size(hpte) >= PGDIR_SIZE) { + pgd = *(pgd_t *)hpte->ptep; + return pgd_present(pgd) && pgd_leaf(pgd); + } else if (hugetlb_pte_size(hpte) >= P4D_SIZE) { + p4d = *(p4d_t *)hpte->ptep; + return p4d_present(p4d) && p4d_leaf(p4d); + } else if (hugetlb_pte_size(hpte) >= PUD_SIZE) { + pud = *(pud_t *)hpte->ptep; + return pud_present(pud) && pud_leaf(pud); + } else if (hugetlb_pte_size(hpte) >= PMD_SIZE) { + pmd = *(pmd_t *)hpte->ptep; + return pmd_present(pmd) && pmd_leaf(pmd); + } else if (hugetlb_pte_size(hpte) >= PAGE_SIZE) + return pte_present(*hpte->ptep); + BUG(); +} + +bool hugetlb_pte_none(const struct hugetlb_pte *hpte) +{ + if (hugetlb_pte_huge(hpte)) + return huge_pte_none(huge_ptep_get(hpte->ptep)); + return pte_none(ptep_get(hpte->ptep)); +} + +bool hugetlb_pte_none_mostly(const struct hugetlb_pte *hpte) +{ + if (hugetlb_pte_huge(hpte)) + return huge_pte_none_mostly(huge_ptep_get(hpte->ptep)); + return pte_none_mostly(ptep_get(hpte->ptep)); +} + +pte_t hugetlb_ptep_get(const struct hugetlb_pte *hpte) +{ + if (hugetlb_pte_huge(hpte)) + return huge_ptep_get(hpte->ptep); + return ptep_get(hpte->ptep); +} + +void hugetlb_pte_clear(struct mm_struct *mm, const struct hugetlb_pte *hpte, + unsigned long address) +{ + BUG_ON(!hpte->ptep); + unsigned long sz = hugetlb_pte_size(hpte); + + if (sz > PAGE_SIZE) + return huge_pte_clear(mm, address, hpte->ptep, sz); + return pte_clear(mm, address, hpte->ptep); +} + static void enqueue_huge_page(struct hstate *h, struct page *page) { int nid = page_to_nid(page); From patchwork Fri Jun 24 17:36:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894933 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D471C433EF for ; Fri, 24 Jun 2022 17:37:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 00FCC8E0246; Fri, 24 Jun 2022 13:37:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E8C888E0244; Fri, 24 Jun 2022 13:37:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C1BFC8E0246; Fri, 24 Jun 2022 13:37:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9C76D8E0244 for ; Fri, 24 Jun 2022 13:37:21 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 6F12160B6F for ; Fri, 24 Jun 2022 17:37:21 +0000 (UTC) X-FDA: 79613835882.02.CC0E3F4 Received: from mail-ua1-f73.google.com (mail-ua1-f73.google.com [209.85.222.73]) by imf30.hostedemail.com (Postfix) with ESMTP id 134E58002C for ; Fri, 24 Jun 2022 17:37:20 +0000 (UTC) Received: by mail-ua1-f73.google.com with SMTP id b10-20020ab0238a000000b0037efa0a4ba0so1013217uan.11 for ; Fri, 24 Jun 2022 10:37:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=LYP6EWO0tHkDZ/Eb/N4ct/rtumW77ceMKYt6OWO28gw=; b=XFGCLL1FPCDch3s8zoHY1/aiVrNxiWrnh/6o6qV/K2RAwhUG1NoqoYSivh5BlEjSSN JSeDqdD7wmJqUHehWTyabP/SFxoPxLcdmJQciamvw0zZEkmrFOLEfH5MKvY7BMKKaRs5 OeQHOUrdz9iPL7xkNs8N8nJEHEVe58C79l9VExLO0snSG9Lzk432UZi5FNR67YMTpjNb nWD0bA1/rl29atfGjcsEE2x8Ah2JCvGLGQcQO7+KX8SrBMuFX9ufi1FM9t5s4yyq73GD HMYrizZfRIn1go+aVU8Gs1V6ny5gGjt1jqAVjKkGM9IiyAl9gtwolhjj4PEcCHbqvzlV R5nQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=LYP6EWO0tHkDZ/Eb/N4ct/rtumW77ceMKYt6OWO28gw=; b=HDcpaw7sGWCXM4ORuGAsY/YXBwDEQC4iXpTkxq0Dux6Uflfh5JkXIu3YOYc+kZb5rf SB98KZP1TPua+o1szL6VhENeiDoSzcOkwHCT+KGwQRuPt3NGw6gYXs4JKXWP8QjcsuYl ibv521NV8uNSxTJfnjhKJqtACg+I0yymlJi3shc59QiEnoowRSSEwyGFgDi+IuntHzsW 1tnLnWn3llyQILwhlVxAg6vKz2bnIjmU69qwTWWY8gjmBFYmNh8FA1eq75Aeokm1AaHl bA+C0b0RxFaQxbhZ746qaZu6wBq/yuHIBBhoA9Yhrdbo03KNos8txEETpubQD8SRMcSJ QH6A== X-Gm-Message-State: AJIora+o5Jun2LSDciKxsB5OIGz2jylwS44SuQ665deMrekh3xdX1/kj a0EckhfygidGssPj6xuhWqXQM8NJ4XM1jggG X-Google-Smtp-Source: AGRyM1ssMLMIRbwDkEduAqDq4DS/T59s9retjKTDy0iTSci5NwCImR9/ISla7A1bU/FQrm45/Cl8LL0Be0TLQc6l X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6102:374d:b0:34c:1f02:aa59 with SMTP id u13-20020a056102374d00b0034c1f02aa59mr21351211vst.21.1656092240411; Fri, 24 Jun 2022 10:37:20 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:38 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-9-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 08/26] hugetlb: add hugetlb_free_range to free PT structures From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092241; a=rsa-sha256; cv=none; b=Di5n7BI33Of2x62fKDQy5JmJpnGEUhiWE2n1GL2ude5mPUVluTyfgAR3mTN7kCgcMXbFxN ZJB9lGlAO2/kPJpdWiGc596ylEH/+gSioKN+nPZnLMASY4V2rQ0/JKctx0/lH+Ls0Yvyea ksFL0/BjRLa42U5XKLN8CWbpcDBX4rE= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=XFGCLL1F; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of 3UPa1YgoKCDEWgUbhTUgbaTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3UPa1YgoKCDEWgUbhTUgbaTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092241; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LYP6EWO0tHkDZ/Eb/N4ct/rtumW77ceMKYt6OWO28gw=; b=NpgaoNaSdB2rel75vhDUc+dXdnkgxzdy9TONXClFPan7SNO0ucTCkpPZlJrnUGu7YYcXoj 2pW/DEKHa9eL28qHo35CvYehse+tVskHRCrOzOY5Vx6lI+iQ541pSSodb7pN4vsdZsyLJk W0ZrHCnziBvgAZmEcjl3Ablmcv/WIDo= X-Stat-Signature: 1ejqgbhoaztf41ff9jyy5qhnsefr49ym X-Rspamd-Queue-Id: 134E58002C Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=XFGCLL1F; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of 3UPa1YgoKCDEWgUbhTUgbaTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3UPa1YgoKCDEWgUbhTUgbaTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--jthoughton.bounces.google.com X-Rspamd-Server: rspam10 X-Rspam-User: X-HE-Tag: 1656092240-564722 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a helper function for freeing the bits of the page table that map a particular HugeTLB PTE. Signed-off-by: James Houghton --- include/linux/hugetlb.h | 2 ++ mm/hugetlb.c | 17 +++++++++++++++++ 2 files changed, 19 insertions(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 1d4ec9dfdebf..33ba48fac551 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -107,6 +107,8 @@ bool hugetlb_pte_none_mostly(const struct hugetlb_pte *hpte); pte_t hugetlb_ptep_get(const struct hugetlb_pte *hpte); void hugetlb_pte_clear(struct mm_struct *mm, const struct hugetlb_pte *hpte, unsigned long address); +void hugetlb_free_range(struct mmu_gather *tlb, const struct hugetlb_pte *hpte, + unsigned long start, unsigned long end); struct hugepage_subpool { spinlock_t lock; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 1a1434e29740..a2d2ffa76173 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1120,6 +1120,23 @@ static bool vma_has_reserves(struct vm_area_struct *vma, long chg) return false; } +void hugetlb_free_range(struct mmu_gather *tlb, const struct hugetlb_pte *hpte, + unsigned long start, unsigned long end) +{ + unsigned long floor = start & hugetlb_pte_mask(hpte); + unsigned long ceiling = floor + hugetlb_pte_size(hpte); + + if (hugetlb_pte_size(hpte) >= PGDIR_SIZE) { + free_p4d_range(tlb, (pgd_t *)hpte->ptep, start, end, floor, ceiling); + } else if (hugetlb_pte_size(hpte) >= P4D_SIZE) { + free_pud_range(tlb, (p4d_t *)hpte->ptep, start, end, floor, ceiling); + } else if (hugetlb_pte_size(hpte) >= PUD_SIZE) { + free_pmd_range(tlb, (pud_t *)hpte->ptep, start, end, floor, ceiling); + } else if (hugetlb_pte_size(hpte) >= PMD_SIZE) { + free_pte_range(tlb, (pmd_t *)hpte->ptep, start); + } +} + bool hugetlb_pte_present_leaf(const struct hugetlb_pte *hpte) { pgd_t pgd; From patchwork Fri Jun 24 17:36:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894934 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60650C433EF for ; Fri, 24 Jun 2022 17:37:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9D2338E0247; Fri, 24 Jun 2022 13:37:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7B94E8E0244; Fri, 24 Jun 2022 13:37:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E14D8E0247; Fri, 24 Jun 2022 13:37:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1ED408E0244 for ; Fri, 24 Jun 2022 13:37:23 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 00ABB806C5 for ; Fri, 24 Jun 2022 17:37:22 +0000 (UTC) X-FDA: 79613835966.19.5C66043 Received: from mail-ua1-f73.google.com (mail-ua1-f73.google.com [209.85.222.73]) by imf30.hostedemail.com (Postfix) with ESMTP id 78EB88002C for ; Fri, 24 Jun 2022 17:37:22 +0000 (UTC) Received: by mail-ua1-f73.google.com with SMTP id b10-20020ab0238a000000b0037efa0a4ba0so1013217uan.11 for ; Fri, 24 Jun 2022 10:37:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=WX+TLc+c32jY73zAmELqfC7scmTotnHMJPgbPcH9Hag=; b=rmtQIA0RTPQYWEYJ8kZRSAxO4ZWdbskuHARii0tUqF87tsg4HJnPzpTHsKmHhi7YHN szZo8r74Pfgxru6pGunO7KfxJtr6Wzabwa1wcCmLVAdMbm+j8fKHC5fV7wwrJG+ncVrB fWoXECRaKY9gz8ZiGg0zDe612OVnVf2pAh18JbfgZ2OytbBwiZCNN/oOGXrLiF3q3rzH fxFhSCJMn3o5octv4QkArXh3RvJqGO05jZSaGjrL9vscpWTsOFD0x47zLoXDuqUkCgEo PIWSrdAA+02ET/Ct0BrCGPnvyDWs7lIPDo0hPsnRg76cZrOaj70ey7iZ/86fIxktGj4r ckXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=WX+TLc+c32jY73zAmELqfC7scmTotnHMJPgbPcH9Hag=; b=VCb4b9ldy2YIOsnRtAQ2OHJTXWbeTksJWDsc8cXZx3RT9SXXJetDPR1pKRCOKPrTNP TbWs3v7Q5kI7waDjZ9Q+bgkrqRGGMlgUdFPVmmDgu/k7CEv2lRo5wR74MtyrPypeW3qi 9Jbkquzi786wM+lkmGH5J+fwkH7YQI1A8jhMBlzuC6KomnjGFkttgtexSj+bPZXZll4g 28sDhYImI/9cqMF1Z3KnU+Po1hSOTpnLPp09EG6TrSf2lJ6r4irh/6l43GH8PmyO9Vdr PHSpaPpgGaTC479b10IKc+Jg2vor/6IQsxOdW+MX46fMCj/kI14W6QomWzJ4PVKqrfJA Syqw== X-Gm-Message-State: AJIora+0uMSLlfE3UWBWIaL3TlojWa3REU+1gGUNlibDvdZMbMUCYndF Any4pICmqWnLqzMMK8X2azuNkrNMYQiRebsL X-Google-Smtp-Source: AGRyM1vX/wsU1Y9GWc4aZF68+l300w5Pdum+7CFc6GFKGc4oec+e4PYz/yVOvKtQiR6acb6V/yx2DV4xq/fD/H/b X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a67:6e47:0:b0:354:52a0:8ac2 with SMTP id j68-20020a676e47000000b0035452a08ac2mr173638vsc.65.1656092242031; Fri, 24 Jun 2022 10:37:22 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:39 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-10-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 09/26] hugetlb: add hugetlb_hgm_enabled From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092242; a=rsa-sha256; cv=none; b=Bf23W/yqRwnYbJceaBMMOILi8Uu0Knfn99I+aJpfWS4uB430dCmd4oWVif6dtOfDK8Ybe5 +UvKz8/tnsMgM1ojZLkhNRqkkAAY8TEoo8M4Kyp2P2WEuZWuJLwXY1STr5lD+d+7iYsXr4 h1ZOvoWw17gTxz0cr+aOnsbL0ocJOVY= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=rmtQIA0R; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of 3Uva1YgoKCDMYiWdjVWidcVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3Uva1YgoKCDMYiWdjVWidcVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092242; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WX+TLc+c32jY73zAmELqfC7scmTotnHMJPgbPcH9Hag=; b=U/RwMQz1V1aO0JcnzTrMJ88UO/o1yS2IxukSRbMyvo1MFPV+MjDhfuZVTZ/xqyovojtOmt yppqSs4Qw4sUhRXgauJO6amuBcSB5EkIjhAMyl6HH/iwjkEnCXTn8bX5iQL0jobUejGt7T Fs23/ZaEXZih0d51m0+Zx5RUUw6DU5c= X-Stat-Signature: 5fbmkcp189w3k3hfzp3bifhfcsekg77n X-Rspamd-Queue-Id: 78EB88002C Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=rmtQIA0R; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of 3Uva1YgoKCDMYiWdjVWidcVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3Uva1YgoKCDMYiWdjVWidcVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--jthoughton.bounces.google.com X-Rspamd-Server: rspam10 X-Rspam-User: X-HE-Tag: 1656092242-264808 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, this is always true if the VMA is shared. In the future, it's possible that private mappings will get some or all HGM functionality. Signed-off-by: James Houghton reviewed-by: manish.mishra@nutanix.com Reviewed-by: Mina Almasry --- include/linux/hugetlb.h | 10 ++++++++++ mm/hugetlb.c | 8 ++++++++ 2 files changed, 18 insertions(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 33ba48fac551..e7a6b944d0cc 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -1174,6 +1174,16 @@ static inline void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, } #endif /* CONFIG_HUGETLB_PAGE */ +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +/* If HugeTLB high-granularity mappings are enabled for this VMA. */ +bool hugetlb_hgm_enabled(struct vm_area_struct *vma); +#else +static inline bool hugetlb_hgm_enabled(struct vm_area_struct *vma) +{ + return false; +} +#endif + static inline spinlock_t *huge_pte_lock(struct hstate *h, struct mm_struct *mm, pte_t *pte) { diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a2d2ffa76173..8b10b941458d 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6983,6 +6983,14 @@ pte_t *huge_pte_offset(struct mm_struct *mm, #endif /* CONFIG_ARCH_WANT_GENERAL_HUGETLB */ +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING +bool hugetlb_hgm_enabled(struct vm_area_struct *vma) +{ + /* All shared VMAs have HGM enabled. */ + return vma->vm_flags & VM_SHARED; +} +#endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */ + /* * These functions are overwritable if your architecture needs its own * behavior. From patchwork Fri Jun 24 17:36:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894935 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1FCCC43334 for ; Fri, 24 Jun 2022 17:37:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 69FDE8E0248; Fri, 24 Jun 2022 13:37:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4043A8E0244; Fri, 24 Jun 2022 13:37:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1B94A8E0248; Fri, 24 Jun 2022 13:37:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E63808E0244 for ; Fri, 24 Jun 2022 13:37:24 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id BB2FC35296 for ; Fri, 24 Jun 2022 17:37:24 +0000 (UTC) X-FDA: 79613836008.10.FCF562C Received: from mail-vk1-f202.google.com (mail-vk1-f202.google.com [209.85.221.202]) by imf12.hostedemail.com (Postfix) with ESMTP id 5C95240019 for ; Fri, 24 Jun 2022 17:37:24 +0000 (UTC) Received: by mail-vk1-f202.google.com with SMTP id p185-20020a1fd8c2000000b0036c453f2ea4so918053vkg.18 for ; Fri, 24 Jun 2022 10:37:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=BHwQikGJ0T9dngSuqajvJzRNJRNB+p1hBOZL6v8gtdU=; b=SJaLm0XrTpsH4nsCpNczOnWA4pAKrfnpfCQD7J/A/w8ZLuyF3qMil1lg2kFawB2M4a itbVC6Cy96FvwmkCOrHBppgxya0l4XK+nbemqmVGl2TtO1nxOedDpeGypnCjJRZS7QQN NDC0hCezqTOYHNT6HpGvGH8b8GjOJaayBlAgzsQ6R71RrjTlRa2+jtXbsXNhdPHlJ6Xj Bf0Xtsg/vvdXkEZqCE7ZRTpn6YWG/VddVaye5OHuJMLgDbbZwZkgCRO6djsmySZP8iDS KKIDvmk/HJlHB4lLRHC4SCJoBQ4oZp4rMf+AAsSXjtVnRk/loJUaiEBO6jLZd1OibD+8 oWQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=BHwQikGJ0T9dngSuqajvJzRNJRNB+p1hBOZL6v8gtdU=; b=JUnaE0FVRMUxBoyJ3d6Ajoz6kK2FmgbyicBtPsLUkbmAeaXfCnxiVaFXvrpZMJRxWa au56A8CcVLdpYaIOeyI3BYlcMRewIkp7/Jm8apKdcyy9EvPGWOHTbVItgm57t7sFGVnn 8d69m9P6+CgoFyZIjaEQlRx4UMIxyQiCcvtKVB5R2KPhtAkYpsCBuqKN2/FXsCT7nKhR phATEuOJY1i5SsdseGThoyqQoxzspdlbsb1PSIwqgeah7ICvF517bhLvWnlXFQtSKyKj 11syfCvpbZZkfI8okFfZ4H2kB5gzPwQ3Bv8jvP6i9hHq/gh2TnuQhyK5SsZeSItRbL1a PT6Q== X-Gm-Message-State: AJIora9jemVFJFHemJRE5dpeZLeImdbM0fcaFZd4qR8ucyzfTY7cUo+x xpV4C9neHvkD0RhUPPutQSZnT0VsMKU/4R6I X-Google-Smtp-Source: AGRyM1vwAeqnR6McfUFoedAHpp3M9GTRsmGOf0kSjUYrMb74mEmizd7GUpekp5x9kN5nii0pCQDDYM1i4RCw9Zq8 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a67:d785:0:b0:354:355b:60cd with SMTP id q5-20020a67d785000000b00354355b60cdmr14256935vsj.65.1656092243596; Fri, 24 Jun 2022 10:37:23 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:40 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-11-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 10/26] hugetlb: add for_each_hgm_shift From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=SJaLm0Xr; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3U_a1YgoKCDQZjXekWXjedWeeWbU.SecbYdkn-ccalQSa.ehW@flex--jthoughton.bounces.google.com designates 209.85.221.202 as permitted sender) smtp.mailfrom=3U_a1YgoKCDQZjXekWXjedWeeWbU.SecbYdkn-ccalQSa.ehW@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092244; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BHwQikGJ0T9dngSuqajvJzRNJRNB+p1hBOZL6v8gtdU=; b=dJQrf05ls5K263dIbr3X1K0tJkr6eC+ajGanplrdJymGifURw2wZPpltTTqO2nKt06guLq 3TwJKvN1hqTbYo2hibMvl3z/+xH3xuiwMVO9zZ2h2BqYAZcMDU8srnm2xBMSqEYHaziKBb CpRO++r3sGkgduJd4PmyYDdmhA0V4Bs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092244; a=rsa-sha256; cv=none; b=GD1FohfoUvv5K4HQ1hy1O4Dr72L88hNS5xendNh5//pL0m3UbDhoyo7F8INCobQgZkusHB 7HpmYmNssjOxAR8k6JGFp8D8sugMDFKyjoPxspw8dGud9bhJCUUJkZwcd1JOiwOKsNdK6J MAp5Makakzz2C/W7i3N2Wg/gxbO+QGc= X-Stat-Signature: ni8aiim9ez44zct6uycyhd5eherd7e4i X-Rspamd-Queue-Id: 5C95240019 X-Rspam-User: Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=SJaLm0Xr; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3U_a1YgoKCDQZjXekWXjedWeeWbU.SecbYdkn-ccalQSa.ehW@flex--jthoughton.bounces.google.com designates 209.85.221.202 as permitted sender) smtp.mailfrom=3U_a1YgoKCDQZjXekWXjedWeeWbU.SecbYdkn-ccalQSa.ehW@flex--jthoughton.bounces.google.com X-Rspamd-Server: rspam12 X-HE-Tag: 1656092244-176499 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a helper macro to loop through all the usable page sizes for a high-granularity-enabled HugeTLB VMA. Given the VMA's hstate, it will loop, in descending order, through the page sizes that HugeTLB supports for this architecture; it always includes PAGE_SIZE. Signed-off-by: James Houghton reviewed-by:manish.mishra@nutanix.com --- mm/hugetlb.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 8b10b941458d..557b0afdb503 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6989,6 +6989,16 @@ bool hugetlb_hgm_enabled(struct vm_area_struct *vma) /* All shared VMAs have HGM enabled. */ return vma->vm_flags & VM_SHARED; } +static unsigned int __shift_for_hstate(struct hstate *h) +{ + if (h >= &hstates[hugetlb_max_hstate]) + return PAGE_SHIFT; + return huge_page_shift(h); +} +#define for_each_hgm_shift(hstate, tmp_h, shift) \ + for ((tmp_h) = hstate; (shift) = __shift_for_hstate(tmp_h), \ + (tmp_h) <= &hstates[hugetlb_max_hstate]; \ + (tmp_h)++) #endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */ /* From patchwork Fri Jun 24 17:36:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894936 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E711CCCA473 for ; Fri, 24 Jun 2022 17:37:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA2108E0249; Fri, 24 Jun 2022 13:37:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CDF4D8E0244; Fri, 24 Jun 2022 13:37:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A6C128E0249; Fri, 24 Jun 2022 13:37:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7D5DD8E0244 for ; Fri, 24 Jun 2022 13:37:26 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 59DB1B70 for ; Fri, 24 Jun 2022 17:37:26 +0000 (UTC) X-FDA: 79613836092.05.9DC1C7B Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf11.hostedemail.com (Postfix) with ESMTP id E2D0A4002A for ; Fri, 24 Jun 2022 17:37:25 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-317b6ecba61so27047867b3.9 for ; Fri, 24 Jun 2022 10:37:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=5N1QqSZZ3sAe7uqZbPvcX1MwgZtWBTxE3Z4UbaM2Ubk=; b=q+mTw9aXiewEEPn69Lo/c9WP/AOoUrPORPuTtAp49sBVoij1KAB/iU1Xsmfcz2RONi 97ZReXD54tQvzGoEYGUk7bqiHAMjMBdSDeN0ZeRlmES9GnpySAERBNIAd4xcYCwSqWB9 7rTzJ5T7Icb9PLM1GoXpIL2LTZ8f3PW3xI9Uskzsm3fmgHe4+UJzlKEwXBwzjTskVRhP jO0DeJtpPQFYq6QRquH/FNG3s47JBo454bcpHh+Nv3bPhNDZ/+ST1no2b8ufY9BNObOU 06CUcUjjHu4D3yaQIgx4SDUgySvd1FhGM2vQVwgOwraU6Bc5o7byEvf4kBsFdHoToBy1 X8+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=5N1QqSZZ3sAe7uqZbPvcX1MwgZtWBTxE3Z4UbaM2Ubk=; b=vkusBbyUxlk5QJnPPUyUZSWWWAOaxshLnFosagI90D9nXzSczo+9Ff7lz4FvQZA0mT 8xHA284anyILFFQ3pNh4ojteWymyrz3BhYev2Tw1mwv2TBxc7U4OmSY1wlWBzwGQK9JQ +w0fy6mMgvEoh7FQeF88j+KugV6R+yZdq2OYuvOW638wcGQ43mkMS8UJ6+lzp7N7uKzm OOmYhPX5uEt7B+z5YDNKhWITbkIcp+1CrmhPE1cRlsL0vL87vQ0WI0rWwXIcFQc8WK8L EJUT1ghdZ8Sc3udx6D1PR7xQSqkQGZiXVamZ1W3CS93CbHC7dHyq2QC7PrlgPaCopcw9 cP8Q== X-Gm-Message-State: AJIora/W8UrE1fbhkUz3TFzNuMdpLjm0hb9JFnnXfsHDnwMEGdbzy/bO kJBii1Fiyne2hRvCSbPp+X5fWHlxstigJMxz X-Google-Smtp-Source: AGRyM1s7zrxwUtB89vtQHWN8oyQ64yZ//LRbGigtUbZGCP3Sqfb2xmvuSoQ6VmyGFOOMey8Bgk7vqBsKoi9rXZ66 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a25:9103:0:b0:668:b7f8:b428 with SMTP id v3-20020a259103000000b00668b7f8b428mr309298ybl.270.1656092245324; Fri, 24 Jun 2022 10:37:25 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:41 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-12-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 11/26] hugetlb: add hugetlb_walk_to to do PT walks From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092245; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5N1QqSZZ3sAe7uqZbPvcX1MwgZtWBTxE3Z4UbaM2Ubk=; b=KWrtP+3+TUpHUuAmAeoe1MPU0qS/Ylm443savQI6oSAZluPl8M/SibYqAAyU087U8bLgHG Y9yS8kKVtnWnlLGxIfmiiTwo+YMGKPA5CcOeLDospkhWFA2Gy/9P/bCoF54o7cba3jQOLA BbVA7V2UDfjTVCFJGyvFr8GnTkybHIs= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=q+mTw9aX; spf=pass (imf11.hostedemail.com: domain of 3Vfa1YgoKCDYblZgmYZlgfYggYdW.Ugedafmp-eecnSUc.gjY@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Vfa1YgoKCDYblZgmYZlgfYggYdW.Ugedafmp-eecnSUc.gjY@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092246; a=rsa-sha256; cv=none; b=sNvdvQMETG5lTvXZTAwywSv/bFx0taw6bya/Yg9ycuSA6UEtCCmQNGaNepJUjVRgRQsWIB zlxxiPQWGtHNILAwap+XtBwwCMlWGdHQkd6dE55VHeC4bqcKIp1Hko4Wj2kXJI9IjZ3r3e BdF2PXnKHW0HpyNKSxvin4ya8H5s9uA= Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=q+mTw9aX; spf=pass (imf11.hostedemail.com: domain of 3Vfa1YgoKCDYblZgmYZlgfYggYdW.Ugedafmp-eecnSUc.gjY@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Vfa1YgoKCDYblZgmYZlgfYggYdW.Ugedafmp-eecnSUc.gjY@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam06 X-Stat-Signature: 97n597t99mqz9iyfctxboszeopsrn41s X-Rspamd-Queue-Id: E2D0A4002A X-HE-Tag: 1656092245-364778 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This adds it for architectures that use GENERAL_HUGETLB, including x86. Signed-off-by: James Houghton --- include/linux/hugetlb.h | 2 ++ mm/hugetlb.c | 45 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index e7a6b944d0cc..605aa19d8572 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -258,6 +258,8 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned long sz); pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long sz); +int hugetlb_walk_to(struct mm_struct *mm, struct hugetlb_pte *hpte, + unsigned long addr, unsigned long sz, bool stop_at_none); int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long *addr, pte_t *ptep); void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 557b0afdb503..3ec2a921ee6f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6981,6 +6981,51 @@ pte_t *huge_pte_offset(struct mm_struct *mm, return (pte_t *)pmd; } +int hugetlb_walk_to(struct mm_struct *mm, struct hugetlb_pte *hpte, + unsigned long addr, unsigned long sz, bool stop_at_none) +{ + pte_t *ptep; + + if (!hpte->ptep) { + pgd_t *pgd = pgd_offset(mm, addr); + + if (!pgd) + return -ENOMEM; + ptep = (pte_t *)p4d_alloc(mm, pgd, addr); + if (!ptep) + return -ENOMEM; + hugetlb_pte_populate(hpte, ptep, P4D_SHIFT); + } + + while (hugetlb_pte_size(hpte) > sz && + !hugetlb_pte_present_leaf(hpte) && + !(stop_at_none && hugetlb_pte_none(hpte))) { + if (hpte->shift == PMD_SHIFT) { + ptep = pte_alloc_map(mm, (pmd_t *)hpte->ptep, addr); + if (!ptep) + return -ENOMEM; + hpte->shift = PAGE_SHIFT; + hpte->ptep = ptep; + } else if (hpte->shift == PUD_SHIFT) { + ptep = (pte_t *)pmd_alloc(mm, (pud_t *)hpte->ptep, + addr); + if (!ptep) + return -ENOMEM; + hpte->shift = PMD_SHIFT; + hpte->ptep = ptep; + } else if (hpte->shift == P4D_SHIFT) { + ptep = (pte_t *)pud_alloc(mm, (p4d_t *)hpte->ptep, + addr); + if (!ptep) + return -ENOMEM; + hpte->shift = PUD_SHIFT; + hpte->ptep = ptep; + } else + BUG(); + } + return 0; +} + #endif /* CONFIG_ARCH_WANT_GENERAL_HUGETLB */ #ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING From patchwork Fri Jun 24 17:36:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894937 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52669C43334 for ; Fri, 24 Jun 2022 17:37:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C0638E024A; Fri, 24 Jun 2022 13:37:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 826568E0244; Fri, 24 Jun 2022 13:37:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 626888E024A; Fri, 24 Jun 2022 13:37:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 47F1F8E0244 for ; Fri, 24 Jun 2022 13:37:28 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id CF71033577 for ; Fri, 24 Jun 2022 17:37:27 +0000 (UTC) X-FDA: 79613836134.20.C624E99 Received: from mail-ua1-f73.google.com (mail-ua1-f73.google.com [209.85.222.73]) by imf12.hostedemail.com (Postfix) with ESMTP id 6FDBC4001D for ; Fri, 24 Jun 2022 17:37:27 +0000 (UTC) Received: by mail-ua1-f73.google.com with SMTP id l7-20020ab053c7000000b003791afd560cso1010426uaa.21 for ; Fri, 24 Jun 2022 10:37:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=9NP33zy0A1MX3vEGrpVIKeFhmsIPx2kWV4buJ37voF4=; b=ZKOKH7vAAREURJ1l8IWF0iXVyZJmt/59w23N609lJlz66JMzYMime9PnmzUaf55ztT vzLMK8s9/fXSSxWnYgNDHKf7LqeMXxNL7KAl5SuDjU3NQJJ60yl5tOO73MRzJCUdunKz 7Lgn3pxPgvpWylL4Lwm2BTczrJtzLYwEI/PWM2Cho00mXJipoxojyLs3eWxEQPPDnMB8 Oxl49ZR9JG0HGfk3WVBD25dYMN0YwU/lye70tN93BwsbS6nd07nY55SUbsCU9gTsjkxw 9k4o1zcXpunPWXdWooquHfqVjKqZmpuvtPY3r1RM36B/t5rz9uVN89mK3zfIIDtHV9hY WUTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=9NP33zy0A1MX3vEGrpVIKeFhmsIPx2kWV4buJ37voF4=; b=qN16YwzqyjcV1hV5BuXMHtCai7/S0tuEOIrtIKppfDyve0l9o2BYjqz/c1YyIoSZ+g 0FdCToqCzbZUMLRnHcHYymmMIXQPuYirhsVN/MNHa2pPTGPk85EwPQ82mVxtqtR9oeJv FUM7tBAKTg5UVDaolFBGYglplIZRJta7g/eng4hLHsV9nVxpfZStsQh9hvjnKcRiIveT 6hEJvS3Rm5xTB2Xr7Lo8Wl2paP0nGjz/ytzz/qJhVvc1rqoQ0kqjoWWlIh0rzGukCvlO YPwXaIwI0OkHVWqgCxVeKNeydSiZHFhp3FIoqIkT8lVGy+hxpyY0A/L7UhG1/KGqV3GU 3uWQ== X-Gm-Message-State: AJIora+6MCeAYm5OiwR3pMosCVUg22uhic+RjcflwBtbFHbUHX6SpizR BFOIduJoDjp5w77v54kLw1SRexZ/79lGvO7j X-Google-Smtp-Source: AGRyM1ud2VpyFSoYe3jfqlLqsCZHAGiyta2WrrFtmqkuRYO0f2uYFJE5sTDJIrzr1fNDbhgGC1mFcexvMmBaf2W2 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a67:ef4a:0:b0:354:6546:8b97 with SMTP id k10-20020a67ef4a000000b0035465468b97mr8260702vsr.83.1656092246792; Fri, 24 Jun 2022 10:37:26 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:42 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-13-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 12/26] hugetlb: add HugeTLB splitting functionality From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092247; a=rsa-sha256; cv=none; b=sX0K3abSYvcomtUXOWYEUaaO1PFXaT9aPV7oo4tAGfOwm/DejMASY2BVrpsebNeLxBP2h7 YizpVOS7TEHKzhJjcDRTuY2drx0VjDV6GXAFx0WL8qS5WObA0Bya/zz/WAyvyE3aAOYxcT Po5LXT8+hthbLw06LmX2p6s8+CFgBok= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092247; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9NP33zy0A1MX3vEGrpVIKeFhmsIPx2kWV4buJ37voF4=; b=R7m4jA7d3VXulY24sGsGdiVgfcUHtYIyOQxNsxUhFd37Q8UwizyEAzIxOGFWHdDKLhZSA2 6Bfk0VjswUWNENpSyHlHZSf0vzLP5TcVnywXc14KFFkhBqQq+QYjy6CJiVc0xKtUV7iN4Y L9dzNtiZmIDRDC2ISmo5dJHhMtpiMik= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ZKOKH7vA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3Vva1YgoKCDccmahnZamhgZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3Vva1YgoKCDccmahnZamhgZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--jthoughton.bounces.google.com X-Stat-Signature: atqtih7exrbfpnjp177nxxr3dumrji69 X-Rspamd-Queue-Id: 6FDBC4001D Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ZKOKH7vA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3Vva1YgoKCDccmahnZamhgZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3Vva1YgoKCDccmahnZamhgZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--jthoughton.bounces.google.com X-Rspam-User: X-Rspamd-Server: rspam04 X-HE-Tag: 1656092247-666214 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The new function, hugetlb_split_to_shift, will optimally split the page table to map a particular address at a particular granularity. This is useful for punching a hole in the mapping and for mapping small sections of a HugeTLB page (via UFFDIO_CONTINUE, for example). Signed-off-by: James Houghton --- mm/hugetlb.c | 122 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 122 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 3ec2a921ee6f..eaffe7b4f67c 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -102,6 +102,18 @@ struct mutex *hugetlb_fault_mutex_table ____cacheline_aligned_in_smp; /* Forward declaration */ static int hugetlb_acct_memory(struct hstate *h, long delta); +/* + * Find the subpage that corresponds to `addr` in `hpage`. + */ +static struct page *hugetlb_find_subpage(struct hstate *h, struct page *hpage, + unsigned long addr) +{ + size_t idx = (addr & ~huge_page_mask(h))/PAGE_SIZE; + + BUG_ON(idx >= pages_per_huge_page(h)); + return &hpage[idx]; +} + static inline bool subpool_is_free(struct hugepage_subpool *spool) { if (spool->count) @@ -7044,6 +7056,116 @@ static unsigned int __shift_for_hstate(struct hstate *h) for ((tmp_h) = hstate; (shift) = __shift_for_hstate(tmp_h), \ (tmp_h) <= &hstates[hugetlb_max_hstate]; \ (tmp_h)++) + +/* + * Given a particular address, split the HugeTLB PTE that currently maps it + * so that, for the given address, the PTE that maps it is `desired_shift`. + * This function will always split the HugeTLB PTE optimally. + * + * For example, given a HugeTLB 1G page that is mapped from VA 0 to 1G. If we + * call this function with addr=0 and desired_shift=PAGE_SHIFT, will result in + * these changes to the page table: + * 1. The PUD will be split into 2M PMDs. + * 2. The first PMD will be split again into 4K PTEs. + */ +static int hugetlb_split_to_shift(struct mm_struct *mm, struct vm_area_struct *vma, + const struct hugetlb_pte *hpte, + unsigned long addr, unsigned long desired_shift) +{ + unsigned long start, end, curr; + unsigned long desired_sz = 1UL << desired_shift; + struct hstate *h = hstate_vma(vma); + int ret; + struct hugetlb_pte new_hpte; + struct mmu_notifier_range range; + struct page *hpage = NULL; + struct page *subpage; + pte_t old_entry; + struct mmu_gather tlb; + + BUG_ON(!hpte->ptep); + BUG_ON(hugetlb_pte_size(hpte) == desired_sz); + + start = addr & hugetlb_pte_mask(hpte); + end = start + hugetlb_pte_size(hpte); + + i_mmap_assert_write_locked(vma->vm_file->f_mapping); + + BUG_ON(!hpte->ptep); + /* This function only works if we are looking at a leaf-level PTE. */ + BUG_ON(!hugetlb_pte_none(hpte) && !hugetlb_pte_present_leaf(hpte)); + + /* + * Clear the PTE so that we will allocate the PT structures when + * walking the page table. + */ + old_entry = huge_ptep_get_and_clear(mm, start, hpte->ptep); + + if (!huge_pte_none(old_entry)) + hpage = pte_page(old_entry); + + BUG_ON(!IS_ALIGNED(start, desired_sz)); + BUG_ON(!IS_ALIGNED(end, desired_sz)); + + for (curr = start; curr < end;) { + struct hstate *tmp_h; + unsigned int shift; + + for_each_hgm_shift(h, tmp_h, shift) { + unsigned long sz = 1UL << shift; + + if (!IS_ALIGNED(curr, sz) || curr + sz > end) + continue; + /* + * If we are including `addr`, we need to make sure + * splitting down to the correct size. Go to a smaller + * size if we are not. + */ + if (curr <= addr && curr + sz > addr && + shift > desired_shift) + continue; + + /* + * Continue the page table walk to the level we want, + * allocate PT structures as we go. + */ + hugetlb_pte_copy(&new_hpte, hpte); + ret = hugetlb_walk_to(mm, &new_hpte, curr, sz, + /*stop_at_none=*/false); + if (ret) + goto err; + BUG_ON(hugetlb_pte_size(&new_hpte) != sz); + if (hpage) { + pte_t new_entry; + + subpage = hugetlb_find_subpage(h, hpage, curr); + new_entry = make_huge_pte_with_shift(vma, subpage, + huge_pte_write(old_entry), + shift); + set_huge_pte_at(mm, curr, new_hpte.ptep, new_entry); + } + curr += sz; + goto next; + } + /* We couldn't find a size that worked. */ + BUG(); +next: + continue; + } + + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, vma->vm_mm, + start, end); + mmu_notifier_invalidate_range_start(&range); + return 0; +err: + tlb_gather_mmu(&tlb, mm); + /* Free any newly allocated page table entries. */ + hugetlb_free_range(&tlb, hpte, start, curr); + /* Restore the old entry. */ + set_huge_pte_at(mm, start, hpte->ptep, old_entry); + tlb_finish_mmu(&tlb); + return ret; +} #endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */ /* From patchwork Fri Jun 24 17:36:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894938 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C950FCCA47E for ; Fri, 24 Jun 2022 17:37:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A47BD8E024B; Fri, 24 Jun 2022 13:37:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9AA358E0244; Fri, 24 Jun 2022 13:37:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7FB698E024B; Fri, 24 Jun 2022 13:37:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5C9808E0244 for ; Fri, 24 Jun 2022 13:37:29 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3D93720749 for ; Fri, 24 Jun 2022 17:37:29 +0000 (UTC) X-FDA: 79613836218.15.BCB0C44 Received: from mail-ua1-f74.google.com (mail-ua1-f74.google.com [209.85.222.74]) by imf06.hostedemail.com (Postfix) with ESMTP id D2C411800A5 for ; Fri, 24 Jun 2022 17:37:28 +0000 (UTC) Received: by mail-ua1-f74.google.com with SMTP id v14-20020ab0768e000000b0037efa637aeeso1007027uaq.23 for ; Fri, 24 Jun 2022 10:37:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=fLhWVrtL3jCVQZTI/SpR/dF9bcFaOpNW6kUek+4VMrQ=; b=h/ysFkhblpzJ+iMXOAuoAW1RCDzHD/+S02IOwgDp+W1ftcj/5GpmKopNXVJGmI11dt mAS2/ecnjScCvn9y+SSrPgkps+68W5ymor9ZeqP24UdBQp7MkKwq9sGbZh9bcS3Kot12 BGqJ5gfYWmse/+yrqaROihgM78qPJlOL2tffwnW0DZUm7UP9c6M6BVGl7wnznm72cE8e Gmgcs44D/bkYNixg8jOWwZMa5AJhJKIcbwa6ooNOCsk7KyEQSYEcelE/7NURJzuTXfld QKmlDWTSdSuaR2EkI87u4H/5x0GVEh50Cju+5MyrG72wa/G6IgHcBFf2GIqziepRDeTo ynXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=fLhWVrtL3jCVQZTI/SpR/dF9bcFaOpNW6kUek+4VMrQ=; b=PsuDsaeFlj2v+MaFRpRyDfFiyj/S1nSDdsr9hj7M2Wv5vfJCsk73CjbZvCy78w22Dk FLDU7baBw4GoMhk0axY0UnJg1h//up127oKc/+yBTgEW7JRey3baD+lDx8KQOsaUCGij rBxnqxSAJbTEclTC485EUQv7/WbabBOQAcuRGV1XgM6w1mOwBNaG57XG4bvf+qcVVINm 8hN6WHy+2rh/hdg+YyKZ0wGXCoNVR6lZCOpPuiU7cVKHtOhYCh+SGJWtX4CTgGeY/g+7 hmGGUrUCbVpOAqRWfUmZ1EwqDzecL17EQl76fR0/3eVZ0GWKi0l9C3uDs9vH3nhpEoXO TBcw== X-Gm-Message-State: AJIora/G1awtQ4NygXic/QNyIPUFOX0dJNFHGf4sqsXTACMvv6ss7cy0 NEYoi4Svov+YMwyR2wqwSwNOr0iPxV3w+sFY X-Google-Smtp-Source: AGRyM1uRSxwvI1ZalFt+ckvzYpzesshV4sVRNlb4t9wpXcehiUcvJo2Hg3rBOz88Imnv+bFCKqP3jsH6tlItOe1M X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a67:c30f:0:b0:354:45bf:748c with SMTP id r15-20020a67c30f000000b0035445bf748cmr351vsj.13.1656092248216; Fri, 24 Jun 2022 10:37:28 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:43 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-14-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 13/26] hugetlb: add huge_pte_alloc_high_granularity From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="h/ysFkhb"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 3WPa1YgoKCDkeocjpbcojibjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--jthoughton.bounces.google.com designates 209.85.222.74 as permitted sender) smtp.mailfrom=3WPa1YgoKCDkeocjpbcojibjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092248; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fLhWVrtL3jCVQZTI/SpR/dF9bcFaOpNW6kUek+4VMrQ=; b=PGf/UoUgJNDF5qc/p59ZifG69ErDa7IHZMHbLu6kQ/s6BWYdWzZmTutF1Yk6xqEpcNJsOB 6GKYSG/yuDBczsfDEhdCAhTb1j6cvXCGabzx4iv/RR+0yFNDLdyDyPb/EFusxP+hvigxsy 8YERIvNbSv4bfEvmUkkZyNKXAGJJGds= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092248; a=rsa-sha256; cv=none; b=50fqXejmGqPb9OV8Uzn7l7xdmfI/pqCG5a+1N0YrJDEvPiIKoXR4Dzji9fuTr2aE6/hVFB gU/N5K9Mv758+P9MHutMcgs+9uQIZceLQXlAZcry/6SGdj5KeSEXw+sEzixYgSHfffEg1F p8UnBuJ+OPLQBa5t7Z8O+b+JWyKXcuo= X-Stat-Signature: bkfgzyxxi65k46x5exeew8ddnshqdc1z X-Rspamd-Queue-Id: D2C411800A5 X-Rspam-User: Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="h/ysFkhb"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 3WPa1YgoKCDkeocjpbcojibjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--jthoughton.bounces.google.com designates 209.85.222.74 as permitted sender) smtp.mailfrom=3WPa1YgoKCDkeocjpbcojibjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--jthoughton.bounces.google.com X-Rspamd-Server: rspam12 X-HE-Tag: 1656092248-939071 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This function is to be used to do a HugeTLB page table walk where we may need to split a leaf-level huge PTE into a new page table level. Consider the case where we want to install 4K inside an empty 1G page: 1. We walk to the PUD and notice that it is pte_none. 2. We split the PUD by calling `hugetlb_split_to_shift`, creating a standard PUD that points to PMDs that are all pte_none. 3. We continue the PT walk to find the PMD. We split it just like we split the PUD. 4. We find the PTE and give it back to the caller. To avoid concurrent splitting operations on the same page table entry, we require that the mapping rwsem is held for writing while collapsing and for reading when doing a high-granularity PT walk. Signed-off-by: James Houghton --- include/linux/hugetlb.h | 23 ++++++++++++++ mm/hugetlb.c | 67 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 90 insertions(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 605aa19d8572..321f5745d87f 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -1176,14 +1176,37 @@ static inline void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, } #endif /* CONFIG_HUGETLB_PAGE */ +enum split_mode { + HUGETLB_SPLIT_NEVER = 0, + HUGETLB_SPLIT_NONE = 1 << 0, + HUGETLB_SPLIT_PRESENT = 1 << 1, + HUGETLB_SPLIT_ALWAYS = HUGETLB_SPLIT_NONE | HUGETLB_SPLIT_PRESENT, +}; #ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING /* If HugeTLB high-granularity mappings are enabled for this VMA. */ bool hugetlb_hgm_enabled(struct vm_area_struct *vma); +int huge_pte_alloc_high_granularity(struct hugetlb_pte *hpte, + struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long addr, + unsigned int desired_sz, + enum split_mode mode, + bool write_locked); #else static inline bool hugetlb_hgm_enabled(struct vm_area_struct *vma) { return false; } +static inline int huge_pte_alloc_high_granularity(struct hugetlb_pte *hpte, + struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long addr, + unsigned int desired_sz, + enum split_mode mode, + bool write_locked) +{ + return -EINVAL; +} #endif static inline spinlock_t *huge_pte_lock(struct hstate *h, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index eaffe7b4f67c..6e0c5fbfe32c 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -7166,6 +7166,73 @@ static int hugetlb_split_to_shift(struct mm_struct *mm, struct vm_area_struct *v tlb_finish_mmu(&tlb); return ret; } + +/* + * Similar to huge_pte_alloc except that this can be used to create or walk + * high-granularity mappings. It will automatically split existing HugeTLB PTEs + * if required by @mode. The resulting HugeTLB PTE will be returned in @hpte. + * + * There are three options for @mode: + * - HUGETLB_SPLIT_NEVER - Never split. + * - HUGETLB_SPLIT_NONE - Split empty PTEs. + * - HUGETLB_SPLIT_PRESENT - Split present PTEs. + * - HUGETLB_SPLIT_ALWAYS - Split both empty and present PTEs. + */ +int huge_pte_alloc_high_granularity(struct hugetlb_pte *hpte, + struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long addr, + unsigned int desired_shift, + enum split_mode mode, + bool write_locked) +{ + struct address_space *mapping = vma->vm_file->f_mapping; + bool has_write_lock = write_locked; + unsigned long desired_sz = 1UL << desired_shift; + int ret; + + BUG_ON(!hpte); + + if (has_write_lock) + i_mmap_assert_write_locked(mapping); + else + i_mmap_assert_locked(mapping); + +retry: + ret = 0; + hugetlb_pte_init(hpte); + + ret = hugetlb_walk_to(mm, hpte, addr, desired_sz, + !(mode & HUGETLB_SPLIT_NONE)); + if (ret || hugetlb_pte_size(hpte) == desired_sz) + goto out; + + if ( + ((mode & HUGETLB_SPLIT_NONE) && hugetlb_pte_none(hpte)) || + ((mode & HUGETLB_SPLIT_PRESENT) && + hugetlb_pte_present_leaf(hpte)) + ) { + if (!has_write_lock) { + i_mmap_unlock_read(mapping); + i_mmap_lock_write(mapping); + has_write_lock = true; + goto retry; + } + ret = hugetlb_split_to_shift(mm, vma, hpte, addr, + desired_shift); + } + +out: + if (has_write_lock && !write_locked) { + /* Drop the write lock. */ + i_mmap_unlock_write(mapping); + i_mmap_lock_read(mapping); + has_write_lock = false; + goto retry; + } + + return ret; +} #endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */ /* From patchwork Fri Jun 24 17:36:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56EE4C433EF for ; Fri, 24 Jun 2022 17:37:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CBA878E024C; Fri, 24 Jun 2022 13:37:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BF3508E0244; Fri, 24 Jun 2022 13:37:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9F7788E024C; Fri, 24 Jun 2022 13:37:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 84FFE8E0244 for ; Fri, 24 Jun 2022 13:37:31 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id DD27660A22 for ; Fri, 24 Jun 2022 17:37:30 +0000 (UTC) X-FDA: 79613836260.30.BAFBCCF Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf25.hostedemail.com (Postfix) with ESMTP id 590B9A002A for ; Fri, 24 Jun 2022 17:37:30 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-31776c7cd7eso27012557b3.5 for ; Fri, 24 Jun 2022 10:37:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=lYqZbxAfBwsss+xA5HxS7xzSZhdiSmDQ+D50F9gXIgQ=; b=AiWn9iagaaEqL24+KVa8KjoC0SIZwamKtMJ35pbIDJTojCuL6VXP0VSfXxYM2GJoiA tXj5pA7zO/MZz+I7fQy4xKFovF8i9LNOHZGdUx3KVD9yK1D4zESZShI0E/gGWUdB3D/2 1ADUjQ+gxpS4V0oB9/bdZGTJt2yorQkzSsS+y6oGy9k77oaO80RuBq2WUTr0EJMY1jAO gLoMWS+xm8OqsYrHKuE+kwcZ2qKoOlUObqWrWZk586l/lOiFFSq9bpKQKyql0c1sWbLX UF71Bd5MqWqGJwsN7Z/xzMqXDGiK3xdOGEQTTNo1TpbEk6rqSZV7mNM6CeMM2SuGbv54 jdJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=lYqZbxAfBwsss+xA5HxS7xzSZhdiSmDQ+D50F9gXIgQ=; b=1BVx1IhEdDcFtznaQiVzyPKQkzNr8g7N712AgMacg+1fTU4fTaA5NNmX1EHhG7HICo MMYFV9oHJ/JIYNa6pem0FHqf3FfUM0cf/jh1nA97OuSuCfxGhAzv7nQm92F6uUd15EkJ tOmqFqMWacWSPlEMpmApwvu00JZEisxtfGNpGE0J6szKMv9rntK2w98C+33+tIoRlZfU LVPXCugANL6XsltuetmcwG8POqAeXJxbXBH1JZVGAgCQrIBi/AjoPTR+tTV7l2d6YxPC 72yrkTm1+J2iw+8NhDJn/ECl0qeHXAeeJnTMrJdnjT+u4wyapLXynvctT6leudz2zMv/ NPdw== X-Gm-Message-State: AJIora9xt8wVmzTSvV0Ev8VgET7VQ1Ne4/NmZjZZtbWuKx9naZ3eutnt sGmq0DE8i7ZQdBu9FmW0Q+NPgNaaRgPe5hFB X-Google-Smtp-Source: AGRyM1tFgx74zurr2iIb2oIw18HoYxXMoONA1Emkdp55kb3soSpWLVNuC13FbAed12gldve16t6gKku4h/EYaeMe X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a5b:cd0:0:b0:668:f06d:df60 with SMTP id e16-20020a5b0cd0000000b00668f06ddf60mr299470ybr.191.1656092249707; Fri, 24 Jun 2022 10:37:29 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:44 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-15-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 14/26] hugetlb: add HGM support for hugetlb_fault and hugetlb_no_page From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=AiWn9iag; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3Wfa1YgoKCDofpdkqcdpkjckkcha.Ykihejqt-iigrWYg.knc@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Wfa1YgoKCDofpdkqcdpkjckkcha.Ykihejqt-iigrWYg.knc@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092250; a=rsa-sha256; cv=none; b=IDBuvKdAWr8m5pFJv7vuS/7x0oG2jXlmCCHyVngeZU4eUu5bibemH1kNqaKXLDw1gQRMkh odHzoWD5g/De3F+2nyjIGQfm+NWZ53vQtU/CXnlz7q9lEG2QkpzEwmL2SYwX4w5wufN5vg v2Z9QPqrBLfs7+DL3AylkYagwRkKkfQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092250; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lYqZbxAfBwsss+xA5HxS7xzSZhdiSmDQ+D50F9gXIgQ=; b=UP06sVhor1BV62ESQdW9jl054lRAo81CcfLqnNjlAxImzgYxXkDcZRxZXo2pQ80mOMF/Eo WEdW/Q67vicNiAMkqA8Zb5+p0BrfVrQOtUAs06OV22J3TrjYBLDZ7AVurlXqa+itoPrkL6 RSv8mw/Pm3UaR+pxMbD1VQGC9kjRZ5k= X-Stat-Signature: jqxk1wc3u1mnxj38iog1arrbd8nu4db3 X-Rspamd-Server: rspam08 X-Rspam-User: X-Rspamd-Queue-Id: 590B9A002A Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=AiWn9iag; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3Wfa1YgoKCDofpdkqcdpkjckkcha.Ykihejqt-iigrWYg.knc@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Wfa1YgoKCDofpdkqcdpkjckkcha.Ykihejqt-iigrWYg.knc@flex--jthoughton.bounces.google.com X-HE-Tag: 1656092250-528803 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This CL is the first main functional HugeTLB change. Together, these changes allow the HugeTLB fault path to handle faults on HGM-enabled VMAs. The two main behaviors that can be done now: 1. Faults can be passed to handle_userfault. (Userspace will want to use UFFD_FEATURE_REAL_ADDRESS to get the real address to know which region they should be call UFFDIO_CONTINUE on later.) 2. Faults on pages that have been partially mapped (and userfaultfd is not being used) will get mapped at the largest possible size. For example, if a 1G page has been partially mapped at 2M, and we fault on an unmapped 2M section, hugetlb_no_page will create a 2M PMD to map the faulting address. This commit does not handle hugetlb_wp right now, and it doesn't handle HugeTLB page migration and swap entries. Signed-off-by: James Houghton --- include/linux/hugetlb.h | 12 ++++ mm/hugetlb.c | 121 +++++++++++++++++++++++++++++++--------- 2 files changed, 106 insertions(+), 27 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 321f5745d87f..ac4ac8fbd901 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -1185,6 +1185,9 @@ enum split_mode { #ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING /* If HugeTLB high-granularity mappings are enabled for this VMA. */ bool hugetlb_hgm_enabled(struct vm_area_struct *vma); +int hugetlb_alloc_largest_pte(struct hugetlb_pte *hpte, struct mm_struct *mm, + struct vm_area_struct *vma, unsigned long start, + unsigned long end); int huge_pte_alloc_high_granularity(struct hugetlb_pte *hpte, struct mm_struct *mm, struct vm_area_struct *vma, @@ -1197,6 +1200,15 @@ static inline bool hugetlb_hgm_enabled(struct vm_area_struct *vma) { return false; } + +static inline +int hugetlb_alloc_largest_pte(struct hugetlb_pte *hpte, struct mm_struct *mm, + struct vm_area_struct *vma, unsigned long start, + unsigned long end) +{ + BUG(); +} + static inline int huge_pte_alloc_high_granularity(struct hugetlb_pte *hpte, struct mm_struct *mm, struct vm_area_struct *vma, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 6e0c5fbfe32c..da30621656b8 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5605,18 +5605,24 @@ static inline vm_fault_t hugetlb_handle_userfault(struct vm_area_struct *vma, static vm_fault_t hugetlb_no_page(struct mm_struct *mm, struct vm_area_struct *vma, struct address_space *mapping, pgoff_t idx, - unsigned long address, pte_t *ptep, + unsigned long address, struct hugetlb_pte *hpte, pte_t old_pte, unsigned int flags) { struct hstate *h = hstate_vma(vma); vm_fault_t ret = VM_FAULT_SIGBUS; int anon_rmap = 0; unsigned long size; - struct page *page; + struct page *page, *subpage; pte_t new_pte; spinlock_t *ptl; unsigned long haddr = address & huge_page_mask(h); + unsigned long haddr_hgm = address & hugetlb_pte_mask(hpte); bool new_page, new_pagecache_page = false; + /* + * This page is getting mapped for the first time, in which case we + * want to increment its mapcount. + */ + bool new_mapping = hpte->shift == huge_page_shift(h); /* * Currently, we are forced to kill the process in the event the @@ -5665,9 +5671,9 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, * here. Before returning error, get ptl and make * sure there really is no pte entry. */ - ptl = huge_pte_lock(h, mm, ptep); + ptl = hugetlb_pte_lock(mm, hpte); ret = 0; - if (huge_pte_none(huge_ptep_get(ptep))) + if (hugetlb_pte_none(hpte)) ret = vmf_error(PTR_ERR(page)); spin_unlock(ptl); goto out; @@ -5731,18 +5737,25 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, vma_end_reservation(h, vma, haddr); } - ptl = huge_pte_lock(h, mm, ptep); + ptl = hugetlb_pte_lock(mm, hpte); ret = 0; /* If pte changed from under us, retry */ - if (!pte_same(huge_ptep_get(ptep), old_pte)) + if (!pte_same(hugetlb_ptep_get(hpte), old_pte)) goto backout; - if (anon_rmap) { - ClearHPageRestoreReserve(page); - hugepage_add_new_anon_rmap(page, vma, haddr); - } else - page_dup_file_rmap(page, true); - new_pte = make_huge_pte(vma, page, ((vma->vm_flags & VM_WRITE) + if (new_mapping) { + /* Only increment this page's mapcount if we are mapping it + * for the first time. + */ + if (anon_rmap) { + ClearHPageRestoreReserve(page); + hugepage_add_new_anon_rmap(page, vma, haddr); + } else + page_dup_file_rmap(page, true); + } + + subpage = hugetlb_find_subpage(h, page, haddr_hgm); + new_pte = make_huge_pte(vma, subpage, ((vma->vm_flags & VM_WRITE) && (vma->vm_flags & VM_SHARED))); /* * If this pte was previously wr-protected, keep it wr-protected even @@ -5750,12 +5763,13 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, */ if (unlikely(pte_marker_uffd_wp(old_pte))) new_pte = huge_pte_wrprotect(huge_pte_mkuffd_wp(new_pte)); - set_huge_pte_at(mm, haddr, ptep, new_pte); + set_huge_pte_at(mm, haddr_hgm, hpte->ptep, new_pte); - hugetlb_count_add(pages_per_huge_page(h), mm); + hugetlb_count_add(hugetlb_pte_size(hpte) / PAGE_SIZE, mm); if ((flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) { + BUG_ON(hugetlb_pte_size(hpte) != huge_page_size(h)); /* Optimization, do the COW without a second fault */ - ret = hugetlb_wp(mm, vma, address, ptep, flags, page, ptl); + ret = hugetlb_wp(mm, vma, address, hpte->ptep, flags, page, ptl); } spin_unlock(ptl); @@ -5816,11 +5830,15 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, u32 hash; pgoff_t idx; struct page *page = NULL; + struct page *subpage = NULL; struct page *pagecache_page = NULL; struct hstate *h = hstate_vma(vma); struct address_space *mapping; int need_wait_lock = 0; unsigned long haddr = address & huge_page_mask(h); + unsigned long haddr_hgm; + bool hgm_enabled = hugetlb_hgm_enabled(vma); + struct hugetlb_pte hpte; ptep = huge_pte_offset(mm, haddr, huge_page_size(h)); if (ptep) { @@ -5866,11 +5884,22 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, hash = hugetlb_fault_mutex_hash(mapping, idx); mutex_lock(&hugetlb_fault_mutex_table[hash]); - entry = huge_ptep_get(ptep); + hugetlb_pte_populate(&hpte, ptep, huge_page_shift(h)); + + if (hgm_enabled) { + ret = hugetlb_walk_to(mm, &hpte, address, + PAGE_SIZE, /*stop_at_none=*/true); + if (ret) { + ret = vmf_error(ret); + goto out_mutex; + } + } + + entry = hugetlb_ptep_get(&hpte); /* PTE markers should be handled the same way as none pte */ - if (huge_pte_none_mostly(entry)) { - ret = hugetlb_no_page(mm, vma, mapping, idx, address, ptep, - entry, flags); + if (hugetlb_pte_none_mostly(&hpte)) { + ret = hugetlb_no_page(mm, vma, mapping, idx, address, &hpte, + entry, flags); goto out_mutex; } @@ -5908,14 +5937,17 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, vma, haddr); } - ptl = huge_pte_lock(h, mm, ptep); + ptl = hugetlb_pte_lock(mm, &hpte); /* Check for a racing update before calling hugetlb_wp() */ - if (unlikely(!pte_same(entry, huge_ptep_get(ptep)))) + if (unlikely(!pte_same(entry, hugetlb_ptep_get(&hpte)))) goto out_ptl; + /* haddr_hgm is the base address of the region that hpte maps. */ + haddr_hgm = address & hugetlb_pte_mask(&hpte); + /* Handle userfault-wp first, before trying to lock more pages */ - if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(ptep)) && + if (userfaultfd_wp(vma) && huge_pte_uffd_wp(hugetlb_ptep_get(&hpte)) && (flags & FAULT_FLAG_WRITE) && !huge_pte_write(entry)) { struct vm_fault vmf = { .vma = vma, @@ -5939,7 +5971,8 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * pagecache_page, so here we need take the former one * when page != pagecache_page or !pagecache_page. */ - page = pte_page(entry); + subpage = pte_page(entry); + page = compound_head(subpage); if (page != pagecache_page) if (!trylock_page(page)) { need_wait_lock = 1; @@ -5950,7 +5983,8 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, if (flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) { if (!huge_pte_write(entry)) { - ret = hugetlb_wp(mm, vma, address, ptep, flags, + BUG_ON(hugetlb_pte_size(&hpte) != huge_page_size(h)); + ret = hugetlb_wp(mm, vma, address, hpte.ptep, flags, pagecache_page, ptl); goto out_put_page; } else if (likely(flags & FAULT_FLAG_WRITE)) { @@ -5958,9 +5992,9 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, } } entry = pte_mkyoung(entry); - if (huge_ptep_set_access_flags(vma, haddr, ptep, entry, + if (huge_ptep_set_access_flags(vma, haddr_hgm, hpte.ptep, entry, flags & FAULT_FLAG_WRITE)) - update_mmu_cache(vma, haddr, ptep); + update_mmu_cache(vma, haddr_hgm, hpte.ptep); out_put_page: if (page != pagecache_page) unlock_page(page); @@ -6951,7 +6985,8 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, pte = (pte_t *)pmd_alloc(mm, pud, addr); } } - BUG_ON(pte && pte_present(*pte) && !pte_huge(*pte)); + if (!hugetlb_hgm_enabled(vma)) + BUG_ON(pte && pte_present(*pte) && !pte_huge(*pte)); return pte; } @@ -7057,6 +7092,38 @@ static unsigned int __shift_for_hstate(struct hstate *h) (tmp_h) <= &hstates[hugetlb_max_hstate]; \ (tmp_h)++) +/* + * Allocate a HugeTLB PTE that maps as much of [start, end) as possible with a + * single page table entry. The allocated HugeTLB PTE is returned in hpte. + */ +int hugetlb_alloc_largest_pte(struct hugetlb_pte *hpte, struct mm_struct *mm, + struct vm_area_struct *vma, unsigned long start, + unsigned long end) +{ + struct hstate *h = hstate_vma(vma), *tmp_h; + unsigned int shift; + int ret; + + for_each_hgm_shift(h, tmp_h, shift) { + unsigned long sz = 1UL << shift; + + if (!IS_ALIGNED(start, sz) || start + sz > end) + continue; + ret = huge_pte_alloc_high_granularity(hpte, mm, vma, start, + shift, HUGETLB_SPLIT_NONE, + /*write_locked=*/false); + if (ret) + return ret; + + if (hpte->shift > shift) + return -EEXIST; + + BUG_ON(hpte->shift != shift); + return 0; + } + return -EINVAL; +} + /* * Given a particular address, split the HugeTLB PTE that currently maps it * so that, for the given address, the PTE that maps it is `desired_shift`. From patchwork Fri Jun 24 17:36:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894940 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 533EDCCA473 for ; Fri, 24 Jun 2022 17:37:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8734C8E024D; Fri, 24 Jun 2022 13:37:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7AD0F8E0244; Fri, 24 Jun 2022 13:37:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B1548E024D; Fri, 24 Jun 2022 13:37:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 3E2BD8E0244 for ; Fri, 24 Jun 2022 13:37:32 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id EE36535476 for ; Fri, 24 Jun 2022 17:37:31 +0000 (UTC) X-FDA: 79613836302.13.659E5FA Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf16.hostedemail.com (Postfix) with ESMTP id A5882180025 for ; Fri, 24 Jun 2022 17:37:31 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-31382419c22so26681707b3.18 for ; Fri, 24 Jun 2022 10:37:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=LoJsmKx0RIJYFCgh3RUmS9FnkZR9nxjSRnqvv6TZGjM=; b=IZS4DM3M3K6LoUVsC8Bl04dfOKEEAWyhZp/PaBKn0d92TIDnzwMDFN7wLf3gZK8KXk Hqp1S+enLGzviCJsO6DpNpGzxKEp+T0yzrdIwA4S415ZdLqTsoPiTRgFb6aOA0kKPBAY zyheLmcN2aPMf7MJVwN4znu5j9nveQnM+HbQGBNnfXtj5p0uhIS5X82WJqumvoFeu78Z kexFoLJRgVhL59k8quEJaOagGrrlQ0qdHEHVyls+GVapUIwLCmrA+dHzKoYHQOi/e8xv vUWXfvm60zyRjMUV+v9MtvRAUv6UlhcNITygQafNxgge+q1x6oNJBsUIR4wKbgjkQNng /GzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=LoJsmKx0RIJYFCgh3RUmS9FnkZR9nxjSRnqvv6TZGjM=; b=mGyYPqq1aVXJCcRcts4hVwZji6fn9XkOf48zCjmvoxMfKQWaRCGapLL3dxhNTnG8F5 FmXbLMsATYodnDTA8qPHu7QWvfZ7/XOZPrQGsnsqihT8hsMM86hXTUJcXT5scBEf5uhv +Jb9QykGWgjm0ynlyc46h+q/Ye7t0lYmQ72muV1+G5U4wc5VM75WIfXh8KStkwzxOFY5 jpw1tUvNtA1vrR/JFQWf0VC2E6vYwt9wibih5e/wzcLnkitW5fBm0t05OuyyOfSr3Uw4 h8rDaPjqKyoTrhLm4x1Ui2O6zBbue02lwDLRyWUkforDfo0Rp7O3gG5AU5YaqPZenySq k9uQ== X-Gm-Message-State: AJIora84uPlFdvH3xNG1yomrNWyIiVD8LPY4etP9gV2wxqwFV+xXwjdj 5jN8LfygtxRByzLj9gZYcMBMaCx9FVavkb4Q X-Google-Smtp-Source: AGRyM1se79AgT0Cox1n1PfeD3monM06aR7cLfdIp8zaSZ2VT02A9SECnyrpKDnZIvwtzr+hFf9tsMPJX4L1dTD28 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a25:7801:0:b0:669:b51b:10d0 with SMTP id t1-20020a257801000000b00669b51b10d0mr331321ybc.204.1656092251057; Fri, 24 Jun 2022 10:37:31 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:45 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-16-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 15/26] hugetlb: make unmapping compatible with high-granularity mappings From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092251; a=rsa-sha256; cv=none; b=PNUM23zcrYzkqW8Tl7YCYnij+O5IafyR8mBM5I3xtNQ9gnL9sVYFL7hUMGGNn8/IORkkFt qfkxW4WBGBm2Igt0mGtyTwY/5TJRKD73KX7iCi0lkrCu29esOV0WnuvNZImiDWzw7wvma9 Eputmd+jE2Tn66lkV5AWyHoqG6t0CjU= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=IZS4DM3M; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 3W_a1YgoKCDwhrfmsefrmlemmejc.amkjglsv-kkitYai.mpe@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3W_a1YgoKCDwhrfmsefrmlemmejc.amkjglsv-kkitYai.mpe@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092251; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LoJsmKx0RIJYFCgh3RUmS9FnkZR9nxjSRnqvv6TZGjM=; b=q1wgQvKH3xe0BiuZN6uWdGclEyrfSKJSGevAXHDNaMHhV+qi6fy1SuvCCtHSRdsCzDK8E+ 1Zc3L97aXJMByFPMHGPMbsxLCD29xrB3owzO0e1XYLZnXUNTA7Yfy6J4xmspXEdLSanJC6 cAw835GSA5DxrbUR4JWKCEEj9XnFvPk= X-Rspam-User: X-Stat-Signature: 1huza4x1j8aa978b8sa9t59m1uurind4 X-Rspamd-Queue-Id: A5882180025 Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=IZS4DM3M; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 3W_a1YgoKCDwhrfmsefrmlemmejc.amkjglsv-kkitYai.mpe@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3W_a1YgoKCDwhrfmsefrmlemmejc.amkjglsv-kkitYai.mpe@flex--jthoughton.bounces.google.com X-Rspamd-Server: rspam03 X-HE-Tag: 1656092251-452077 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This enlightens __unmap_hugepage_range to deal with high-granularity mappings. This doesn't change its API; it still must be called with hugepage alignment, but it will correctly unmap hugepages that have been mapped at high granularity. Analogous to the mapcount rules introduced by hugetlb_no_page, we only drop mapcount in this case if we are unmapping an entire hugepage in one operation. This is the case when a VMA is destroyed. Eventually, functionality here can be expanded to allow users to call MADV_DONTNEED on PAGE_SIZE-aligned sections of a hugepage, but that is not done here. Signed-off-by: James Houghton --- include/asm-generic/tlb.h | 6 +-- mm/hugetlb.c | 85 ++++++++++++++++++++++++++------------- 2 files changed, 59 insertions(+), 32 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index ff3e82553a76..8daa3ae460d9 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -562,9 +562,9 @@ static inline void tlb_flush_p4d_range(struct mmu_gather *tlb, __tlb_remove_tlb_entry(tlb, ptep, address); \ } while (0) -#define tlb_remove_huge_tlb_entry(h, tlb, ptep, address) \ +#define tlb_remove_huge_tlb_entry(tlb, hpte, address) \ do { \ - unsigned long _sz = huge_page_size(h); \ + unsigned long _sz = hugetlb_pte_size(&hpte); \ if (_sz >= P4D_SIZE) \ tlb_flush_p4d_range(tlb, address, _sz); \ else if (_sz >= PUD_SIZE) \ @@ -573,7 +573,7 @@ static inline void tlb_flush_p4d_range(struct mmu_gather *tlb, tlb_flush_pmd_range(tlb, address, _sz); \ else \ tlb_flush_pte_range(tlb, address, _sz); \ - __tlb_remove_tlb_entry(tlb, ptep, address); \ + __tlb_remove_tlb_entry(tlb, hpte.ptep, address);\ } while (0) /** diff --git a/mm/hugetlb.c b/mm/hugetlb.c index da30621656b8..51fc1d3f122f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5120,24 +5120,20 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct { struct mm_struct *mm = vma->vm_mm; unsigned long address; - pte_t *ptep; + struct hugetlb_pte hpte; pte_t pte; spinlock_t *ptl; - struct page *page; + struct page *hpage, *subpage; struct hstate *h = hstate_vma(vma); unsigned long sz = huge_page_size(h); struct mmu_notifier_range range; bool force_flush = false; + bool hgm_enabled = hugetlb_hgm_enabled(vma); WARN_ON(!is_vm_hugetlb_page(vma)); BUG_ON(start & ~huge_page_mask(h)); BUG_ON(end & ~huge_page_mask(h)); - /* - * This is a hugetlb vma, all the pte entries should point - * to huge page. - */ - tlb_change_page_size(tlb, sz); tlb_start_vma(tlb, vma); /* @@ -5148,25 +5144,43 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct adjust_range_if_pmd_sharing_possible(vma, &range.start, &range.end); mmu_notifier_invalidate_range_start(&range); address = start; - for (; address < end; address += sz) { - ptep = huge_pte_offset(mm, address, sz); - if (!ptep) + + while (address < end) { + pte_t *ptep = huge_pte_offset(mm, address, sz); + + if (!ptep) { + address += sz; continue; + } + hugetlb_pte_populate(&hpte, ptep, huge_page_shift(h)); + if (hgm_enabled) { + int ret = huge_pte_alloc_high_granularity( + &hpte, mm, vma, address, PAGE_SHIFT, + HUGETLB_SPLIT_NEVER, + /*write_locked=*/true); + /* + * We will never split anything, so this should always + * succeed. + */ + BUG_ON(ret); + } - ptl = huge_pte_lock(h, mm, ptep); - if (huge_pmd_unshare(mm, vma, &address, ptep)) { + ptl = hugetlb_pte_lock(mm, &hpte); + if (!hgm_enabled && huge_pmd_unshare( + mm, vma, &address, hpte.ptep)) { spin_unlock(ptl); tlb_flush_pmd_range(tlb, address & PUD_MASK, PUD_SIZE); force_flush = true; - continue; + goto next_hpte; } - pte = huge_ptep_get(ptep); - if (huge_pte_none(pte)) { + if (hugetlb_pte_none(&hpte)) { spin_unlock(ptl); - continue; + goto next_hpte; } + pte = hugetlb_ptep_get(&hpte); + /* * Migrating hugepage or HWPoisoned hugepage is already * unmapped and its refcount is dropped, so just clear pte here. @@ -5180,24 +5194,27 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct */ if (pte_swp_uffd_wp_any(pte) && !(zap_flags & ZAP_FLAG_DROP_MARKER)) - set_huge_pte_at(mm, address, ptep, + set_huge_pte_at(mm, address, hpte.ptep, make_pte_marker(PTE_MARKER_UFFD_WP)); else - huge_pte_clear(mm, address, ptep, sz); + huge_pte_clear(mm, address, hpte.ptep, + hugetlb_pte_size(&hpte)); spin_unlock(ptl); - continue; + goto next_hpte; } - page = pte_page(pte); + subpage = pte_page(pte); + BUG_ON(!subpage); + hpage = compound_head(subpage); /* * If a reference page is supplied, it is because a specific * page is being unmapped, not a range. Ensure the page we * are about to unmap is the actual page of interest. */ if (ref_page) { - if (page != ref_page) { + if (hpage != ref_page) { spin_unlock(ptl); - continue; + goto next_hpte; } /* * Mark the VMA as having unmapped its page so that @@ -5207,25 +5224,35 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct set_vma_resv_flags(vma, HPAGE_RESV_UNMAPPED); } - pte = huge_ptep_get_and_clear(mm, address, ptep); - tlb_remove_huge_tlb_entry(h, tlb, ptep, address); + pte = huge_ptep_get_and_clear(mm, address, hpte.ptep); + tlb_change_page_size(tlb, hugetlb_pte_size(&hpte)); + tlb_remove_huge_tlb_entry(tlb, hpte, address); if (huge_pte_dirty(pte)) - set_page_dirty(page); + set_page_dirty(hpage); /* Leave a uffd-wp pte marker if needed */ if (huge_pte_uffd_wp(pte) && !(zap_flags & ZAP_FLAG_DROP_MARKER)) - set_huge_pte_at(mm, address, ptep, + set_huge_pte_at(mm, address, hpte.ptep, make_pte_marker(PTE_MARKER_UFFD_WP)); - hugetlb_count_sub(pages_per_huge_page(h), mm); - page_remove_rmap(page, vma, true); + + hugetlb_count_sub(hugetlb_pte_size(&hpte)/PAGE_SIZE, mm); + + /* + * If we are unmapping the entire page, remove it from the + * rmap. + */ + if (IS_ALIGNED(address, sz) && address + sz <= end) + page_remove_rmap(hpage, vma, true); spin_unlock(ptl); - tlb_remove_page_size(tlb, page, huge_page_size(h)); + tlb_remove_page_size(tlb, subpage, hugetlb_pte_size(&hpte)); /* * Bail out after unmapping reference page if supplied */ if (ref_page) break; +next_hpte: + address += hugetlb_pte_size(&hpte); } mmu_notifier_invalidate_range_end(&range); tlb_end_vma(tlb, vma); From patchwork Fri Jun 24 17:36:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894941 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12139C43334 for ; Fri, 24 Jun 2022 17:37:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BF62D8E024E; Fri, 24 Jun 2022 13:37:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B7DB18E0244; Fri, 24 Jun 2022 13:37:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 95B1A8E024E; Fri, 24 Jun 2022 13:37:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7ABF58E0244 for ; Fri, 24 Jun 2022 13:37:33 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 5E21535476 for ; Fri, 24 Jun 2022 17:37:33 +0000 (UTC) X-FDA: 79613836386.25.0FD11A4 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf20.hostedemail.com (Postfix) with ESMTP id F0CB91C001F for ; Fri, 24 Jun 2022 17:37:32 +0000 (UTC) Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-3178a95ec78so27001427b3.4 for ; Fri, 24 Jun 2022 10:37:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=ndW8f2/FMhLCn5Y2u5fQS3kI5t4rS7D1K1+5Zv5C8rs=; b=czqXqvIE/8k7tydJPozB7auTFGn2IWAhFN0QqqSLWSloUnswocGDGgUeo23aTjkMei kJVRiKE5/tiYCuHldEVJHKMPdnIu1VB4kfdTvkuHI+ANDJBJq4WBaukBrLfqF7kOY/tK MT6za/4MfqzLFtBFRgSQ5jACJHp8mD5dBxqaUXvHHjfCB4WTLOVkleis0beQo12/qieV 0OirUq/zaPSgClUBBFEAredAa0pJS7lZV3B53aynse7fEXzlquNVE8V9XBV/gY5amf1H Zy8GYpedNEgd7iU2Q+yG3Q5EJxgpMOpVeMlp5AHPW2AVdClXpdfObc+IfRZlB5gzBNda kxBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ndW8f2/FMhLCn5Y2u5fQS3kI5t4rS7D1K1+5Zv5C8rs=; b=QflOH6QaXcmmCuXUZRNhyaPYBBnQnugGv5R2LD9bBDQoWFxMRUk4ZKGszupukytGKP ef8Rd3tfSBf6LCo3krieBIuQwQQLSbCFiiqj/FE1W40/HyJbk2Bdvkj/9WA4lsWd1ZWb 8w9N2xskYbL/1R8SiT7OhDSm3ELI4UHho2d3xtw26i68lYChbe0LoYUXt29Gvh6PxA7d d+R5d/WZNSs7uA9vrCIR6Nl62/5Bc2bfyHzvY4cnG5RP/501X+VfDabCuwVHUfT9iXUo tWTdCRN30hM3G2n37GqA9jNJIeedkLuAs71SCb5LltYoOnO4oKtjF9ST3ah1FzUtjmEq 1QXQ== X-Gm-Message-State: AJIora9QgB7sj2JhHBeKuLff+6QP6Hedm317e4m6gfqWIkKyKLqJ+odA YzS48b8K7q+XQTB713nwLZFpmOrp9bUplliI X-Google-Smtp-Source: AGRyM1vf9VAPWkUYJ/WFD/esjoiahsG2GN/fHJroFrmfPTAWRMvqzCQsNWId7C9YuKC2vmXazC7krxj0p+91Xop2 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a25:f30f:0:b0:664:af3b:f83f with SMTP id c15-20020a25f30f000000b00664af3bf83fmr292571ybs.516.1656092252403; Fri, 24 Jun 2022 10:37:32 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:46 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-17-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 16/26] hugetlb: make hugetlb_change_protection compatible with HGM From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092253; a=rsa-sha256; cv=none; b=2cyNOrtyXFnpzrCh51dQK65iUtGPXIveSb+rff4UQRGH9ICmbB6abmTC8hKV6fSkfvp9St +amkJmc9gizp2MJnkVzwL7fvGM8f4ywSM/XgEqPfhXr8WiMNMOq/8tAQ7aHdcKVkHqD6vF SvNFZ+jyETT3ImUBVg9ExO4qsJhUulc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092253; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ndW8f2/FMhLCn5Y2u5fQS3kI5t4rS7D1K1+5Zv5C8rs=; b=OAG8BsrCxKnVpE+Fr3Red7mImikgs2SHxtT4I3pYarDplrHG/htKZBILEFdGG4j+xhsglM 7FtxRJTsZ4Yarh4rMIVLd24jC2aS0TgA8asE3r+mfoX7/YMsrjsB2RU8ntM1E/pM4HAXuc ons7y+SHxWrO6gqL9g+w1wPcLIaWx14= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=czqXqvIE; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3XPa1YgoKCD0isgntfgsnmfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--jthoughton.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3XPa1YgoKCD0isgntfgsnmfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--jthoughton.bounces.google.com X-Stat-Signature: qp4iyo7te95zx37fo659zn3ecorjqxs7 X-Rspamd-Queue-Id: F0CB91C001F Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=czqXqvIE; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3XPa1YgoKCD0isgntfgsnmfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--jthoughton.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3XPa1YgoKCD0isgntfgsnmfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--jthoughton.bounces.google.com X-Rspam-User: X-Rspamd-Server: rspam04 X-HE-Tag: 1656092252-175365 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: HugeTLB is now able to change the protection of hugepages that are mapped at high granularity. I need to add more of the HugeTLB PTE wrapper functions to clean up this patch. I'll do this in the next version. Signed-off-by: James Houghton --- mm/hugetlb.c | 91 +++++++++++++++++++++++++++++++++++----------------- 1 file changed, 62 insertions(+), 29 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 51fc1d3f122f..f9c7daa6c090 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6476,14 +6476,15 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, { struct mm_struct *mm = vma->vm_mm; unsigned long start = address; - pte_t *ptep; pte_t pte; struct hstate *h = hstate_vma(vma); - unsigned long pages = 0, psize = huge_page_size(h); + unsigned long base_pages = 0, psize = huge_page_size(h); bool shared_pmd = false; struct mmu_notifier_range range; bool uffd_wp = cp_flags & MM_CP_UFFD_WP; bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE; + struct hugetlb_pte hpte; + bool hgm_enabled = hugetlb_hgm_enabled(vma); /* * In the case of shared PMDs, the area to flush could be beyond @@ -6499,28 +6500,38 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, mmu_notifier_invalidate_range_start(&range); i_mmap_lock_write(vma->vm_file->f_mapping); - for (; address < end; address += psize) { + while (address < end) { spinlock_t *ptl; - ptep = huge_pte_offset(mm, address, psize); - if (!ptep) + pte_t *ptep = huge_pte_offset(mm, address, huge_page_size(h)); + + if (!ptep) { + address += huge_page_size(h); continue; - ptl = huge_pte_lock(h, mm, ptep); - if (huge_pmd_unshare(mm, vma, &address, ptep)) { + } + hugetlb_pte_populate(&hpte, ptep, huge_page_shift(h)); + if (hgm_enabled) { + int ret = hugetlb_walk_to(mm, &hpte, address, PAGE_SIZE, + /*stop_at_none=*/true); + BUG_ON(ret); + } + + ptl = hugetlb_pte_lock(mm, &hpte); + if (huge_pmd_unshare(mm, vma, &address, hpte.ptep)) { /* * When uffd-wp is enabled on the vma, unshare * shouldn't happen at all. Warn about it if it * happened due to some reason. */ WARN_ON_ONCE(uffd_wp || uffd_wp_resolve); - pages++; + base_pages += hugetlb_pte_size(&hpte) / PAGE_SIZE; spin_unlock(ptl); shared_pmd = true; - continue; + goto next_hpte; } - pte = huge_ptep_get(ptep); + pte = hugetlb_ptep_get(&hpte); if (unlikely(is_hugetlb_entry_hwpoisoned(pte))) { spin_unlock(ptl); - continue; + goto next_hpte; } if (unlikely(is_hugetlb_entry_migration(pte))) { swp_entry_t entry = pte_to_swp_entry(pte); @@ -6540,12 +6551,13 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, newpte = pte_swp_mkuffd_wp(newpte); else if (uffd_wp_resolve) newpte = pte_swp_clear_uffd_wp(newpte); - set_huge_swap_pte_at(mm, address, ptep, - newpte, psize); - pages++; + set_huge_swap_pte_at(mm, address, hpte.ptep, + newpte, + hugetlb_pte_size(&hpte)); + base_pages += hugetlb_pte_size(&hpte) / PAGE_SIZE; } spin_unlock(ptl); - continue; + goto next_hpte; } if (unlikely(pte_marker_uffd_wp(pte))) { /* @@ -6553,21 +6565,40 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, * no need for huge_ptep_modify_prot_start/commit(). */ if (uffd_wp_resolve) - huge_pte_clear(mm, address, ptep, psize); + huge_pte_clear(mm, address, hpte.ptep, psize); } - if (!huge_pte_none(pte)) { + if (!hugetlb_pte_none(&hpte)) { pte_t old_pte; - unsigned int shift = huge_page_shift(hstate_vma(vma)); - - old_pte = huge_ptep_modify_prot_start(vma, address, ptep); - pte = huge_pte_modify(old_pte, newprot); - pte = arch_make_huge_pte(pte, shift, vma->vm_flags); - if (uffd_wp) - pte = huge_pte_mkuffd_wp(huge_pte_wrprotect(pte)); - else if (uffd_wp_resolve) - pte = huge_pte_clear_uffd_wp(pte); - huge_ptep_modify_prot_commit(vma, address, ptep, old_pte, pte); - pages++; + unsigned int shift = hpte.shift; + /* + * This is ugly. This will be cleaned up in a future + * version of this series. + */ + if (shift > PAGE_SHIFT) { + old_pte = huge_ptep_modify_prot_start( + vma, address, hpte.ptep); + pte = huge_pte_modify(old_pte, newprot); + pte = arch_make_huge_pte( + pte, shift, vma->vm_flags); + if (uffd_wp) + pte = huge_pte_mkuffd_wp(huge_pte_wrprotect(pte)); + else if (uffd_wp_resolve) + pte = huge_pte_clear_uffd_wp(pte); + huge_ptep_modify_prot_commit( + vma, address, hpte.ptep, + old_pte, pte); + } else { + old_pte = ptep_modify_prot_start( + vma, address, hpte.ptep); + pte = pte_modify(old_pte, newprot); + if (uffd_wp) + pte = pte_mkuffd_wp(pte_wrprotect(pte)); + else if (uffd_wp_resolve) + pte = pte_clear_uffd_wp(pte); + ptep_modify_prot_commit( + vma, address, hpte.ptep, old_pte, pte); + } + base_pages += hugetlb_pte_size(&hpte) / PAGE_SIZE; } else { /* None pte */ if (unlikely(uffd_wp)) @@ -6576,6 +6607,8 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, make_pte_marker(PTE_MARKER_UFFD_WP)); } spin_unlock(ptl); +next_hpte: + address += hugetlb_pte_size(&hpte); } /* * Must flush TLB before releasing i_mmap_rwsem: x86's huge_pmd_unshare @@ -6597,7 +6630,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, i_mmap_unlock_write(vma->vm_file->f_mapping); mmu_notifier_invalidate_range_end(&range); - return pages << h->order; + return base_pages; } /* Return true if reservation was successful, false otherwise. */ From patchwork Fri Jun 24 17:36:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894942 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94E42CCA473 for ; Fri, 24 Jun 2022 17:37:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A2978E024F; Fri, 24 Jun 2022 13:37:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 405828E0244; Fri, 24 Jun 2022 13:37:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 195E58E024F; Fri, 24 Jun 2022 13:37:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E76B28E0244 for ; Fri, 24 Jun 2022 13:37:34 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id C792233713 for ; Fri, 24 Jun 2022 17:37:34 +0000 (UTC) X-FDA: 79613836428.30.1A1C868 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf25.hostedemail.com (Postfix) with ESMTP id 4B1D0A002A for ; Fri, 24 Jun 2022 17:37:34 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-31776c7cd7eso27012557b3.5 for ; Fri, 24 Jun 2022 10:37:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=4vXUhYS1gVPcFf2LfwHq3p4PeYS4NxljSjceZ18tGG4=; b=SANzcf+LwCVGpRM55Cay++FK4cUCDPSiB1XYFVIRVPd8bp9OmeUxkkbKxAjSLEPaec rO0L74FDJGW8GBsbPMOk/D6MIlmhHhIDSjtWw7C+OlohYel2WaCeh5uufn8zu/z6hULm /YPEqs3V1KLPyQPBtO4T1n3x5IEA0mIDDhG4OM6wA+fQpkRtkK3I4ZlOCnhtXfYJJBvT zqcEyFkmPgpGY0oBWs35n3cyP752+a4rX4v1dipphzs54KYnjLeWO2uVKbQ1AeddxDAb M9UdJg2b1CXLL7GgWGFeWr1dEZLZQzO1Y1PjZaxJ6F/EO+UtB2CbCiS/IWX6f6NMt489 ntQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=4vXUhYS1gVPcFf2LfwHq3p4PeYS4NxljSjceZ18tGG4=; b=zuPjW4zhLbOIkcGgYe4PchFFROCyRFwbLJqj54HKgFcGHRpsf+mn7ccv98KpKEClAT 5UXbYDXylgZR4Or99Uy0H4hPx85aZ3wCxpu5tKW3MYx2/Q2/VyL9IJ0nd02I/ObwhYfZ SaPszMw6Y5DyuE0Wb8auuPJkOMFLpqr0mJLvWI0d94SPZ6MYWP+Ux11ObLHb7ahUWZcJ rtP0vnixi95bL/0h0P/kIt5lDUmcTaOPZK+7Q626XQefqA8PVjRd74RmxhwvemLu+BSV p63TCHM9ji8ATZ7Hk9/gOnlVBKGbj/3SmcrVixH1fXMqYE1hVRK9f5wvG1oiMauzllo0 7gGQ== X-Gm-Message-State: AJIora80v1djF69VamtGrZGYlpKYXRBCssVbVTFYdcJwIJKnyWcjWuSM HQE2WrxEX9++FmD1gvXdDAxqDAm511rm9BDc X-Google-Smtp-Source: AGRyM1ukQX4IYJ9JitkrwX+RpgB7i6QAoj2im0rwaYwc2vW9f/595sRIyJPCfzsFwuJfCBggqm9wh7FmHwoLS8un X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a81:b622:0:b0:317:af00:63c3 with SMTP id u34-20020a81b622000000b00317af0063c3mr24103ywh.298.1656092253828; Fri, 24 Jun 2022 10:37:33 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:47 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-18-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 17/26] hugetlb: update follow_hugetlb_page to support HGM From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=SANzcf+L; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3Xfa1YgoKCD4oymtzlmytslttlqj.htrqnsz2-rrp0fhp.twl@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Xfa1YgoKCD4oymtzlmytslttlqj.htrqnsz2-rrp0fhp.twl@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092254; a=rsa-sha256; cv=none; b=qUISRroulgOElMN4ZBKLDLhm4OxD4J/OJYVFJZ4sFB5GW6xauFHWRoUdcMF17vtm5rkY6n GuwslJoSrUaIeOcvjxF7gtwRmJwWURk4sN7/r4JDq5VpGlGG+FMsysNy1Phmim7UQ7kFyi rijGPirtILuGqLXFvj7n8/gz4rQCbNE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092254; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4vXUhYS1gVPcFf2LfwHq3p4PeYS4NxljSjceZ18tGG4=; b=EhTqIao2Taek0U4UH7ONkKRPQvnboLV535z00RTLi4cvOf1tfS5ZuFXEokJomNtq4T93Lk /YJwYkjw3rHa2avM4DkLIQ3d9v3sPqJBuUKU8vnP/PkH+m8ZwWjnhlIhpGr1riXnO+ncCy 2WqRGTsY/GY2XO9aVQ2OGnmEW0up84Q= X-Stat-Signature: w6dpbgbjrt651taco3qk4kmqngocw3yj X-Rspamd-Server: rspam08 X-Rspam-User: X-Rspamd-Queue-Id: 4B1D0A002A Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=SANzcf+L; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3Xfa1YgoKCD4oymtzlmytslttlqj.htrqnsz2-rrp0fhp.twl@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Xfa1YgoKCD4oymtzlmytslttlqj.htrqnsz2-rrp0fhp.twl@flex--jthoughton.bounces.google.com X-HE-Tag: 1656092254-623049 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This enables support for GUP, and it is needed for the KVM demand paging self-test to work. One important change here is that, before, we never needed to grab the i_mmap_sem, but now, to prevent someone from collapsing the page tables out from under us, we grab it for reading when doing high-granularity PT walks. Signed-off-by: James Houghton --- mm/hugetlb.c | 70 ++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 57 insertions(+), 13 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index f9c7daa6c090..aadfcee947cf 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6298,14 +6298,18 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long vaddr = *position; unsigned long remainder = *nr_pages; struct hstate *h = hstate_vma(vma); + struct address_space *mapping = vma->vm_file->f_mapping; int err = -EFAULT, refs; + bool has_i_mmap_sem = false; while (vaddr < vma->vm_end && remainder) { pte_t *pte; spinlock_t *ptl = NULL; bool unshare = false; int absent; + unsigned long pages_per_hpte; struct page *page; + struct hugetlb_pte hpte; /* * If we have a pending SIGKILL, don't keep faulting pages and @@ -6325,9 +6329,23 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, */ pte = huge_pte_offset(mm, vaddr & huge_page_mask(h), huge_page_size(h)); - if (pte) - ptl = huge_pte_lock(h, mm, pte); - absent = !pte || huge_pte_none(huge_ptep_get(pte)); + if (pte) { + hugetlb_pte_populate(&hpte, pte, huge_page_shift(h)); + if (hugetlb_hgm_enabled(vma)) { + BUG_ON(has_i_mmap_sem); + i_mmap_lock_read(mapping); + /* + * Need to hold the mapping semaphore for + * reading to do a HGM walk. + */ + has_i_mmap_sem = true; + hugetlb_walk_to(mm, &hpte, vaddr, PAGE_SIZE, + /*stop_at_none=*/true); + } + ptl = hugetlb_pte_lock(mm, &hpte); + } + + absent = !pte || hugetlb_pte_none(&hpte); /* * When coredumping, it suits get_dump_page if we just return @@ -6338,8 +6356,13 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, */ if (absent && (flags & FOLL_DUMP) && !hugetlbfs_pagecache_present(h, vma, vaddr)) { - if (pte) + if (pte) { + if (has_i_mmap_sem) { + i_mmap_unlock_read(mapping); + has_i_mmap_sem = false; + } spin_unlock(ptl); + } remainder = 0; break; } @@ -6359,8 +6382,13 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, vm_fault_t ret; unsigned int fault_flags = 0; - if (pte) + if (pte) { + if (has_i_mmap_sem) { + i_mmap_unlock_read(mapping); + has_i_mmap_sem = false; + } spin_unlock(ptl); + } if (flags & FOLL_WRITE) fault_flags |= FAULT_FLAG_WRITE; else if (unshare) @@ -6403,8 +6431,11 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, continue; } - pfn_offset = (vaddr & ~huge_page_mask(h)) >> PAGE_SHIFT; - page = pte_page(huge_ptep_get(pte)); + pfn_offset = (vaddr & ~hugetlb_pte_mask(&hpte)) >> PAGE_SHIFT; + page = pte_page(hugetlb_ptep_get(&hpte)); + pages_per_hpte = hugetlb_pte_size(&hpte) / PAGE_SIZE; + if (hugetlb_hgm_enabled(vma)) + page = compound_head(page); VM_BUG_ON_PAGE((flags & FOLL_PIN) && PageAnon(page) && !PageAnonExclusive(page), page); @@ -6414,17 +6445,21 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, * and skip the same_page loop below. */ if (!pages && !vmas && !pfn_offset && - (vaddr + huge_page_size(h) < vma->vm_end) && - (remainder >= pages_per_huge_page(h))) { - vaddr += huge_page_size(h); - remainder -= pages_per_huge_page(h); - i += pages_per_huge_page(h); + (vaddr + pages_per_hpte < vma->vm_end) && + (remainder >= pages_per_hpte)) { + vaddr += pages_per_hpte; + remainder -= pages_per_hpte; + i += pages_per_hpte; spin_unlock(ptl); + if (has_i_mmap_sem) { + has_i_mmap_sem = false; + i_mmap_unlock_read(mapping); + } continue; } /* vaddr may not be aligned to PAGE_SIZE */ - refs = min3(pages_per_huge_page(h) - pfn_offset, remainder, + refs = min3(pages_per_hpte - pfn_offset, remainder, (vma->vm_end - ALIGN_DOWN(vaddr, PAGE_SIZE)) >> PAGE_SHIFT); if (pages || vmas) @@ -6447,6 +6482,10 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, if (WARN_ON_ONCE(!try_grab_folio(pages[i], refs, flags))) { spin_unlock(ptl); + if (has_i_mmap_sem) { + has_i_mmap_sem = false; + i_mmap_unlock_read(mapping); + } remainder = 0; err = -ENOMEM; break; @@ -6458,8 +6497,13 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, i += refs; spin_unlock(ptl); + if (has_i_mmap_sem) { + has_i_mmap_sem = false; + i_mmap_unlock_read(mapping); + } } *nr_pages = remainder; + BUG_ON(has_i_mmap_sem); /* * setting position is actually required only if remainder is * not zero but it's faster not to add a "if (remainder)" From patchwork Fri Jun 24 17:36:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894943 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FECDC43334 for ; Fri, 24 Jun 2022 17:37:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9FB198E0250; Fri, 24 Jun 2022 13:37:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 934F08E0244; Fri, 24 Jun 2022 13:37:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6C3BC8E0250; Fri, 24 Jun 2022 13:37:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 535E58E0244 for ; Fri, 24 Jun 2022 13:37:36 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 318A435476 for ; Fri, 24 Jun 2022 17:37:36 +0000 (UTC) X-FDA: 79613836512.25.05D9180 Received: from mail-ua1-f73.google.com (mail-ua1-f73.google.com [209.85.222.73]) by imf20.hostedemail.com (Postfix) with ESMTP id C3C481C0003 for ; Fri, 24 Jun 2022 17:37:35 +0000 (UTC) Received: by mail-ua1-f73.google.com with SMTP id v19-20020ab05593000000b0037ed9894dfbso1009091uaa.19 for ; Fri, 24 Jun 2022 10:37:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=PksJjdHLiQ+Ffd6zQef9Iz4jywxKAioMqz57hWInkDg=; b=CVSOUoyNCnP0rQqtT4BE26Dr49QTmhaHr+EPb5lqChGIPzWFaKi2n/SA62lBxC6G/B 38JLajUBFsj5yNNTJYJpPlG3zt4pbxEEo2Rz5Lr/JGAlR2IwlDVrB8t1N7Y+51hiEkuD gcnSn3SJ6wF8gQl0GgUcoaWzUop7tCNZg00gOvNwMpyc7M5qd3SPq9S8bqdni3sMcV35 kXFlC8wCSqKpXC2OZoAMiwb0Uh8BsCsg9QMBeTSA8RKx5u8y//crTwU2xRxSAPBpji+j 8CrB/r/r4To8R+R8wlCzmWqlOMTd2sXPyHS+kZrv02x6KWrEb//6NFB/h8l/58GK1WFV fg6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=PksJjdHLiQ+Ffd6zQef9Iz4jywxKAioMqz57hWInkDg=; b=qA7Ivo9VmloOHhbHu/8wgzRoVg8LkCWt6GWjjjEUh8ZGIIosG9Ohwt4piSbBDUL9mG pWwvqImfdc63Ufna/JC05ljifqVowf43yQJxR5YJyZS+KaSBAZ16ab1TBr5174NNPxRd EvHxz6Mm4UbMWK0GJTxWWapnCPovo8o3eZCts5gmw9zmN/kzuT61LThHPq7w5bpHgVkT Ipacwqv5rovpy1B+q5Lqu8KEtF3IKaso5xDOSdleCbB0LJjOkfdC4FYNKdMmGhxTd4qb PWXBFtUxC22DbhvLMtAH8b9fRa8Q0tkf3iGHjwCSJE4U1uB/fvX8NUkYy+x65IOAGDks Z/Iw== X-Gm-Message-State: AJIora+8JakYZADh33K42zUQkcJJJ37FHqjhye8tmC5G8Slrx2Gp9ifh 6ahlpt2Cli/fDFFfyAkJxwMTCzlkWLha0Mf5 X-Google-Smtp-Source: AGRyM1uzsUo/NuKSmF0jRF4mJpyyTbf5p62c3XbnqsReOpJTYqhsntb/uUYVHZ67smUJaqTgbmvBTzx9NYHbRHzK X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6122:10c7:b0:36c:1d34:16bc with SMTP id l7-20020a05612210c700b0036c1d3416bcmr17244vko.22.1656092255089; Fri, 24 Jun 2022 10:37:35 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:48 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-19-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 18/26] hugetlb: use struct hugetlb_pte for walk_hugetlb_range From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092255; a=rsa-sha256; cv=none; b=ACGk2MXMlEVewyWD0zl7MN+n9LTsGXKPpBBdCPC3wGI9DJBCl0Ec827hsWG+qgryaFhfG6 X3fjY7Doa6VgBtevea1cXaW4yDKbEDX9EYEkEStYbLeKtDgugIxax01PJklyGE4VI1ke/S LRqwOl8g7Au1bn4DF0UAjzTnbbfCt0A= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=CVSOUoyN; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3X_a1YgoKCEAlvjqwijvqpiqqing.eqonkpwz-oomxcem.qti@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3X_a1YgoKCEAlvjqwijvqpiqqing.eqonkpwz-oomxcem.qti@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092255; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PksJjdHLiQ+Ffd6zQef9Iz4jywxKAioMqz57hWInkDg=; b=TOl/LCh+drMedZroiDPyKpQ+ZnxNzk1/Snagbb02p6OqIKEoW+kdexaDNuWGMiVTjZsA+f FZ91IFfAm4ccP/MnOj6YQv/NE/STBBCnTFQKG0H4pnTtba0Zs/npSnEZAOpYusVYj0zXJ4 e1slWh1RlsqVWr1HTb/H9MKGie7Ep2w= X-Stat-Signature: ccuuixcg8dt9zj3zsyk5pa7nrnk5smc7 X-Rspamd-Queue-Id: C3C481C0003 Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=CVSOUoyN; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3X_a1YgoKCEAlvjqwijvqpiqqing.eqonkpwz-oomxcem.qti@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3X_a1YgoKCEAlvjqwijvqpiqqing.eqonkpwz-oomxcem.qti@flex--jthoughton.bounces.google.com X-Rspamd-Server: rspam10 X-Rspam-User: X-HE-Tag: 1656092255-64276 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Although this change is large, it is somewhat straightforward. Before, all users of walk_hugetlb_range could get the size of the PTE just be checking the hmask or the mm_walk struct. With HGM, that information is held in the hugetlb_pte struct, so we provide that instead of the raw pte_t*. Signed-off-by: James Houghton --- arch/s390/mm/gmap.c | 8 ++++++-- fs/proc/task_mmu.c | 35 +++++++++++++++++++---------------- include/linux/pagewalk.h | 3 ++- mm/damon/vaddr.c | 34 ++++++++++++++++++---------------- mm/hmm.c | 7 ++++--- mm/mempolicy.c | 11 ++++++++--- mm/mincore.c | 4 ++-- mm/mprotect.c | 6 +++--- mm/pagewalk.c | 18 ++++++++++++++++-- 9 files changed, 78 insertions(+), 48 deletions(-) diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index b8ae4a4aa2ba..518cebfd72cd 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -2620,10 +2620,14 @@ static int __s390_enable_skey_pmd(pmd_t *pmd, unsigned long addr, return 0; } -static int __s390_enable_skey_hugetlb(pte_t *pte, unsigned long addr, - unsigned long hmask, unsigned long next, +static int __s390_enable_skey_hugetlb(struct hugetlb_pte *hpte, + unsigned long addr, unsigned long next, struct mm_walk *walk) { + if (!hugetlb_pte_present_leaf(hpte) || + hugetlb_pte_size(hpte) != PMD_SIZE) + return -EINVAL; + pmd_t *pmd = (pmd_t *)pte; unsigned long start, end; struct page *page = pmd_page(*pmd); diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 2d04e3470d4c..b2d683f99fa9 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -714,18 +714,19 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma) } #ifdef CONFIG_HUGETLB_PAGE -static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, +static int smaps_hugetlb_range(struct hugetlb_pte *hpte, unsigned long addr, unsigned long end, struct mm_walk *walk) { struct mem_size_stats *mss = walk->private; struct vm_area_struct *vma = walk->vma; struct page *page = NULL; + pte_t pte = hugetlb_ptep_get(hpte); - if (pte_present(*pte)) { - page = vm_normal_page(vma, addr, *pte); - } else if (is_swap_pte(*pte)) { - swp_entry_t swpent = pte_to_swp_entry(*pte); + if (hugetlb_pte_present_leaf(hpte)) { + page = vm_normal_page(vma, addr, pte); + } else if (is_swap_pte(pte)) { + swp_entry_t swpent = pte_to_swp_entry(pte); if (is_pfn_swap_entry(swpent)) page = pfn_swap_entry_to_page(swpent); @@ -734,9 +735,9 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, int mapcount = page_mapcount(page); if (mapcount >= 2) - mss->shared_hugetlb += huge_page_size(hstate_vma(vma)); + mss->shared_hugetlb += hugetlb_pte_size(hpte); else - mss->private_hugetlb += huge_page_size(hstate_vma(vma)); + mss->private_hugetlb += hugetlb_pte_size(hpte); } return 0; } @@ -1535,7 +1536,7 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, #ifdef CONFIG_HUGETLB_PAGE /* This function walks within one hugetlb entry in the single call */ -static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask, +static int pagemap_hugetlb_range(struct hugetlb_pte *hpte, unsigned long addr, unsigned long end, struct mm_walk *walk) { @@ -1543,13 +1544,13 @@ static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask, struct vm_area_struct *vma = walk->vma; u64 flags = 0, frame = 0; int err = 0; - pte_t pte; + unsigned long hmask = hugetlb_pte_mask(hpte); if (vma->vm_flags & VM_SOFTDIRTY) flags |= PM_SOFT_DIRTY; - pte = huge_ptep_get(ptep); - if (pte_present(pte)) { + if (hugetlb_pte_present_leaf(hpte)) { + pte_t pte = hugetlb_ptep_get(hpte); struct page *page = pte_page(pte); if (!PageAnon(page)) @@ -1565,7 +1566,7 @@ static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask, if (pm->show_pfn) frame = pte_pfn(pte) + ((addr & ~hmask) >> PAGE_SHIFT); - } else if (pte_swp_uffd_wp_any(pte)) { + } else if (pte_swp_uffd_wp_any(hugetlb_ptep_get(hpte))) { flags |= PM_UFFD_WP; } @@ -1869,17 +1870,19 @@ static int gather_pte_stats(pmd_t *pmd, unsigned long addr, return 0; } #ifdef CONFIG_HUGETLB_PAGE -static int gather_hugetlb_stats(pte_t *pte, unsigned long hmask, - unsigned long addr, unsigned long end, struct mm_walk *walk) +static int gather_hugetlb_stats(struct hugetlb_pte *hpte, unsigned long addr, + unsigned long end, struct mm_walk *walk) { - pte_t huge_pte = huge_ptep_get(pte); + pte_t huge_pte = hugetlb_ptep_get(hpte); struct numa_maps *md; struct page *page; - if (!pte_present(huge_pte)) + if (!hugetlb_pte_present_leaf(hpte)) return 0; page = pte_page(huge_pte); + if (page != compound_head(page)) + return 0; md = walk->private; gather_stats(page, md, pte_dirty(huge_pte), 1); diff --git a/include/linux/pagewalk.h b/include/linux/pagewalk.h index ac7b38ad5903..0d21e25df37f 100644 --- a/include/linux/pagewalk.h +++ b/include/linux/pagewalk.h @@ -3,6 +3,7 @@ #define _LINUX_PAGEWALK_H #include +#include struct mm_walk; @@ -47,7 +48,7 @@ struct mm_walk_ops { unsigned long next, struct mm_walk *walk); int (*pte_hole)(unsigned long addr, unsigned long next, int depth, struct mm_walk *walk); - int (*hugetlb_entry)(pte_t *pte, unsigned long hmask, + int (*hugetlb_entry)(struct hugetlb_pte *hpte, unsigned long addr, unsigned long next, struct mm_walk *walk); int (*test_walk)(unsigned long addr, unsigned long next, diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 59e1653799f8..ce50b937dcf2 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -324,14 +324,15 @@ static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned long addr, } #ifdef CONFIG_HUGETLB_PAGE -static void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm, +static void damon_hugetlb_mkold(struct hugetlb_pte *hpte, struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr) { bool referenced = false; pte_t entry = huge_ptep_get(pte); struct page *page = pte_page(entry); + struct page *hpage = compound_head(page); - get_page(page); + get_page(hpage); if (pte_young(entry)) { referenced = true; @@ -342,18 +343,18 @@ static void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm, #ifdef CONFIG_MMU_NOTIFIER if (mmu_notifier_clear_young(mm, addr, - addr + huge_page_size(hstate_vma(vma)))) + addr + hugetlb_pte_size(hpte)); referenced = true; #endif /* CONFIG_MMU_NOTIFIER */ if (referenced) - set_page_young(page); + set_page_young(hpage); - set_page_idle(page); - put_page(page); + set_page_idle(hpage); + put_page(hpage); } -static int damon_mkold_hugetlb_entry(pte_t *pte, unsigned long hmask, +static int damon_mkold_hugetlb_entry(struct hugetlb_pte *hpte, unsigned long addr, unsigned long end, struct mm_walk *walk) { @@ -361,12 +362,12 @@ static int damon_mkold_hugetlb_entry(pte_t *pte, unsigned long hmask, spinlock_t *ptl; pte_t entry; - ptl = huge_pte_lock(h, walk->mm, pte); - entry = huge_ptep_get(pte); + ptl = huge_pte_lock_shift(hpte->shift, walk->mm, hpte->ptep); + entry = huge_ptep_get(hpte->ptep); if (!pte_present(entry)) goto out; - damon_hugetlb_mkold(pte, walk->mm, walk->vma, addr); + damon_hugetlb_mkold(hpte, walk->mm, walk->vma, addr); out: spin_unlock(ptl); @@ -474,31 +475,32 @@ static int damon_young_pmd_entry(pmd_t *pmd, unsigned long addr, } #ifdef CONFIG_HUGETLB_PAGE -static int damon_young_hugetlb_entry(pte_t *pte, unsigned long hmask, +static int damon_young_hugetlb_entry(struct hugetlb_pte *hpte, unsigned long addr, unsigned long end, struct mm_walk *walk) { struct damon_young_walk_private *priv = walk->private; struct hstate *h = hstate_vma(walk->vma); - struct page *page; + struct page *page, *hpage; spinlock_t *ptl; pte_t entry; - ptl = huge_pte_lock(h, walk->mm, pte); + ptl = huge_pte_lock_shift(hpte->shift, walk->mm, hpte->ptep); entry = huge_ptep_get(pte); if (!pte_present(entry)) goto out; page = pte_page(entry); - get_page(page); + hpage = compound_head(page); + get_page(hpage); - if (pte_young(entry) || !page_is_idle(page) || + if (pte_young(entry) || !page_is_idle(hpage) || mmu_notifier_test_young(walk->mm, addr)) { *priv->page_sz = huge_page_size(h); priv->young = true; } - put_page(page); + put_page(hpage); out: spin_unlock(ptl); diff --git a/mm/hmm.c b/mm/hmm.c index 3fd3242c5e50..1ad5d76fa8be 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -472,7 +472,7 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end, #endif #ifdef CONFIG_HUGETLB_PAGE -static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask, +static int hmm_vma_walk_hugetlb_entry(struct hugetlb_pte *hpte, unsigned long start, unsigned long end, struct mm_walk *walk) { @@ -483,11 +483,12 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask, unsigned int required_fault; unsigned long pfn_req_flags; unsigned long cpu_flags; + unsigned long hmask = hugetlb_pte_mask(hpte); spinlock_t *ptl; pte_t entry; - ptl = huge_pte_lock(hstate_vma(vma), walk->mm, pte); - entry = huge_ptep_get(pte); + ptl = huge_pte_lock_shift(hpte->shift, walk->mm, hpte->ptep); + entry = huge_ptep_get(hpte->ptep); i = (start - range->start) >> PAGE_SHIFT; pfn_req_flags = range->hmm_pfns[i]; diff --git a/mm/mempolicy.c b/mm/mempolicy.c index d39b01fd52fe..a1d82db7c19f 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -559,7 +559,7 @@ static int queue_pages_pte_range(pmd_t *pmd, unsigned long addr, return addr != end ? -EIO : 0; } -static int queue_pages_hugetlb(pte_t *pte, unsigned long hmask, +static int queue_pages_hugetlb(struct hugetlb_pte *hpte, unsigned long addr, unsigned long end, struct mm_walk *walk) { @@ -571,8 +571,13 @@ static int queue_pages_hugetlb(pte_t *pte, unsigned long hmask, spinlock_t *ptl; pte_t entry; - ptl = huge_pte_lock(hstate_vma(walk->vma), walk->mm, pte); - entry = huge_ptep_get(pte); + /* We don't migrate high-granularity HugeTLB mappings for now. */ + if (hugetlb_pte_size(hpte) != + huge_page_size(hstate_vma(walk->vma))) + return -EINVAL; + + ptl = hugetlb_pte_lock(walk->mm, hpte); + entry = hugetlb_ptep_get(hpte); if (!pte_present(entry)) goto unlock; page = pte_page(entry); diff --git a/mm/mincore.c b/mm/mincore.c index fa200c14185f..dc1717dc6a2c 100644 --- a/mm/mincore.c +++ b/mm/mincore.c @@ -22,7 +22,7 @@ #include #include "swap.h" -static int mincore_hugetlb(pte_t *pte, unsigned long hmask, unsigned long addr, +static int mincore_hugetlb(struct hugetlb_pte *hpte, unsigned long addr, unsigned long end, struct mm_walk *walk) { #ifdef CONFIG_HUGETLB_PAGE @@ -33,7 +33,7 @@ static int mincore_hugetlb(pte_t *pte, unsigned long hmask, unsigned long addr, * Hugepages under user process are always in RAM and never * swapped out, but theoretically it needs to be checked. */ - present = pte && !huge_pte_none(huge_ptep_get(pte)); + present = hpte->ptep && !hugetlb_pte_none(hpte); for (; addr != end; vec++, addr += PAGE_SIZE) *vec = present; walk->private = vec; diff --git a/mm/mprotect.c b/mm/mprotect.c index ba5592655ee3..9c5a35a1c0eb 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -476,12 +476,12 @@ static int prot_none_pte_entry(pte_t *pte, unsigned long addr, 0 : -EACCES; } -static int prot_none_hugetlb_entry(pte_t *pte, unsigned long hmask, +static int prot_none_hugetlb_entry(struct hugetlb_pte *hpte, unsigned long addr, unsigned long next, struct mm_walk *walk) { - return pfn_modify_allowed(pte_pfn(*pte), *(pgprot_t *)(walk->private)) ? - 0 : -EACCES; + return pfn_modify_allowed(pte_pfn(*hpte->ptep), + *(pgprot_t *)(walk->private)) ? 0 : -EACCES; } static int prot_none_test(unsigned long addr, unsigned long next, diff --git a/mm/pagewalk.c b/mm/pagewalk.c index 9b3db11a4d1d..f8e24a0a0179 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -3,6 +3,7 @@ #include #include #include +#include /* * We want to know the real level where a entry is located ignoring any @@ -301,13 +302,26 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end, pte_t *pte; const struct mm_walk_ops *ops = walk->ops; int err = 0; + struct hugetlb_pte hpte; do { - next = hugetlb_entry_end(h, addr, end); pte = huge_pte_offset(walk->mm, addr & hmask, sz); + if (!pte) { + next = hugetlb_entry_end(h, addr, end); + } else { + hugetlb_pte_populate(&hpte, pte, huge_page_shift(h)); + if (hugetlb_hgm_enabled(vma)) { + err = hugetlb_walk_to(walk->mm, &hpte, addr, + PAGE_SIZE, + /*stop_at_none=*/true); + if (err) + break; + } + next = min(addr + hugetlb_pte_size(&hpte), end); + } if (pte) - err = ops->hugetlb_entry(pte, hmask, addr, next, walk); + err = ops->hugetlb_entry(&hpte, addr, next, walk); else if (ops->pte_hole) err = ops->pte_hole(addr, next, -1, walk); From patchwork Fri Jun 24 17:36:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894944 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E7C7CCA47E for ; Fri, 24 Jun 2022 17:37:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B31AF8E0251; Fri, 24 Jun 2022 13:37:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ABB8A8E0244; Fri, 24 Jun 2022 13:37:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8714A8E0251; Fri, 24 Jun 2022 13:37:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6DC108E0244 for ; Fri, 24 Jun 2022 13:37:38 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4B33AB70 for ; Fri, 24 Jun 2022 17:37:38 +0000 (UTC) X-FDA: 79613836596.27.5CBD609 Received: from mail-ua1-f74.google.com (mail-ua1-f74.google.com [209.85.222.74]) by imf23.hostedemail.com (Postfix) with ESMTP id 5032014002A for ; Fri, 24 Jun 2022 17:37:37 +0000 (UTC) Received: by mail-ua1-f74.google.com with SMTP id j14-20020ab01d0e000000b0037f3ad22193so1059647uak.0 for ; Fri, 24 Jun 2022 10:37:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=BEnHzeA8IqLcgv72Sg9jh80/umPdhd6MjUiQ8hqfSFI=; b=GdzVxrX7wkYjgErfDdqCIkYSvWPbbIDKSxW5Wzna9kLimEZLIF7v/kjfzJ0/MJJI8C Nan7vqP8mjs7qbiftpzOotx0G0VhhFdl+a3xoAwOSm1ekZLV4zffUeKakFvQsIpRI1Zt tqdgg1fUIcbUxn4sKjY6xnWu7yiFtN8U8XUowuMeteG3pczq9XwOEfTu1w4q9aFihIch Gea+lKdHnWOnA7lB1uuW7+Uv8ZFug+g0pv2kbNnErXzZs6E84L9Efj3yIVwEZKkFg8tY dzRXPhe7pxjxGRQKU8w+ukahEWnGGDq1tUSkIQMfu89Z2YI4CkLnSQBu54xqTuD+UzJw axPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=BEnHzeA8IqLcgv72Sg9jh80/umPdhd6MjUiQ8hqfSFI=; b=7OkGp0Jf7oBD4noDCrESY8h5UFNSGmJ7lJHyDnq1F+q6B1Yv6bSLF7LNUuWNpi+YZW fAcH6GpTlvGW5DMN87UEHTeXmx86pZBhAup0Xe6iSOhhztNwX+qOpDeX4+ZUErFUAx5X c7XkF3GLHTWC1HLs+qpW6K2KoRhYE4BNUOobiEudEkeLmBlWSTVES7EnvY8ntXx+RTUx kxa9LZ8yxoOipDtKs11GSdXOh+eqvxrolteXKG9ruN8TUuzcWTklHWCtuas7yycJpH8/ g2t5wHBSU5l38guUixADk4YlekpMR2FsR6EKa3R+aSe95RE5RMaYOLc5msCe0M8p7PxQ kx0w== X-Gm-Message-State: AJIora9Sa02xeJYAIven7o6zjyonq4Nu+GZIfvFaGD9yhrbRqXI2UiFG bLkrycrtjiS1/9u8aOmKqIQLLkGi6Co3dIdd X-Google-Smtp-Source: AGRyM1uf1g766JLfRQ4yT/c5CHXe9SFwBo1833UUdGMaiNufztGVkND1yyG58uFyUB+Rm94Hkh8sbhU0SS7vep8r X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a1f:3456:0:b0:368:ca4e:1bd1 with SMTP id b83-20020a1f3456000000b00368ca4e1bd1mr4581vka.36.1656092256495; Fri, 24 Jun 2022 10:37:36 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:49 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-20-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 19/26] hugetlb: add HGM support for copy_hugetlb_page_range From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=GdzVxrX7; spf=pass (imf23.hostedemail.com: domain of 3YPa1YgoKCEEmwkrxjkwrqjrrjoh.frpolqx0-ppnydfn.ruj@flex--jthoughton.bounces.google.com designates 209.85.222.74 as permitted sender) smtp.mailfrom=3YPa1YgoKCEEmwkrxjkwrqjrrjoh.frpolqx0-ppnydfn.ruj@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092258; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BEnHzeA8IqLcgv72Sg9jh80/umPdhd6MjUiQ8hqfSFI=; b=SL8XZujXyH2+yXoxM+e9ZnM00lVP5GCHUanJ3NVW/MdvihglPx3tecyXpUijY75oKJczGI kiR5l26WTlZjsRoqCecl3ABSdgh3JxsumuuBfX38PofBUayUf7DKkIT7kG2KAwRGt3IjDX BnCkF8rs7s0YaSmLhtKDOs/qy6zpATs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092258; a=rsa-sha256; cv=none; b=zi+sPyd9ehl4q4OtG/WOKNcm/qjynzp4IUdYUSJHZQjWs8cV1rZPT1vNv7LudQoSu01S5b DtMksK4WhWRnd1uJVauY4Cl5z6umasNthnMaw+eyp5o0mdjB5fN5sN91V7FPb/8UiSn7kQ pTTUDgH+RgJeEy5TW8SCzTod0cQgiQA= X-Stat-Signature: q5d8rroyzygij91fkbfwxe8b6cb38o1x X-Rspamd-Queue-Id: 5032014002A X-Rspam-User: Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=GdzVxrX7; spf=pass (imf23.hostedemail.com: domain of 3YPa1YgoKCEEmwkrxjkwrqjrrjoh.frpolqx0-ppnydfn.ruj@flex--jthoughton.bounces.google.com designates 209.85.222.74 as permitted sender) smtp.mailfrom=3YPa1YgoKCEEmwkrxjkwrqjrrjoh.frpolqx0-ppnydfn.ruj@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam02 X-HE-Tag: 1656092257-989069 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This allows fork() to work with high-granularity mappings. The page table structure is copied such that partially mapped regions will remain partially mapped in the same way for the new process. Signed-off-by: James Houghton --- mm/hugetlb.c | 74 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 59 insertions(+), 15 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index aadfcee947cf..0ec2f231524e 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4851,7 +4851,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, struct vm_area_struct *src_vma) { pte_t *src_pte, *dst_pte, entry, dst_entry; - struct page *ptepage; + struct hugetlb_pte src_hpte, dst_hpte; + struct page *ptepage, *hpage; unsigned long addr; bool cow = is_cow_mapping(src_vma->vm_flags); struct hstate *h = hstate_vma(src_vma); @@ -4878,17 +4879,44 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, i_mmap_lock_read(mapping); } - for (addr = src_vma->vm_start; addr < src_vma->vm_end; addr += sz) { + addr = src_vma->vm_start; + while (addr < src_vma->vm_end) { spinlock_t *src_ptl, *dst_ptl; + unsigned long hpte_sz; src_pte = huge_pte_offset(src, addr, sz); - if (!src_pte) + if (!src_pte) { + addr += sz; continue; + } dst_pte = huge_pte_alloc(dst, dst_vma, addr, sz); if (!dst_pte) { ret = -ENOMEM; break; } + hugetlb_pte_populate(&src_hpte, src_pte, huge_page_shift(h)); + hugetlb_pte_populate(&dst_hpte, dst_pte, huge_page_shift(h)); + + if (hugetlb_hgm_enabled(src_vma)) { + BUG_ON(!hugetlb_hgm_enabled(dst_vma)); + ret = hugetlb_walk_to(src, &src_hpte, addr, + PAGE_SIZE, /*stop_at_none=*/true); + if (ret) + break; + ret = huge_pte_alloc_high_granularity( + &dst_hpte, dst, dst_vma, addr, + hugetlb_pte_shift(&src_hpte), + HUGETLB_SPLIT_NONE, + /*write_locked=*/false); + if (ret) + break; + + src_pte = src_hpte.ptep; + dst_pte = dst_hpte.ptep; + } + + hpte_sz = hugetlb_pte_size(&src_hpte); + /* * If the pagetables are shared don't copy or take references. * dst_pte == src_pte is the common case of src/dest sharing. @@ -4899,16 +4927,19 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, * after taking the lock below. */ dst_entry = huge_ptep_get(dst_pte); - if ((dst_pte == src_pte) || !huge_pte_none(dst_entry)) + if ((dst_pte == src_pte) || !hugetlb_pte_none(&dst_hpte)) { + addr += hugetlb_pte_size(&src_hpte); continue; + } - dst_ptl = huge_pte_lock(h, dst, dst_pte); - src_ptl = huge_pte_lockptr(huge_page_shift(h), src, src_pte); + dst_ptl = hugetlb_pte_lock(dst, &dst_hpte); + src_ptl = hugetlb_pte_lockptr(src, &src_hpte); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); entry = huge_ptep_get(src_pte); dst_entry = huge_ptep_get(dst_pte); again: - if (huge_pte_none(entry) || !huge_pte_none(dst_entry)) { + if (hugetlb_pte_none(&src_hpte) || + !hugetlb_pte_none(&dst_hpte)) { /* * Skip if src entry none. Also, skip in the * unlikely case dst entry !none as this implies @@ -4931,11 +4962,12 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, if (userfaultfd_wp(src_vma) && uffd_wp) entry = huge_pte_mkuffd_wp(entry); set_huge_swap_pte_at(src, addr, src_pte, - entry, sz); + entry, hpte_sz); } if (!userfaultfd_wp(dst_vma) && uffd_wp) entry = huge_pte_clear_uffd_wp(entry); - set_huge_swap_pte_at(dst, addr, dst_pte, entry, sz); + set_huge_swap_pte_at(dst, addr, dst_pte, entry, + hpte_sz); } else if (unlikely(is_pte_marker(entry))) { /* * We copy the pte marker only if the dst vma has @@ -4946,7 +4978,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, } else { entry = huge_ptep_get(src_pte); ptepage = pte_page(entry); - get_page(ptepage); + hpage = compound_head(ptepage); + get_page(hpage); /* * Failing to duplicate the anon rmap is a rare case @@ -4959,9 +4992,16 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, * sleep during the process. */ if (!PageAnon(ptepage)) { - page_dup_file_rmap(ptepage, true); + /* Only dup_rmap once for a page */ + if (IS_ALIGNED(addr, sz)) + page_dup_file_rmap(hpage, true); } else if (page_try_dup_anon_rmap(ptepage, true, src_vma)) { + if (hugetlb_hgm_enabled(src_vma)) { + ret = -EINVAL; + break; + } + BUG_ON(!IS_ALIGNED(addr, hugetlb_pte_size(&src_hpte))); pte_t src_pte_old = entry; struct page *new; @@ -4970,13 +5010,13 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, /* Do not use reserve as it's private owned */ new = alloc_huge_page(dst_vma, addr, 1); if (IS_ERR(new)) { - put_page(ptepage); + put_page(hpage); ret = PTR_ERR(new); break; } - copy_user_huge_page(new, ptepage, addr, dst_vma, + copy_user_huge_page(new, hpage, addr, dst_vma, npages); - put_page(ptepage); + put_page(hpage); /* Install the new huge page if src pte stable */ dst_ptl = huge_pte_lock(h, dst, dst_pte); @@ -4994,6 +5034,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, hugetlb_install_page(dst_vma, dst_pte, addr, new); spin_unlock(src_ptl); spin_unlock(dst_ptl); + addr += hugetlb_pte_size(&src_hpte); continue; } @@ -5010,10 +5051,13 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, } set_huge_pte_at(dst, addr, dst_pte, entry); - hugetlb_count_add(npages, dst); + hugetlb_count_add( + hugetlb_pte_size(&dst_hpte) / PAGE_SIZE, + dst); } spin_unlock(src_ptl); spin_unlock(dst_ptl); + addr += hugetlb_pte_size(&src_hpte); } if (cow) { From patchwork Fri Jun 24 17:36:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894945 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A52EC43334 for ; Fri, 24 Jun 2022 17:37:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 84BA78E0252; Fri, 24 Jun 2022 13:37:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7D4298E0244; Fri, 24 Jun 2022 13:37:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 58A348E0252; Fri, 24 Jun 2022 13:37:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 407308E0244 for ; Fri, 24 Jun 2022 13:37:39 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0C93C209B7 for ; Fri, 24 Jun 2022 17:37:39 +0000 (UTC) X-FDA: 79613836638.16.A9AAAE4 Received: from mail-ua1-f73.google.com (mail-ua1-f73.google.com [209.85.222.73]) by imf30.hostedemail.com (Postfix) with ESMTP id 8E61C8000A for ; Fri, 24 Jun 2022 17:37:38 +0000 (UTC) Received: by mail-ua1-f73.google.com with SMTP id b10-20020ab0238a000000b0037efa0a4ba0so1013520uan.11 for ; Fri, 24 Jun 2022 10:37:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=eidAN3wZMLr6mteQRN/ZtvQWu3ENTPRsJrFQKIye6Xw=; b=ifSq4WGIJ2MCNV5i5c6CLm1k1Z7n2P0GnMgBr+kmdUIdSKCtXRvtwj81oWCo0YoMrQ IcDr8n7yQfUrrSxAkN1YXwEqM5IfWAKR1PNsnVeRNVm2rrNwgsnyKXFCZPsKFz18eJJi AjC4fr/5aG8LaYEcmLNbETyNpFrp12C+EhOfYQCwYxaqx8OG+RRnBg3h+rYInBsaXiGS KRXgjFMiQk5PwqNRHbBuU728wz+WYRDzZh3FNG4M1JnAhfkHDi44T6MgrmY0SGp9Cbs2 7hO2DcqZyx877IoqKOW+5z0GVcwNVXUbeq0rR28lZa597DKuk/BcTnZ/dqDW4zwXaf5J ajig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=eidAN3wZMLr6mteQRN/ZtvQWu3ENTPRsJrFQKIye6Xw=; b=aN5Xd6mAnrQA4qCzigc9e/qvPdxlimVwKujPDfoDhTYhowHhgm1yISD0j5yX1xLbGt vMGML9LgcC6jEF4HEfey/oX3N8DuR3eDh6gRt6p0EcTugOFFcDAk+yeUcX4ciuVP/xkY baY426/cgcjR4gzOwbtW0j3NAZ/7z9MDxk6Y1daWXPBGWRBS/TmemUP9N7+Xk1xnULnn iYnp2FjG1hMBEMxbA4dB8Q1zBVTwReegKaUtVp/qmiJLmOtn9uXe6Q21g+x37QxjuvM6 D1PbuNUufuvpL3NdeEIP27PBvfGhitihvix7Io8kozpuFs2fvkMlHsTGRiMFqloMc915 kPxQ== X-Gm-Message-State: AJIora9gjlGsnnVZ0rqFg3sYPAZt1BPTNhxh8p95u33HBnbfBtQwtlep fteCX2WYWUKxKtswcTRccVr0kxVXDx9VOAiB X-Google-Smtp-Source: AGRyM1vWVTZJ5pVWjn7x2PEWVIBLimy1j0CTpYliO/gy+6d80cSvRPXbRG7kYQht6u2HA8oY0zQFqOyEJ9POXahm X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a67:5c41:0:b0:356:20ab:2f29 with SMTP id q62-20020a675c41000000b0035620ab2f29mr1454923vsb.63.1656092257795; Fri, 24 Jun 2022 10:37:37 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:50 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-21-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 20/26] hugetlb: add support for high-granularity UFFDIO_CONTINUE From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092258; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eidAN3wZMLr6mteQRN/ZtvQWu3ENTPRsJrFQKIye6Xw=; b=EQRPAkh8iJCZXfiO8PNVkiz09xH0o0R82txgBho17DHj9uVHacP5dQz0gw4h/WlhbKyaPt 1pmnA/IL+ivd/SYZEeko5glRTnAHvIN5OvUjuyqaJcnG09xt2USff0MvkJSFV+QZVXZXNX o6WNedOe2GBXBppUZHkYbsQOqJd7d0w= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092258; a=rsa-sha256; cv=none; b=hUBMKg8IzVaHu1kzXatFFnyAZxMgopX/3PwaXqBVUsNfLJ7niIetIV3S5BFV8gDrps6oqs y52iSfMN3QZvT/0mYCyZsVIw9Xky1H+Nk0MWeC1cOlNcxOXFyEHKbJLsDBRnfCakLXkwuH rVJ7ULjpeqrMUn7QEwPdHOBePoMGgFo= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ifSq4WGI; spf=pass (imf30.hostedemail.com: domain of 3Yfa1YgoKCEInxlsyklxsrksskpi.gsqpmry1-qqozego.svk@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3Yfa1YgoKCEInxlsyklxsrksskpi.gsqpmry1-qqozego.svk@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ifSq4WGI; spf=pass (imf30.hostedemail.com: domain of 3Yfa1YgoKCEInxlsyklxsrksskpi.gsqpmry1-qqozego.svk@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3Yfa1YgoKCEInxlsyklxsrksskpi.gsqpmry1-qqozego.svk@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 8E61C8000A X-Stat-Signature: 9kwcapxhmh5868z58z77p1ecf99wed73 X-HE-Tag: 1656092258-424301 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The changes here are very similar to the changes made to hugetlb_no_page, where we do a high-granularity page table walk and do accounting slightly differently because we are mapping only a piece of a page. Signed-off-by: James Houghton --- fs/userfaultfd.c | 3 +++ include/linux/hugetlb.h | 6 +++-- mm/hugetlb.c | 54 +++++++++++++++++++++----------------- mm/userfaultfd.c | 57 +++++++++++++++++++++++++++++++---------- 4 files changed, 82 insertions(+), 38 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index e943370107d0..77c1b8a7d0b9 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -245,6 +245,9 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx, if (!ptep) goto out; + if (hugetlb_hgm_enabled(vma)) + goto out; + ret = false; pte = huge_ptep_get(ptep); diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index ac4ac8fbd901..c207b1ac6195 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -221,13 +221,15 @@ unsigned long hugetlb_total_pages(void); vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, unsigned int flags); #ifdef CONFIG_USERFAULTFD -int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, pte_t *dst_pte, +int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, + struct hugetlb_pte *dst_hpte, struct vm_area_struct *dst_vma, unsigned long dst_addr, unsigned long src_addr, enum mcopy_atomic_mode mode, struct page **pagep, - bool wp_copy); + bool wp_copy, + bool new_mapping); #endif /* CONFIG_USERFAULTFD */ bool hugetlb_reserve_pages(struct inode *inode, long from, long to, struct vm_area_struct *vma, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 0ec2f231524e..09fa57599233 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5808,6 +5808,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, vma_end_reservation(h, vma, haddr); } + /* This lock will get pretty expensive at 4K. */ ptl = hugetlb_pte_lock(mm, hpte); ret = 0; /* If pte changed from under us, retry */ @@ -6098,24 +6099,26 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * modifications for huge pages. */ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, - pte_t *dst_pte, + struct hugetlb_pte *dst_hpte, struct vm_area_struct *dst_vma, unsigned long dst_addr, unsigned long src_addr, enum mcopy_atomic_mode mode, struct page **pagep, - bool wp_copy) + bool wp_copy, + bool new_mapping) { bool is_continue = (mode == MCOPY_ATOMIC_CONTINUE); struct hstate *h = hstate_vma(dst_vma); struct address_space *mapping = dst_vma->vm_file->f_mapping; + unsigned long haddr = dst_addr & huge_page_mask(h); pgoff_t idx = vma_hugecache_offset(h, dst_vma, dst_addr); unsigned long size; int vm_shared = dst_vma->vm_flags & VM_SHARED; pte_t _dst_pte; spinlock_t *ptl; int ret = -ENOMEM; - struct page *page; + struct page *page, *subpage; int writable; bool page_in_pagecache = false; @@ -6130,12 +6133,12 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, * a non-missing case. Return -EEXIST. */ if (vm_shared && - hugetlbfs_pagecache_present(h, dst_vma, dst_addr)) { + hugetlbfs_pagecache_present(h, dst_vma, haddr)) { ret = -EEXIST; goto out; } - page = alloc_huge_page(dst_vma, dst_addr, 0); + page = alloc_huge_page(dst_vma, haddr, 0); if (IS_ERR(page)) { ret = -ENOMEM; goto out; @@ -6151,13 +6154,13 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, /* Free the allocated page which may have * consumed a reservation. */ - restore_reserve_on_error(h, dst_vma, dst_addr, page); + restore_reserve_on_error(h, dst_vma, haddr, page); put_page(page); /* Allocate a temporary page to hold the copied * contents. */ - page = alloc_huge_page_vma(h, dst_vma, dst_addr); + page = alloc_huge_page_vma(h, dst_vma, haddr); if (!page) { ret = -ENOMEM; goto out; @@ -6171,14 +6174,14 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, } } else { if (vm_shared && - hugetlbfs_pagecache_present(h, dst_vma, dst_addr)) { + hugetlbfs_pagecache_present(h, dst_vma, haddr)) { put_page(*pagep); ret = -EEXIST; *pagep = NULL; goto out; } - page = alloc_huge_page(dst_vma, dst_addr, 0); + page = alloc_huge_page(dst_vma, haddr, 0); if (IS_ERR(page)) { ret = -ENOMEM; *pagep = NULL; @@ -6216,8 +6219,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, page_in_pagecache = true; } - ptl = huge_pte_lockptr(huge_page_shift(h), dst_mm, dst_pte); - spin_lock(ptl); + ptl = hugetlb_pte_lock(dst_mm, dst_hpte); /* * Recheck the i_size after holding PT lock to make sure not @@ -6239,14 +6241,16 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, * registered, we firstly wr-protect a none pte which has no page cache * page backing it, then access the page. */ - if (!huge_pte_none_mostly(huge_ptep_get(dst_pte))) + if (!hugetlb_pte_none_mostly(dst_hpte)) goto out_release_unlock; - if (vm_shared) { - page_dup_file_rmap(page, true); - } else { - ClearHPageRestoreReserve(page); - hugepage_add_new_anon_rmap(page, dst_vma, dst_addr); + if (new_mapping) { + if (vm_shared) { + page_dup_file_rmap(page, true); + } else { + ClearHPageRestoreReserve(page); + hugepage_add_new_anon_rmap(page, dst_vma, haddr); + } } /* @@ -6258,7 +6262,11 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, else writable = dst_vma->vm_flags & VM_WRITE; - _dst_pte = make_huge_pte(dst_vma, page, writable); + subpage = hugetlb_find_subpage(h, page, dst_addr); + if (subpage != page) + BUG_ON(!hugetlb_hgm_enabled(dst_vma)); + + _dst_pte = make_huge_pte(dst_vma, subpage, writable); /* * Always mark UFFDIO_COPY page dirty; note that this may not be * extremely important for hugetlbfs for now since swapping is not @@ -6271,14 +6279,14 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, if (wp_copy) _dst_pte = huge_pte_mkuffd_wp(_dst_pte); - set_huge_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte); + set_huge_pte_at(dst_mm, dst_addr, dst_hpte->ptep, _dst_pte); - (void)huge_ptep_set_access_flags(dst_vma, dst_addr, dst_pte, _dst_pte, - dst_vma->vm_flags & VM_WRITE); - hugetlb_count_add(pages_per_huge_page(h), dst_mm); + (void)huge_ptep_set_access_flags(dst_vma, dst_addr, dst_hpte->ptep, + _dst_pte, dst_vma->vm_flags & VM_WRITE); + hugetlb_count_add(hugetlb_pte_size(dst_hpte) / PAGE_SIZE, dst_mm); /* No need to invalidate - it was non-present before */ - update_mmu_cache(dst_vma, dst_addr, dst_pte); + update_mmu_cache(dst_vma, dst_addr, dst_hpte->ptep); spin_unlock(ptl); if (!is_continue) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 4f4892a5f767..ee40d98068bf 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -310,14 +310,16 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, { int vm_shared = dst_vma->vm_flags & VM_SHARED; ssize_t err; - pte_t *dst_pte; unsigned long src_addr, dst_addr; long copied; struct page *page; - unsigned long vma_hpagesize; + unsigned long vma_hpagesize, vma_altpagesize; pgoff_t idx; u32 hash; struct address_space *mapping; + bool use_hgm = hugetlb_hgm_enabled(dst_vma) && + mode == MCOPY_ATOMIC_CONTINUE; + struct hstate *h = hstate_vma(dst_vma); /* * There is no default zero huge page for all huge page sizes as @@ -335,12 +337,16 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, copied = 0; page = NULL; vma_hpagesize = vma_kernel_pagesize(dst_vma); + if (use_hgm) + vma_altpagesize = PAGE_SIZE; + else + vma_altpagesize = vma_hpagesize; /* * Validate alignment based on huge page size */ err = -EINVAL; - if (dst_start & (vma_hpagesize - 1) || len & (vma_hpagesize - 1)) + if (dst_start & (vma_altpagesize - 1) || len & (vma_altpagesize - 1)) goto out_unlock; retry: @@ -361,6 +367,8 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, vm_shared = dst_vma->vm_flags & VM_SHARED; } + BUG_ON(!vm_shared && use_hgm); + /* * If not shared, ensure the dst_vma has a anon_vma. */ @@ -371,11 +379,13 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, } while (src_addr < src_start + len) { + struct hugetlb_pte hpte; + bool new_mapping; BUG_ON(dst_addr >= dst_start + len); /* * Serialize via i_mmap_rwsem and hugetlb_fault_mutex. - * i_mmap_rwsem ensures the dst_pte remains valid even + * i_mmap_rwsem ensures the hpte.ptep remains valid even * in the case of shared pmds. fault mutex prevents * races with other faulting threads. */ @@ -383,27 +393,47 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, i_mmap_lock_read(mapping); idx = linear_page_index(dst_vma, dst_addr); hash = hugetlb_fault_mutex_hash(mapping, idx); + /* This lock will get expensive at 4K. */ mutex_lock(&hugetlb_fault_mutex_table[hash]); - err = -ENOMEM; - dst_pte = huge_pte_alloc(dst_mm, dst_vma, dst_addr, vma_hpagesize); - if (!dst_pte) { + err = 0; + + pte_t *ptep = huge_pte_alloc(dst_mm, dst_vma, dst_addr, + vma_hpagesize); + if (!ptep) + err = -ENOMEM; + else { + hugetlb_pte_populate(&hpte, ptep, + huge_page_shift(h)); + /* + * If the hstate-level PTE is not none, then a mapping + * was previously established. + * The per-hpage mutex prevents double-counting. + */ + new_mapping = hugetlb_pte_none(&hpte); + if (use_hgm) + err = hugetlb_alloc_largest_pte(&hpte, dst_mm, dst_vma, + dst_addr, + dst_start + len); + } + + if (err) { mutex_unlock(&hugetlb_fault_mutex_table[hash]); i_mmap_unlock_read(mapping); goto out_unlock; } if (mode != MCOPY_ATOMIC_CONTINUE && - !huge_pte_none_mostly(huge_ptep_get(dst_pte))) { + !hugetlb_pte_none_mostly(&hpte)) { err = -EEXIST; mutex_unlock(&hugetlb_fault_mutex_table[hash]); i_mmap_unlock_read(mapping); goto out_unlock; } - err = hugetlb_mcopy_atomic_pte(dst_mm, dst_pte, dst_vma, + err = hugetlb_mcopy_atomic_pte(dst_mm, &hpte, dst_vma, dst_addr, src_addr, mode, &page, - wp_copy); + wp_copy, new_mapping); mutex_unlock(&hugetlb_fault_mutex_table[hash]); i_mmap_unlock_read(mapping); @@ -413,6 +443,7 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, if (unlikely(err == -ENOENT)) { mmap_read_unlock(dst_mm); BUG_ON(!page); + BUG_ON(hpte.shift != huge_page_shift(h)); err = copy_huge_page_from_user(page, (const void __user *)src_addr, @@ -430,9 +461,9 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm, BUG_ON(page); if (!err) { - dst_addr += vma_hpagesize; - src_addr += vma_hpagesize; - copied += vma_hpagesize; + dst_addr += hugetlb_pte_size(&hpte); + src_addr += hugetlb_pte_size(&hpte); + copied += hugetlb_pte_size(&hpte); if (fatal_signal_pending(current)) err = -EINTR; From patchwork Fri Jun 24 17:36:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894946 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83091CCA473 for ; Fri, 24 Jun 2022 17:37:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E3ED78E0253; Fri, 24 Jun 2022 13:37:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DC84A8E0244; Fri, 24 Jun 2022 13:37:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE27E8E0253; Fri, 24 Jun 2022 13:37:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 970568E0244 for ; Fri, 24 Jun 2022 13:37:40 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 796E11204A4 for ; Fri, 24 Jun 2022 17:37:40 +0000 (UTC) X-FDA: 79613836680.05.55AC3DD Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf17.hostedemail.com (Postfix) with ESMTP id 05BE340024 for ; Fri, 24 Jun 2022 17:37:39 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id d6-20020a256806000000b00668a3d90e95so2784645ybc.2 for ; Fri, 24 Jun 2022 10:37:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=qZvlqd2OvHLqh9twYpoJb6ja2/RhkWxX/89zPHTcn/8=; b=HypVRxxS8PLtYfgYxeV9SmynEG7TPz9Ce9bLT5zNY+BqsBZ2dR4Yu4comEv+XUubMf Tmv+PSPoTEgf52wvkhnc+37jzKxSIz2QTdTgR7IQrEhC+hxTC0PzE5yPLYSLsF+ypT+o 1OeHChfqJn7nF+ydLkcR+31+kCdErcxCg8RBgd1BXnunKktxWtdnnRX0+mbYnTgG3k0r JrUchYTC7GVTsL0xlXQAvH8al5VjVjNSgxGZ6hOsGdc8TeDaNczkB4rHzNv7Qy1NkMg2 e5uS8qHvOfJ8S8UR/qFyiENp/V1amoyWL0FT1OODwP+y6Qxkpk2El/2APigbVyzdYVIR wp2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=qZvlqd2OvHLqh9twYpoJb6ja2/RhkWxX/89zPHTcn/8=; b=zaso7KLsK7ULZxBPH2Io/DKLGWW24XF04gKCK97cF6MMMQrrFrW2BUqhVSWlWEF7Uf muEBhtOKYC3qy1uW8v9031zpdLZCGmE/X8RTJTqtqXrJQpmUdByETQ7JJbtVfXugRIAh fnGPhAxZlIy7xlVc32tmc2gYkkcmu5FblIACq/Fnz2QhIY14j7Pd7FwyDFhxiZEhK1to NVZfCBxGMXgBDyUOWISryydObJv/SE5renpfdnMYXm2q2ubwihYrVLiEHn4DVCAz333B E8aR+L1XpymvVY7NyEz44XwlctRfGdi1DlsoK7NsSwig3OFgAD5KXi0LPsM1+NIpfRb1 92Mg== X-Gm-Message-State: AJIora8CPaDmEEqD90dfC0quP+WqHcUJp08MvqukbUl5AfFeDZ7w8gbs q715TbllQ4+p0nTHEw3baUK/V0Ek0TAiHVwh X-Google-Smtp-Source: AGRyM1vcrrrISl0XVFny3ew+g+bNEBjQOnnF9F+zSGfcQuntVV3TKgvjvU6TjNDpiUuNx/38+gEcAslpNt52T6De X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a25:c7cb:0:b0:664:67aa:36a7 with SMTP id w194-20020a25c7cb000000b0066467aa36a7mr298952ybe.548.1656092259292; Fri, 24 Jun 2022 10:37:39 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:51 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-22-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 21/26] hugetlb: add hugetlb_collapse From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092260; a=rsa-sha256; cv=none; b=AFp4t6d4mdmGH0rWrPEd2ymg2Lf5DFXhLHZfX0nyUjp3hAtIXtfONvDvRUsbmJzCCI4hhy eHQEPdBMWutu+aTeBPWwjEBznDfeUupM6dddPMcfBZSRHJ6ik8myneupNxYQVYvB/KsxKU 4ZlehUt8W6to2nyi+lvbx+7iIpwNOqc= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=HypVRxxS; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3Y_a1YgoKCEQpznu0mnzutmuumrk.iusrot03-ssq1giq.uxm@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3Y_a1YgoKCEQpznu0mnzutmuumrk.iusrot03-ssq1giq.uxm@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092260; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qZvlqd2OvHLqh9twYpoJb6ja2/RhkWxX/89zPHTcn/8=; b=BELVpox2EmL2V5Xw8DuQNGj+iPNnB8E9L88J4BvZJy0eneakgx3cV0/NQ6EiKNLaSk+LJP fYVWVXxb/xcHkFAuYopnw5XKaM7htQpPqcGjS/81p5S7iJ6siuKGT5ukZ8GRCRA3kMK1En Qi4nshyWW5G+c9Gaz0b4q6iZZDBo/RI= X-Stat-Signature: zo1pb53acds7kfz9zastaq8jh5iiex89 X-Rspamd-Queue-Id: 05BE340024 Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=HypVRxxS; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3Y_a1YgoKCEQpznu0mnzutmuumrk.iusrot03-ssq1giq.uxm@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3Y_a1YgoKCEQpznu0mnzutmuumrk.iusrot03-ssq1giq.uxm@flex--jthoughton.bounces.google.com X-Rspamd-Server: rspam10 X-Rspam-User: X-HE-Tag: 1656092259-614059 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is what implements MADV_COLLAPSE for HugeTLB pages. This is a necessary extension to the UFFDIO_CONTINUE changes. When userspace finishes mapping an entire hugepage with UFFDIO_CONTINUE, the kernel has no mechanism to automatically collapse the page table to map the whole hugepage normally. We require userspace to inform us that they would like the hugepages to be collapsed; they do this with MADV_COLLAPSE. If userspace has not mapped all of a hugepage with UFFDIO_CONTINUE, but only some, hugetlb_collapse will cause the requested range to be mapped as if it were UFFDIO_CONTINUE'd already. Signed-off-by: James Houghton --- include/linux/hugetlb.h | 7 ++++ mm/hugetlb.c | 88 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index c207b1ac6195..438057dc3b75 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -1197,6 +1197,8 @@ int huge_pte_alloc_high_granularity(struct hugetlb_pte *hpte, unsigned int desired_sz, enum split_mode mode, bool write_locked); +int hugetlb_collapse(struct mm_struct *mm, struct vm_area_struct *vma, + unsigned long start, unsigned long end); #else static inline bool hugetlb_hgm_enabled(struct vm_area_struct *vma) { @@ -1221,6 +1223,11 @@ static inline int huge_pte_alloc_high_granularity(struct hugetlb_pte *hpte, { return -EINVAL; } +int hugetlb_collapse(struct mm_struct *mm, struct vm_area_struct *vma, + unsigned long start, unsigned long end) +{ + return -EINVAL; +} #endif static inline spinlock_t *huge_pte_lock(struct hstate *h, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 09fa57599233..70bb3a1342d9 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -7280,6 +7280,94 @@ int hugetlb_alloc_largest_pte(struct hugetlb_pte *hpte, struct mm_struct *mm, return -EINVAL; } +/* + * Collapse the address range from @start to @end to be mapped optimally. + * + * This is only valid for shared mappings. The main use case for this function + * is following UFFDIO_CONTINUE. If a user UFFDIO_CONTINUEs an entire hugepage + * by calling UFFDIO_CONTINUE once for each 4K region, the kernel doesn't know + * to collapse the mapping after the final UFFDIO_CONTINUE. Instead, we leave + * it up to userspace to tell us to do so, via MADV_COLLAPSE. + * + * Any holes in the mapping will be filled. If there is no page in the + * pagecache for a region we're collapsing, the PTEs will be cleared. + */ +int hugetlb_collapse(struct mm_struct *mm, struct vm_area_struct *vma, + unsigned long start, unsigned long end) +{ + struct hstate *h = hstate_vma(vma); + struct address_space *mapping = vma->vm_file->f_mapping; + struct mmu_notifier_range range; + struct mmu_gather tlb; + struct hstate *tmp_h; + unsigned int shift; + unsigned long curr = start; + int ret = 0; + struct page *hpage, *subpage; + pgoff_t idx; + bool writable = vma->vm_flags & VM_WRITE; + bool shared = vma->vm_flags & VM_SHARED; + pte_t entry; + + /* + * This is only supported for shared VMAs, because we need to look up + * the page to use for any PTEs we end up creating. + */ + if (!shared) + return -EINVAL; + + i_mmap_assert_write_locked(mapping); + + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, + start, end); + mmu_notifier_invalidate_range_start(&range); + tlb_gather_mmu(&tlb, mm); + + while (curr < end) { + for_each_hgm_shift(h, tmp_h, shift) { + unsigned long sz = 1UL << shift; + struct hugetlb_pte hpte; + + if (!IS_ALIGNED(curr, sz) || curr + sz > end) + continue; + + hugetlb_pte_init(&hpte); + ret = hugetlb_walk_to(mm, &hpte, curr, sz, + /*stop_at_none=*/false); + if (ret) + goto out; + if (hugetlb_pte_size(&hpte) >= sz) + goto hpte_finished; + + idx = vma_hugecache_offset(h, vma, curr); + hpage = find_lock_page(mapping, idx); + hugetlb_free_range(&tlb, &hpte, curr, + curr + hugetlb_pte_size(&hpte)); + if (!hpage) { + hugetlb_pte_clear(mm, &hpte, curr); + goto hpte_finished; + } + + subpage = hugetlb_find_subpage(h, hpage, curr); + entry = make_huge_pte_with_shift(vma, subpage, + writable, shift); + set_huge_pte_at(mm, curr, hpte.ptep, entry); + unlock_page(hpage); +hpte_finished: + curr += hugetlb_pte_size(&hpte); + goto next; + } + ret = -EINVAL; + goto out; +next: + continue; + } +out: + tlb_finish_mmu(&tlb); + mmu_notifier_invalidate_range_end(&range); + return ret; +} + /* * Given a particular address, split the HugeTLB PTE that currently maps it * so that, for the given address, the PTE that maps it is `desired_shift`. From patchwork Fri Jun 24 17:36:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894947 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36ED0C43334 for ; Fri, 24 Jun 2022 17:37:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A2C88E0254; Fri, 24 Jun 2022 13:37:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3DD3D8E0244; Fri, 24 Jun 2022 13:37:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1BAAA8E0254; Fri, 24 Jun 2022 13:37:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id F33048E0244 for ; Fri, 24 Jun 2022 13:37:41 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D4CE260B6F for ; Fri, 24 Jun 2022 17:37:41 +0000 (UTC) X-FDA: 79613836722.25.0C5FA14 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf29.hostedemail.com (Postfix) with ESMTP id 6263B120010 for ; Fri, 24 Jun 2022 17:37:41 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-317ab78a345so26751437b3.10 for ; Fri, 24 Jun 2022 10:37:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=HisKs0OAYNP82NuB/Frk0iPqJLb2lc+sAZjNm287jcQ=; b=P/aHZ9fJlVwvkJfaZVFajZm98KOIfMj2IYppFM5LuQ1zf9EsSZ8qMdlGw+rfUUepxT Rk4rneqzgNI3hBkLTvsDOw/sovXOPg7QHxBlzbSMDv947qVGHCYD/AKf6PpAucF+SB8+ CPR/TNLo08h/tyhtM460hIdHHPcABjdzuulR5lTk4iFehrldnIL1GfqZda2In7YFtggB usvpcPQWqZ0WphYnX3EhkMBx+Typu02xyZyvlY2VIUMvBbNVFv8qTl1N/MQr/QOO/hLC HQXC2pLyWYxCtUrqnUx0Na56WcsA/KgE6sPe3nZndOHiUWkRsKxAtTwz+BZWdqDY7Yot OYtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=HisKs0OAYNP82NuB/Frk0iPqJLb2lc+sAZjNm287jcQ=; b=PUhnka9CJFqcMn3z/UG5Zbep+CjOKZ5H8p35XZ7P69YUNadsHFXfQ/585McCJ3Gdb3 hwj6K1IytY0qrD5GRiah3C6wJSSDVds7rkjBvxigl4VobZpUx81TU2/u6h8PRu7ls9hQ dClN/h6VU0L4KCaJpztAU4kwgZrlWDSj01sAIFeTmIjFBc2asIfOG+1MsM/B6FcBA2X7 cwUtKc2/sRpcwbJhedBH+9VoRgeXa1DF7mx06xHe1k+hV0IdUuM82Jj2OBNWALqpRLKC QDaP/GtYy3w/kyTKcJwcC1FWSIlmSCc4/OKnkimJ2+T7YyBpXwLLUvYEg1rn0GpdkulL DIaw== X-Gm-Message-State: AJIora+daE/j9Iy6ikeSyAJIFnra1EyQ2I5FO9zVQwQfFIJEWEOK4CQo D2l7lXloYGm8/M2A/K75ds/HsjEyale9Xyfy X-Google-Smtp-Source: AGRyM1tzk7GXyftSzcieFt/kefrccg/jUZ3bHYFjHpboz2q85ML1dIMb8Z3beFS8X7RsMAKgtTvTxLeNH4OzIoqy X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a0d:e60e:0:b0:317:893c:90a3 with SMTP id p14-20020a0de60e000000b00317893c90a3mr18394744ywe.241.1656092260720; Fri, 24 Jun 2022 10:37:40 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:52 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-23-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 22/26] madvise: add uapi for HugeTLB HGM collapse: MADV_COLLAPSE From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092261; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HisKs0OAYNP82NuB/Frk0iPqJLb2lc+sAZjNm287jcQ=; b=69kDR0zIqgx9yX1JCkXot1yqVFDYINbYW7Qh548ADUKFGCQoZ3UqwIzQYfXvOznl6AOCks S6Ps46SqOhCQuNJKQ2Gb0Zir/zh2sRf/GYlhDLvtW5dTLMRn89Lu6wumWx1UASz2SwyNEj ZXFNo14DZwzIQFao1sL7KAgPd6/8dOY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092261; a=rsa-sha256; cv=none; b=y2X4YzzmlKRq2d8UpvOA48vloNzCJXf0usO0L0cnuFgL9iKuZnuQA/OhM2DoV1BkPhknBY XZ4mrKtxkPgQe9m6L3iCbL0jTXi6yQjLBlH6+ecqU17ylsp3f04NmpsEs86HKC/lSN00AO FbTP/Afu1zW32nvAVCOdqMTcdTb2/5w= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="P/aHZ9fJ"; spf=pass (imf29.hostedemail.com: domain of 3ZPa1YgoKCEUq0ov1no0vunvvnsl.jvtspu14-ttr2hjr.vyn@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3ZPa1YgoKCEUq0ov1no0vunvvnsl.jvtspu14-ttr2hjr.vyn@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="P/aHZ9fJ"; spf=pass (imf29.hostedemail.com: domain of 3ZPa1YgoKCEUq0ov1no0vunvvnsl.jvtspu14-ttr2hjr.vyn@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3ZPa1YgoKCEUq0ov1no0vunvvnsl.jvtspu14-ttr2hjr.vyn@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 6263B120010 X-Stat-Signature: iuhmha9z7ror8ojffboi435yw3xs8sso X-HE-Tag: 1656092261-121895 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This commit is co-opting the same madvise mode that is being introduced by zokeefe@google.com to manually collapse THPs[1]. As with the rest of the high-granularity mapping support, MADV_COLLAPSE is only supported for shared VMAs right now. [1] https://lore.kernel.org/linux-mm/20220604004004.954674-10-zokeefe@google.com/ Signed-off-by: James Houghton --- include/uapi/asm-generic/mman-common.h | 2 ++ mm/madvise.c | 23 +++++++++++++++++++++++ 2 files changed, 25 insertions(+) diff --git a/include/uapi/asm-generic/mman-common.h b/include/uapi/asm-generic/mman-common.h index 6c1aa92a92e4..b686920ca731 100644 --- a/include/uapi/asm-generic/mman-common.h +++ b/include/uapi/asm-generic/mman-common.h @@ -77,6 +77,8 @@ #define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */ +#define MADV_COLLAPSE 25 /* collapse an address range into hugepages */ + /* compatibility flags */ #define MAP_FILE 0 diff --git a/mm/madvise.c b/mm/madvise.c index d7b4f2602949..c624c0f02276 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -59,6 +59,7 @@ static int madvise_need_mmap_write(int behavior) case MADV_FREE: case MADV_POPULATE_READ: case MADV_POPULATE_WRITE: + case MADV_COLLAPSE: return 0; default: /* be safe, default to 1. list exceptions explicitly */ @@ -981,6 +982,20 @@ static long madvise_remove(struct vm_area_struct *vma, return error; } +static int madvise_collapse(struct vm_area_struct *vma, + struct vm_area_struct **prev, + unsigned long start, unsigned long end) +{ + bool shared = vma->vm_flags & VM_SHARED; + *prev = vma; + + /* Only allow collapsing for HGM-enabled, shared mappings. */ + if (!is_vm_hugetlb_page(vma) || !hugetlb_hgm_enabled(vma) || !shared) + return -EINVAL; + + return hugetlb_collapse(vma->vm_mm, vma, start, end); +} + /* * Apply an madvise behavior to a region of a vma. madvise_update_vma * will handle splitting a vm area into separate areas, each area with its own @@ -1011,6 +1026,8 @@ static int madvise_vma_behavior(struct vm_area_struct *vma, case MADV_POPULATE_READ: case MADV_POPULATE_WRITE: return madvise_populate(vma, prev, start, end, behavior); + case MADV_COLLAPSE: + return madvise_collapse(vma, prev, start, end); case MADV_NORMAL: new_flags = new_flags & ~VM_RAND_READ & ~VM_SEQ_READ; break; @@ -1158,6 +1175,9 @@ madvise_behavior_valid(int behavior) #ifdef CONFIG_MEMORY_FAILURE case MADV_SOFT_OFFLINE: case MADV_HWPOISON: +#endif +#ifdef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING + case MADV_COLLAPSE: #endif return true; @@ -1351,6 +1371,9 @@ int madvise_set_anon_name(struct mm_struct *mm, unsigned long start, * triggering read faults if required * MADV_POPULATE_WRITE - populate (prefault) page tables writable by * triggering write faults if required + * MADV_COLLAPSE - collapse a high-granularity HugeTLB mapping into huge + * mappings. This is useful after an entire hugepage has been + * mapped with individual small UFFDIO_CONTINUE operations. * * return values: * zero - success From patchwork Fri Jun 24 17:36:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894948 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB3F1C433EF for ; Fri, 24 Jun 2022 17:37:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 714698E0255; Fri, 24 Jun 2022 13:37:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6262F8E0244; Fri, 24 Jun 2022 13:37:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 452158E0255; Fri, 24 Jun 2022 13:37:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 2905F8E0244 for ; Fri, 24 Jun 2022 13:37:43 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0F135B82 for ; Fri, 24 Jun 2022 17:37:43 +0000 (UTC) X-FDA: 79613836806.29.6B4FC17 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf22.hostedemail.com (Postfix) with ESMTP id 9D719C001B for ; Fri, 24 Jun 2022 17:37:42 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id j11-20020a05690212cb00b006454988d225so2747957ybu.10 for ; Fri, 24 Jun 2022 10:37:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=fAGdwktIt8KMFEeY6rcBub88vVCunW5aTmCo6YUaLic=; b=Vv9tNv7i10/uP/TErfNMavUn+CONiybyM+rrVRU58xqDyKcuGlAEDDrYb4H8klnBqK 6JRS9rF1Ax96bUDo9E8Q4msDjYxTQRaGDpgp0PfneuVYFjBRZnNER/PyTbYsKCHGqcfZ wZWHRDAniodz7pO51uXdkiH13sfi6Axk4M+P3ph2OFO2iVkw+faALWo+6f5rgiVVvomg DpY1Vfkw9K59AUl480vwthENZtLgRMWXBf+ShceS/WJ9l4dL6v1IQjWdqA0Jh9uJzr7B 6UZ1PTjJenNdIGVZeRP15MnVvJZeuiEXKSxtsNuQDFnn+yupWoF7oiY8vsG+FIoUiDIP Hx5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=fAGdwktIt8KMFEeY6rcBub88vVCunW5aTmCo6YUaLic=; b=zRB0XzcxwD7VilMIezMsXb9nrm2p38wHErSJ3OhEOBuHAwGPE/pKciUn703DXF6/X/ 7ACy3dUQuD7pE5duRsLS2nQE1F0E1mzHGe//0dPcr5M/MdN7UQtCMNmfuGEJA+gy9B4P n1jF0Itt4ddTEfttVWSC6BGRxRJ6FqtVMK6M5w2RHqbU9/3GZW3fCakZnl4nkeiyYxSl uub6pQwIPtqOv8vMZ4291rKDwQrlpqXCN+TqwGTcFcd23AaPrPCC4R6cPufRMsWtqaBk Eagc3t8Ci8mzwglahJPAPAviBL4jlfpiqRb1ApkxN1Yy+1D55pHlriqMQ+8l8K8dVDQl hhVg== X-Gm-Message-State: AJIora97SwRowjCb6NJ+F9QYkqaKdt950u/T81tdMNHnhhKPaXUB8P4q cANFi6s+gwiQCsGenCcYOIkhPEfK2i3lG0SH X-Google-Smtp-Source: AGRyM1s9InsMUuonAQPM2rPsoKgLt0oAO15w8QDXDBNTwbyi4uqbeJxT6CBZesWu39P67C1HtW9QkiD1Grt/J/X2 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a81:64f:0:b0:318:8841:b44e with SMTP id 76-20020a81064f000000b003188841b44emr29471ywg.216.1656092262019; Fri, 24 Jun 2022 10:37:42 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:53 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-24-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 23/26] userfaultfd: add UFFD_FEATURE_MINOR_HUGETLBFS_HGM From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092262; a=rsa-sha256; cv=none; b=7OAC45d09VreWnlXIi7itqptSn75NxUE2mdOIAgQkEwF1T38OoghUdkA4aBaMii2kDWxf+ AT76JvSqQ2XDhlKu7QEW67tWiN8koK8fL1PJ4CGSeq2VpWkNfHIczpIOGLZ5ryTPMxgiPC NmuRRSCq8bWHp/xOLNEZD3000ivIMKU= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Vv9tNv7i; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3Zva1YgoKCEcs2qx3pq2xwpxxpun.lxvurw36-vvt4jlt.x0p@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Zva1YgoKCEcs2qx3pq2xwpxxpun.lxvurw36-vvt4jlt.x0p@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092262; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fAGdwktIt8KMFEeY6rcBub88vVCunW5aTmCo6YUaLic=; b=KwCij3egO0UUqRi4+h21LGPAHwnCGbV+HukPf23wRfgjUb/TOBWmiwwd9V9eQGXSd8H0oP 9EKcDo+B//ZyWCE//K+pithnRsWdwts4RpTnXaE29Jsd40NGVx8/S/ChkA6AVmcNULnIGs fYyMCPAZWqruj72QQr0sA1syB/+ymb4= X-Rspam-User: X-Stat-Signature: kfcc1euct4xo4wwnpo98ys4sd71fk9yt X-Rspamd-Queue-Id: 9D719C001B Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Vv9tNv7i; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3Zva1YgoKCEcs2qx3pq2xwpxxpun.lxvurw36-vvt4jlt.x0p@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Zva1YgoKCEcs2qx3pq2xwpxxpun.lxvurw36-vvt4jlt.x0p@flex--jthoughton.bounces.google.com X-Rspamd-Server: rspam03 X-HE-Tag: 1656092262-882068 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is so that userspace is aware that their kernel was compiled with HugeTLB high-granularity mapping and that UFFDIO_CONTINUE down to PAGE_SIZE-aligned chunks are valid. Signed-off-by: James Houghton --- fs/userfaultfd.c | 7 ++++++- include/uapi/linux/userfaultfd.h | 2 ++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 77c1b8a7d0b9..59bfdb7a67e0 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -1935,10 +1935,15 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, goto err_out; /* report all available features and ioctls to userland */ uffdio_api.features = UFFD_API_FEATURES; + #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR uffdio_api.features &= ~(UFFD_FEATURE_MINOR_HUGETLBFS | UFFD_FEATURE_MINOR_SHMEM); -#endif +#ifndef CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING + uffdio_api.features &= ~UFFD_FEATURE_MINOR_HUGETLBFS_HGM; +#endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */ +#endif /* CONFIG_HAVE_ARCH_USERFAULTFD_MINOR */ + #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_WP uffdio_api.features &= ~UFFD_FEATURE_PAGEFAULT_FLAG_WP; #endif diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h index 7d32b1e797fb..50fbcb0bcba0 100644 --- a/include/uapi/linux/userfaultfd.h +++ b/include/uapi/linux/userfaultfd.h @@ -32,6 +32,7 @@ UFFD_FEATURE_SIGBUS | \ UFFD_FEATURE_THREAD_ID | \ UFFD_FEATURE_MINOR_HUGETLBFS | \ + UFFD_FEATURE_MINOR_HUGETLBFS_HGM | \ UFFD_FEATURE_MINOR_SHMEM | \ UFFD_FEATURE_EXACT_ADDRESS | \ UFFD_FEATURE_WP_HUGETLBFS_SHMEM) @@ -213,6 +214,7 @@ struct uffdio_api { #define UFFD_FEATURE_MINOR_SHMEM (1<<10) #define UFFD_FEATURE_EXACT_ADDRESS (1<<11) #define UFFD_FEATURE_WP_HUGETLBFS_SHMEM (1<<12) +#define UFFD_FEATURE_MINOR_HUGETLBFS_HGM (1<<13) __u64 features; __u64 ioctls; From patchwork Fri Jun 24 17:36:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894949 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86EF2C43334 for ; Fri, 24 Jun 2022 17:37:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B58CF8E0256; Fri, 24 Jun 2022 13:37:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A466B8E0244; Fri, 24 Jun 2022 13:37:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 738228E0256; Fri, 24 Jun 2022 13:37:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5803E8E0244 for ; Fri, 24 Jun 2022 13:37:44 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 38811B82 for ; Fri, 24 Jun 2022 17:37:44 +0000 (UTC) X-FDA: 79613836848.12.57E39CF Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf14.hostedemail.com (Postfix) with ESMTP id D005510000A for ; Fri, 24 Jun 2022 17:37:43 +0000 (UTC) Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-317597643bfso26552317b3.20 for ; Fri, 24 Jun 2022 10:37:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=+v5vFcEkCeptJMisViVmMo9Bs1ZO+THs4heiTRIsd+o=; b=NNXX2wgNjzsnd7NQe74D6jQQ3JbhNS1xVYTChLiQX5nqdIcnMSKpyyj3i3EvpGCsEm RHq6youwXY0rUgRMsMTqDuY+7MTgv+Ntfb9USuGpLdovFk8QaUNnj/5hu4tOf0Mr8Ypa tM+TcTpJN5YH+/RNf30TRecLDgM5H/Zp5MJnxEVgf0YteupwUS+Lg1C1mvGu3kxmJaAP xgJKg9LuXjQrzoqn63vyXZqC4Ckt6dxdIBoGoTX+pIFxhc2k6bw9SxJVfy3KdgryLAmu 8iaVOe1LIhnXOC1SpxD4kIV9TfjqwXSaMv6b0r1/9KFQEay+2Tk+s5gsM4+trTfyjtP4 tSMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=+v5vFcEkCeptJMisViVmMo9Bs1ZO+THs4heiTRIsd+o=; b=g4V6/xbZInHbhH0OKGaj0+/N4ts/Ca48EJKO+jgM6PMqFgHZvLh3nfFsVuM3sA+zKC wTr9ZAU9YKUErmQ27tu7MYrY3zkebF3PiyZEsQlh+6rjELIyp8+BMg4XsedqvOnMF+/F 4CMc0mBwU5y5JWciq4/jhZPFcsADcYPmeaeYDPjPqKTIpzx+BrsDThunXmLoKj0WEu+Y ERusn2kt/ogz0XlaYVLpDOuD4Hx7ImNvWfSgt1C6/bZ11OA44JQCGo189sgakIfGrwzS sXjltWxFf4oT7FB9TfBpjV1RiIDyVzJ1C9Wydk3TcpHCcnxZ6OfSVkXNMjpcFyyqxOdc AECw== X-Gm-Message-State: AJIora8obsqYnGaLedc9CBc6bo22THnnCKefY96d5iEs6mExoRvO512U JRoFgxcpxJXNjL+/4dYy1A/wrZpwmWR+1t6T X-Google-Smtp-Source: AGRyM1uKY4yvxaVhU0NJgxRylqUzzNHeETpBTfmI2EV5YsAV33yJ/2TcaSldBOT8fH//yUIflu07auIqQ61eftvT X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a81:5e42:0:b0:31b:6254:1c2b with SMTP id s63-20020a815e42000000b0031b62541c2bmr3298ywb.35.1656092263238; Fri, 24 Jun 2022 10:37:43 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:54 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-25-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 24/26] arm64/hugetlb: add support for high-granularity mappings From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092263; a=rsa-sha256; cv=none; b=iCbwRhwrubgZT8Xwo3+k/EWmxVRNkb81/UjYKX1HiCKWPRSx0GhQ0mlP8n82Kh0EsZSLMp JT8O8Sk7hPJiHVtQhev/RpDtY9dw7kvtXsnpG6RsFn6hRVKAY1xiAAjbMCCjDBwgfXpzLv zVX2ZtL+v9rDOoCEsZPBSQMSXN+9Uhs= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=NNXX2wgN; spf=pass (imf14.hostedemail.com: domain of 3Z_a1YgoKCEgt3ry4qr3yxqyyqvo.mywvsx47-wwu5kmu.y1q@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Z_a1YgoKCEgt3ry4qr3yxqyyqvo.mywvsx47-wwu5kmu.y1q@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092263; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+v5vFcEkCeptJMisViVmMo9Bs1ZO+THs4heiTRIsd+o=; b=ktooe5+VKmpPvMewHQ7ekNme6EIrwfHJtRHoouPOrkkuJZ0cY05eITHc22JD60FtBJvWgk cAQB9Vb/0OP3bf0GkDd/hacWN1lIKKbNUyzCVQuMJbkFJ6N9VoVmszoUD8rX+5OAATrCmb zAerso2ArrMDu6Y20MM3In2SXe0c0H4= X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: D005510000A Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=NNXX2wgN; spf=pass (imf14.hostedemail.com: domain of 3Z_a1YgoKCEgt3ry4qr3yxqyyqvo.mywvsx47-wwu5kmu.y1q@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Z_a1YgoKCEgt3ry4qr3yxqyyqvo.mywvsx47-wwu5kmu.y1q@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Stat-Signature: wp7pz38r5k1prmjkdwyhtjmr9d1znm8b X-HE-Tag: 1656092263-886014 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is included in this RFC to demonstrate how an architecture that doesn't use ARCH_WANT_GENERAL_HUGETLB can be updated to support HugeTLB high-granularity mappings: an architecture just needs to implement hugetlb_walk_to. Signed-off-by: James Houghton --- arch/arm64/Kconfig | 1 + arch/arm64/mm/hugetlbpage.c | 63 +++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 1652a9800ebe..74108713a99a 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -99,6 +99,7 @@ config ARM64 select ARCH_WANT_FRAME_POINTERS select ARCH_WANT_HUGE_PMD_SHARE if ARM64_4K_PAGES || (ARM64_16K_PAGES && !ARM64_VA_BITS_36) select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP + select ARCH_HAS_SPECIAL_HUGETLB_HGM select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_NO_INSTR select ARCH_HAS_UBSAN_SANITIZE_ALL diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index e2a5ec9fdc0d..1901818bed9d 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -281,6 +281,69 @@ void set_huge_swap_pte_at(struct mm_struct *mm, unsigned long addr, set_pte(ptep, pte); } +int hugetlb_walk_to(struct mm_struct *mm, struct hugetlb_pte *hpte, + unsigned long addr, unsigned long sz, bool stop_at_none) +{ + pgd_t *pgdp; + p4d_t *p4dp; + pte_t *ptep; + + if (!hpte->ptep) { + pgdp = pgd_offset(mm, addr); + p4dp = p4d_offset(pgdp, addr); + if (!p4dp) + return -ENOMEM; + hugetlb_pte_populate(hpte, (pte_t *)p4dp, P4D_SHIFT); + } + + while (hugetlb_pte_size(hpte) > sz && + !hugetlb_pte_present_leaf(hpte) && + !(stop_at_none && hugetlb_pte_none(hpte))) { + if (hpte->shift == PMD_SHIFT) { + unsigned long rounded_addr = sz == CONT_PTE_SIZE + ? addr & CONT_PTE_MASK + : addr; + + ptep = pte_offset_kernel((pmd_t *)hpte->ptep, + rounded_addr); + if (!ptep) + return -ENOMEM; + if (sz == CONT_PTE_SIZE) + hpte->shift = CONT_PTE_SHIFT; + else + hpte->shift = pte_cont(*ptep) ? CONT_PTE_SHIFT + : PAGE_SHIFT; + hpte->ptep = ptep; + } else if (hpte->shift == PUD_SHIFT) { + pud_t *pudp = (pud_t *)hpte->ptep; + + ptep = (pte_t *)pmd_alloc(mm, pudp, addr); + + if (!ptep) + return -ENOMEM; + if (sz == CONT_PMD_SIZE) + hpte->shift = CONT_PMD_SHIFT; + else + hpte->shift = pte_cont(*ptep) ? CONT_PMD_SHIFT + : PMD_SHIFT; + hpte->ptep = ptep; + } else if (hpte->shift == P4D_SHIFT) { + ptep = (pte_t *)pud_alloc(mm, (p4d_t *)hpte->ptep, addr); + if (!ptep) + return -ENOMEM; + hpte->shift = PUD_SHIFT; + hpte->ptep = ptep; + } else + /* + * This also catches the cases of CONT_PMD_SHIFT and + * CONT_PTE_SHIFT. Those PTEs should always be leaves. + */ + BUG(); + } + + return 0; +} + pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned long sz) { From patchwork Fri Jun 24 17:36:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894950 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A0CEC43334 for ; Fri, 24 Jun 2022 17:37:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 232458E0257; Fri, 24 Jun 2022 13:37:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0D1618E0244; Fri, 24 Jun 2022 13:37:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E15718E0257; Fri, 24 Jun 2022 13:37:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C314D8E0244 for ; Fri, 24 Jun 2022 13:37:45 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id A43BA33596 for ; Fri, 24 Jun 2022 17:37:45 +0000 (UTC) X-FDA: 79613836890.12.429CE4D Received: from mail-vk1-f202.google.com (mail-vk1-f202.google.com [209.85.221.202]) by imf01.hostedemail.com (Postfix) with ESMTP id 5622A40030 for ; Fri, 24 Jun 2022 17:37:45 +0000 (UTC) Received: by mail-vk1-f202.google.com with SMTP id g136-20020a1f208e000000b0036c645241f1so925196vkg.5 for ; Fri, 24 Jun 2022 10:37:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=SIBOH6opXUz/XwGM5Y6ya4CpHP7LmoYYYTh8F0m81no=; b=Pbkkfo9sSsWByEA3MDQOtz+Q2704OLexkMuoEPgStT8U3UpSblQ3IEPUJoumDhRsCS KF3OzIl1hpNYAOnJFSoAUINBxdMD3KyitvNSBwfbQwAIVinkxf2E5fS++Nptgx5omNHK MhGNV51KMhOAPw8/F/4bz6597P0y0Tjf0Quez0cg4ldgJsyCR4bIHrUcoZUbXkNnkJTy Y9FpzlIt/V/d7jDwUfIW+aYewo3REz+w4TXtV8Ubu/W8DxnbGVsCZKID9paL3I07nVHN ibi7C91OOK5NPVkKxk8pfTxCZhen97wIWN591SFKIs0pk37m97cZ299uot1663T0JWat EJYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=SIBOH6opXUz/XwGM5Y6ya4CpHP7LmoYYYTh8F0m81no=; b=vhAXQF7nXdIgRZaMaBYrMrWqt2rL7pHPJKB6RCC1g7Z+VMtlVtdm++4InM18isTWww nzWT5oN+7SaHd1Oi0LI+2u1BGoUaK3022a21MVoQY+2qA9Y/tVXKSNnhknY+Qkiot9uM yPOk837hxmVbSLMz11082FH52PX9k6piAXhHBs6FeN5p69UXMZkiiC4C+KlssfzfTusx WabT2vjbotX6liTV8RZ2rM+ScWVEa/M+tA5l0PRnt52F77GvfTzfCC6AgTJQ+Dm4mUvu s2XLjCP9pO5F1+hd6UkhpAt/CtjaV887JovG4lG08bdQBjYGsCxHKmrTPstYkRH+2YN5 2VCA== X-Gm-Message-State: AJIora/sqM0XSbYjOEJ8BdZX9SA68HgqrkbjsZC4PDZtT81t8VsrAbqu vRTwxabYvhCSIanQSnxAWFykNGgzcfNZJpAB X-Google-Smtp-Source: AGRyM1vSDDKOe0+ex+JDy0gPJ7CNOXQ1Vy1CORyh/oXQDqJ/7aNJp/5naeafubXWqtHGG6rbV8KIUOgKlN+dxbOo X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6102:d4:b0:356:1e42:b3d8 with SMTP id u20-20020a05610200d400b003561e42b3d8mr1691552vsp.82.1656092264711; Fri, 24 Jun 2022 10:37:44 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:55 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-26-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 25/26] selftests: add HugeTLB HGM to userfaultfd selftest From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092265; a=rsa-sha256; cv=none; b=QM4/eKRn6XzX0scZjWxy8WtXJnJkRc8ZJSYXPCm8YlxJKhERB4qqGvB8Rae5prqOuO1+Nv ahkwS4lg4MTE39ZULFqOO3C11GNnElQLt3OROYs0o6BdHhDmrrv3YdN4zUHvzdzeYgoWtR 7eqlqxGMzscnpHmyy9NP2gq3P8QNDX4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092265; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SIBOH6opXUz/XwGM5Y6ya4CpHP7LmoYYYTh8F0m81no=; b=ZylB6DL2z2sstTcY+gk5m1PR75bHmgxnxkJe2ieXOtA8MFljjYpf/LXJGapoxN9Ibq0NJb 2EtKTFT1WC6C8qYCdz4h0SOpXn0/3rDtAy0asNevrTgJCJpkOP2UPaUmclEZs8Gr5C6u/K xY3gmCkTYojMIKGxSTl6ayQM7TX8nbc= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Pbkkfo9s; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 3aPa1YgoKCEku4sz5rs4zyrzzrwp.nzxwty58-xxv6lnv.z2r@flex--jthoughton.bounces.google.com designates 209.85.221.202 as permitted sender) smtp.mailfrom=3aPa1YgoKCEku4sz5rs4zyrzzrwp.nzxwty58-xxv6lnv.z2r@flex--jthoughton.bounces.google.com X-Stat-Signature: go4eztq4cc8bbbrgkkqsdkjknjcq5y1k X-Rspamd-Queue-Id: 5622A40030 Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Pbkkfo9s; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 3aPa1YgoKCEku4sz5rs4zyrzzrwp.nzxwty58-xxv6lnv.z2r@flex--jthoughton.bounces.google.com designates 209.85.221.202 as permitted sender) smtp.mailfrom=3aPa1YgoKCEku4sz5rs4zyrzzrwp.nzxwty58-xxv6lnv.z2r@flex--jthoughton.bounces.google.com X-Rspam-User: X-Rspamd-Server: rspam04 X-HE-Tag: 1656092265-801347 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It behaves just like the regular shared HugeTLB configuration, except that it uses 4K instead of hugepages. This doesn't test collapsing yet. I'll add a test for that for v1. Signed-off-by: James Houghton --- tools/testing/selftests/vm/userfaultfd.c | 61 ++++++++++++++++++++---- 1 file changed, 51 insertions(+), 10 deletions(-) diff --git a/tools/testing/selftests/vm/userfaultfd.c b/tools/testing/selftests/vm/userfaultfd.c index 0bdfc1955229..9cbb959519a6 100644 --- a/tools/testing/selftests/vm/userfaultfd.c +++ b/tools/testing/selftests/vm/userfaultfd.c @@ -64,7 +64,7 @@ #ifdef __NR_userfaultfd -static unsigned long nr_cpus, nr_pages, nr_pages_per_cpu, page_size; +static unsigned long nr_cpus, nr_pages, nr_pages_per_cpu, page_size, hpage_size; #define BOUNCE_RANDOM (1<<0) #define BOUNCE_RACINGFAULTS (1<<1) @@ -72,9 +72,10 @@ static unsigned long nr_cpus, nr_pages, nr_pages_per_cpu, page_size; #define BOUNCE_POLL (1<<3) static int bounces; -#define TEST_ANON 1 -#define TEST_HUGETLB 2 -#define TEST_SHMEM 3 +#define TEST_ANON 1 +#define TEST_HUGETLB 2 +#define TEST_HUGETLB_HGM 3 +#define TEST_SHMEM 4 static int test_type; /* exercise the test_uffdio_*_eexist every ALARM_INTERVAL_SECS */ @@ -85,6 +86,7 @@ static volatile bool test_uffdio_zeropage_eexist = true; static bool test_uffdio_wp = true; /* Whether to test uffd minor faults */ static bool test_uffdio_minor = false; +static bool test_uffdio_copy = true; static bool map_shared; static int shm_fd; @@ -140,12 +142,17 @@ static void usage(void) fprintf(stderr, "\nUsage: ./userfaultfd " "[hugetlbfs_file]\n\n"); fprintf(stderr, "Supported : anon, hugetlb, " - "hugetlb_shared, shmem\n\n"); + "hugetlb_shared, hugetlb_shared_hgm, shmem\n\n"); fprintf(stderr, "Examples:\n\n"); fprintf(stderr, "%s", examples); exit(1); } +static bool test_is_hugetlb(void) +{ + return test_type == TEST_HUGETLB || test_type == TEST_HUGETLB_HGM; +} + #define _err(fmt, ...) \ do { \ int ret = errno; \ @@ -348,7 +355,7 @@ static struct uffd_test_ops *uffd_test_ops; static inline uint64_t uffd_minor_feature(void) { - if (test_type == TEST_HUGETLB && map_shared) + if (test_is_hugetlb() && map_shared) return UFFD_FEATURE_MINOR_HUGETLBFS; else if (test_type == TEST_SHMEM) return UFFD_FEATURE_MINOR_SHMEM; @@ -360,7 +367,7 @@ static uint64_t get_expected_ioctls(uint64_t mode) { uint64_t ioctls = UFFD_API_RANGE_IOCTLS; - if (test_type == TEST_HUGETLB) + if (test_is_hugetlb()) ioctls &= ~(1 << _UFFDIO_ZEROPAGE); if (!((mode & UFFDIO_REGISTER_MODE_WP) && test_uffdio_wp)) @@ -1116,6 +1123,12 @@ static int userfaultfd_events_test(void) char c; struct uffd_stats stats = { 0 }; + if (!test_uffdio_copy) { + printf("Skipping userfaultfd events test " + "(test_uffdio_copy=false)\n"); + return 0; + } + printf("testing events (fork, remap, remove): "); fflush(stdout); @@ -1169,6 +1182,12 @@ static int userfaultfd_sig_test(void) char c; struct uffd_stats stats = { 0 }; + if (!test_uffdio_copy) { + printf("Skipping userfaultfd signal test " + "(test_uffdio_copy=false)\n"); + return 0; + } + printf("testing signal delivery: "); fflush(stdout); @@ -1438,6 +1457,12 @@ static int userfaultfd_stress(void) pthread_attr_init(&attr); pthread_attr_setstacksize(&attr, 16*1024*1024); + if (!test_uffdio_copy) { + printf("Skipping userfaultfd stress test " + "(test_uffdio_copy=false)\n"); + bounces = 0; + } + while (bounces--) { printf("bounces: %d, mode:", bounces); if (bounces & BOUNCE_RANDOM) @@ -1598,6 +1623,13 @@ static void set_test_type(const char *type) uffd_test_ops = &hugetlb_uffd_test_ops; /* Minor faults require shared hugetlb; only enable here. */ test_uffdio_minor = true; + } else if (!strcmp(type, "hugetlb_shared_hgm")) { + map_shared = true; + test_type = TEST_HUGETLB_HGM; + uffd_test_ops = &hugetlb_uffd_test_ops; + /* Minor faults require shared hugetlb; only enable here. */ + test_uffdio_minor = true; + test_uffdio_copy = false; } else if (!strcmp(type, "shmem")) { map_shared = true; test_type = TEST_SHMEM; @@ -1607,8 +1639,10 @@ static void set_test_type(const char *type) err("Unknown test type: %s", type); } + hpage_size = default_huge_page_size(); if (test_type == TEST_HUGETLB) - page_size = default_huge_page_size(); + // TEST_HUGETLB_HGM gets small pages. + page_size = hpage_size; else page_size = sysconf(_SC_PAGE_SIZE); @@ -1658,19 +1692,26 @@ int main(int argc, char **argv) nr_cpus = sysconf(_SC_NPROCESSORS_ONLN); nr_pages_per_cpu = atol(argv[2]) * 1024*1024 / page_size / nr_cpus; + if (test_type == TEST_HUGETLB_HGM) + /* + * `page_size` refers to the page_size we can use in + * UFFDIO_CONTINUE. We still need nr_pages to be appropriately + * aligned, so align it here. + */ + nr_pages_per_cpu -= nr_pages_per_cpu % (hpage_size / page_size); if (!nr_pages_per_cpu) { _err("invalid MiB"); usage(); } + nr_pages = nr_pages_per_cpu * nr_cpus; bounces = atoi(argv[3]); if (bounces <= 0) { _err("invalid bounces"); usage(); } - nr_pages = nr_pages_per_cpu * nr_cpus; - if (test_type == TEST_HUGETLB && map_shared) { + if (test_is_hugetlb() && map_shared) { if (argc < 5) usage(); huge_fd = open(argv[4], O_CREAT | O_RDWR, 0755); From patchwork Fri Jun 24 17:36:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 12894951 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C83DCCA47E for ; Fri, 24 Jun 2022 17:37:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 654AC8E0258; Fri, 24 Jun 2022 13:37:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 53D308E0244; Fri, 24 Jun 2022 13:37:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3DC8D8E0258; Fri, 24 Jun 2022 13:37:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 20D2F8E0244 for ; Fri, 24 Jun 2022 13:37:47 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id EF126120801 for ; Fri, 24 Jun 2022 17:37:46 +0000 (UTC) X-FDA: 79613836932.19.C0F3358 Received: from mail-ua1-f73.google.com (mail-ua1-f73.google.com [209.85.222.73]) by imf11.hostedemail.com (Postfix) with ESMTP id 8D4FA4002A for ; Fri, 24 Jun 2022 17:37:46 +0000 (UTC) Received: by mail-ua1-f73.google.com with SMTP id v13-20020ab0200d000000b0037f1212efd9so1022804uak.9 for ; Fri, 24 Jun 2022 10:37:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=u+0c8IGQItY5NCCATolNVsWOPdUIr5MDdbCK/q0TIvo=; b=D90ixVslphWwRS1UyK82N/Fu4Pbp7RfYXJAXQqDLm0AYhlK59/wd1jqEmTakmfoAdM bCaJEmUAqS6tCX2dak5OsRfQg3/S32jkFNOOv5AhzS6KIq2Y8/bxRLDG5Ynhh7Q4ZWRf T28FEgQTuQAmTAzbinSDCUL5FqalKu06/tPW5zgoGnZaHOERoYdGxV1amlIWNDhV6gNx Hux7xxh8o8EA2S1TUMwn3Pd1RB6gqb3P03MxNrYVTbnyXFz2pdn7Ecfkw2C38+x5Tq4t 6G6ZSXt7n/jc3VeCFkh0qVe2rMcVgaPEC639yccX83zFJnO8qJjsZQZj8m3ROTU/7Otn uqtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=u+0c8IGQItY5NCCATolNVsWOPdUIr5MDdbCK/q0TIvo=; b=DBvs5c0ODZfR7Z2oKX3h5hO45U7PTniPGNj86iKoky50HrnqnV4CiOIIodtPxNgtIZ YYtbn6MnMi2MT7lhY1fFCEecFhlpZD1YFTsasVIh2kWifoFdNA0p5821zT6cxIUc3X4l IEw1nY8++m5rTiW81ke5Nxu6dXMcrcUAXUeHM0Yix+1t6o7gjYs4bMj8VPZ3qnHpNO+0 yrIU24mePBECKlgG9RcN0Aoa5YPs2z9gKCxoK/PY+eKngfayu7lYUa8NYyhMA5bFjduc BFO8CtoxLRmVFGsC2GEOE3uKfq/1ifbf9BaRuTIKknew4OuYvm/PodHJGA+FJsDVXuAu JPcw== X-Gm-Message-State: AJIora9U4p1b1XbcA99gRsAeHEOYJwz6hh7DpXOT9m9h78Pt0YrI1ie7 xhLyEy/IIczKk7Ogw2ftTsjiEbohQ77klrEt X-Google-Smtp-Source: AGRyM1sXBK2BOgwtHhnCdm6QzU6aNINH2NrcD5ngkogaDCkYiXeckpXlq4vGXhiBI+c1PtjBBhz07CDWB+WagWPT X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6122:1990:b0:36b:f70c:ba55 with SMTP id bv16-20020a056122199000b0036bf70cba55mr26413vkb.12.1656092265972; Fri, 24 Jun 2022 10:37:45 -0700 (PDT) Date: Fri, 24 Jun 2022 17:36:56 +0000 In-Reply-To: <20220624173656.2033256-1-jthoughton@google.com> Message-Id: <20220624173656.2033256-27-jthoughton@google.com> Mime-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> X-Mailer: git-send-email 2.37.0.rc0.161.g10f37bed90-goog Subject: [RFC PATCH 26/26] selftests: add HugeTLB HGM to KVM demand paging selftest From: James Houghton To: Mike Kravetz , Muchun Song , Peter Xu Cc: David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , Manish Mishra , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, James Houghton ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656092266; a=rsa-sha256; cv=none; b=24QBzanNinTKOgZcnVMyVgcbDbeN8RkqzrWu7PnBizgUBpSiJl4h/0nmWpuGUXiepbGsyh +GdDFlpjYUAanUcdaKBqQUD/bY9lAYYYGvd9ERV8NM864ZMj+QRgc6TGCwCLXGKKkk1Eeu eab28d4OT0A8HrPIg7tUPyrx2DX/R4w= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=D90ixVsl; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of 3afa1YgoKCEov5t06st50zs00sxq.o0yxuz69-yyw7mow.03s@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3afa1YgoKCEov5t06st50zs00sxq.o0yxuz69-yyw7mow.03s@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656092266; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=u+0c8IGQItY5NCCATolNVsWOPdUIr5MDdbCK/q0TIvo=; b=tvlvcNVGw/o8BRH4qcIlN/NuG41GXHL4IuiOHynU5wBddyyoShss6v3MnpBJ9cVpWTwfi0 KnQHyD3HqbB/q/U2xe33YlI7PyHQtMdjDYQkA6rMRt5XFQAzfZ9kVLXVeev7BO6noZVgV0 MrZBm/R1Mzoef3idf4H4cSiDySoLPPg= X-Rspam-User: X-Stat-Signature: rdq54e8ajccmae35ukm3f9yqopi7b46n X-Rspamd-Queue-Id: 8D4FA4002A Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=D90ixVsl; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of 3afa1YgoKCEov5t06st50zs00sxq.o0yxuz69-yyw7mow.03s@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3afa1YgoKCEov5t06st50zs00sxq.o0yxuz69-yyw7mow.03s@flex--jthoughton.bounces.google.com X-Rspamd-Server: rspam03 X-HE-Tag: 1656092266-658455 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This doesn't address collapsing yet, and it only works with the MINOR mode (UFFDIO_CONTINUE). Signed-off-by: James Houghton --- tools/testing/selftests/kvm/include/test_util.h | 2 ++ tools/testing/selftests/kvm/lib/kvm_util.c | 2 +- tools/testing/selftests/kvm/lib/test_util.c | 14 ++++++++++++++ 3 files changed, 17 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/kvm/include/test_util.h b/tools/testing/selftests/kvm/include/test_util.h index 99e0dcdc923f..6209e44981a7 100644 --- a/tools/testing/selftests/kvm/include/test_util.h +++ b/tools/testing/selftests/kvm/include/test_util.h @@ -87,6 +87,7 @@ enum vm_mem_backing_src_type { VM_MEM_SRC_ANONYMOUS_HUGETLB_16GB, VM_MEM_SRC_SHMEM, VM_MEM_SRC_SHARED_HUGETLB, + VM_MEM_SRC_SHARED_HUGETLB_HGM, NUM_SRC_TYPES, }; @@ -105,6 +106,7 @@ size_t get_def_hugetlb_pagesz(void); const struct vm_mem_backing_src_alias *vm_mem_backing_src_alias(uint32_t i); size_t get_backing_src_pagesz(uint32_t i); bool is_backing_src_hugetlb(uint32_t i); +bool is_backing_src_shared_hugetlb(enum vm_mem_backing_src_type src_type); void backing_src_help(const char *flag); enum vm_mem_backing_src_type parse_backing_src_type(const char *type_name); long get_run_delay(void); diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c index 1665a220abcb..382f8fb75b7f 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -993,7 +993,7 @@ void vm_userspace_mem_region_add(struct kvm_vm *vm, region->fd = -1; if (backing_src_is_shared(src_type)) region->fd = kvm_memfd_alloc(region->mmap_size, - src_type == VM_MEM_SRC_SHARED_HUGETLB); + is_backing_src_shared_hugetlb(src_type)); region->mmap_start = mmap(NULL, region->mmap_size, PROT_READ | PROT_WRITE, diff --git a/tools/testing/selftests/kvm/lib/test_util.c b/tools/testing/selftests/kvm/lib/test_util.c index 6d23878bbfe1..710dc42077fe 100644 --- a/tools/testing/selftests/kvm/lib/test_util.c +++ b/tools/testing/selftests/kvm/lib/test_util.c @@ -254,6 +254,13 @@ const struct vm_mem_backing_src_alias *vm_mem_backing_src_alias(uint32_t i) */ .flag = MAP_SHARED, }, + [VM_MEM_SRC_SHARED_HUGETLB_HGM] = { + /* + * Identical to shared_hugetlb except for the name. + */ + .name = "shared_hugetlb_hgm", + .flag = MAP_SHARED, + }, }; _Static_assert(ARRAY_SIZE(aliases) == NUM_SRC_TYPES, "Missing new backing src types?"); @@ -272,6 +279,7 @@ size_t get_backing_src_pagesz(uint32_t i) switch (i) { case VM_MEM_SRC_ANONYMOUS: case VM_MEM_SRC_SHMEM: + case VM_MEM_SRC_SHARED_HUGETLB_HGM: return getpagesize(); case VM_MEM_SRC_ANONYMOUS_THP: return get_trans_hugepagesz(); @@ -288,6 +296,12 @@ bool is_backing_src_hugetlb(uint32_t i) return !!(vm_mem_backing_src_alias(i)->flag & MAP_HUGETLB); } +bool is_backing_src_shared_hugetlb(enum vm_mem_backing_src_type src_type) +{ + return src_type == VM_MEM_SRC_SHARED_HUGETLB || + src_type == VM_MEM_SRC_SHARED_HUGETLB_HGM; +} + static void print_available_backing_src_types(const char *prefix) { int i;