From patchwork Mon Oct 21 04:22:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13843541 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A294D3C93D for ; Mon, 21 Oct 2024 04:22:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 02E086B0088; Mon, 21 Oct 2024 00:22:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EA6706B0089; Mon, 21 Oct 2024 00:22:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD19D6B008A; Mon, 21 Oct 2024 00:22:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id AF7C16B0088 for ; Mon, 21 Oct 2024 00:22:30 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id DA72A12153F for ; Mon, 21 Oct 2024 04:22:16 +0000 (UTC) X-FDA: 82696312440.24.FB69079 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf19.hostedemail.com (Postfix) with ESMTP id BBF0E1A0003 for ; Mon, 21 Oct 2024 04:22:10 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=qiC2Uaxr; spf=pass (imf19.hostedemail.com: domain of 3A9cVZwYKCEg849rkyqyyqvo.mywvsx47-wwu5kmu.y1q@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3A9cVZwYKCEg849rkyqyyqvo.mywvsx47-wwu5kmu.y1q@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729484350; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kQVfzlUfyOXzQNiM6rrgf1IxWXC/TuVQ/9/mg1gNSUk=; b=Hr9fqPVXrGNtyaadnZzaiNgs53BVf8ATgfEsrka6p2MGIMvTMNyFwsvK2jloV9l1+ijcVr fIi0D/7SbVboRDS7R/2LUz3rid6WxDTMjtZYW2T0mboFS03o3p3wb2JN69D1Ady0XJrAqS I0Qkk9qwvTYypC/3BGhkpg8OBt2kGxk= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=qiC2Uaxr; spf=pass (imf19.hostedemail.com: domain of 3A9cVZwYKCEg849rkyqyyqvo.mywvsx47-wwu5kmu.y1q@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3A9cVZwYKCEg849rkyqyyqvo.mywvsx47-wwu5kmu.y1q@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729484350; a=rsa-sha256; cv=none; b=YJMpgbjO+j94UNCs/2+o0/5mriwxuDeht7H3xDKrEqx3T8nvr5zMBezPIwllqTZTWZG3kz WCoYLoQMoPq+TwdD5jOrhhfhgW3SJrUI8hZ4q+GQp/8SNrY8DVZ0hvDjm3TxlI4NRPJpvR Dxr/JVxeYoWm4VmW7TyOkAZcexJ5RsQ= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6e315a5b199so64481337b3.2 for ; Sun, 20 Oct 2024 21:22:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729484548; x=1730089348; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kQVfzlUfyOXzQNiM6rrgf1IxWXC/TuVQ/9/mg1gNSUk=; b=qiC2UaxrWj+cpqLemRVCyqmi9lqpQevy1nqq3sVjNNpwJMuS2jXGiPX/YqjY+IKCcP H7fRnddBaupa7cX5MfgXWP5mW/0DLwYQxXuoK2ok2SqNRuPlzQz3wNABkwd5Me+AfSz3 VXlGZrsclulFONZZ2y5c5O16n5i0fxFQiHLV4bBqsjTNN0n2KR2tIzFC0rjPt6x9ZnmO aMa3qG7egaEPpBZvbeh6AvT2kixR4y239ZxRc+nRzFRTELYIHAlizvh8EP2j3zrj4CwR 8lzMqw+G/DMdIoQfLmn52mbbXchAKhkIJcVqOqYWBCPIpg2qsxHaiybHpeYjWx/uMyha E3CQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729484548; x=1730089348; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kQVfzlUfyOXzQNiM6rrgf1IxWXC/TuVQ/9/mg1gNSUk=; b=CI9KRRm3ll2DJfAdnZwez8GUFPkpY2tibPDoFHCA+951Z590a+6fS8C7Xo18dXIlHl mAvPg15OAXnbbiL5zI6y9aR6xIZM8HVvbVZDeHga3fCZeC6OYXxB4ZAY9RScISp2mO/E nQ7SGvcJDUTvFyTkBGchaYmeg9QQsjbDAAtLasSlNDddPdHn12Wzzq99v3yf/oVKycU7 YqUchQxIwLmA02qR9jL6/Vwndo0c5RSg3fQuTioF2Z+ncjhl/fWA4vdYsFPLNhabklj4 aMb12+eRshcQ+iUZ7ODW0WI8qNGjg6hn1wqLWNaDkcTDX/NkuVlKalMzOluc0tuLB7CF eW7w== X-Forwarded-Encrypted: i=1; AJvYcCWuS+iT4E++JZ0N5i3TaphJ7Ltusnj1BDZhEbKYtypfTWqDj6tv31I45Uy9pGOXNSzXmXb5972IZA==@kvack.org X-Gm-Message-State: AOJu0YyUUoHOeBB35Xk8l5E73sAO3buu44wvSTj72REDBMsyzoNMC7xS Y7fKOyW/oNm05y5iOFarZsGhE0sN8SQDq2qhsGvW+jWDJZ4JP3cN7Feg4a7jaGVqbuRPDXkCOy7 LGA== X-Google-Smtp-Source: AGHT+IFa+RAhMnCTBGawKhIBMzr1i4wNva1osbWPtOgVISCaAsioJIQoe2S7YEzQTgFhpJ04zIjZfcK8uDQ= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:1569:9ef4:20ab:abf9]) (user=yuzhao job=sendgmr) by 2002:a05:690c:6383:b0:663:ddc1:eab8 with SMTP id 00721157ae682-6e5bfbeb2e0mr2568607b3.4.1729484547736; Sun, 20 Oct 2024 21:22:27 -0700 (PDT) Date: Sun, 20 Oct 2024 22:22:14 -0600 In-Reply-To: <20241021042218.746659-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241021042218.746659-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241021042218.746659-3-yuzhao@google.com> Subject: [PATCH v1 2/6] mm/hugetlb_vmemmap: add arch-independent helpers From: Yu Zhao To: Andrew Morton , Catalin Marinas , Marc Zyngier , Muchun Song , Thomas Gleixner , Will Deacon Cc: Douglas Anderson , Mark Rutland , Nanyong Sun , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Yu Zhao X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: BBF0E1A0003 X-Stat-Signature: ptu7hhxrajc9eo1if4p37i4e6p3pwqyh X-Rspam-User: X-HE-Tag: 1729484530-899358 X-HE-Meta: U2FsdGVkX19H8ZWLhC37mjmxqeQGijvs5Del5Vow0ItuDKdKpizi7/q9c3HBPRJcUGqMGJbRRepfoQJ99ebZKlTWSz1/dvXvWpXKTbDmDgAacRs2y9wtxLTkCTj17D3LU8rD4zC2eRAF4E55m9CLfeyNMoRuXb/gHM5PFRB+QXMKnDZ5hkPsntKW+YHgpyoYITRQ7GzeJrxxY4iHK9yeSI6rQLPhdZe6x64W/HyqpTEAttfnmdG4SFHVWr+mPetKI7zJx4eJlmKvH9Y/BVeD056HqTX28Hpn1n+oPckvhArYdMvBvb89FdxjMvxMg0teq0haMr/U68vNl7I84EkYOynR6E+xDWPPXT0E1RmSyieVuutXBp+cztFMco5j1ZtuDQBtepHZWWFY6Id0mrkpMDfx0ynlyqj2OQcXKVkUDnhskwyd6hDXzMsiBUdmXduZ2OmboXApjMgHS1sZc+1IcP7VDnOqjesAWvA+KS2Vnf4C8krxbfJBZ1XvbNEnQ6JpBnKhGrZQoYRlOgTXcfT4XIhiXHyoqJhau5a7L7vC7sYP5mkeUEDSxtu9TtuSQNakXQipGvI46M2liNINOPzaNkO6Jt6evgOhoBITvoKrLumRs2MAKNBl9C6vCMzUtYXQa6wHogQC39JmuoPCmMaAEGAQhhyuzW45nx+iFRs0yzQpSgFPUCPfGDuSUhOJQNfnJYEFrgYgSf+/orOLDRW5we/L7X64nofVQ+KZ59/KSSo03qrpORqCBlg5SVA6/kUHx5tpdL0B36L6mrBf/wJu+k8xopf1NM6A3dN71yZNz6cSlYIjYhJZVUmbkAIU6mfsRH9b3E02bDs3cMeORPj8pbWjjM52KVRR0bKLIAZehv9+wm14Zi0ZiVAkp1yskD6Ya1YN27PcM/7nddTqFgX5Am+63c30+KIIFFEdc/dNnHTHOBgTyVjDcFAvb7tY+Evn86pMbybzE+xFc1hKmI9 h147oQKG R1pyvxykalm4PPtP56qRjiJ2mM0eyk2zfA/oO1mTIFqWVzflhAOIGe0VmkrJxn6MwHoAq8Hj28jfgm15rLI1Ed0cl/hvseyn2luB0UdIl0fr30xATsBXGugdDyFHxHWCTpijA0Y7ksJDMlV/PTYQerDT5JnXhC9vHzmvF+JMrLrXPpRoCMxk4zt7ZZNJp7nJGb2yNOYnhw0C9k/inlTuQ631YslVzE/8ayAEy6BYOkjCo110/RyxMFuDh6thEFINiOML5r2ALN9aMrBegdZeL5q919u+r9gkC9kYwcN0UB/9tnUfa015jtVj/8rIpePxQa3zPrCsgiiTpesJbPl52rgs7kNPrNqIH2K2YLJ50FIg+JMyHEdJo62/Mrrxa8DzeKPB7DPorqdmpqHloMJ9ryOy82hAuGFEBEdWqOHRycbW9KHhrNXY3vauHF8ZfVgMWH06SDEkgSZTV9CcjOp/cf377RSwGlQYbtsHOwyxjU2knrRHC4WjE1EJy4X2Lff6cqfEDJfZZvGPXOXsyiRB5w8+Jkrylp8TTghqFLAsnmP488t/EEsYwob8EJqDYW5V/mCY8ntTGDBunZbM+BKdlgNbcMA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add architecture-independent helpers to allow individual architectures to work around their own limitations when updating vmemmap. Specifically, the current remap workflow requires break-before-make (BBM) on arm64. By overriding the default helpers later in this series, arm64 will be able to support the current HVO implementation. Signed-off-by: Yu Zhao --- include/linux/mm_types.h | 7 +++ mm/hugetlb_vmemmap.c | 99 ++++++++++++++++++++++++++++++++++------ 2 files changed, 92 insertions(+), 14 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 6e3bdf8e38bc..0f3ae6e173f6 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1499,4 +1499,11 @@ enum { /* See also internal only FOLL flags in mm/internal.h */ }; +/* Skip the TLB flush when we split the PMD */ +#define VMEMMAP_SPLIT_NO_TLB_FLUSH BIT(0) +/* Skip the TLB flush when we remap the PTE */ +#define VMEMMAP_REMAP_NO_TLB_FLUSH BIT(1) +/* synchronize_rcu() to avoid writes from page_ref_add_unless() */ +#define VMEMMAP_SYNCHRONIZE_RCU BIT(2) + #endif /* _LINUX_MM_TYPES_H */ diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 46befab48d41..e50a196399f5 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -38,16 +38,56 @@ struct vmemmap_remap_walk { struct page *reuse_page; unsigned long reuse_addr; struct list_head *vmemmap_pages; - -/* Skip the TLB flush when we split the PMD */ -#define VMEMMAP_SPLIT_NO_TLB_FLUSH BIT(0) -/* Skip the TLB flush when we remap the PTE */ -#define VMEMMAP_REMAP_NO_TLB_FLUSH BIT(1) -/* synchronize_rcu() to avoid writes from page_ref_add_unless() */ -#define VMEMMAP_SYNCHRONIZE_RCU BIT(2) unsigned long flags; }; +#ifndef VMEMMAP_ARCH_TLB_FLUSH_FLAGS +#define VMEMMAP_ARCH_TLB_FLUSH_FLAGS 0 +#endif + +#ifndef vmemmap_update_supported +static bool vmemmap_update_supported(void) +{ + return true; +} +#endif + +#ifndef vmemmap_update_lock +static void vmemmap_update_lock(void) +{ +} +#endif + +#ifndef vmemmap_update_unlock +static void vmemmap_update_unlock(void) +{ +} +#endif + +#ifndef vmemmap_update_pte_range_start +static void vmemmap_update_pte_range_start(pte_t *pte, unsigned long start, unsigned long end) +{ +} +#endif + +#ifndef vmemmap_update_pte_range_end +static void vmemmap_update_pte_range_end(void) +{ +} +#endif + +#ifndef vmemmap_update_pmd_range_start +static void vmemmap_update_pmd_range_start(pmd_t *pmd, unsigned long start, unsigned long end) +{ +} +#endif + +#ifndef vmemmap_update_pmd_range_end +static void vmemmap_update_pmd_range_end(void) +{ +} +#endif + static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start, struct vmemmap_remap_walk *walk) { @@ -83,7 +123,9 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start, /* Make pte visible before pmd. See comment in pmd_install(). */ smp_wmb(); + vmemmap_update_pmd_range_start(pmd, start, start + PMD_SIZE); pmd_populate_kernel(&init_mm, pmd, pgtable); + vmemmap_update_pmd_range_end(); if (!(walk->flags & VMEMMAP_SPLIT_NO_TLB_FLUSH)) flush_tlb_kernel_range(start, start + PMD_SIZE); } else { @@ -164,10 +206,12 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, VM_BUG_ON(!PAGE_ALIGNED(start | end)); + vmemmap_update_lock(); mmap_read_lock(&init_mm); ret = walk_page_range_novma(&init_mm, start, end, &vmemmap_remap_ops, NULL, walk); mmap_read_unlock(&init_mm); + vmemmap_update_unlock(); if (ret) return ret; @@ -228,6 +272,8 @@ static void vmemmap_remap_pte_range(pte_t *pte, unsigned long start, unsigned lo smp_wmb(); } + vmemmap_update_pte_range_start(pte, start, end); + for (i = 0; i < nr_pages; i++) { pte_t val; @@ -242,6 +288,8 @@ static void vmemmap_remap_pte_range(pte_t *pte, unsigned long start, unsigned lo set_pte_at(&init_mm, start + PAGE_SIZE * i, pte + i, val); } + + vmemmap_update_pte_range_end(); } /* @@ -287,6 +335,8 @@ static void vmemmap_restore_pte_range(pte_t *pte, unsigned long start, unsigned */ smp_wmb(); + vmemmap_update_pte_range_start(pte, start, end); + for (i = 0; i < nr_pages; i++) { pte_t val; @@ -296,6 +346,8 @@ static void vmemmap_restore_pte_range(pte_t *pte, unsigned long start, unsigned val = mk_pte(page, PAGE_KERNEL); set_pte_at(&init_mm, start + PAGE_SIZE * i, pte + i, val); } + + vmemmap_update_pte_range_end(); } /** @@ -513,7 +565,8 @@ static int __hugetlb_vmemmap_restore_folio(const struct hstate *h, */ int hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio) { - return __hugetlb_vmemmap_restore_folio(h, folio, VMEMMAP_SYNCHRONIZE_RCU); + return __hugetlb_vmemmap_restore_folio(h, folio, + VMEMMAP_SYNCHRONIZE_RCU | VMEMMAP_ARCH_TLB_FLUSH_FLAGS); } /** @@ -553,7 +606,7 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h, list_move(&folio->lru, non_hvo_folios); } - if (restored) + if (restored && !(VMEMMAP_ARCH_TLB_FLUSH_FLAGS & VMEMMAP_REMAP_NO_TLB_FLUSH)) flush_tlb_all(); if (!ret) ret = restored; @@ -641,7 +694,8 @@ void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio) { LIST_HEAD(vmemmap_pages); - __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, VMEMMAP_SYNCHRONIZE_RCU); + __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, + VMEMMAP_SYNCHRONIZE_RCU | VMEMMAP_ARCH_TLB_FLUSH_FLAGS); free_vmemmap_page_list(&vmemmap_pages); } @@ -683,7 +737,8 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l break; } - flush_tlb_all(); + if (!(VMEMMAP_ARCH_TLB_FLUSH_FLAGS & VMEMMAP_SPLIT_NO_TLB_FLUSH)) + flush_tlb_all(); list_for_each_entry(folio, folio_list, lru) { int ret; @@ -701,24 +756,35 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l * allowing more vmemmap remaps to occur. */ if (ret == -ENOMEM && !list_empty(&vmemmap_pages)) { - flush_tlb_all(); + if (!(VMEMMAP_ARCH_TLB_FLUSH_FLAGS & VMEMMAP_REMAP_NO_TLB_FLUSH)) + flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); INIT_LIST_HEAD(&vmemmap_pages); __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, flags); } } - flush_tlb_all(); + if (!(VMEMMAP_ARCH_TLB_FLUSH_FLAGS & VMEMMAP_REMAP_NO_TLB_FLUSH)) + flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); } +static int hugetlb_vmemmap_sysctl(const struct ctl_table *ctl, int write, + void *buffer, size_t *lenp, loff_t *ppos) +{ + if (!vmemmap_update_supported()) + return -ENODEV; + + return proc_dobool(ctl, write, buffer, lenp, ppos); +} + static struct ctl_table hugetlb_vmemmap_sysctls[] = { { .procname = "hugetlb_optimize_vmemmap", .data = &vmemmap_optimize_enabled, .maxlen = sizeof(vmemmap_optimize_enabled), .mode = 0644, - .proc_handler = proc_dobool, + .proc_handler = hugetlb_vmemmap_sysctl, }, }; @@ -729,6 +795,11 @@ static int __init hugetlb_vmemmap_init(void) /* HUGETLB_VMEMMAP_RESERVE_SIZE should cover all used struct pages */ BUILD_BUG_ON(__NR_USED_SUBPAGE > HUGETLB_VMEMMAP_RESERVE_PAGES); + if (READ_ONCE(vmemmap_optimize_enabled) && !vmemmap_update_supported()) { + pr_warn("HugeTLB: disabling HVO due to missing support.\n"); + WRITE_ONCE(vmemmap_optimize_enabled, false); + } + for_each_hstate(h) { if (hugetlb_vmemmap_optimizable(h)) { register_sysctl_init("vm", hugetlb_vmemmap_sysctls);