From patchwork Mon Oct 21 04:22:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13843544 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F0B8D3C933 for ; Mon, 21 Oct 2024 04:22:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 113ED6B0095; Mon, 21 Oct 2024 00:22:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C5526B0096; Mon, 21 Oct 2024 00:22:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA6E36B0098; Mon, 21 Oct 2024 00:22:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C298B6B0095 for ; Mon, 21 Oct 2024 00:22:37 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E911C141605 for ; Mon, 21 Oct 2024 04:22:21 +0000 (UTC) X-FDA: 82696312482.13.92ED3AD Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf12.hostedemail.com (Postfix) with ESMTP id 156F14000E for ; Mon, 21 Oct 2024 04:22:28 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=XxSsBpXJ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3CtcVZwYKCE8FBGyr5x55x2v.t532z4BE-331Crt1.58x@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3CtcVZwYKCE8FBGyr5x55x2v.t532z4BE-331Crt1.58x@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729484519; a=rsa-sha256; cv=none; b=RAxiArMo6O6XNskFidA2RDGJZ7RcJcDSoBI89QD220zfNKy+yuf4gp0vmfWtNtbDTNk2tW jd5M5qenOVdm4YLspFmlhU3s5snpyXS2xJ99WBVOyS/R6gySH8BL3b5vis+hNT4apyhtPa HhgT9TM6EIuxXdTMD3DpcKlZUMddLKo= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=XxSsBpXJ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of 3CtcVZwYKCE8FBGyr5x55x2v.t532z4BE-331Crt1.58x@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3CtcVZwYKCE8FBGyr5x55x2v.t532z4BE-331Crt1.58x@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729484519; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=toVG9Bv1ZFXM9w/cWVF/q6arTJn/fxOOJss0eISMw6g=; b=VGgqNntGH62+Ch0Ws7c41iFo+jdsjWHyU3uopkqOmhyZ9AKsheIXeAUnpYpVpw7znyi2f2 wtm3EDxr+DNq5iqWP1owcvrVGgszYmowb39bhLP6iSLlS3yMeWCgewEmzqM4m9TisT+tzk wkH1oIcLm0fkC7xSHSK9M6m8Dk7rhik= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e293150c2c6so8378662276.1 for ; Sun, 20 Oct 2024 21:22:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729484555; x=1730089355; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=toVG9Bv1ZFXM9w/cWVF/q6arTJn/fxOOJss0eISMw6g=; b=XxSsBpXJn0hWnEbW6lIoteEbYB+8hG6wQwBVXRxsMoAtJVuCJKjbKxXuygnYvYYa22 u5R2OjwlIHCkqonFX7VnY4avcviA2F0EkkUboF50hhYy7iYAUJ9serlCo0TGjsGDQO/y mrtl2p9IK27ZZT1q/3iGrcgszFVCA1TGvaZ36kzAvul1mugJ0ENYntVQ71fxIh5XB5zC 6veOa4y+rfsZDIO4PxQ5AbA5NyPCyPXu6igPuFeAnSB5uaJAfTgpgvnyx7bq9MSTmLEu v1sdWpCljeVYeSpq+o8IYHCkt+gpvISWG4L9muU+HUezt5TmefpIy1DXXuf9N15H/+Ku WPAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729484555; x=1730089355; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=toVG9Bv1ZFXM9w/cWVF/q6arTJn/fxOOJss0eISMw6g=; b=b8KdonWmZVOspyAutlWnhkA4h1P8Vj/6798PGzxSrA17UCeIh6nk4gIIAxOClNooiX I+8VVRt8muidsuom/zMNLh+hfvmG/RXu1cWNrEc8vUUyP4wSB0f27wY0vuPG15fmt1yJ Ss1/Z18HvR7Jq0Dk9JXREIZexSUi2zEayjehYHSCDv27cf0eLFwPfWAHb+u2nMuTK14S Cf89yKx1bdNZc5QHoAd++CP+9lKtEOs7hvqz2nPA8d4ij0amUPO3IHE0cIgvMFQ2DJIs oxoyfK3d/644vBZvcYvi8kVubiZDl9HA1+qcedYpkaji86AN7CJ7xV6HzTZg073221Si ZKGg== X-Forwarded-Encrypted: i=1; AJvYcCUJJ9VEYFRCCD9OWyxTJOdzEYEy5+fuQ/k6ZrBYYi3NweHPzFjdBDemE0WGLQUkM63PFDtpC914VQ==@kvack.org X-Gm-Message-State: AOJu0Yz59sDXQjBUI49jdvcKkQlsk/XdCdFVT+ndcG/Niy+bZ8GdJ5ee Gi39EhGEQ4aJFYOyauGysMDtRP0hQMBNIcAHHmwGy3T2O1yxR6zhRoRaLSkjWXoUbgUzB6SjRzh lYA== X-Google-Smtp-Source: AGHT+IFk9uamhIyVPiPVkC9AAkl4R2cZvpHrX9RuV4B2/rBtrreDBZ8gNEYfOVDCArI/WEZD+WVYAn+KZ5o= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:1569:9ef4:20ab:abf9]) (user=yuzhao job=sendgmr) by 2002:a25:7453:0:b0:e28:eee0:aaa1 with SMTP id 3f1490d57ef6-e2bb12db7e5mr5668276.4.1729484554722; Sun, 20 Oct 2024 21:22:34 -0700 (PDT) Date: Sun, 20 Oct 2024 22:22:17 -0600 In-Reply-To: <20241021042218.746659-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241021042218.746659-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241021042218.746659-6-yuzhao@google.com> Subject: [PATCH v1 5/6] arm64: pause remote CPUs to update vmemmap From: Yu Zhao To: Andrew Morton , Catalin Marinas , Marc Zyngier , Muchun Song , Thomas Gleixner , Will Deacon Cc: Douglas Anderson , Mark Rutland , Nanyong Sun , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Yu Zhao X-Rspam-User: X-Stat-Signature: 1ycjaynp8c9bzapuc1jzo8om6kq6sfjs X-Rspamd-Queue-Id: 156F14000E X-Rspamd-Server: rspam02 X-HE-Tag: 1729484548-926349 X-HE-Meta: U2FsdGVkX1+i9sIgUN+sL1LdXqT6/OGELYaZ2Xpb2jIGDz3N4Qur5G5+SwayiGgvsGZzHcnSFmXRKEtqy7kPbkgcp0ajZJRK0UCmSH8Bds2hXOzXr2FkyGtcFYMQT7sH9yKhVagSOV+zhY5nSeSvGf+yLfKZSrblQ18NE4Yzy4WH9MIdw0JrHHijqm3BhOB8ZJ6XeKHjNFIWz6wjyPuRRKvkqFNWqY/GT2jMyW2SCjsA4MkrOG8HOHjP1ZK9lL7Xvt+dXRfUCkSG7DkRBtT7otjUj1pW+gG7YbuCDIkH9/mJA8WcUoJMgt5mLVbWlMtYDlkYLq/1ZZx+HFsW5y3DjB53s9vDjJyl71UmjaGfL4sXMBTLv1NT3w8erXIa8A7yMJcY3HTeV6duvGVttu20OVnwVhv4lii/zVFu18RYlafskKXxuUEIPbhm/uJhqsvd4UkKxJXF8tJU9dd0r1yxhv8K1eaPj9yq7JYyKE9bqSajjYs5yNh2K3HCtkRUka6C2FeAzuCLCe2I6+tkUicZlJSxwWFas4P9BzvanB061PNytQqe0t3YqAMYU5llP1peYSgR9McqfSLPb9TmpW078oqJFPEIs1RMmtRMoeTiz/q9V1kjvIHXT01iIUNvtm0RCJXiRbvFiAOQOO1Sv57RYeqUmpKzDYPm7Y195VhWlowY7esXV7cTNoekNM1rjLKIubwolTiSyp64jXVFG7jQl0mRYuiEXn+8giuM7ckiVo3unWkZm4RMDGhexU0CBdrbjjTZpoRMJqfIGZH9nPOsJlFB9oO6zhWERhECxHj044hUDbFHu3BuTQbd8schJ3gI1dV5jNotdixFITK4rhOQV1muhMJV1j4fJR6QO5griuNHsSY/F6dR7S/Jl2d8DYnN+sjWwYF8W72ismCJN/R27DJiylxOu3Kfqma+FA8xasPD0i4ezzoo1qj/yyEiqiQWnLbd8N2zjepMJmm1Y8P vdLTkoBR LbOey4kuppD3PR18dj/wn9RL0suJtf7GkBqpjYUMwcP9KJ1lptVu7sXJKaprXqJRjDGtoHsgEwn2A2pya3s/Z91GuuYNfGRia5VUa2tDSW/MxVXLqAL4Ql+Q+mxyuN/ZWQT23dmqN+Wh6VygloHPbJNOqnz8lstKAq37H2dYgDciDLu2RblNURAy2NN2dCNcRgnG6d/J08Gs3eNK56ZSFckZd8Hc6r8vhWD06djjSczoD+7T3lGg+HO/8EvrFLz2gKCxLRz1LjjcwGOj8qmKXBtHwHVEBl0shP21jR/8e8YKrGIFvjtm8so2bqmM0EdaBqG82izVo/EMMq+obiLEQjJqTW2IOJ5klsTkvMQUOO0yqqs67nRkiAAwbyAlC77HLq1fUCcB8wl590IBcjRF/6X1eQxRmUBX28Wahh59WXqn6YafMaxpU3WMzNvFHYg6OEiYg/p45kTmYWGoT8JXL24cn3GaUNTM1p+fBJz0t0OPh2k5K2wnOCc0M5OXbou5CSCFUAusn2mdcCutTJRQKknwdTKd+3wtidBugr5BnCgdoBXq5iS//hAUCDkMcwHudEz8tUMCDICjl1YE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Pause remote CPUs so that the local CPU can follow the proper BBM sequence to safely update the vmemmap mapping `struct page` areas. While updating the vmemmap, it is guaranteed that neither the local CPU nor the remote ones will access the `struct page` area being updated, and therefore they should not trigger (non-spurious) kernel PFs. Signed-off-by: Yu Zhao --- arch/arm64/include/asm/pgalloc.h | 69 ++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 8ff5f2a2579e..f50f79f57c1e 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -12,6 +12,7 @@ #include #include #include +#include #define __HAVE_ARCH_PGD_FREE #define __HAVE_ARCH_PUD_FREE @@ -137,4 +138,72 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep) __pmd_populate(pmdp, page_to_phys(ptep), PMD_TYPE_TABLE | PMD_TABLE_PXN); } +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP + +#define VMEMMAP_ARCH_TLB_FLUSH_FLAGS (VMEMMAP_SPLIT_NO_TLB_FLUSH | VMEMMAP_REMAP_NO_TLB_FLUSH) + +#define vmemmap_update_supported vmemmap_update_supported +static inline bool vmemmap_update_supported(void) +{ + return system_uses_irq_prio_masking(); +} + +#define vmemmap_update_lock vmemmap_update_lock +static inline void vmemmap_update_lock(void) +{ + cpus_read_lock(); +} + +#define vmemmap_update_unlock vmemmap_update_unlock +static inline void vmemmap_update_unlock(void) +{ + cpus_read_unlock(); +} + +#define vmemmap_update_pte_range_start vmemmap_update_pte_range_start +static inline void vmemmap_update_pte_range_start(pte_t *pte, + unsigned long start, unsigned long end) +{ + unsigned long addr; + + local_irq_disable(); + pause_remote_cpus(); + + for (addr = start; addr != end; addr += PAGE_SIZE, pte++) + pte_clear(&init_mm, addr, pte); + + flush_tlb_kernel_range(start, end); +} + +#define vmemmap_update_pte_range_end vmemmap_update_pte_range_end +static inline void vmemmap_update_pte_range_end(void) +{ + resume_remote_cpus(); + local_irq_enable(); +} + +#define vmemmap_update_pmd_range_start vmemmap_update_pmd_range_start +static inline void vmemmap_update_pmd_range_start(pmd_t *pmd, + unsigned long start, unsigned long end) +{ + unsigned long addr; + + local_irq_disable(); + pause_remote_cpus(); + + for (addr = start; addr != end; addr += PMD_SIZE, pmd++) + pmd_clear(pmd); + + flush_tlb_kernel_range(start, end); +} + +#define vmemmap_update_pmd_range_end vmemmap_update_pmd_range_end +static inline void vmemmap_update_pmd_range_end(void) +{ + resume_remote_cpus(); + local_irq_enable(); +} + +#endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ + #endif