From patchwork Tue Dec 19 17:50:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jisheng Zhang X-Patchwork-Id: 13498796 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3256C41535 for ; Tue, 19 Dec 2023 18:03:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5577E6B0072; Tue, 19 Dec 2023 13:03:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 506A66B0080; Tue, 19 Dec 2023 13:03:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3CDA46B0078; Tue, 19 Dec 2023 13:03:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 280978D0001 for ; Tue, 19 Dec 2023 13:03:29 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id DADB88041D for ; Tue, 19 Dec 2023 18:03:28 +0000 (UTC) X-FDA: 81584340096.08.9C80B89 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf20.hostedemail.com (Postfix) with ESMTP id CE5901C002B for ; Tue, 19 Dec 2023 18:03:26 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="Tx6P/DJj"; spf=pass (imf20.hostedemail.com: domain of jszhang@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=jszhang@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703009007; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=DRTfjTzKB6pmfIrGrFKfi8dTXOFErbhekmtAC+88+/4=; b=GK+7Hqj3MzbZCyex73HDA1LK+CkmBwTRtQAyy2bzQZxvxK+LJ0kbaypjVP74Zlf0LkVbbj UZ5XmG4/4RIkcHR1FXZ/1K3SZgMslaM3ryQ7RGGZsoiWCaiEzJLraT029TeB4Big1g2Us+ 9YXGpDQv3F7VdkGZ7M27ORn15ugBM+U= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703009007; a=rsa-sha256; cv=none; b=1MXeSI9+BBsQDZqFsKJDNaWpv6hA9EUnW1ctUA9ovaikI+iOYoiGYlEIcrML/41qoS5Okl 79xxjpaNq80lMtBeNX5HypSM90gXhasFqjQlgvGknjyKRkGCeoXoKmamsKrqwrm4z9fLOT PfOx0IdeETeTEoPZGven1x53eVDnjcw= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="Tx6P/DJj"; spf=pass (imf20.hostedemail.com: domain of jszhang@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=jszhang@kernel.org; dmarc=pass (policy=none) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 33F18CE1AA3; Tue, 19 Dec 2023 18:03:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 248D2C433C7; Tue, 19 Dec 2023 18:03:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1703009002; bh=UL2kffNuUbkPlnJBN+qpCzpP5kui14dRs7DZfONdDT0=; h=From:To:Cc:Subject:Date:From; b=Tx6P/DJjFhfS2hJiQefr5BQzr7JnNvODIRRy7UQlV7ksSfFMQfUEGi2O7ZYHnX46p 7zZWYPi/ssCCH0Z/Xb/HvTsstn+U5ywq4poEVpAPlmY6tUmyi39QY7UPJV0EdHJBJK h0auiil7imG1WcV2nDuuziUeVKwCTF7BrLxBlSrcxkeJBHc6uElnSju2pnjNw0TYZp I7zsC2VlkFB6MNWsigTWEybThY55+MS4U5/x/cM3+JUt6CUBm0K1KrUnhmeG94M1Ft SCXWVZSV2TaN4upbjHB+1U/m0Kt3CkLh+nuKeRkxaQ0T4ePmCHdLqAh0Us6nZH2L+K ncYMPB83i9ceg== From: Jisheng Zhang To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Will Deacon , "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 0/4] riscv: support fast gup Date: Wed, 20 Dec 2023 01:50:42 +0800 Message-Id: <20231219175046.2496-1-jszhang@kernel.org> X-Mailer: git-send-email 2.40.0 MIME-Version: 1.0 X-Rspamd-Queue-Id: CE5901C002B X-Rspam-User: X-Stat-Signature: 5mh5p7ctffz9d5buknmwmorcmamcg1ex X-Rspamd-Server: rspam03 X-HE-Tag: 1703009006-826542 X-HE-Meta: U2FsdGVkX19cFZAWjP15uUCaAo1Z0qHBH9MjmdZXP6qqFh0LoYq/nkU5B7lwfA7ZW5Ro3dUJ5AsskDQUcGmgwXFPf/0BeAF7tUhSHWJCjJM+rPZxdmuoOEFw70hs0N7cylJ4CLM4bcreccMxZ+02GP+7CLVUYLlvM+ygcbe2pjvUODzbWKW6roOxmJaPn5Vkj58oE/25mushX+raN/iyIfMaqbo2uhI10JbCbzGSVFRnNme01wqPGs4LqLjvbfkyzcP2qkzuggoLmcoWFmnK/sqIqW3PmSlv2EpMEXlk1czJR1UBhY3VhL5ImRkbO0Rh1gU8CAYwaFTQ+9yU/blplXUSAyxuR+uQ18BsonIkNKmo9NVUM3UYd6EjMOeckQzDPGJzUeSBnfDrYjHOWGJid44FVVcP/NIEYJ+J4mALbn7IrT24xCprd68gWrvrGLWpN3BapMNAbJUopUjBS3WQgv4ac214yFuPhWIuYkTSz2/Lh8q0lQ9ZSsDPppJaS4D1h6Che393CojsUTQP3U1hgXMgsYg7NuYXikVe2OiK4z7Z4rvbDBc5KrJaQ5/DX5hJzb5aFcCUqBXFsWczk5NQUx5gbrxjLy0C37gyNib8TyON6DOEdyPX04ldQrPvY3nVZ//qd+zqsQNYDg2MTeESdYuYhkNUe7wo3STrPEbT7/bpxBCeP7ugPP00Ox3Vfk1Pn49ILbfwE4cNH+PH11d20BYC/SnHyhgjNvw9XkXxJVD4bPy0u9kVM1vlKJikUS4bjRtHyhLgk4BMoIXYIGS8Gr6r2A/Noo2g5/gjpPFX1pitDaKhSQsBGrLsrdXaJwffYIorasrxrwnzEaP/2deLDt6ib6ziOwdmw2plXw070qeQ/ptNAuN+kYiMBGsmeiXYuf52o3OK75V7nOUikaW6z09sqdF6ZfUDk8o00SLHS09ohNIxytt7ONfD1yR9bE4xk3pkabJMa7QJ6w9b4EM +dHpSXt+ gJsvqpj43/+9Zo1oynUPN28Hs8exOarbvRWXMMU/aN8eW6MTmW2gPFqDZ/57l65Lg7M3VeRpxgkoC7L3EPVeJnzkrtuUocfrlyMZYAU5uMXqA87/6ho+NbHCmepFpu6xXKP/rbzByDeXZXHXa28RqQquWu86eRiudfiWed48Bu1bqDDTUN7LRDK1QbA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This series adds fast gup support to riscv. The First patch fixes a bug in __p*d_free_tlb(). Per the riscv privileged spec, if non-leaf PTEs I.E pmd, pud or p4d is modified, a sfence.vma is a must. The 2nd patch is a preparation patch. The last two patches do the real work: In order to implement fast gup we need to ensure that the page table walker is protected from page table pages being freed from under it. riscv situation is more complicated than other architectures: some riscv platforms may use IPI to perform TLB shootdown, for example, those platforms which support AIA, usually the riscv_ipi_for_rfence is true on these platforms; some riscv platforms may rely on the SBI to perform TLB shootdown, usually the riscv_ipi_for_rfence is false on these platforms. To keep software pagetable walkers safe in this case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h for more details. This patch enables MMU_GATHER_RCU_TABLE_FREE, then use *tlb_remove_page_ptdesc() for those platforms which use IPI to perform TLB shootdown; *tlb_remove_ptdesc() for those platforms which use SBI to perform TLB shootdown; Both case mean that disabling interrupts will block the free and protect the fast gup page walker. So after the 3rd patch, everything is well prepared, let's select HAVE_FAST_GUP if MMU. Jisheng Zhang (4): riscv: tlb: fix __p*d_free_tlb() riscv: tlb: convert __p*d_free_tlb() to inline functions riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU riscv: enable HAVE_FAST_GUP if MMU arch/riscv/Kconfig | 2 ++ arch/riscv/include/asm/pgalloc.h | 53 +++++++++++++++++++++++++++----- arch/riscv/include/asm/pgtable.h | 6 ++++ arch/riscv/include/asm/tlb.h | 18 +++++++++++ 4 files changed, 71 insertions(+), 8 deletions(-)