From patchwork Fri Jan 10 18:40:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13935235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5DF83E77188 for ; Fri, 10 Jan 2025 18:41:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4E6F36B00C1; Fri, 10 Jan 2025 13:41:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 46E5F6B00C2; Fri, 10 Jan 2025 13:41:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 270B96B00C3; Fri, 10 Jan 2025 13:41:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 00D1A6B00C1 for ; Fri, 10 Jan 2025 13:41:02 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id ACFF9C0E00 for ; Fri, 10 Jan 2025 18:41:02 +0000 (UTC) X-FDA: 82992409164.01.15FEB30 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) by imf08.hostedemail.com (Postfix) with ESMTP id CB8ED16001D for ; Fri, 10 Jan 2025 18:41:00 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=MoaxMIRc; spf=pass (imf08.hostedemail.com: domain of 3u2mBZwgKCNYB24CE2F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=3u2mBZwgKCNYB24CE2F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736534460; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kYL7XeCfcudeBLlUDEGoh0o6eIbGWBwJtwhvHOdo8c0=; b=CjMHSSTU1L7bSVuoap5N5YKLJVookR9YGhqz3vnKwqaKUVCldG/OaNdgrB3kgdfNL7mvMA ZUnz7g/veNJXjeBBK0lCRXHnpfuONiHt3gKDryij21h3s6TR3ry7VA5XFOZwMXGkL0y/P4 Em2OniMNOKla1BVpiirhEjS1+81W8T8= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=MoaxMIRc; spf=pass (imf08.hostedemail.com: domain of 3u2mBZwgKCNYB24CE2F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=3u2mBZwgKCNYB24CE2F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736534460; a=rsa-sha256; cv=none; b=cl5o17uaQNUfjnupaYL6h5iPB/vC6wG1h9QSvPjJITjW1bJ3ZQOqrPsbkrFox882SkVNvX KnxG3FjQSIpEZdZPt0qbQ+9mM/lyPGlCDPTkevoyjgk914RRLhWNk8aDe2AH5Vr42cdDV4 Gudd32oF0mKSMZlAPbPAWnEv+ikCoOY= Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-4361ecebc5bso12445175e9.1 for ; Fri, 10 Jan 2025 10:41:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736534459; x=1737139259; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kYL7XeCfcudeBLlUDEGoh0o6eIbGWBwJtwhvHOdo8c0=; b=MoaxMIRcciz8ejfW4n38yXtQQnI4/7gutY7SmiWkbPHm/2VVFSqTCiYZqiCh9UaKhI 6R4D6F3+d3zvpbJWCqy+AcOA/Uah/90dD6Rq47Q0O5SRJFHkLiTJgMTCVL3dqLODPYuU 4IpYIdUvTRNcnnhon3hSgXqQb3kIQOkw03ukbvXI6jTgu9KZSR2gj8nPbwDDLM19s2ia mdkmT4gXiHqzjNliih4Sw0Wu9cicW41Orwv5H12ZN7ZQwvOxG2ObbB221rPez8uGsB7h Mri63LS5XKmtvVq1ncw8MFjJVXVesPdlQV13K8tR8ZZLjYydrpufLGaAIds1oiPGHBIi h1Sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736534459; x=1737139259; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kYL7XeCfcudeBLlUDEGoh0o6eIbGWBwJtwhvHOdo8c0=; b=JP61wip3b8MdX83l/tfxdehvTRpRZ/lK9Dh+yjxfM+c02pRpHKbHg0km6cCELHgyA6 RUQS4+f8D3TiYczeyx34emGxD81KgAJ7JXcsBWwgLmOfADA85FrF8hMwrTi8Z55OIlcD bWUknVvcLa25O28+T/rREouEi+bZDjRrpCpl6p7Q3aU2QkSd4lU/7v7Oq3DUVoRgG6vL KQJx5uFP6ggba9Ma4WXyooioUMUpYZp70VVZ9nTm0oblHcFj7wS0tRFdj+HUSlHEnDj3 YCuMsdff9DIxfbtZC3DrGYrkDBpnI49hO4MFPOuSgvT1Hm9JUNEuYexRkn0V7OkC5TAT a7iw== X-Forwarded-Encrypted: i=1; AJvYcCVIUdJgTlxh/xpOVu8+EF/ZnPu49IhJ4z5yEa8m99GswW2XJmnN9hXi82ZsGn0pyvgxZQjI5PPn5A==@kvack.org X-Gm-Message-State: AOJu0Yytt8i+rDnPu+h6oBMo93rVqCza07bXkhB5OqW6VL/oiayNH8gh razqWWywXJs3W96BpGcl4tuKVXpcP6RbLTvZHlpjI2Y+gjHP6aCUwTvAONOCaj7cRZqv4Tn3MNu HR1exyXSlwQ== X-Google-Smtp-Source: AGHT+IH0DyTQGUBeXAkb8pkdNHIPyF1JMCOQ9V4x2A26uY/IXi2bUp3/F2mGP9HMTIoPxxyBiM7AkavMfpYEag== X-Received: from wmdn10.prod.google.com ([2002:a05:600c:294a:b0:436:d819:e4eb]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4f09:b0:434:fafe:edb with SMTP id 5b1f17b1804b1-436e26f00fdmr96576145e9.24.1736534459144; Fri, 10 Jan 2025 10:40:59 -0800 (PST) Date: Fri, 10 Jan 2025 18:40:32 +0000 In-Reply-To: <20250110-asi-rfc-v2-v2-0-8419288bc805@google.com> Mime-Version: 1.0 References: <20250110-asi-rfc-v2-v2-0-8419288bc805@google.com> X-Mailer: b4 0.15-dev Message-ID: <20250110-asi-rfc-v2-v2-6-8419288bc805@google.com> Subject: [PATCH RFC v2 06/29] mm: asi: Use separate PCIDs for restricted address spaces From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Richard Henderson , Matt Turner , Vineet Gupta , Russell King , Catalin Marinas , Will Deacon , Guo Ren , Brian Cain , Huacai Chen , WANG Xuerui , Geert Uytterhoeven , Michal Simek , Thomas Bogendoerfer , Dinh Nguyen , Jonas Bonn , Stefan Kristiansson , Stafford Horne , "James E.J. Bottomley" , Helge Deller , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Naveen N Rao , Madhavan Srinivasan , Paul Walmsley , Palmer Dabbelt , Albert Ou , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Yoshinori Sato , Rich Felker , John Paul Adrian Glaubitz , "David S. Miller" , Andreas Larsson , Richard Weinberger , Anton Ivanov , Johannes Berg , Chris Zankel , Max Filippov , Arnd Bergmann , Andrew Morton , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Uladzislau Rezki , Christoph Hellwig , Masami Hiramatsu , Mathieu Desnoyers , Mike Rapoport , Arnaldo Carvalho de Melo , Namhyung Kim , Mark Rutland , Alexander Shishkin , Jiri Olsa , Ian Rogers , Adrian Hunter , Dennis Zhou , Tejun Heo , Christoph Lameter , Sean Christopherson , Paolo Bonzini , Ard Biesheuvel , Josh Poimboeuf , Pawan Gupta Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-alpha@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-um@lists.infradead.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, kvm@vger.kernel.org, linux-efi@vger.kernel.org, Brendan Jackman , Yosry Ahmed , Junaid Shahid X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: CB8ED16001D X-Rspam-User: X-Stat-Signature: uebcyqt4e7coaujucek5tzkhg9hqrwrk X-HE-Tag: 1736534460-925707 X-HE-Meta: U2FsdGVkX1/wxscHzruO2tQU+Eo6UENp33j6hEaKFSMdxrQ/2ObyFYRWkbbbwo5MKUTeuR1v6OMzddhswNzh2P2lEANo4xLoYQAhnaC7Kobq1Lt6tyAQCwS/VP5jw7q/UlHfbG9tsdqqIpxJgTfeNP7r7cpUzeYtryTL2+E36urRcpNR2eXmxKz9P/S+UJSwXWBzB45B3WuXBfLivM2AnY/059tNASECJGMD6DDB3JmBQAKrB6dfrwrO/G+kSCXkelhg+Xi8DJEzI+sdyQEDAO89UXDUmNRXG5vsg/UKHAedjbCyorO92peo+72LIOIXmu6CuYVdpXj5yysPZSRD67ENTnCifaf4naI6OnTeEvNtYfMQqUauUsLAO9CnqtcAi/8QPybHBG+Aea4R/yzR3RBXGJhu6U5afL66RpknJTF9UKf8voVf1fRUxq2HDhebr8/QUrYZa0JspUcwxqUEdu19dN8dcqf4rG0nDPVSuc6ywNZ8LL1hpJjdsN78HX3OO0lIN251grAKw08FX93pNnVRMSYA5K3WKsIiIA7Vew5kqrUt/7Lf7N6SY+NcV9sMQJ0nryT8Fvxfj02PlI0HxYsgR67PxvYOneJRbxq2otuvs8McvXTYX9p31b9GcoKpPcaTB9fMYTlNMuhoRo0rCIj/iLfUP4y6SghPV9iVrg1S2AVG0Nc+VGLaBLTq4dej8tTosZYHKHgCoxknoZawj5mQ1/u6DUoL9+16GKilc8K85Qfqb7w14KGpMja/XiZDBsiewDOFdRvHPbWC9ctK9P4TNbwzpqM6pgrZzAnAoWCWnJkA2wtBLYVSj4lsVnm/x3EQdwvfkPSgDFkmyNFseQoV8jhVlNJQwcznoQOgy/dYuK9RuSiZaNgWKqBQ/wdpzn22mNMFF4ir7H6gDyEvR6YwbTgQ+UzDwFti80yoWiTR7rS3lmy3AhZDvCMdJbwAmcqGc8giHdDlN4LAGU5 thkaPMpN 9xSahKc496JVIXOtsxkKl+qd9ZINRwO9KHx7FRAx8+38/pBEDCN9Hx8XKwJZQ2+oQImjp73mBAlsCp7JzobrWFa1zGZq3IOTS9qWd5PUWxYWZS4ly5vOH4u3S+VPsZ2J2cSdfQM6mDwrvstRICQ6oahg41vlV8Ee4oqCOzyW7IFeUnnKnFobT6rAk+UZ55KanY7t1Qk5lAxUQ+PwwMcSPeF5M86AqQ7ANWi37vQ9/nfR/LQkcFnIK/043l55qYRDI4+JX/1T6kaoGaxblqcuazvPIoPtuHQeUkhaQbTZDCIsYVTHKGhC/wY36co0VfT5vq/+pQuK+8f4X2ilH6Zhr3IoRddjo7Vh1NQGK9r0QWG74cQgQA5H2DoGyzSO+zHaiF/X/JAvlzoz/1TO6CH/FCqYe9XLdn+UN2ltY9F2SKo2Zdosa/jsrZVEDvcolcpylhBDTtzOChWtKQf1rcvUwDA8rqJCt5NNliMqER1q0OXq6T/ftJj7kR7PZhAJJNoOr1fBmIvrAlw2ZCZQCeaAmqOCNamYWmnZ63Xv1dDSjmX+Efs+j8SXQHhXCHg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Yosry Ahmed Each restricted address space is assigned a separate PCID. Since currently only one ASI instance per-class exists for a given process, the PCID is just derived from the class index. This commit only sets the appropriate PCID when switching CR3, but does not actually use the NOFLUSH bit. That will be done by later patches. Co-developed-by: Junaid Shahid Signed-off-by: Junaid Shahid Signed-off-by: Yosry Ahmed Signed-off-by: Brendan Jackman --- arch/x86/include/asm/asi.h | 4 +-- arch/x86/include/asm/processor-flags.h | 24 +++++++++++++++++ arch/x86/include/asm/tlbflush.h | 3 +++ arch/x86/mm/asi.c | 10 +++---- arch/x86/mm/tlb.c | 49 +++++++++++++++++++++++++++++++--- include/asm-generic/asi.h | 2 ++ 6 files changed, 81 insertions(+), 11 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index 9a9a139518289fc65f26a4d1cd311aa52cc5357f..a55e73f1b2bc84c41b9ab25f642a4d5f1aa6ba90 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -4,13 +4,13 @@ #include -#include - #include #include #include #include +#include + #ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION /* diff --git a/arch/x86/include/asm/processor-flags.h b/arch/x86/include/asm/processor-flags.h index e5f204b9b33dfaa92ed1b05faa6b604e50d5f2f3..42c5acb67c2d2a6b03deb548fe3dd088baa88842 100644 --- a/arch/x86/include/asm/processor-flags.h +++ b/arch/x86/include/asm/processor-flags.h @@ -55,4 +55,28 @@ # define X86_CR3_PTI_PCID_USER_BIT 11 #endif +/* + * An ASI identifier is included in the higher bits of PCID to use a different + * PCID for each restricted address space, different from the PCID of the + * unrestricted address space (see asi_pcid()). We use the bits directly after + * the bit used by PTI (if any). + */ +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + +#define X86_CR3_ASI_PCID_BITS 2 + +/* Use the highest available PCID bits after the PTI bit (if any) */ +#ifdef CONFIG_MITIGATION_PAGE_TABLE_ISOLATION +#define X86_CR3_ASI_PCID_END_BIT (X86_CR3_PTI_PCID_USER_BIT - 1) +#else +#define X86_CR3_ASI_PCID_END_BIT (X86_CR3_PCID_BITS - 1) +#endif + +#define X86_CR3_ASI_PCID_BITS_SHIFT (X86_CR3_ASI_PCID_END_BIT - X86_CR3_ASI_PCID_BITS + 1) +#define X86_CR3_ASI_PCID_MASK (((1UL << X86_CR3_ASI_PCID_BITS) - 1) << X86_CR3_ASI_PCID_BITS_SHIFT) + +#else +#define X86_CR3_ASI_PCID_BITS 0 +#endif + #endif /* _ASM_X86_PROCESSOR_FLAGS_H */ diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index c884174a44e119a3c027c44ada6c5cdba14d1282..f167feb5ebdfc7faba26b8b18ac65888cd9b0494 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -425,5 +425,8 @@ static inline void __native_tlb_flush_global(unsigned long cr4) } unsigned long build_cr3_noinstr(pgd_t *pgd, u16 asid, unsigned long lam); +unsigned long build_cr3_pcid_noinstr(pgd_t *pgd, u16 pcid, unsigned long lam, bool noflush); + +u16 asi_pcid(struct asi *asi, u16 asid); #endif /* _ASM_X86_TLBFLUSH_H */ diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index 054315d566c082c0925a00ce3a0877624c8b9957..8d060c633be68b508847e2c1c111761df1da92af 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -238,6 +238,7 @@ static __always_inline void maybe_flush_data(struct asi *next_asi) noinstr void __asi_enter(void) { u64 asi_cr3; + u16 pcid; struct asi *target = asi_get_target(current); /* @@ -266,9 +267,8 @@ noinstr void __asi_enter(void) this_cpu_write(curr_asi, target); maybe_flush_control(target); - asi_cr3 = build_cr3_noinstr(target->pgd, - this_cpu_read(cpu_tlbstate.loaded_mm_asid), - tlbstate_lam_cr3_mask()); + pcid = asi_pcid(target, this_cpu_read(cpu_tlbstate.loaded_mm_asid)); + asi_cr3 = build_cr3_pcid_noinstr(target->pgd, pcid, tlbstate_lam_cr3_mask(), false); write_cr3(asi_cr3); maybe_flush_data(target); @@ -335,8 +335,8 @@ noinstr void asi_exit(void) unrestricted_cr3 = build_cr3_noinstr(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, - this_cpu_read(cpu_tlbstate.loaded_mm_asid), - tlbstate_lam_cr3_mask()); + this_cpu_read(cpu_tlbstate.loaded_mm_asid), + tlbstate_lam_cr3_mask()); /* Tainting first makes reentrancy easier to reason about. */ this_cpu_or(asi_taints, ASI_TAINT_KERNEL_DATA); diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 7c2309996d1d5a7cac23bd122f7b56a869d67d6a..2601beed83aef182d88800c09d70e4c5e95e7ed0 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -13,6 +13,7 @@ #include #include +#include #include #include #include @@ -96,7 +97,10 @@ # define PTI_CONSUMED_PCID_BITS 0 #endif -#define CR3_AVAIL_PCID_BITS (X86_CR3_PCID_BITS - PTI_CONSUMED_PCID_BITS) +#define CR3_AVAIL_PCID_BITS (X86_CR3_PCID_BITS - PTI_CONSUMED_PCID_BITS - \ + X86_CR3_ASI_PCID_BITS) + +static_assert(BIT(CR3_AVAIL_PCID_BITS) > TLB_NR_DYN_ASIDS); /* * ASIDs are zero-based: 0->MAX_AVAIL_ASID are valid. -1 below to account @@ -125,6 +129,11 @@ static __always_inline u16 kern_pcid(u16 asid) */ VM_WARN_ON_ONCE(asid & (1 << X86_CR3_PTI_PCID_USER_BIT)); #endif + +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + BUILD_BUG_ON(TLB_NR_DYN_ASIDS >= (1 << X86_CR3_ASI_PCID_BITS_SHIFT)); + VM_WARN_ON_ONCE(asid & X86_CR3_ASI_PCID_MASK); +#endif /* * The dynamically-assigned ASIDs that get passed in are small * (class_id + 1) << X86_CR3_ASI_PCID_BITS_SHIFT); + // return kern_pcid(asid) | ((asi->index + 1) << X86_CR3_ASI_PCID_BITS_SHIFT); +} + +#else /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ + +u16 asi_pcid(struct asi *asi, u16 asid) { return kern_pcid(asid); } + +#endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ + void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int stride_shift, bool freed_tables) diff --git a/include/asm-generic/asi.h b/include/asm-generic/asi.h index 7f542c59c2b8a2b74432e4edb7199f9171db8a84..f777a6cf604b0656fb39087f6eba08f980b2cb6f 100644 --- a/include/asm-generic/asi.h +++ b/include/asm-generic/asi.h @@ -2,6 +2,7 @@ #ifndef __ASM_GENERIC_ASI_H #define __ASM_GENERIC_ASI_H +#include #include #ifndef _ASSEMBLY_ @@ -16,6 +17,7 @@ enum asi_class_id { #endif ASI_MAX_NUM_CLASSES, }; +static_assert(order_base_2(X86_CR3_ASI_PCID_BITS) <= ASI_MAX_NUM_CLASSES); typedef u8 asi_taints_t;