From patchwork Fri Jan 10 18:40:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13935247 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9349E7719C for ; Fri, 10 Jan 2025 18:41:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 46FA98D0009; Fri, 10 Jan 2025 13:41:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 380718D0005; Fri, 10 Jan 2025 13:41:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 186308D0009; Fri, 10 Jan 2025 13:41:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E7FC48D0005 for ; Fri, 10 Jan 2025 13:41:29 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A5CD41C7EEE for ; Fri, 10 Jan 2025 18:41:29 +0000 (UTC) X-FDA: 82992410298.14.B001445 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) by imf14.hostedemail.com (Postfix) with ESMTP id CAB4A100007 for ; Fri, 10 Jan 2025 18:41:27 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="AtW31Er/"; spf=pass (imf14.hostedemail.com: domain of 31mmBZwgKCPEcTVdfTgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--jackmanb.bounces.google.com designates 209.85.221.73 as permitted sender) smtp.mailfrom=31mmBZwgKCPEcTVdfTgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736534487; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+fqV4f6s/Eo+EZ+PE2OCyE0kBm/vfnsLcFKRz+1otrE=; b=8VtxCJoR5sIvJUaSkcMJ02h9goFkm4sMMhK8QAg2kNrV6oWu+ubh1U1SfnosvzkIp3PTT2 DiuFwsObLuhJWelJ6fJ3yofmYdPHmLzkGirodz8a81IJAg6cwwEfDAADzbYuYDDTZtwnTY y+KJ4hzb0FhlHAduYfSpy52N/H9vaCU= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="AtW31Er/"; spf=pass (imf14.hostedemail.com: domain of 31mmBZwgKCPEcTVdfTgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--jackmanb.bounces.google.com designates 209.85.221.73 as permitted sender) smtp.mailfrom=31mmBZwgKCPEcTVdfTgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736534487; a=rsa-sha256; cv=none; b=A3v/3N1y59ytIjHvGL9V1MExaugnE5zX5wN7p8ssvhL0oJiOHyAajw2tR1035HuA3rwdfM iNuUvOQP/BwNhuQxtRFifE9VTiotyeKsNImMnvoZ+Gai/2HC08DxY369mAd1+pU15PLxyw bTVpQ8MNIvJrkdrnddvDbMOK1zR3I80= Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-385e49efd59so941630f8f.0 for ; Fri, 10 Jan 2025 10:41:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736534486; x=1737139286; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+fqV4f6s/Eo+EZ+PE2OCyE0kBm/vfnsLcFKRz+1otrE=; b=AtW31Er/SKhLzzVYw3eMqpemQtPbQ082gHAykfkIoXhe5cDV0nHrTVmFNihpzT5JQQ o7HN35Z4tO+2k4bOOT0tNMVnuforNuBFZQnWWDjAkwEU+VsLNvRkhXUwZAsOtfg336Qd 3AVtVS+RVPAJdL+q7gPGL15lSyg9xjzHk+IaI/CRH7JGt3H/Y1AkL7LvFVE7lst5syyW 9C5si4D5m5qRuq6cJ1GL2OCQ6iyoMqbRuiV5cQvjyRr+n52uIFB0ZFOGglc6c701/bNb BjiF/kUrvDIYw8nc1FgHDzJ37nQp03lUEO2XUXooXL3NSP1JgfDjuDjNzX8WQs/4ECKl DPCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736534486; x=1737139286; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+fqV4f6s/Eo+EZ+PE2OCyE0kBm/vfnsLcFKRz+1otrE=; b=HTl5MMFDtplq3xtYHjLDpwEhLudltvsS0RKWMPHyoZK2GeuW2c+ZAWika/9D+FYG5L WHeekZ/ji/HSpY67W2JAS31R3A0CmazJJR5dfHdN5MYN0YXMHuvgs2kf2PGsVkAWAxCy EnS4yWqvRxzFI3yfpflkV1mGWUeglZqA08/acXBhwQGJ1DZUlsvD9vQju3TjYaABePLn dhWEyYEKI9EjSbwbRVWgLIwLshp3RPfWxJmPPIeCJMJHh1zepC+U4UIzsm8pIHcrric+ Rf2MCtgQ5y6/882J012nc/qwQ029GCGK/lK+6vGcy2g5h/7+WnjQcHUUJPuvZ0DiApXD VPsg== X-Forwarded-Encrypted: i=1; AJvYcCXK/BWeDB1pos769glo9oPzT0JvXsRNlELBFER/tFtPCxN/u6FSMSoz6VvyyC+FqpnIH/rwScWu4A==@kvack.org X-Gm-Message-State: AOJu0Yy34ouxtT6cmYBdry8SN67rCjl4eh9PK5VAFYSAOWRkfqYz83ck Uu9GXboUXPfh87IeWCajRsULeTo9jNL2W9yfxw2c17/7RNWoyMSSryoQILSjl6cKksiKudOXrVG YFkZxrFW0kQ== X-Google-Smtp-Source: AGHT+IFgJUpClA++DtJe4xQ45Yc9kKsVSyJCXKqF0cxpPlCBfuMButRzIe/JLM7ynzshFcOsnG5hhjluySNsEA== X-Received: from wrbeh5.prod.google.com ([2002:a05:6000:4105:b0:382:4235:c487]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:adf:8b5b:0:b0:38a:88bc:aea4 with SMTP id ffacd0b85a97d-38a88bcaebfmr7355114f8f.30.1736534486264; Fri, 10 Jan 2025 10:41:26 -0800 (PST) Date: Fri, 10 Jan 2025 18:40:44 +0000 In-Reply-To: <20250110-asi-rfc-v2-v2-0-8419288bc805@google.com> Mime-Version: 1.0 References: <20250110-asi-rfc-v2-v2-0-8419288bc805@google.com> X-Mailer: b4 0.15-dev Message-ID: <20250110-asi-rfc-v2-v2-18-8419288bc805@google.com> Subject: [PATCH RFC v2 18/29] mm: asi: Map dynamic percpu memory as nonsensitive From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Richard Henderson , Matt Turner , Vineet Gupta , Russell King , Catalin Marinas , Will Deacon , Guo Ren , Brian Cain , Huacai Chen , WANG Xuerui , Geert Uytterhoeven , Michal Simek , Thomas Bogendoerfer , Dinh Nguyen , Jonas Bonn , Stefan Kristiansson , Stafford Horne , "James E.J. Bottomley" , Helge Deller , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Naveen N Rao , Madhavan Srinivasan , Paul Walmsley , Palmer Dabbelt , Albert Ou , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Yoshinori Sato , Rich Felker , John Paul Adrian Glaubitz , "David S. Miller" , Andreas Larsson , Richard Weinberger , Anton Ivanov , Johannes Berg , Chris Zankel , Max Filippov , Arnd Bergmann , Andrew Morton , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Uladzislau Rezki , Christoph Hellwig , Masami Hiramatsu , Mathieu Desnoyers , Mike Rapoport , Arnaldo Carvalho de Melo , Namhyung Kim , Mark Rutland , Alexander Shishkin , Jiri Olsa , Ian Rogers , Adrian Hunter , Dennis Zhou , Tejun Heo , Christoph Lameter , Sean Christopherson , Paolo Bonzini , Ard Biesheuvel , Josh Poimboeuf , Pawan Gupta Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-alpha@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-um@lists.infradead.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, kvm@vger.kernel.org, linux-efi@vger.kernel.org, Brendan Jackman , Reiji Watanabe , Junaid Shahid X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: CAB4A100007 X-Rspam-User: X-Stat-Signature: 5f5jhu167wb3e36656oidqsbdhunjqef X-HE-Tag: 1736534487-779672 X-HE-Meta: U2FsdGVkX18uLKHUZ/v0fRN1EEaY5HFd3f4I5EYaRyPToX/fwSyfg1OApZUw/IXINWz9mhrigztWCofSYxGYfbhIpy1z32aLZpsPqZn/1kNWAzjz1AQd+dubUKbDu4eZIkeC73nfP+iF7HRYObX5v1U0k+ulAd5ED1EhrE8IG2SJQgI2gR0d53kHIqvSO6bCy1jFml8ENhwYEGonMIj7f7YBtJqkgI/SodEx6yrQQqco+Ea6fYVBRFYBwJ90NjOVYYtUOBDIN/ZzmJMlEVFKb+sDxzKj6RQAehk0UqW2TETVL21umt8zGc0TnvWn67+ZuDIg/V3zE7nl6s6SAP+jnZwPzwsNlqE6Nr/1ezQQHs8EYikusYxA4SbeB2lU53Diaisx/iVbqnJTD7H2Mwp8UXvLCGvhgfPk6eID0Uwm7fTZjcmcLpkzUiBOayiTPCjPfgvib8UdtiObfXzRNHhEUVzbLTK5HyGBa1JpSHYyb/+/H4Ngo2OtBYsxki6v5nDWmK2YPwUXMALrTv17EQGbpeUUg3oUFAqZNlaRtaAiv+Jzypc5y8kkMTtwPnbMwOTiLheIHulkxsTpMrrPetlcY6nG1rvVdXH6o2lviFRvTXVBBNqJNJByonIe4xPYTUn09NNaZisZ0MdvLxF3zZvEkxKcxBTraM+/A+j/b4TaUhwJC3Y0i+hCRdevwxECQQlZU6xsiDZt8wr97nj1D6yzG1DI4bIGLbZkB33eQ0L0vlon+yqVvNdgqOj9Zsy+b4bmNKsUCsvWspt/pD7H6Vjy3OQjFCFTUt3vxfY0Vh6eHlYbKMP351KBCOgwNIpVKKOnbOuqTOU0XCtvBjdOAZne+HOhBLVKPNmsctccTFe3v1KlL+SdCW2cLR37SmYKq6Pxx/E1lF87PbYl59pNT9S4EpAEfzRDumL76ZxmlNNLpq1K1I7UkVJ29zzeOlb+0hsF5JrGfdaQ8BplWAnhLcb G9+bwZ1L d9TCu5ofv6oiLbh3q6wqfbXvsxjVN5dJ6U4eA7nIx4KA54GRS+xcNj1PHSNrRp9LT59kT9PRqzWO7ugoRB+TUKiVkbR49J/fLDIK2V/y5bWretIMEv/IDuGheVErHHW6eOu81PHJWquNsZsHg5E5EocA7LAU0qlDtdSeMKvMzOzCHtD0kDykWuHN2mkXxs6t9Le9HyK1EHT95271jXjVun29NMlAouw5dZBxaLfp1m8aCitwjCSpAzl7mt4gzQrCT37tV9IeY3oB0DKk39PPQDFhO3TsaFzKcMnmyjB0KIiToTpuDDOqmVqSRUnFW3USLxu3hLHOJq0IRRx/jl47aJi/XyBONTrHsquA7tkKi3xeTOhFz7Br8+oMfNcPCSyS4X3lWEqGh0c7EmKJqjSnOyG6rLbumlrvV0wFy/IZeJTiOCmgJiXdr4GAAwj5VjzySwyQcrB0+GlHT8WqOmfwi0LXKmQMwjsmoIz3U1BQ5fQwC6P1+GIryDACl+U4wdHnpddaW7oauMMvhdNAdf/aHdyRIMmyvSups1pINw32TsrOql2D2a60f7mDXfQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Reiji Watanabe Currently, all dynamic percpu memory is implicitly (and unintentionally) treated as sensitive memory. Unconditionally map pages for dynamically allocated percpu memory as global nonsensitive memory, other than pages that are allocated for pcpu_{first,reserved}_chunk during early boot via memblock allocator (these will be taken care by the following patch). We don't support sensitive percpu memory allocation yet. Co-developed-by: Junaid Shahid Signed-off-by: Junaid Shahid Signed-off-by: Reiji Watanabe Signed-off-by: Brendan Jackman WIP: Drop VM_SENSITIVE checks from percpu code --- mm/percpu-vm.c | 50 ++++++++++++++++++++++++++++++++++++++++++++------ mm/percpu.c | 4 ++-- 2 files changed, 46 insertions(+), 8 deletions(-) diff --git a/mm/percpu-vm.c b/mm/percpu-vm.c index cd69caf6aa8d8eded2395eb4bc4051b78ec6aa33..2935d7fbac41548819a94dcc60566bd18cde819a 100644 --- a/mm/percpu-vm.c +++ b/mm/percpu-vm.c @@ -132,11 +132,20 @@ static void pcpu_pre_unmap_flush(struct pcpu_chunk *chunk, pcpu_chunk_addr(chunk, pcpu_high_unit_cpu, page_end)); } -static void __pcpu_unmap_pages(unsigned long addr, int nr_pages) +static void ___pcpu_unmap_pages(unsigned long addr, int nr_pages) { vunmap_range_noflush(addr, addr + (nr_pages << PAGE_SHIFT)); } +static void __pcpu_unmap_pages(unsigned long addr, int nr_pages, + unsigned long vm_flags) +{ + unsigned long size = nr_pages << PAGE_SHIFT; + + asi_unmap(ASI_GLOBAL_NONSENSITIVE, (void *)addr, size); + ___pcpu_unmap_pages(addr, nr_pages); +} + /** * pcpu_unmap_pages - unmap pages out of a pcpu_chunk * @chunk: chunk of interest @@ -153,6 +162,8 @@ static void __pcpu_unmap_pages(unsigned long addr, int nr_pages) static void pcpu_unmap_pages(struct pcpu_chunk *chunk, struct page **pages, int page_start, int page_end) { + struct vm_struct **vms = (struct vm_struct **)chunk->data; + unsigned long vm_flags = vms ? vms[0]->flags : VM_ALLOC; unsigned int cpu; int i; @@ -165,7 +176,7 @@ static void pcpu_unmap_pages(struct pcpu_chunk *chunk, pages[pcpu_page_idx(cpu, i)] = page; } __pcpu_unmap_pages(pcpu_chunk_addr(chunk, cpu, page_start), - page_end - page_start); + page_end - page_start, vm_flags); } } @@ -190,13 +201,38 @@ static void pcpu_post_unmap_tlb_flush(struct pcpu_chunk *chunk, pcpu_chunk_addr(chunk, pcpu_high_unit_cpu, page_end)); } -static int __pcpu_map_pages(unsigned long addr, struct page **pages, - int nr_pages) +/* + * __pcpu_map_pages() should not be called during the percpu initialization, + * as asi_map() depends on the page allocator (which isn't available yet + * during percpu initialization). Instead, ___pcpu_map_pages() can be used + * during the percpu initialization. But, any pages that are mapped with + * ___pcpu_map_pages() will be treated as sensitive memory, unless + * they are explicitly mapped with asi_map() later. + */ +static int ___pcpu_map_pages(unsigned long addr, struct page **pages, + int nr_pages) { return vmap_pages_range_noflush(addr, addr + (nr_pages << PAGE_SHIFT), PAGE_KERNEL, pages, PAGE_SHIFT); } +static int __pcpu_map_pages(unsigned long addr, struct page **pages, + int nr_pages, unsigned long vm_flags) +{ + unsigned long size = nr_pages << PAGE_SHIFT; + int err; + + err = ___pcpu_map_pages(addr, pages, nr_pages); + if (err) + return err; + + /* + * If this fails, pcpu_map_pages()->__pcpu_unmap_pages() will call + * asi_unmap() and clean up any partial mappings. + */ + return asi_map(ASI_GLOBAL_NONSENSITIVE, (void *)addr, size); +} + /** * pcpu_map_pages - map pages into a pcpu_chunk * @chunk: chunk of interest @@ -214,13 +250,15 @@ static int __pcpu_map_pages(unsigned long addr, struct page **pages, static int pcpu_map_pages(struct pcpu_chunk *chunk, struct page **pages, int page_start, int page_end) { + struct vm_struct **vms = (struct vm_struct **)chunk->data; + unsigned long vm_flags = vms ? vms[0]->flags : VM_ALLOC; unsigned int cpu, tcpu; int i, err; for_each_possible_cpu(cpu) { err = __pcpu_map_pages(pcpu_chunk_addr(chunk, cpu, page_start), &pages[pcpu_page_idx(cpu, page_start)], - page_end - page_start); + page_end - page_start, vm_flags); if (err < 0) goto err; @@ -232,7 +270,7 @@ static int pcpu_map_pages(struct pcpu_chunk *chunk, err: for_each_possible_cpu(tcpu) { __pcpu_unmap_pages(pcpu_chunk_addr(chunk, tcpu, page_start), - page_end - page_start); + page_end - page_start, vm_flags); if (tcpu == cpu) break; } diff --git a/mm/percpu.c b/mm/percpu.c index da21680ff294cb53dfb42bf0d3b3bbd2654d2cfa..c2d913c579bf07892957ac7f601a6a71defadc4b 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -3273,8 +3273,8 @@ int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_t pcpu_populate_pte(unit_addr + (i << PAGE_SHIFT)); /* pte already populated, the following shouldn't fail */ - rc = __pcpu_map_pages(unit_addr, &pages[unit * unit_pages], - unit_pages); + rc = ___pcpu_map_pages(unit_addr, &pages[unit * unit_pages], + unit_pages); if (rc < 0) panic("failed to map percpu area, err=%d\n", rc);