From patchwork Thu Sep 15 15:03:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 12977527 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F29BCC6FA89 for ; Thu, 15 Sep 2022 15:05:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 898A080009; Thu, 15 Sep 2022 11:05:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 847108D0001; Thu, 15 Sep 2022 11:05:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E8C380009; Thu, 15 Sep 2022 11:05:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 60D7C8D0001 for ; Thu, 15 Sep 2022 11:05:05 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2DD931C6110 for ; Thu, 15 Sep 2022 15:05:05 +0000 (UTC) X-FDA: 79914642570.04.E34DB3D Received: from mail-ej1-f74.google.com (mail-ej1-f74.google.com [209.85.218.74]) by imf31.hostedemail.com (Postfix) with ESMTP id 61D39200CF for ; Thu, 15 Sep 2022 15:05:04 +0000 (UTC) Received: by mail-ej1-f74.google.com with SMTP id qa33-20020a17090786a100b0077a69976d24so6989985ejc.7 for ; Thu, 15 Sep 2022 08:05:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=PTsTfo/gJDV7iQzWmhln0w3yo3kBxDNSc9M3IZOTBoE=; b=JPgXbB5IFWEFHgGxJgBDcBqsn7OA+OOmod3TkAJCtmEW3tmDIEAaPKco4V3rSfeVUT GdXM8f5AP9MppFWIxBTCdhy9ugH7naOnesHXUn34A53RkPodiqw3X1l45i+8mKggb98k Rfas/CBCSU/l2+KKyy5Yy+k2HFZReEtFK6RUw8luzU4yWnpDATvBrwpXvH5uKPEFz/CY fMEUsRqiEpjIjSm6svk+PcgL/IEq9k1aaJenW+l4RBh8IWYn4eFeXv42PrPIL6LXSCM+ D6Wc5KgbtUAixY99TEj/8stcJrQCa9sE6zMp90t6pz7+Avl4xRSk9+TMfjnHy6he5k4v 4d0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=PTsTfo/gJDV7iQzWmhln0w3yo3kBxDNSc9M3IZOTBoE=; b=fsld6lnjmZ4I3SEtMLmixjIvqTPcR6YMcerF1pVRIIvYaFn3FCfcaYmGWYFFZPygQq g7YZcIT2wFxrBFLYbRNxmPvroZAqj5nkqOgNJAl4jb4TG+ez1yrE4a+vbZT86vY7mi9h 8KbEHJ7oQzvDMoQ6idfExyYOZfGDeycVnIHJWnFxSEPKnUJCdq+x/xO4gqHQLNu7jcBG L5NZAiNnxoKQ60amJP5lpgw53bc8BFwetyhc7g5dbI1abyYtyF6ivy1XA+pQwUoJ6r7Z Uh+yWi19ZVPXZYGxdhc5RviQWQqP8X/PWbVJLgJlsn+EIlKJuUzbXOCd07EuducBW1lc 54+g== X-Gm-Message-State: ACrzQf3ohhLtDUcyRg3cDeoQYoWUWZTRi4+WCLbEhl27XHK4kGkomjvU ns7thobrAomUXv1CAHIx/OHxTrtlWiU= X-Google-Smtp-Source: AMsMyM4p6KqskMugQKLr3d2QoJoMFJ9ItcCVTDioHI8BQ0089p953E4XlBHrGmtCk9sUiS5VpTwgREGXwKI= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:686d:27b5:495:85b7]) (user=glider job=sendgmr) by 2002:a17:907:2cce:b0:77a:6958:5aaa with SMTP id hg14-20020a1709072cce00b0077a69585aaamr280253ejc.245.1663254303074; Thu, 15 Sep 2022 08:05:03 -0700 (PDT) Date: Thu, 15 Sep 2022 17:03:43 +0200 In-Reply-To: <20220915150417.722975-1-glider@google.com> Mime-Version: 1.0 References: <20220915150417.722975-1-glider@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220915150417.722975-10-glider@google.com> Subject: [PATCH v7 09/43] x86: kmsan: pgtable: reduce vmalloc space From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Alexei Starovoitov , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Biggers , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Stephen Rothwell , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663254304; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PTsTfo/gJDV7iQzWmhln0w3yo3kBxDNSc9M3IZOTBoE=; b=1K+HDNf5Z+l5B5hGZBC/6yu7mEELD7J6hzal1esjnmwZB021tJkIlTQco/XCEEZqOhkHFa IOSiqYMCmwwlBiTbbk6NExLmkJnQIHG8yqPe05GzPNkKV2NNGaauxoVcNhumP2kPA80tZb MyeCIPp/+s0V/3QGkT0LnyDvWgF6uDY= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=JPgXbB5I; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf31.hostedemail.com: domain of 3Hz8jYwYKCEosxupq3s00sxq.o0yxuz69-yyw7mow.03s@flex--glider.bounces.google.com designates 209.85.218.74 as permitted sender) smtp.mailfrom=3Hz8jYwYKCEosxupq3s00sxq.o0yxuz69-yyw7mow.03s@flex--glider.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663254304; a=rsa-sha256; cv=none; b=SDUllGRQP82gMQPuuUme5V/A5BmrIsVr+y5GVhlBYLRcS7gmTbFAXn/+vb0FLxdXhQQ68J vLrj+oS4f4s68e6OtLplvT/tNtofKADnQ+dBwCH8IvWqUN5591MhTgHwIq264kdXnNwJ4Z LtfBdSN3w3/f/sAwtXkgA4rsBuyK2ks= Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=JPgXbB5I; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf31.hostedemail.com: domain of 3Hz8jYwYKCEosxupq3s00sxq.o0yxuz69-yyw7mow.03s@flex--glider.bounces.google.com designates 209.85.218.74 as permitted sender) smtp.mailfrom=3Hz8jYwYKCEosxupq3s00sxq.o0yxuz69-yyw7mow.03s@flex--glider.bounces.google.com X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 61D39200CF X-Stat-Signature: iyu36is3kp4emjstfzzhtefjcquekrfk X-HE-Tag: 1663254304-268384 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: KMSAN is going to use 3/4 of existing vmalloc space to hold the metadata, therefore we lower VMALLOC_END to make sure vmalloc() doesn't allocate past the first 1/4. Signed-off-by: Alexander Potapenko --- v2: -- added x86: to the title v5: -- add comment for VMEMORY_END Link: https://linux-review.googlesource.com/id/I9d8b7f0a88a639f1263bc693cbd5c136626f7efd --- arch/x86/include/asm/pgtable_64_types.h | 47 ++++++++++++++++++++++++- arch/x86/mm/init_64.c | 2 +- 2 files changed, 47 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/pgtable_64_types.h b/arch/x86/include/asm/pgtable_64_types.h index 70e360a2e5fb7..04f36063ad546 100644 --- a/arch/x86/include/asm/pgtable_64_types.h +++ b/arch/x86/include/asm/pgtable_64_types.h @@ -139,7 +139,52 @@ extern unsigned int ptrs_per_p4d; # define VMEMMAP_START __VMEMMAP_BASE_L4 #endif /* CONFIG_DYNAMIC_MEMORY_LAYOUT */ -#define VMALLOC_END (VMALLOC_START + (VMALLOC_SIZE_TB << 40) - 1) +/* + * End of the region for which vmalloc page tables are pre-allocated. + * For non-KMSAN builds, this is the same as VMALLOC_END. + * For KMSAN builds, VMALLOC_START..VMEMORY_END is 4 times bigger than + * VMALLOC_START..VMALLOC_END (see below). + */ +#define VMEMORY_END (VMALLOC_START + (VMALLOC_SIZE_TB << 40) - 1) + +#ifndef CONFIG_KMSAN +#define VMALLOC_END VMEMORY_END +#else +/* + * In KMSAN builds vmalloc area is four times smaller, and the remaining 3/4 + * are used to keep the metadata for virtual pages. The memory formerly + * belonging to vmalloc area is now laid out as follows: + * + * 1st quarter: VMALLOC_START to VMALLOC_END - new vmalloc area + * 2nd quarter: KMSAN_VMALLOC_SHADOW_START to + * VMALLOC_END+KMSAN_VMALLOC_SHADOW_OFFSET - vmalloc area shadow + * 3rd quarter: KMSAN_VMALLOC_ORIGIN_START to + * VMALLOC_END+KMSAN_VMALLOC_ORIGIN_OFFSET - vmalloc area origins + * 4th quarter: KMSAN_MODULES_SHADOW_START to KMSAN_MODULES_ORIGIN_START + * - shadow for modules, + * KMSAN_MODULES_ORIGIN_START to + * KMSAN_MODULES_ORIGIN_START + MODULES_LEN - origins for modules. + */ +#define VMALLOC_QUARTER_SIZE ((VMALLOC_SIZE_TB << 40) >> 2) +#define VMALLOC_END (VMALLOC_START + VMALLOC_QUARTER_SIZE - 1) + +/* + * vmalloc metadata addresses are calculated by adding shadow/origin offsets + * to vmalloc address. + */ +#define KMSAN_VMALLOC_SHADOW_OFFSET VMALLOC_QUARTER_SIZE +#define KMSAN_VMALLOC_ORIGIN_OFFSET (VMALLOC_QUARTER_SIZE << 1) + +#define KMSAN_VMALLOC_SHADOW_START (VMALLOC_START + KMSAN_VMALLOC_SHADOW_OFFSET) +#define KMSAN_VMALLOC_ORIGIN_START (VMALLOC_START + KMSAN_VMALLOC_ORIGIN_OFFSET) + +/* + * The shadow/origin for modules are placed one by one in the last 1/4 of + * vmalloc space. + */ +#define KMSAN_MODULES_SHADOW_START (VMALLOC_END + KMSAN_VMALLOC_ORIGIN_OFFSET + 1) +#define KMSAN_MODULES_ORIGIN_START (KMSAN_MODULES_SHADOW_START + MODULES_LEN) +#endif /* CONFIG_KMSAN */ #define MODULES_VADDR (__START_KERNEL_map + KERNEL_IMAGE_SIZE) /* The module sections ends with the start of the fixmap */ diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 0fe690ebc269b..39b6bfcaa0ed4 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -1287,7 +1287,7 @@ static void __init preallocate_vmalloc_pages(void) unsigned long addr; const char *lvl; - for (addr = VMALLOC_START; addr <= VMALLOC_END; addr = ALIGN(addr + 1, PGDIR_SIZE)) { + for (addr = VMALLOC_START; addr <= VMEMORY_END; addr = ALIGN(addr + 1, PGDIR_SIZE)) { pgd_t *pgd = pgd_offset_k(addr); p4d_t *p4d; pud_t *pud;