From patchwork Tue Dec 14 16:20:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 12676343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E714C433F5 for ; Tue, 14 Dec 2021 16:27:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F36DC6B0082; Tue, 14 Dec 2021 11:22:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EBB4E6B0085; Tue, 14 Dec 2021 11:22:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C4C316B0083; Tue, 14 Dec 2021 11:22:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0180.hostedemail.com [216.40.44.180]) by kanga.kvack.org (Postfix) with ESMTP id A9D836B0082 for ; Tue, 14 Dec 2021 11:22:32 -0500 (EST) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 5D6058248D52 for ; Tue, 14 Dec 2021 16:22:22 +0000 (UTC) X-FDA: 78916917324.16.85C30EE Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) by imf27.hostedemail.com (Postfix) with ESMTP id 98A2A40009 for ; Tue, 14 Dec 2021 16:22:21 +0000 (UTC) Received: by mail-wm1-f74.google.com with SMTP id 144-20020a1c0496000000b003305ac0e03aso13380262wme.8 for ; Tue, 14 Dec 2021 08:22:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=ITyhCHO1UiX2DPCFkkqWNUvg7KfOdfMNTQoZ+QNE7bU=; b=VBMgNGVHDe18Og0TwNncVl3ew6zMtrxStb1uB6st2XRmRBQ3tEQXjuXrYuoSm0prGY K20SlkDHnXzsSFK83A1uOoPf7s8MoCjkMO60Pz4Bsrw5Zv3EFffTyroafyTpApie3f5m nG6BxjJm7YHC0e0oJK8lMT2uRqxC1uWTQHEppiGgKAXQKqpaYr/YGuy4I6cjhoRPfjGj aJvrRtPNJ75tOy8zCBClXxAQIrWbUSjbUhHVVIs8uDdSTyy0Rb+tOXzVFasYYZt5TqIc i0ARL88+0gAlT2wgHGVsXU2gLTSMd4ju6zO7fLp36fAYYiZyasvP1IysAaRLHtGTDfz/ h1sA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ITyhCHO1UiX2DPCFkkqWNUvg7KfOdfMNTQoZ+QNE7bU=; b=bwbjCX3fBC87Pr0sRyQIipuYL/7Kp9IQ22zR3MO2RHmBCfJGTLJBAaawHQMo6D3atX UMiNoQLb/YdnX9wL4nQsDSTZui2WO2uTxcxX66Qh7FEpBnvoKLEjXxbNqSR//IZeSfg+ L3M09bTcXhEpQE8T1SLxhpz0vsLnsoEcHL4BuL6N8BcBWvHJUA2Ny0WAAbHb67CqCUZ7 aEr4PhAXVeJXlHpXR/y9hvoesNAPyHdXTEYz/S9wuPcYgZWq1/GqwtCJ7wLEwARvtkhK 5DitLvPJF6TTg4f/XosQK/XaMP3d7zBowFxfF433PawlSaupjzYrJC7LC0+xfSvgjAdD Y4yg== X-Gm-Message-State: AOAM530+H6XurZehMfysAK1Ig4scR19oV0/RoDf4GVTS6UDlXY4czXMn xVTGt6tyVYZ1Gml2P6Z6aPg9kX2zk9c= X-Google-Smtp-Source: ABdhPJynp7SO7nkkbS3IYqSUnN1vnEfdIZ95PUH2Mr7X6JqBGNbKrX/KoqtS/NjRGYPpNTG+FE367wmyggw= X-Received: from glider.muc.corp.google.com ([2a00:79e0:15:13:357e:2b9d:5b13:a652]) (user=glider job=sendgmr) by 2002:a5d:6ac2:: with SMTP id u2mr6665311wrw.486.1639498940724; Tue, 14 Dec 2021 08:22:20 -0800 (PST) Date: Tue, 14 Dec 2021 17:20:17 +0100 In-Reply-To: <20211214162050.660953-1-glider@google.com> Message-Id: <20211214162050.660953-11-glider@google.com> Mime-Version: 1.0 References: <20211214162050.660953-1-glider@google.com> X-Mailer: git-send-email 2.34.1.173.g76aa8bc2d0-goog Subject: [PATCH 10/43] kmsan: pgtable: reduce vmalloc space From: Alexander Potapenko To: glider@google.com Cc: Alexander Viro , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Ard Biesheuvel , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Marco Elver , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org X-Rspamd-Queue-Id: 98A2A40009 X-Stat-Signature: bznt1nw5x3tzaqqjme4yn1wjg1ey4h5s Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=VBMgNGVH; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of 3vMS4YQYKCD0fkhcdqfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--glider.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3vMS4YQYKCD0fkhcdqfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--glider.bounces.google.com X-Rspamd-Server: rspam11 X-HE-Tag: 1639498941-627071 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: KMSAN is going to use 3/4 of existing vmalloc space to hold the metadata, therefore we lower VMALLOC_END to make sure vmalloc() doesn't allocate past the first 1/4. Signed-off-by: Alexander Potapenko --- Link: https://linux-review.googlesource.com/id/I9d8b7f0a88a639f1263bc693cbd5c136626f7efd --- arch/x86/include/asm/pgtable_64_types.h | 41 ++++++++++++++++++++++++- arch/x86/mm/init_64.c | 2 +- 2 files changed, 41 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/pgtable_64_types.h b/arch/x86/include/asm/pgtable_64_types.h index 91ac106545703..7f15d43754a34 100644 --- a/arch/x86/include/asm/pgtable_64_types.h +++ b/arch/x86/include/asm/pgtable_64_types.h @@ -139,7 +139,46 @@ extern unsigned int ptrs_per_p4d; # define VMEMMAP_START __VMEMMAP_BASE_L4 #endif /* CONFIG_DYNAMIC_MEMORY_LAYOUT */ -#define VMALLOC_END (VMALLOC_START + (VMALLOC_SIZE_TB << 40) - 1) +#define VMEMORY_END (VMALLOC_START + (VMALLOC_SIZE_TB << 40) - 1) + +#ifndef CONFIG_KMSAN +#define VMALLOC_END VMEMORY_END +#else +/* + * In KMSAN builds vmalloc area is four times smaller, and the remaining 3/4 + * are used to keep the metadata for virtual pages. The memory formerly + * belonging to vmalloc area is now laid out as follows: + * + * 1st quarter: VMALLOC_START to VMALLOC_END - new vmalloc area + * 2nd quarter: KMSAN_VMALLOC_SHADOW_START to + * VMALLOC_END+KMSAN_VMALLOC_SHADOW_OFFSET - vmalloc area shadow + * 3rd quarter: KMSAN_VMALLOC_ORIGIN_START to + * VMALLOC_END+KMSAN_VMALLOC_ORIGIN_OFFSET - vmalloc area origins + * 4th quarter: KMSAN_MODULES_SHADOW_START to KMSAN_MODULES_ORIGIN_START + * - shadow for modules, + * KMSAN_MODULES_ORIGIN_START to + * KMSAN_MODULES_ORIGIN_START + MODULES_LEN - origins for modules. + */ +#define VMALLOC_QUARTER_SIZE ((VMALLOC_SIZE_TB << 40) >> 2) +#define VMALLOC_END (VMALLOC_START + VMALLOC_QUARTER_SIZE - 1) + +/* + * vmalloc metadata addresses are calculated by adding shadow/origin offsets + * to vmalloc address. + */ +#define KMSAN_VMALLOC_SHADOW_OFFSET VMALLOC_QUARTER_SIZE +#define KMSAN_VMALLOC_ORIGIN_OFFSET (VMALLOC_QUARTER_SIZE << 1) + +#define KMSAN_VMALLOC_SHADOW_START (VMALLOC_START + KMSAN_VMALLOC_SHADOW_OFFSET) +#define KMSAN_VMALLOC_ORIGIN_START (VMALLOC_START + KMSAN_VMALLOC_ORIGIN_OFFSET) + +/* + * The shadow/origin for modules are placed one by one in the last 1/4 of + * vmalloc space. + */ +#define KMSAN_MODULES_SHADOW_START (VMALLOC_END + KMSAN_VMALLOC_ORIGIN_OFFSET + 1) +#define KMSAN_MODULES_ORIGIN_START (KMSAN_MODULES_SHADOW_START + MODULES_LEN) +#endif /* CONFIG_KMSAN */ #define MODULES_VADDR (__START_KERNEL_map + KERNEL_IMAGE_SIZE) /* The module sections ends with the start of the fixmap */ diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 36098226a9573..8e884e44a8d1e 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -1287,7 +1287,7 @@ static void __init preallocate_vmalloc_pages(void) unsigned long addr; const char *lvl; - for (addr = VMALLOC_START; addr <= VMALLOC_END; addr = ALIGN(addr + 1, PGDIR_SIZE)) { + for (addr = VMALLOC_START; addr <= VMEMORY_END; addr = ALIGN(addr + 1, PGDIR_SIZE)) { pgd_t *pgd = pgd_offset_k(addr); p4d_t *p4d; pud_t *pud;