From patchwork Wed Feb 17 20:56:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrey Konovalov X-Patchwork-Id: 12092329 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 713EFC433E0 for ; Wed, 17 Feb 2021 20:58:59 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2993E64D5D for ; Wed, 17 Feb 2021 20:58:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2993E64D5D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:Mime-Version:Message-Id:Date: Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender :Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=CeiGCrZ94NmJzijg9SZ8mTi1Iuf2uugYG9Nr01CgFmw=; b=bxjDrsijKzFlKZP8i0Ls6UN5l0 EW2f4xM+fn82IMI3qyZstUPLATIo9INRNAakOAtZ016U5Oh83TV38r98SPBV35EEKUqXVUP/Zh8OS pqJYHQHlJuPMtgNOWguXQ9vYNyohCIHhqTHE3etOKfuAsF2MADH36vzgCJ1qv1ufMNd2eNP6Gq38w CWLWFAkOSdTa/ioB0u0IVI/gB8cO4jMNIH/vGVNtgRQSDGXA79Ls1D1QX4+/S1aHRNw8yAtmd5dJV 0grLKPj1Cc4ZT05ueTp+VDUw4zKLk8haccUesLbf/Q96jRy//NjhRWu+Pwfvs+2DUo5xg43mi8W/7 7JQhd4JA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1lCTsc-00043N-4k; Wed, 17 Feb 2021 20:57:02 +0000 Received: from mail-wm1-x34a.google.com ([2a00:1450:4864:20::34a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1lCTsX-000424-PG for linux-arm-kernel@lists.infradead.org; Wed, 17 Feb 2021 20:56:58 +0000 Received: by mail-wm1-x34a.google.com with SMTP id u186so2657378wmb.0 for ; Wed, 17 Feb 2021 12:56:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:message-id:mime-version:subject:from:to:cc; bh=SFShG9HSiA/a5wZPjw/2OI6suzBuDo/vPg1sGXiNPU4=; b=uq+kvVIzeR3An4gN/CAYw6QE6pVQHn27D86IFwunejdmU2OjZaTb9XiR41vGvWVf1m DksVLOj8BdFHTVsw2bj45IJjx43mEoGKUZA6z2RJ8uqSk/sjjs9qy/YvYTfc78kfmHKw ur3Pwdw66rmu5Fo43k0P81tGop/9P/X9cep/Alz9YOKrijW7UIjvvnikPc5vPke+4zXc oBsnCBHRKMjugKkcpV4PRP1ZwVB352IYfVqSk6lqqJtCp2DNfCCl0Fi3P+J0Ydy/z84l cXjO9FMDbiHmosEuqU9+ZzFuSrBXaRZ+eUr4rk9bU5OtGytxWDGj7MwqEkWhh+/Uyc5Q ADNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:message-id:mime-version:subject:from :to:cc; bh=SFShG9HSiA/a5wZPjw/2OI6suzBuDo/vPg1sGXiNPU4=; b=Nxbb3kf71mN0QVBb863spAmgF8r2ojjhffHBd+ygFu3ZI0zPYV4YEpLFYEKl3lQSz0 PREpyonhRcVfH7bkn0yDdeauHztFTExscWRJJimt261B/7E4AQMW6zfkF1GKBz0QaQTr +35fg0erFOfOl4RySFKnAlEPc5B/BJ/cvE7uiQU0f5R6q+agTAKckYS+5S4Mj+PS9n4O p8BIrQt6SxdKIxNd/vEkvDYt3vrA3IFE9HfEymkhSHnhAIfq6/PBzvd/OqJB9Jo853KA R1yDfliqGY+4bGIr/VcBKCCeT1PG7eg9/75zR7FDQ8Fta1OlJZTYp1uikMEokXM30Vkg JbsQ== X-Gm-Message-State: AOAM530kJNL6aA1a9pRxQauTtJsccqfuVfR7RQLdSVQI1IDq5x4FN7K7 5MmUCAo0b6sZ7otOau8Njzs8LOl2/EK3fArU X-Google-Smtp-Source: ABdhPJzpzJFCBeHD/qAy4KRbvT1c+dX2xurE+gdmzIVLGnbTZurQCDQ/kd3T+GmwRfTjrwYjKetpJ1RUziqKPKwg X-Received: from andreyknvl3.muc.corp.google.com ([2a00:79e0:15:13:fc35:c4d:59c2:bb21]) (user=andreyknvl job=sendgmr) by 2002:a05:600c:4314:: with SMTP id p20mr604956wme.52.1613595413677; Wed, 17 Feb 2021 12:56:53 -0800 (PST) Date: Wed, 17 Feb 2021 21:56:32 +0100 Message-Id: <487751e1ccec8fcd32e25a06ce000617e96d7ae1.1613595269.git.andreyknvl@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.30.0.478.g8a0d178c01-goog Subject: [PATCH] mm, kasan: don't poison boot memory From: Andrey Konovalov To: Andrew Morton , Catalin Marinas , Vincenzo Frascino X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210217_155657_846449_F1C61985 X-CRM114-Status: GOOD ( 19.09 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arm-kernel@lists.infradead.org, Marco Elver , Andrey Konovalov , Kevin Brodsky , Will Deacon , Branislav Rankov , kasan-dev@googlegroups.com, linux-kernel@vger.kernel.org, Christoph Hellwig , linux-mm@kvack.org, Alexander Potapenko , Evgenii Stepanov , Andrey Ryabinin , Peter Collingbourne , Dmitry Vyukov Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org During boot, all non-reserved memblock memory is exposed to the buddy allocator. Poisoning all that memory with KASAN lengthens boot time, especially on systems with large amount of RAM. This patch makes page_alloc to not call kasan_free_pages() on all new memory. __free_pages_core() is used when exposing fresh memory during system boot and when onlining memory during hotplug. This patch adds a new FPI_SKIP_KASAN_POISON flag and passes it to __free_pages_ok() through free_pages_prepare() from __free_pages_core(). This has little impact on KASAN memory tracking. Assuming that there are no references to newly exposed pages before they are ever allocated, there won't be any intended (but buggy) accesses to that memory that KASAN would normally detect. However, with this patch, KASAN stops detecting wild and large out-of-bounds accesses that happen to land on a fresh memory page that was never allocated. This is taken as an acceptable trade-off. All memory allocated normally when the boot is over keeps getting poisoned as usual. Signed-off-by: Andrey Konovalov Change-Id: Iae6b1e4bb8216955ffc14af255a7eaaa6f35324d --- mm/page_alloc.c | 43 ++++++++++++++++++++++++++++++++----------- 1 file changed, 32 insertions(+), 11 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0b55c9c95364..f10966e3b4a5 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -108,6 +108,17 @@ typedef int __bitwise fpi_t; */ #define FPI_TO_TAIL ((__force fpi_t)BIT(1)) +/* + * Don't poison memory with KASAN. + * During boot, all non-reserved memblock memory is exposed to the buddy + * allocator. Poisoning all that memory lengthens boot time, especially on + * systems with large amount of RAM. This flag is used to skip that poisoning. + * Assuming that there are no references to those newly exposed pages before + * they are ever allocated, this has little effect on KASAN memory tracking. + * All memory allocated normally after boot gets poisoned as usual. + */ +#define FPI_SKIP_KASAN_POISON ((__force fpi_t)BIT(2)) + /* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ static DEFINE_MUTEX(pcp_batch_high_lock); #define MIN_PERCPU_PAGELIST_FRACTION (8) @@ -384,10 +395,14 @@ static DEFINE_STATIC_KEY_TRUE(deferred_pages); * on-demand allocation and then freed again before the deferred pages * initialization is done, but this is not likely to happen. */ -static inline void kasan_free_nondeferred_pages(struct page *page, int order) +static inline void kasan_free_nondeferred_pages(struct page *page, int order, + fpi_t fpi_flags) { - if (!static_branch_unlikely(&deferred_pages)) - kasan_free_pages(page, order); + if (static_branch_unlikely(&deferred_pages)) + return; + if (fpi_flags & FPI_SKIP_KASAN_POISON) + return; + kasan_free_pages(page, order); } /* Returns true if the struct page for the pfn is uninitialised */ @@ -438,7 +453,13 @@ defer_init(int nid, unsigned long pfn, unsigned long end_pfn) return false; } #else -#define kasan_free_nondeferred_pages(p, o) kasan_free_pages(p, o) +static inline void kasan_free_nondeferred_pages(struct page *page, int order, + fpi_t fpi_flags) +{ + if (fpi_flags & FPI_SKIP_KASAN_POISON) + return; + kasan_free_pages(page, order); +} static inline bool early_page_uninitialised(unsigned long pfn) { @@ -1216,7 +1237,7 @@ static void kernel_init_free_pages(struct page *page, int numpages) } static __always_inline bool free_pages_prepare(struct page *page, - unsigned int order, bool check_free) + unsigned int order, bool check_free, fpi_t fpi_flags) { int bad = 0; @@ -1290,7 +1311,7 @@ static __always_inline bool free_pages_prepare(struct page *page, debug_pagealloc_unmap_pages(page, 1 << order); - kasan_free_nondeferred_pages(page, order); + kasan_free_nondeferred_pages(page, order, fpi_flags); return true; } @@ -1303,7 +1324,7 @@ static __always_inline bool free_pages_prepare(struct page *page, */ static bool free_pcp_prepare(struct page *page) { - return free_pages_prepare(page, 0, true); + return free_pages_prepare(page, 0, true, FPI_NONE); } static bool bulkfree_pcp_prepare(struct page *page) @@ -1323,9 +1344,9 @@ static bool bulkfree_pcp_prepare(struct page *page) static bool free_pcp_prepare(struct page *page) { if (debug_pagealloc_enabled_static()) - return free_pages_prepare(page, 0, true); + return free_pages_prepare(page, 0, true, FPI_NONE); else - return free_pages_prepare(page, 0, false); + return free_pages_prepare(page, 0, false, FPI_NONE); } static bool bulkfree_pcp_prepare(struct page *page) @@ -1533,7 +1554,7 @@ static void __free_pages_ok(struct page *page, unsigned int order, int migratetype; unsigned long pfn = page_to_pfn(page); - if (!free_pages_prepare(page, order, true)) + if (!free_pages_prepare(page, order, true, fpi_flags)) return; migratetype = get_pfnblock_migratetype(page, pfn); @@ -1570,7 +1591,7 @@ void __free_pages_core(struct page *page, unsigned int order) * Bypass PCP and place fresh pages right to the tail, primarily * relevant for memory onlining. */ - __free_pages_ok(page, order, FPI_TO_TAIL); + __free_pages_ok(page, order, FPI_TO_TAIL | FPI_SKIP_KASAN_POISON); } #ifdef CONFIG_NEED_MULTIPLE_NODES