From patchwork Thu May 23 14:08:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 10957751 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5E20315A6 for ; Thu, 23 May 2019 14:09:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4530428562 for ; Thu, 23 May 2019 14:09:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 38FEC285CD; Thu, 23 May 2019 14:09:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A975A285FD for ; Thu, 23 May 2019 14:09:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 938416B000C; Thu, 23 May 2019 10:09:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8E9086B000D; Thu, 23 May 2019 10:09:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B1EB6B000E; Thu, 23 May 2019 10:09:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by kanga.kvack.org (Postfix) with ESMTP id 54EE06B000C for ; Thu, 23 May 2019 10:09:07 -0400 (EDT) Received: by mail-qk1-f197.google.com with SMTP id p7so5515086qkk.19 for ; Thu, 23 May 2019 07:09:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=gdPkgObKlWOcdk8e46Oik49xVTWYBX7bQXMpu7mlMGs=; b=LG8gEHmHz/uRFmk8gPUUq291r4QGoURSO10IM8gAgwiXjYKCRopwIOP98FlQ3txMiL atY/jbNzyVYKUuN7UmTAcmbOQ21ALtsQi271lwsDzVVMmuMHts6NENL+QSVOnUt6M+3b ZNJOl4spmcummR5+XMWS9s84Sofq6vDs2TcQ/vt3KcIWpyb4EYX1m5f7sxfzkBFlysTZ OH6FkoQ5YYnHKhbzZZSFO3P7um5yw003qDBoxJb+zocavSxGVIMNB3qOjRvZBxFADC7o rh1E38DtEj1oJMZMxSC27soRc3Haj8a4AVM7o+3qXEhRNpJT9YBEp/z8anraigT9g61y PmWA== X-Gm-Message-State: APjAAAWxsa8qpEsaLo+1P6VgqM5id+4AKcEKFQuOgBBRZBi0GD8d6YIo Lof8xgEjT1dTio6mQVEhJaaSwL3xBuoGz0jzCDzzpCNwYgYrgkN6X4iV3Bz007nZ3PHFe9OXDrh 5cUb7pi1CTwLUjn2g7PZKepy/VzTrdS1HN4Kkyf2NjYLna+PbCnhrN2BAldVru4mjbA== X-Received: by 2002:a37:4ad7:: with SMTP id x206mr62273146qka.129.1558620547030; Thu, 23 May 2019 07:09:07 -0700 (PDT) X-Received: by 2002:a37:4ad7:: with SMTP id x206mr62272999qka.129.1558620545722; Thu, 23 May 2019 07:09:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558620545; cv=none; d=google.com; s=arc-20160816; b=dNHvOdl8vVzrI0beTm/7jJ/B3Al+hqCYQvgYthuMBEzjzDRAPBO+0ym0pUbe0RXygS uvqHar2GZ8OHQ3byafWr54X58eKEWMV0Hjry2lz/i+4XdUwtzDzCGudiXLFu0ffdzADf mo37N2k7Fce5lqpF9cUl3rpJ1/FlorT0PnOnCeueyp/8AVlUw1p6XKCq4xpEzU+tMLrs qkluWwg/nHdTUcR7LmfXkYbMsquaNqxifoeUo+a40c0+U1Ej3yiOUYaAGjGoOUG8l+CR LLS3DOBhR2ioBvzJQkGJJUxgJCvyfNbZUB7e4BlOt1FD2tdA5imlPcISPl2ItZBrOJjd F3sA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=gdPkgObKlWOcdk8e46Oik49xVTWYBX7bQXMpu7mlMGs=; b=00W6fAu/JhWUzvDCrCRXgqfs89GIMoXff35lNUnsT7sKDz2HO3c9FDg1fF+4pkfuqG hOTw7O6r8qFBxp38TRMUg5d9fC8yoNtlQDCpicqfkNpAjq1cHxbUx57UbQR8sEGt0rXQ 1zfIYfbu7OuMzBkVCoCQ+XE+1fpbLcsgBhEusD5r8Eqn4ebVUS9OtT3xxGTyg1IJ4HaA 9OX1sw+MGMqkMfjUikAgNz4IYxD8s+Byg6USP+neKjbNSoUdBtPYif3AmGp4sePv/tcV BiGnls98hJ4Mtz7dtBbnuKJEcYN7xLaeI1VxR0KtiBiBlyD5pty/xvGiqjsDDkFQZlOQ RLTA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b="rXDxY0v/"; spf=pass (google.com: domain of 3ganmxaykcfwafc78laiiaf8.6igfchor-ggep46e.ila@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3ganmXAYKCFwAFC78LAIIAF8.6IGFCHOR-GGEP46E.ILA@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id j1sor35761828qtn.1.2019.05.23.07.09.05 for (Google Transport Security); Thu, 23 May 2019 07:09:05 -0700 (PDT) Received-SPF: pass (google.com: domain of 3ganmxaykcfwafc78laiiaf8.6igfchor-ggep46e.ila@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b="rXDxY0v/"; spf=pass (google.com: domain of 3ganmxaykcfwafc78laiiaf8.6igfchor-ggep46e.ila@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3ganmXAYKCFwAFC78LAIIAF8.6IGFCHOR-GGEP46E.ILA@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=gdPkgObKlWOcdk8e46Oik49xVTWYBX7bQXMpu7mlMGs=; b=rXDxY0v/lcLDjFplx5Y7/QPVkoA04qniNWemRKZT0Zdqy5OBEhJjVJuUUaV29pBTB3 VS8jo8tOdan672tQQIhQnAGT3cmdY6wBeUeLQSj4sADPKAINNSV/21V8SWKqq1X5LzuV GYRwl821kh4NHSLftKaqcyzjagUgP5K4JXc3zu/LmfvjZjJOgxiAa/kawy+JXOpskcTt 20e6VYa67N/9/K9VQ6bw6sMXSFZnnstfWtGU6+zroCYSfBqwg1mbIDPo0NLzcEOQrpKS gZpnlYbcCfC/qKaxkL86vIe2Vp4whKOnpmGejWpxfWdlhsGdJm3j+LJy1WEmBUUulK5D Plkw== X-Google-Smtp-Source: APXvYqwf2z6zZcNlfdnPdndyRBGBPUUEGL4p18pe71ljCZ4mCuadqwo7nQ1dTlYgVSfyU+Ol0TYu5b6w9rM= X-Received: by 2002:ac8:1b33:: with SMTP id y48mr80781800qtj.176.1558620545363; Thu, 23 May 2019 07:09:05 -0700 (PDT) Date: Thu, 23 May 2019 16:08:42 +0200 In-Reply-To: <20190523140844.132150-1-glider@google.com> Message-Id: <20190523140844.132150-2-glider@google.com> Mime-Version: 1.0 References: <20190523140844.132150-1-glider@google.com> X-Mailer: git-send-email 2.21.0.1020.gf2820cf01a-goog Subject: [PATCH v4 1/3] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options From: Alexander Potapenko To: akpm@linux-foundation.org, cl@linux.com, keescook@chromium.org Cc: kernel-hardening@lists.openwall.com, linux-mm@kvack.org, linux-security-module@vger.kernel.org, Masahiro Yamada , Michal Hocko , James Morris , "Serge E. Hallyn" , Nick Desaulniers , Kostya Serebryany , Dmitry Vyukov , Sandeep Patil , Laura Abbott , Randy Dunlap , Jann Horn , Mark Rutland X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic. init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed. init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses. Both init_on_alloc=1 and init_on_free=1 guarantee that the allocator returns zeroed memory. The only exception is slab caches with constructors. Those are never zero-initialized to preserve their semantics. For SLOB allocator init_on_free=1 also implies init_on_alloc=1 behavior, i.e. objects are zeroed at both allocation and deallocation time. This is done because SLOB may otherwise return multiple freelist pointers in the allocated object. For SLAB and SLUB enabling either init_on_alloc or init_on_free leads to one-time initialization of the object. Both init_on_alloc and init_on_free default to zero, but those defaults can be overridden with CONFIG_INIT_ON_ALLOC_DEFAULT_ON and CONFIG_INIT_ON_FREE_DEFAULT_ON. Slowdown for the new features compared to init_on_free=0, init_on_alloc=0: hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%) Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%) Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%) Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%) The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error. The new features are also going to pave the way for hardware memory tagging (e.g. arm64's MTE), which will require both on_alloc and on_free hooks to set the tags for heap objects. With MTE, tagging will have the same cost as memory initialization. Although init_on_free is rather costly, there are paranoid use-cases where in-memory data lifetime is desired to be minimized. There are various arguments for/against the realism of the associated threat models, but given that we'll need the infrastructre for MTE anyway, and there are people who want wipe-on-free behavior no matter what the performance cost, it seems reasonable to include it in this series. Signed-off-by: Alexander Potapenko To: Andrew Morton To: Christoph Lameter To: Kees Cook Cc: Masahiro Yamada Cc: Michal Hocko Cc: James Morris Cc: "Serge E. Hallyn" Cc: Nick Desaulniers Cc: Kostya Serebryany Cc: Dmitry Vyukov Cc: Sandeep Patil Cc: Laura Abbott Cc: Randy Dunlap Cc: Jann Horn Cc: Mark Rutland Cc: linux-mm@kvack.org Cc: linux-security-module@vger.kernel.org Cc: kernel-hardening@lists.openwall.com --- v2: - unconditionally initialize pages in kernel_init_free_pages() - comment from Randy Dunlap: drop 'default false' lines from Kconfig.hardening v3: - don't call kernel_init_free_pages() from memblock_free_pages() - adopted some Kees' comments for the patch description v4: - use NULL instead of 0 in slab_alloc_node() (found by kbuild test robot) - don't write to NULL object in slab_alloc_node() (found by Android testing) --- .../admin-guide/kernel-parameters.txt | 8 +++ drivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/mm.h | 22 +++++++ kernel/kexec_core.c | 2 +- mm/dmapool.c | 2 +- mm/page_alloc.c | 63 ++++++++++++++++--- mm/slab.c | 16 ++++- mm/slab.h | 16 +++++ mm/slob.c | 22 ++++++- mm/slub.c | 27 ++++++-- net/core/sock.c | 2 +- security/Kconfig.hardening | 14 +++++ 12 files changed, 175 insertions(+), 21 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 52e6fbb042cc..68fb6fa41cc1 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1673,6 +1673,14 @@ initrd= [BOOT] Specify the location of the initial ramdisk + init_on_alloc= [MM] Fill newly allocated pages and heap objects with + zeroes. + Format: 0 | 1 + Default set by CONFIG_INIT_ON_ALLOC_DEFAULT_ON. + init_on_free= [MM] Fill freed pages and heap objects with zeroes. + Format: 0 | 1 + Default set by CONFIG_INIT_ON_FREE_DEFAULT_ON. + init_pkru= [x86] Specify the default memory protection keys rights register contents for all processes. 0x55555554 by default (disallow access to all but pkey 0). Can diff --git a/drivers/infiniband/core/uverbs_ioctl.c b/drivers/infiniband/core/uverbs_ioctl.c index 829b0c6944d8..61758201d9b2 100644 --- a/drivers/infiniband/core/uverbs_ioctl.c +++ b/drivers/infiniband/core/uverbs_ioctl.c @@ -127,7 +127,7 @@ __malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size, res = (void *)pbundle->internal_buffer + pbundle->internal_used; pbundle->internal_used = ALIGN(new_used, sizeof(*pbundle->internal_buffer)); - if (flags & __GFP_ZERO) + if (want_init_on_alloc(flags)) memset(res, 0, size); return res; } diff --git a/include/linux/mm.h b/include/linux/mm.h index 0e8834ac32b7..7733a341c0c4 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2685,6 +2685,28 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { } #endif +#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE(init_on_alloc); +#else +DECLARE_STATIC_KEY_FALSE(init_on_alloc); +#endif +static inline bool want_init_on_alloc(gfp_t flags) +{ + if (static_branch_unlikely(&init_on_alloc)) + return true; + return flags & __GFP_ZERO; +} + +#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE(init_on_free); +#else +DECLARE_STATIC_KEY_FALSE(init_on_free); +#endif +static inline bool want_init_on_free(void) +{ + return static_branch_unlikely(&init_on_free); +} + extern bool _debug_pagealloc_enabled; static inline bool debug_pagealloc_enabled(void) diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c index fd5c95ff9251..2f75dd0d0d81 100644 --- a/kernel/kexec_core.c +++ b/kernel/kexec_core.c @@ -315,7 +315,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order) arch_kexec_post_alloc_pages(page_address(pages), count, gfp_mask); - if (gfp_mask & __GFP_ZERO) + if (want_init_on_alloc(gfp_mask)) for (i = 0; i < count; i++) clear_highpage(pages + i); } diff --git a/mm/dmapool.c b/mm/dmapool.c index 76a160083506..493d151067cb 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -381,7 +381,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags, #endif spin_unlock_irqrestore(&pool->lock, flags); - if (mem_flags & __GFP_ZERO) + if (want_init_on_alloc(mem_flags)) memset(retval, 0, pool->size); return retval; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3b13d3914176..14ded6620aa0 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -135,6 +135,48 @@ unsigned long totalcma_pages __read_mostly; int percpu_pagelist_fraction; gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK; +#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON +DEFINE_STATIC_KEY_TRUE(init_on_alloc); +#else +DEFINE_STATIC_KEY_FALSE(init_on_alloc); +#endif +#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON +DEFINE_STATIC_KEY_TRUE(init_on_free); +#else +DEFINE_STATIC_KEY_FALSE(init_on_free); +#endif + +static int __init early_init_on_alloc(char *buf) +{ + int ret; + bool bool_result; + + if (!buf) + return -EINVAL; + ret = kstrtobool(buf, &bool_result); + if (bool_result) + static_branch_enable(&init_on_alloc); + else + static_branch_disable(&init_on_alloc); + return ret; +} +early_param("init_on_alloc", early_init_on_alloc); + +static int __init early_init_on_free(char *buf) +{ + int ret; + bool bool_result; + + if (!buf) + return -EINVAL; + ret = kstrtobool(buf, &bool_result); + if (bool_result) + static_branch_enable(&init_on_free); + else + static_branch_disable(&init_on_free); + return ret; +} +early_param("init_on_free", early_init_on_free); /* * A cached value of the page's pageblock's migratetype, used when the page is @@ -1089,6 +1131,14 @@ static int free_tail_pages_check(struct page *head_page, struct page *page) return ret; } +static void kernel_init_free_pages(struct page *page, int numpages) +{ + int i; + + for (i = 0; i < numpages; i++) + clear_highpage(page + i); +} + static __always_inline bool free_pages_prepare(struct page *page, unsigned int order, bool check_free) { @@ -1141,6 +1191,8 @@ static __always_inline bool free_pages_prepare(struct page *page, } arch_free_page(page, order); kernel_poison_pages(page, 1 << order, 0); + if (want_init_on_free()) + kernel_init_free_pages(page, 1 << order); if (debug_pagealloc_enabled()) kernel_map_pages(page, 1 << order, 0); @@ -2019,8 +2071,8 @@ static inline int check_new_page(struct page *page) static inline bool free_pages_prezeroed(void) { - return IS_ENABLED(CONFIG_PAGE_POISONING_ZERO) && - page_poisoning_enabled(); + return (IS_ENABLED(CONFIG_PAGE_POISONING_ZERO) && + page_poisoning_enabled()) || want_init_on_free(); } #ifdef CONFIG_DEBUG_VM @@ -2074,13 +2126,10 @@ inline void post_alloc_hook(struct page *page, unsigned int order, static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags, unsigned int alloc_flags) { - int i; - post_alloc_hook(page, order, gfp_flags); - if (!free_pages_prezeroed() && (gfp_flags & __GFP_ZERO)) - for (i = 0; i < (1 << order); i++) - clear_highpage(page + i); + if (!free_pages_prezeroed() && want_init_on_alloc(gfp_flags)) + kernel_init_free_pages(page, 1 << order); if (order && (gfp_flags & __GFP_COMP)) prep_compound_page(page, order); diff --git a/mm/slab.c b/mm/slab.c index 2915d912e89a..d42eb11f8f50 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1853,6 +1853,14 @@ static bool set_objfreelist_slab_cache(struct kmem_cache *cachep, cachep->num = 0; + /* + * If slab auto-initialization on free is enabled, store the freelist + * off-slab, so that its contents don't end up in one of the allocated + * objects. + */ + if (unlikely(slab_want_init_on_free(cachep))) + return false; + if (cachep->ctor || flags & SLAB_TYPESAFE_BY_RCU) return false; @@ -3293,7 +3301,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, local_irq_restore(save_flags); ptr = cache_alloc_debugcheck_after(cachep, flags, ptr, caller); - if (unlikely(flags & __GFP_ZERO) && ptr) + if (unlikely(slab_want_init_on_alloc(flags, cachep)) && ptr) memset(ptr, 0, cachep->object_size); slab_post_alloc_hook(cachep, flags, 1, &ptr); @@ -3350,7 +3358,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) objp = cache_alloc_debugcheck_after(cachep, flags, objp, caller); prefetchw(objp); - if (unlikely(flags & __GFP_ZERO) && objp) + if (unlikely(slab_want_init_on_alloc(flags, cachep)) && objp) memset(objp, 0, cachep->object_size); slab_post_alloc_hook(cachep, flags, 1, &objp); @@ -3471,6 +3479,8 @@ void ___cache_free(struct kmem_cache *cachep, void *objp, struct array_cache *ac = cpu_cache_get(cachep); check_irq_off(); + if (unlikely(slab_want_init_on_free(cachep))) + memset(objp, 0, cachep->object_size); kmemleak_free_recursive(objp, cachep->flags); objp = cache_free_debugcheck(cachep, objp, caller); @@ -3558,7 +3568,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, cache_alloc_debugcheck_after_bulk(s, flags, size, p, _RET_IP_); /* Clear memory outside IRQ disabled section */ - if (unlikely(flags & __GFP_ZERO)) + if (unlikely(slab_want_init_on_alloc(flags, s))) for (i = 0; i < size; i++) memset(p[i], 0, s->object_size); diff --git a/mm/slab.h b/mm/slab.h index 43ac818b8592..24ae887359b8 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -524,4 +524,20 @@ static inline int cache_random_seq_create(struct kmem_cache *cachep, static inline void cache_random_seq_destroy(struct kmem_cache *cachep) { } #endif /* CONFIG_SLAB_FREELIST_RANDOM */ +static inline bool slab_want_init_on_alloc(gfp_t flags, struct kmem_cache *c) +{ + if (static_branch_unlikely(&init_on_alloc)) + return !(c->ctor); + else + return flags & __GFP_ZERO; +} + +static inline bool slab_want_init_on_free(struct kmem_cache *c) +{ + if (static_branch_unlikely(&init_on_free)) + return !(c->ctor); + else + return false; +} + #endif /* MM_SLAB_H */ diff --git a/mm/slob.c b/mm/slob.c index 84aefd9b91ee..1b565ee7f479 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -212,6 +212,19 @@ static void slob_free_pages(void *b, int order) free_pages((unsigned long)b, order); } +/* + * init_on_free=1 also implies initialization at allocation time. + * This is because newly allocated objects may contain freelist pointers + * somewhere in the middle. + */ +static inline bool slob_want_init_on_alloc(gfp_t flags, struct kmem_cache *c) +{ + if (static_branch_unlikely(&init_on_alloc) || + static_branch_unlikely(&init_on_free)) + return c ? (!c->ctor) : true; + return flags & __GFP_ZERO; +} + /* * slob_page_alloc() - Allocate a slob block within a given slob_page sp. * @sp: Page to look in. @@ -353,8 +366,6 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node) BUG_ON(!b); spin_unlock_irqrestore(&slob_lock, flags); } - if (unlikely(gfp & __GFP_ZERO)) - memset(b, 0, size); return b; } @@ -389,6 +400,9 @@ static void slob_free(void *block, int size) return; } + if (unlikely(want_init_on_free())) + memset(block, 0, size); + if (!slob_page_free(sp)) { /* This slob page is about to become partially free. Easy! */ sp->units = units; @@ -484,6 +498,8 @@ __do_kmalloc_node(size_t size, gfp_t gfp, int node, unsigned long caller) } kmemleak_alloc(ret, size, 1, gfp); + if (unlikely(slob_want_init_on_alloc(gfp, 0))) + memset(ret, 0, size); return ret; } @@ -582,6 +598,8 @@ static void *slob_alloc_node(struct kmem_cache *c, gfp_t flags, int node) WARN_ON_ONCE(flags & __GFP_ZERO); c->ctor(b); } + if (unlikely(slob_want_init_on_alloc(flags, c))) + memset(b, 0, c->size); kmemleak_alloc_recursive(b, c->size, 1, c->flags, flags); return b; diff --git a/mm/slub.c b/mm/slub.c index cd04dbd2b5d0..db8ed569a5cf 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1424,6 +1424,19 @@ static __always_inline bool slab_free_hook(struct kmem_cache *s, void *x) static inline bool slab_free_freelist_hook(struct kmem_cache *s, void **head, void **tail) { + + void *object; + void *next = *head; + void *old_tail = *tail ? *tail : *head; + + if (slab_want_init_on_free(s)) + do { + object = next; + next = get_freepointer(s, object); + memset(object, 0, s->size); + set_freepointer(s, object, next); + } while (object != old_tail); + /* * Compiler cannot detect this function can be removed if slab_free_hook() * evaluates to nothing. Thus, catch all relevant config debug options here. @@ -1433,9 +1446,7 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s, defined(CONFIG_DEBUG_OBJECTS_FREE) || \ defined(CONFIG_KASAN) - void *object; - void *next = *head; - void *old_tail = *tail ? *tail : *head; + next = *head; /* Head and tail of the reconstructed freelist */ *head = NULL; @@ -2741,8 +2752,14 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, prefetch_freepointer(s, next_object); stat(s, ALLOC_FASTPATH); } + /* + * If the object has been wiped upon free, make sure it's fully + * initialized by zeroing out freelist pointer. + */ + if (unlikely(slab_want_init_on_free(s)) && object) + *(void **)object = NULL; - if (unlikely(gfpflags & __GFP_ZERO) && object) + if (unlikely(slab_want_init_on_alloc(gfpflags, s)) && object) memset(object, 0, s->object_size); slab_post_alloc_hook(s, gfpflags, 1, &object); @@ -3163,7 +3180,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, local_irq_enable(); /* Clear memory outside IRQ disabled fastpath loop */ - if (unlikely(flags & __GFP_ZERO)) { + if (unlikely(slab_want_init_on_alloc(flags, s))) { int j; for (j = 0; j < i; j++) diff --git a/net/core/sock.c b/net/core/sock.c index 75b1c950b49f..9ceb90c875bc 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -1602,7 +1602,7 @@ static struct sock *sk_prot_alloc(struct proto *prot, gfp_t priority, sk = kmem_cache_alloc(slab, priority & ~__GFP_ZERO); if (!sk) return sk; - if (priority & __GFP_ZERO) + if (want_init_on_alloc(priority)) sk_prot_clear_nulls(sk, prot->obj_size); } else sk = kmalloc(prot->obj_size, priority); diff --git a/security/Kconfig.hardening b/security/Kconfig.hardening index 0a1d4ca314f4..87883e3e3c2a 100644 --- a/security/Kconfig.hardening +++ b/security/Kconfig.hardening @@ -159,6 +159,20 @@ config STACKLEAK_RUNTIME_DISABLE runtime to control kernel stack erasing for kernels built with CONFIG_GCC_PLUGIN_STACKLEAK. +config INIT_ON_ALLOC_DEFAULT_ON + bool "Set init_on_alloc=1 by default" + help + Enable init_on_alloc=1 by default, making the kernel initialize every + page and heap allocation with zeroes. + init_on_alloc can be overridden via command line. + +config INIT_ON_FREE_DEFAULT_ON + bool "Set init_on_free=1 by default" + help + Enable init_on_free=1 by default, making the kernel initialize freed + pages and slab memory with zeroes. + init_on_free can be overridden via command line. + endmenu endmenu From patchwork Thu May 23 14:08:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 10957753 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A4F8916C1 for ; Thu, 23 May 2019 14:09:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8CBE028562 for ; Thu, 23 May 2019 14:09:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 81531285FD; Thu, 23 May 2019 14:09:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D15E7285FB for ; Thu, 23 May 2019 14:09:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1E4EA6B000D; Thu, 23 May 2019 10:09:10 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 149F56B000E; Thu, 23 May 2019 10:09:10 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 03A6E6B0010; Thu, 23 May 2019 10:09:09 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by kanga.kvack.org (Postfix) with ESMTP id D4A4D6B000D for ; Thu, 23 May 2019 10:09:09 -0400 (EDT) Received: by mail-qk1-f198.google.com with SMTP id v2so4948437qkd.11 for ; Thu, 23 May 2019 07:09:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=NCAxoLMPldvKQMidzNTr8hyOEHq1jSHIcOt+CxAESVs=; b=qhnHCAtijSCNn26wV4yMPm3jkOLzRQ/9bsrbyHuf06HT54z7HHkh/VOzxfySsvb2/H nqLIRmoyJ2qP3dFPcAzBFAk3LmXbqSWdlw5KHCiz5Rr6RkOn0by2OAZKhp5R68Zvk/yz 3IpYc713rQkQvqevi5kq666SPjBdC6gDHc7lKNjwH+5BpEpdtkQ17W6IMv3BdmlFQcgs vCnoZnz3O1GqmrMp5MTGmNb0j/B6retRTEtbnXna7CenmlloL/aWt11hr1MlpcyCt9kr sTmd2rcS7FM0X55qqLEhnltIN0aOvSsHG7YDaPZtEayBwsysbsg485S0iXzuGJ/Qnvo8 O4aQ== X-Gm-Message-State: APjAAAX5nKPukam7qvNRcJoSgd/Lr15Xcr1QE+1e+AJ91cAOnLpUmQbT BtRWKOjUXrM3UBtLfqlzNGXBfMIfEYTjPynnauI236+kBuBdNn783HBUJ8azeyOIBJITyIksnni bNwpu9yK/Y1v8foXfVDxwstnlIpBjVtgnUt1LsKkuKi7i5wt2pF+yjXkZQNTbT2zkyQ== X-Received: by 2002:a0c:9679:: with SMTP id 54mr67094652qvy.168.1558620549474; Thu, 23 May 2019 07:09:09 -0700 (PDT) X-Received: by 2002:a0c:9679:: with SMTP id 54mr67094570qvy.168.1558620548730; Thu, 23 May 2019 07:09:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558620548; cv=none; d=google.com; s=arc-20160816; b=UK5jtB52IDE86XYXYGFdsX6LHWqXep7hq0vBw3dRkLA/bOV02FlcKu7F6xyxTWgxtt hit76HhCtWQHcDAHWRN250CtZWADfqacJGoeHFRKDVBY8LETSeptrz9q6vAj1PUlLyei 020MqSFaDWowrIK8ipibXhdsmmfFof6UFYOcrlwZMd30yrv6eO3GmpRebCLWd6EVTWze PdHc2F2K93Lz1PtNP6LBzRPQ7rWADo1yryOC+TdLvLhBV0M8hzArk056kJ4gCQkMpx+c FNeuH0MjtM8LnukXPjZRxZcUWPhwvTt/NSyRm/nyKq8JVq1dg0ofNsDzIf6afQPhXwsr MEiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=NCAxoLMPldvKQMidzNTr8hyOEHq1jSHIcOt+CxAESVs=; b=va16Dh+qz5TNIt9eksLRq2DgLelwfrVMHEI3N4xL6liuMfX8Z8kWVxTniqb1fVUHxM O0NRSLKlh4q3V3VeClbNmXJtAPwPgk7PlhWhBcyhknK1lJ6bgLNBg4a6rILUecweb/yL np3gETsiErGHjt0V6WXqQNbYSWK35+EmfC9q6ljGsnuhTzbxehYLOvn6quaOJ0nQc2Ok PJASk/eAJiQ+GGOro6W9ieAFEFe9sSTvulIccBAKDetBGtK9YsHUE0o06GXmcwRLXm+0 25/dic3CzlMM07sb9/8dfnkEeFKwnnB+G/QTC3nswh68FYZe3ySjkHrDTPaJIf214HNf 2Uvg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=S6wSpgNj; spf=pass (google.com: domain of 3hknmxaykcf8difabodlldib.9ljifkru-jjhs79h.lod@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3hKnmXAYKCF8DIFABODLLDIB.9LJIFKRU-JJHS79H.LOD@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id l18sor6603346qvq.27.2019.05.23.07.09.08 for (Google Transport Security); Thu, 23 May 2019 07:09:08 -0700 (PDT) Received-SPF: pass (google.com: domain of 3hknmxaykcf8difabodlldib.9ljifkru-jjhs79h.lod@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=S6wSpgNj; spf=pass (google.com: domain of 3hknmxaykcf8difabodlldib.9ljifkru-jjhs79h.lod@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3hKnmXAYKCF8DIFABODLLDIB.9LJIFKRU-JJHS79H.LOD@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=NCAxoLMPldvKQMidzNTr8hyOEHq1jSHIcOt+CxAESVs=; b=S6wSpgNjmbCtFUus2QjJRqTyIBPMAaNkKeh9nwyrwXr2Pzte3ZJmjZTd35nf8Jiwp8 G9GAkzUCb8+zWUG4YDPxD0Nv5k03HZgq8/Ta/e/7PgHhNu8mfR0SwqT1rxSGDK8qDO6C hD/3qvtk57/g0uj0dMoh9/zw8YXxL4ZepOVtxBed7NZvBb59J/onvCvzUkf7SQfdhutw i5udRpMuQdrbEORaIzsRsj3j8E5MX0ytZg/ufaaKzcaTASlDtmnjOO5IdJfOHR00/Cjw 0b+LHRaugR4iGA1B8XlmuIQqf18DU/ewjMiyqblDsLmwtdShh7TI8ujY/H31JUehVwT4 zbaQ== X-Google-Smtp-Source: APXvYqxq75VICyIA1vT5UP2e06N1oVHwb0PX7bkc+F7+9NiaaTyuWWOp9etgrukRuGPGuDjKjiOg4kbrotM= X-Received: by 2002:a0c:d2ae:: with SMTP id q43mr68355733qvh.96.1558620548425; Thu, 23 May 2019 07:09:08 -0700 (PDT) Date: Thu, 23 May 2019 16:08:43 +0200 In-Reply-To: <20190523140844.132150-1-glider@google.com> Message-Id: <20190523140844.132150-3-glider@google.com> Mime-Version: 1.0 References: <20190523140844.132150-1-glider@google.com> X-Mailer: git-send-email 2.21.0.1020.gf2820cf01a-goog Subject: [PATCH v4 2/3] mm: init: report memory auto-initialization features at boot time From: Alexander Potapenko To: akpm@linux-foundation.org, cl@linux.com, keescook@chromium.org Cc: kernel-hardening@lists.openwall.com, linux-mm@kvack.org, linux-security-module@vger.kernel.org, Dmitry Vyukov , James Morris , Jann Horn , Kostya Serebryany , Laura Abbott , Mark Rutland , Masahiro Yamada , Matthew Wilcox , Nick Desaulniers , Randy Dunlap , Sandeep Patil , "Serge E. Hallyn" , Souptick Joarder X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Print the currently enabled stack and heap initialization modes. The possible options for stack are: - "all" for CONFIG_INIT_STACK_ALL; - "byref_all" for CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL; - "byref" for CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF; - "__user" for CONFIG_GCC_PLUGIN_STRUCTLEAK_USER; - "off" otherwise. Depending on the values of init_on_alloc and init_on_free boottime options we also report "heap alloc" and "heap free" as "on"/"off". In the init_on_free mode initializing pages at boot time may take some time, so print a notice about that as well. Signed-off-by: Alexander Potapenko Suggested-by: Kees Cook To: Andrew Morton To: Christoph Lameter Cc: Dmitry Vyukov Cc: James Morris Cc: Jann Horn Cc: Kostya Serebryany Cc: Laura Abbott Cc: Mark Rutland Cc: Masahiro Yamada Cc: Matthew Wilcox Cc: Nick Desaulniers Cc: Randy Dunlap Cc: Sandeep Patil Cc: "Serge E. Hallyn" Cc: Souptick Joarder Cc: kernel-hardening@lists.openwall.com Cc: linux-mm@kvack.org Cc: linux-security-module@vger.kernel.org --- init/main.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/init/main.c b/init/main.c index 5a2c69b4d7b3..90f721c58e61 100644 --- a/init/main.c +++ b/init/main.c @@ -519,6 +519,29 @@ static inline void initcall_debug_enable(void) } #endif +/* Report memory auto-initialization states for this boot. */ +void __init report_meminit(void) +{ + const char *stack; + + if (IS_ENABLED(CONFIG_INIT_STACK_ALL)) + stack = "all"; + else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL)) + stack = "byref_all"; + else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF)) + stack = "byref"; + else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_USER)) + stack = "__user"; + else + stack = "off"; + + pr_info("mem auto-init: stack:%s, heap alloc:%s, heap free:%s\n", + stack, want_init_on_alloc(GFP_KERNEL) ? "on" : "off", + want_init_on_free() ? "on" : "off"); + if (want_init_on_free()) + pr_info("Clearing system memory may take some time...\n"); +} + /* * Set up kernel memory allocators */ @@ -529,6 +552,7 @@ static void __init mm_init(void) * bigger than MAX_ORDER unless SPARSEMEM. */ page_ext_init_flatmem(); + report_meminit(); mem_init(); kmem_cache_init(); pgtable_init(); From patchwork Thu May 23 14:08:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 10957757 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 03FA21395 for ; Thu, 23 May 2019 14:09:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DFD4B26E76 for ; Thu, 23 May 2019 14:09:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CB08528620; Thu, 23 May 2019 14:09:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 14AD8285EB for ; Thu, 23 May 2019 14:09:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E68ED6B000E; Thu, 23 May 2019 10:09:12 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E1AF66B0010; Thu, 23 May 2019 10:09:12 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D09226B0266; Thu, 23 May 2019 10:09:12 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by kanga.kvack.org (Postfix) with ESMTP id B20E16B000E for ; Thu, 23 May 2019 10:09:12 -0400 (EDT) Received: by mail-qt1-f198.google.com with SMTP id n21so5423104qtp.15 for ; Thu, 23 May 2019 07:09:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=GrPd0Ehzkwgs8flG0McuJvkTnr/DIjfrIhDeOSvXnoY=; b=R32LxICex7oNv1qnM9cB4rhn+mmT9LSXEBUVQ2zWQG5WRfndBWlFMUULK4WYAnffIn 2764Ug5F7HUqH66G5wSMc0q46bJAHWO84hdfWI65yL03a5VX3HldeCqQS5YQI9VA3yhW c09FeZG0EHoY7jzOrneCAgVbrQNbN6UaIjFygu5CEJJVQgh8+VKjrIYwpb/VzJ3SZizk 5Y7K+zUpiz0kTytAkXIGDfF6sJEKZG3agdcr8AxSWtfURrDwVsLjoVaBvMbJKixGwkaU QhjDHu7NUu76fReBo+awZwmziPXhZN8s3zcSFiSpf3rd4JIebX76QdYIt3DWFdwOxfrK I9eg== X-Gm-Message-State: APjAAAUdmRegYSZtZpf6upGbLn72d9QyNxO5rWVdcbPf371cwqQ71QdM acyFQ6hvHRNb12+gghJMlxAQ+4rA7NOg0OWcV4pRzKmsz6Z1GEZs68VFDtu8YZ0D2JSIMUodA4a A9u/Q7Gg/wvbDIpCsPjf7kvToWL2QXM69KMTc0uBS8FCxXJIjjTYG4HZjFnUXmPR3BA== X-Received: by 2002:ac8:5246:: with SMTP id y6mr79010251qtn.242.1558620552488; Thu, 23 May 2019 07:09:12 -0700 (PDT) X-Received: by 2002:ac8:5246:: with SMTP id y6mr79010173qtn.242.1558620551740; Thu, 23 May 2019 07:09:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558620551; cv=none; d=google.com; s=arc-20160816; b=I2oHK7FAdoNBe9fSJTSePHw/QA7vcuVHP5XxxXedRJNFxGbN579ORsmZEasFrfjPSd nqHD/Q65MxzNNIxLFPkP7KCddj/Uxv1x5W2x7ltJhCi4gWSakrmqY0DbXpPTBdBs3ewS MEzu39Eg2crxIpfvn7ASOHNInAyl2mlVhKPd2OsVXK5nvn//JdQ8wfk9QuSCw6IRwnYX 2bdNIyE1QJRyo3wimKuF06QcZtwgO9Krleq7tRbJFIUaYVjXJlufcxeI55clhA70ilW8 3NZMHXAqPeFRpyil3/BPPa9ExiNSkvQHuz0QB2LSmaqjX9RDGDEOjLL4VOG260Vb0O4o lVvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=GrPd0Ehzkwgs8flG0McuJvkTnr/DIjfrIhDeOSvXnoY=; b=LaZ38kuqXTU3gFw1SxxI0gMkiR0n+yrkrvTjC/SLU2/Qeb1e4GbiN3IPokr9PAd2kM CGmkV3Xcy4fDppsPvvfZNhmcNGdTb+TD4uEv/6k9CpdceNpikfWV9X4BxOD4D1fjePH/ VRRy8RbXjIcK9TW3URjoJvs3a4j62GEkpeqEaigfs7PzWgRE7EC8rHZsOXX4x7GGMaAZ VJ9f7TQ3rq5P+giSlRlldFqn03ZUIF9QDnphon1qGKvmnzjuQXNcmzKS1mTaOrSHRJsd Muq4BROD+SrB8Sn7Upqr2rbSLHV2NvpF472u2WZopytDf3idNMxUQz1muglXMM6twrXO SLzQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=HxdHAURk; spf=pass (google.com: domain of 3h6nmxaykcgiglidergoogle.comlinux-mmkvack.org@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3h6nmXAYKCGIGLIDERGOOGLE.COMLINUX-MMKVACK.ORG@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id p30sor21696609qvc.71.2019.05.23.07.09.11 for (Google Transport Security); Thu, 23 May 2019 07:09:11 -0700 (PDT) Received-SPF: pass (google.com: domain of 3h6nmxaykcgiglidergoogle.comlinux-mmkvack.org@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=HxdHAURk; spf=pass (google.com: domain of 3h6nmxaykcgiglidergoogle.comlinux-mmkvack.org@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3h6nmXAYKCGIGLIDERGOOGLE.COMLINUX-MMKVACK.ORG@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=GrPd0Ehzkwgs8flG0McuJvkTnr/DIjfrIhDeOSvXnoY=; b=HxdHAURkSl4n27A0yrQIRU1ITTYPBWK1In1KGlmB43xzTe8tfXSa/dWul7q4I/3UzU zAXTkoEFSDGlQlwPxRKEPIsS4gx1GWROBjhSdaDe+osjq2IOHCxE8AK8vVLls9jsiCni Ah0o+mZ5lkt/KWNdF1YGyCq0Tn9kR2BizpDzBrgZFEG+PKOtey/cz+kMH+/+NJpItHf2 vzgiCe1wjnIMjaRo26wo/zwzLMMxV+51y6MNX1nzooG5KC1QggYgSZIVxhYV85G0/IFG eyM/9zRs+5DIGCsdt23RjOyg9gSR0BPyH9rKZ0GdAzhvalKjcPMUFbug9urWslsi51qX M31w== X-Google-Smtp-Source: APXvYqwTc2z+TMaFN0UPPlnm9rius4RP7zoq7ED5Y5T2qgqHTis5h9UUibnhY+/7LWm7HuiK8P97MvS3qoo= X-Received: by 2002:a0c:95d5:: with SMTP id t21mr70859708qvt.215.1558620551451; Thu, 23 May 2019 07:09:11 -0700 (PDT) Date: Thu, 23 May 2019 16:08:44 +0200 In-Reply-To: <20190523140844.132150-1-glider@google.com> Message-Id: <20190523140844.132150-4-glider@google.com> Mime-Version: 1.0 References: <20190523140844.132150-1-glider@google.com> X-Mailer: git-send-email 2.21.0.1020.gf2820cf01a-goog Subject: [PATCH v4 3/3] lib: introduce test_meminit module From: Alexander Potapenko To: akpm@linux-foundation.org, cl@linux.com, keescook@chromium.org Cc: kernel-hardening@lists.openwall.com, linux-mm@kvack.org, linux-security-module@vger.kernel.org, Nick Desaulniers , Kostya Serebryany , Dmitry Vyukov , Sandeep Patil , Laura Abbott , Jann Horn X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add tests for heap and pagealloc initialization. These can be used to check init_on_alloc and init_on_free implementations as well as other approaches to initialization. Expected test output in the case the kernel provides heap initialization (e.g. when running with either init_on_alloc=1 or init_on_free=1): test_meminit: all 10 tests in test_pages passed test_meminit: all 40 tests in test_kvmalloc passed test_meminit: all 20 tests in test_kmemcache passed test_meminit: all 70 tests passed! Signed-off-by: Alexander Potapenko To: Kees Cook To: Andrew Morton To: Christoph Lameter Cc: Nick Desaulniers Cc: Kostya Serebryany Cc: Dmitry Vyukov Cc: Sandeep Patil Cc: Laura Abbott Cc: Jann Horn Cc: linux-mm@kvack.org Cc: linux-security-module@vger.kernel.org Cc: kernel-hardening@lists.openwall.com --- v3: - added example test output to the description - fixed a missing include spotted by kbuild test robot - added a missing MODULE_LICENSE - call do_kmem_cache_size() with size >= sizeof(void*) to unbreak debug builds --- lib/Kconfig.debug | 8 ++ lib/Makefile | 1 + lib/test_meminit.c | 208 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 217 insertions(+) create mode 100644 lib/test_meminit.c diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index fdfa173651eb..036e8ef03831 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2043,6 +2043,14 @@ config TEST_STACKINIT If unsure, say N. +config TEST_MEMINIT + tristate "Test level of heap/page initialization" + help + Test if the kernel is zero-initializing heap and page allocations. + This can be useful to test init_on_alloc and init_on_free features. + + If unsure, say N. + endif # RUNTIME_TESTING_MENU config MEMTEST diff --git a/lib/Makefile b/lib/Makefile index fb7697031a79..05980c802500 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -91,6 +91,7 @@ obj-$(CONFIG_TEST_DEBUG_VIRTUAL) += test_debug_virtual.o obj-$(CONFIG_TEST_MEMCAT_P) += test_memcat_p.o obj-$(CONFIG_TEST_OBJAGG) += test_objagg.o obj-$(CONFIG_TEST_STACKINIT) += test_stackinit.o +obj-$(CONFIG_TEST_MEMINIT) += test_meminit.o obj-$(CONFIG_TEST_LIVEPATCH) += livepatch/ diff --git a/lib/test_meminit.c b/lib/test_meminit.c new file mode 100644 index 000000000000..d46e2b8c8e8e --- /dev/null +++ b/lib/test_meminit.c @@ -0,0 +1,208 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Test cases for SL[AOU]B/page initialization at alloc/free time. + */ +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#include +#include +#include +#include +#include +#include +#include + +#define GARBAGE_INT (0x09A7BA9E) +#define GARBAGE_BYTE (0x9E) + +#define REPORT_FAILURES_IN_FN() \ + do { \ + if (failures) \ + pr_info("%s failed %d out of %d times\n", \ + __func__, failures, num_tests); \ + else \ + pr_info("all %d tests in %s passed\n", \ + num_tests, __func__); \ + } while (0) + +/* Calculate the number of uninitialized bytes in the buffer. */ +static int count_nonzero_bytes(void *ptr, size_t size) +{ + int i, ret = 0; + unsigned char *p = (unsigned char *)ptr; + + for (i = 0; i < size; i++) + if (p[i]) + ret++; + return ret; +} + +static void fill_with_garbage(void *ptr, size_t size) +{ + unsigned int *p = (unsigned int *)ptr; + int i = 0; + + while (size >= sizeof(*p)) { + p[i] = GARBAGE_INT; + i++; + size -= sizeof(*p); + } + if (size) + memset(&p[i], GARBAGE_BYTE, size); +} + +static int __init do_alloc_pages_order(int order, int *total_failures) +{ + struct page *page; + void *buf; + size_t size = PAGE_SIZE << order; + + page = alloc_pages(GFP_KERNEL, order); + buf = page_address(page); + fill_with_garbage(buf, size); + __free_pages(page, order); + + page = alloc_pages(GFP_KERNEL, order); + buf = page_address(page); + if (count_nonzero_bytes(buf, size)) + (*total_failures)++; + fill_with_garbage(buf, size); + __free_pages(page, order); + return 1; +} + +static int __init test_pages(int *total_failures) +{ + int failures = 0, num_tests = 0; + int i; + + for (i = 0; i < 10; i++) + num_tests += do_alloc_pages_order(i, &failures); + + REPORT_FAILURES_IN_FN(); + *total_failures += failures; + return num_tests; +} + +static int __init do_kmalloc_size(size_t size, int *total_failures) +{ + void *buf; + + buf = kmalloc(size, GFP_KERNEL); + fill_with_garbage(buf, size); + kfree(buf); + + buf = kmalloc(size, GFP_KERNEL); + if (count_nonzero_bytes(buf, size)) + (*total_failures)++; + fill_with_garbage(buf, size); + kfree(buf); + return 1; +} + +static int __init do_vmalloc_size(size_t size, int *total_failures) +{ + void *buf; + + buf = vmalloc(size); + fill_with_garbage(buf, size); + vfree(buf); + + buf = vmalloc(size); + if (count_nonzero_bytes(buf, size)) + (*total_failures)++; + fill_with_garbage(buf, size); + vfree(buf); + return 1; +} + +static int __init test_kvmalloc(int *total_failures) +{ + int failures = 0, num_tests = 0; + int i, size; + + for (i = 0; i < 20; i++) { + size = 1 << i; + num_tests += do_kmalloc_size(size, &failures); + num_tests += do_vmalloc_size(size, &failures); + } + + REPORT_FAILURES_IN_FN(); + *total_failures += failures; + return num_tests; +} + +#define CTOR_BYTES 4 +/* Initialize the first 4 bytes of the object. */ +void some_ctor(void *obj) +{ + memset(obj, 'A', CTOR_BYTES); +} + +static int __init do_kmem_cache_size(size_t size, bool want_ctor, + int *total_failures) +{ + struct kmem_cache *c; + void *buf; + int iter, bytes = 0; + int fail = 0; + + c = kmem_cache_create("test_cache", size, 1, 0, + want_ctor ? some_ctor : NULL); + for (iter = 0; iter < 10; iter++) { + buf = kmem_cache_alloc(c, GFP_KERNEL); + if (!want_ctor || iter == 0) + bytes = count_nonzero_bytes(buf, size); + if (want_ctor) { + /* + * Newly initialized memory must be initialized using + * the constructor. + */ + if (iter == 0 && bytes < CTOR_BYTES) + fail = 1; + } else { + if (bytes) + fail = 1; + } + fill_with_garbage(buf, size); + kmem_cache_free(c, buf); + } + kmem_cache_destroy(c); + + *total_failures += fail; + return 1; +} + +static int __init test_kmemcache(int *total_failures) +{ + int failures = 0, num_tests = 0; + int i, size; + + for (i = 0; i < 10; i++) { + size = 8 << i; + num_tests += do_kmem_cache_size(size, false, &failures); + num_tests += do_kmem_cache_size(size, true, &failures); + } + REPORT_FAILURES_IN_FN(); + *total_failures += failures; + return num_tests; +} + +static int __init test_meminit_init(void) +{ + int failures = 0, num_tests = 0; + + num_tests += test_pages(&failures); + num_tests += test_kvmalloc(&failures); + num_tests += test_kmemcache(&failures); + + if (failures == 0) + pr_info("all %d tests passed!\n", num_tests); + else + pr_info("failures: %d out of %d\n", failures, num_tests); + + return failures ? -EINVAL : 0; +} +module_init(test_meminit_init); + +MODULE_LICENSE("GPL");