From patchwork Wed May 8 15:37:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 10936107 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1A483912 for ; Wed, 8 May 2019 15:38:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0A83820881 for ; Wed, 8 May 2019 15:38:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F0D7426246; Wed, 8 May 2019 15:38:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A5A4428390 for ; Wed, 8 May 2019 15:38:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A0D26B02BD; Wed, 8 May 2019 11:38:15 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 653266B02C0; Wed, 8 May 2019 11:38:15 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 541176B02C1; Wed, 8 May 2019 11:38:15 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-oi1-f198.google.com (mail-oi1-f198.google.com [209.85.167.198]) by kanga.kvack.org (Postfix) with ESMTP id 223416B02BD for ; Wed, 8 May 2019 11:38:15 -0400 (EDT) Received: by mail-oi1-f198.google.com with SMTP id w13so3711280oih.22 for ; Wed, 08 May 2019 08:38:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=T6bgGq2KcJ4pPFly2xX3mqEDKu0BtnCpkaNCQLXnqH4=; b=RC4m7d0zbLED2G1LaxHwlAFeHDCd6NsnF0ImyFaghfC8/3VhmFg5MIFQVQtcyMJwEP w2ps2qrMMS0NjbgiHhU63LWN5f5YJ8OxPDUo61xjA61uglRXJQzrVG+BaXckCL9vgeFR aR1ZDcSowaYQSmYvOtIpSJ+8GK5KE7cOH/aS5Q9KdZLiU44LNt7SVx9D9vb5onGxDtZE Uago7Cb8ICWNCDsyr5EH3elJ4GrkOoW2wfkY4p5sGobPNhatZJ9izAuJBRzXvuefz/2b Ho0uwDjef6rpLfg9lxHkp4a09Kax9ln4HycmvIzVX0XcocUVPKwn3wpf0YwM74dZ5g0o mUqg== X-Gm-Message-State: APjAAAVVKK6pmaRsUOU3o/UFwu1xvKH6+HfEtzoiDsGrSKbHpPwuQxEK p6KF+ux7Y+8epePpQv2nOvLVkt9/0QZBlg4xzcGmoXlhJEEtseA5v94UWJw0V458gsc9OcDnWvr YJtpEv9TvvV8bkSeTqFQSw9ONcwMemRL2ia5aRXGpTGCm42f/EmwyIImcBn5lgRCJmA== X-Received: by 2002:aca:4455:: with SMTP id r82mr2488232oia.165.1557329894715; Wed, 08 May 2019 08:38:14 -0700 (PDT) X-Received: by 2002:aca:4455:: with SMTP id r82mr2488175oia.165.1557329893532; Wed, 08 May 2019 08:38:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557329893; cv=none; d=google.com; s=arc-20160816; b=eTdXYKbzKWHaACDqiVmwbzyBMopBxTv46WyuecxLFRhKzNQVbchmUO/lE+Hm1Ladm/ Aqki9BhC6l3+LWM6AIKkxACEk2lBa9XaQUrCnnwmTAH2guZ5Ot6W1mYmcPTYSvk3JKT4 xcyoSrtIpgqlnpbAIGx2LLL2mqGM3U0nhiz2jxKimOOZvNgNd7HVntVwj9QLBhKYDpOh M1UbKqbSZ0dD9dzbjn823WpEo1eQMUkcRx0XC1TQr346F4tzaL3ZXUI+Bz9ixsu44qYb BbsiVa5kOcWw9UhrF5GTysIPECorXrX4hyFL+EdtWg9W5orgCb8abxiXIjDelpTqld0d 3DgA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=T6bgGq2KcJ4pPFly2xX3mqEDKu0BtnCpkaNCQLXnqH4=; b=iyMgW27yjDmGJnd3RqJK8cgty4YLLHUVRt4NQSlqPY2LKyCRnAjr0+lh2h+HggzQIf 4wHHSOLP+pOSLU41q7xrtDLyRKnAebblsLpvvhdzynFjN27JFRH0X6fWjO3AelOfuah8 tzMcabePOp+b+JK/7SUC2jz/Aiy7Q1pYmoK6zqE1HUNyt/032Cv95nvvWsDWcaBsLMtz UGFTAVAKjT1oF0l5ygQL1aChZf+mDX/Z6e7BZ9XzeuijU6sSF8/cXpZJvLDkkhBv1ggH oMLcVwZe2wssCnSPt4nfsTM/FrT3HmJI3ALaXkBB2XyPiE0K6QNnlBlZMu8X9hvTYKvV oIZg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=wH2CxJQ0; spf=pass (google.com: domain of 35ffsxaykca4uzwrs5u22uzs.q20zw18b-00y9oqy.25u@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=35ffSXAYKCA4uzwrs5u22uzs.q20zw18B-00y9oqy.25u@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id b68sor7875886otc.167.2019.05.08.08.38.13 for (Google Transport Security); Wed, 08 May 2019 08:38:13 -0700 (PDT) Received-SPF: pass (google.com: domain of 35ffsxaykca4uzwrs5u22uzs.q20zw18b-00y9oqy.25u@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=wH2CxJQ0; spf=pass (google.com: domain of 35ffsxaykca4uzwrs5u22uzs.q20zw18b-00y9oqy.25u@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=35ffSXAYKCA4uzwrs5u22uzs.q20zw18B-00y9oqy.25u@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=T6bgGq2KcJ4pPFly2xX3mqEDKu0BtnCpkaNCQLXnqH4=; b=wH2CxJQ0LJBXY1TvGY3OcOpyfwtylgUuk2+9DyjsTA+ma+0tYYjwIBVZuz7/E3XME7 3piZenwMsFocc9ZSXKEiBVamiIX11Y46IZ4lX1HOfexWkW3t4aQcCoLZfZaQjyYM88fh 2Hech/znC431NNtHcp60xHhbOh+4yL9JdiES2f5pPtPHkwKcOOhoVLN8hwqY9CWexBhI o9s+tpG1bSCDvnqirQ5d9OkbiGSKFnYU1MzrVyacUOkblJAP6ZGhuQy/J0i8vX/of+Er BgsmArWRXxZRJ7LhHJZ2U3YTjM9QFdnW0UnrlMZnNqQKywgHwQDpVJ+qh0PC37EF13c7 adfw== X-Google-Smtp-Source: APXvYqwR2q6ouUYtuF+R5DXO7FsieffV/w4i42wkW9TNYoHHZPE0YWZDP1jRRK5WG09x2hALr9NWFX8Vpg4= X-Received: by 2002:a9d:37ca:: with SMTP id x68mr7031896otb.347.1557329893204; Wed, 08 May 2019 08:38:13 -0700 (PDT) Date: Wed, 8 May 2019 17:37:33 +0200 In-Reply-To: <20190508153736.256401-1-glider@google.com> Message-Id: <20190508153736.256401-2-glider@google.com> Mime-Version: 1.0 References: <20190508153736.256401-1-glider@google.com> X-Mailer: git-send-email 2.21.0.1020.gf2820cf01a-goog Subject: [PATCH 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options From: Alexander Potapenko To: akpm@linux-foundation.org, cl@linux.com, keescook@chromium.org, labbott@redhat.com Cc: linux-mm@kvack.org, linux-security-module@vger.kernel.org, kernel-hardening@lists.openwall.com, yamada.masahiro@socionext.com, jmorris@namei.org, serge@hallyn.com, ndesaulniers@google.com, kcc@google.com, dvyukov@google.com, sspatil@android.com, rdunlap@infradead.org, jannh@google.com, mark.rutland@arm.com X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic. init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed. init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses. Both init_on_alloc=1 and init_on_free=1 guarantee that the allocator returns zeroed memory. The only exception is slab caches with constructors. Those are never zero-initialized to preserve their semantics. For SLOB allocator init_on_free=1 also implies init_on_alloc=1 behavior, i.e. objects are zeroed at both allocation and deallocation time. This is done because SLOB may otherwise return multiple freelist pointers in the allocated object. For SLAB and SLUB enabling either init_on_alloc or init_on_free leads to one-time initialization of the object. Both init_on_alloc and init_on_free default to zero, but those defaults can be overridden with CONFIG_INIT_ON_ALLOC_DEFAULT_ON and CONFIG_INIT_ON_FREE_DEFAULT_ON. Slowdown for the new features compared to init_on_free=0, init_on_alloc=0: hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%) Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%) Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%) Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%) The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error. Signed-off-by: Alexander Potapenko Cc: Andrew Morton Cc: Christoph Lameter Cc: Masahiro Yamada Cc: James Morris Cc: "Serge E. Hallyn" Cc: Nick Desaulniers Cc: Kostya Serebryany Cc: Dmitry Vyukov Cc: Kees Cook Cc: Sandeep Patil Cc: Laura Abbott Cc: Randy Dunlap Cc: Jann Horn Cc: Mark Rutland Cc: linux-mm@kvack.org Cc: linux-security-module@vger.kernel.org Cc: kernel-hardening@lists.openwall.com --- .../admin-guide/kernel-parameters.txt | 8 +++ drivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/mm.h | 22 +++++++ kernel/kexec_core.c | 2 +- mm/dmapool.c | 2 +- mm/page_alloc.c | 62 +++++++++++++++++-- mm/slab.c | 16 ++++- mm/slab.h | 16 +++++ mm/slob.c | 22 ++++++- mm/slub.c | 27 ++++++-- net/core/sock.c | 2 +- security/Kconfig.hardening | 16 +++++ 12 files changed, 179 insertions(+), 18 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 2b8ee90bb644..be1b66685784 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1671,6 +1671,14 @@ initrd= [BOOT] Specify the location of the initial ramdisk + init_on_alloc= [MM] Fill newly allocated pages and heap objects with + zeroes. + Format: 0 | 1 + Default set by CONFIG_INIT_ON_ALLOC_DEFAULT_ON. + init_on_free= [MM] Fill freed pages and heap objects with zeroes. + Format: 0 | 1 + Default set by CONFIG_INIT_ON_FREE_DEFAULT_ON. + init_pkru= [x86] Specify the default memory protection keys rights register contents for all processes. 0x55555554 by default (disallow access to all but pkey 0). Can diff --git a/drivers/infiniband/core/uverbs_ioctl.c b/drivers/infiniband/core/uverbs_ioctl.c index e1379949e663..c03c92cdd1a2 100644 --- a/drivers/infiniband/core/uverbs_ioctl.c +++ b/drivers/infiniband/core/uverbs_ioctl.c @@ -127,7 +127,7 @@ __malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size, res = (void *)pbundle->internal_buffer + pbundle->internal_used; pbundle->internal_used = ALIGN(new_used, sizeof(*pbundle->internal_buffer)); - if (flags & __GFP_ZERO) + if (want_init_on_alloc(flags)) memset(res, 0, size); return res; } diff --git a/include/linux/mm.h b/include/linux/mm.h index 6b10c21630f5..ee1a1092679c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2610,6 +2610,28 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { } #endif +#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE(init_on_alloc); +#else +DECLARE_STATIC_KEY_FALSE(init_on_alloc); +#endif +static inline bool want_init_on_alloc(gfp_t flags) +{ + if (static_branch_unlikely(&init_on_alloc)) + return true; + return flags & __GFP_ZERO; +} + +#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE(init_on_free); +#else +DECLARE_STATIC_KEY_FALSE(init_on_free); +#endif +static inline bool want_init_on_free(void) +{ + return static_branch_unlikely(&init_on_free); +} + #ifdef CONFIG_DEBUG_PAGEALLOC extern bool _debug_pagealloc_enabled; extern void __kernel_map_pages(struct page *page, int numpages, int enable); diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c index d7140447be75..f19d1a91190b 100644 --- a/kernel/kexec_core.c +++ b/kernel/kexec_core.c @@ -315,7 +315,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order) arch_kexec_post_alloc_pages(page_address(pages), count, gfp_mask); - if (gfp_mask & __GFP_ZERO) + if (want_init_on_alloc(gfp_mask)) for (i = 0; i < count; i++) clear_highpage(pages + i); } diff --git a/mm/dmapool.c b/mm/dmapool.c index 76a160083506..493d151067cb 100644 --- a/mm/dmapool.c +++ b/mm/dmapool.c @@ -381,7 +381,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags, #endif spin_unlock_irqrestore(&pool->lock, flags); - if (mem_flags & __GFP_ZERO) + if (want_init_on_alloc(mem_flags)) memset(retval, 0, pool->size); return retval; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c02cff1ed56e..d8b5bf9da08a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -133,6 +133,48 @@ unsigned long totalcma_pages __read_mostly; int percpu_pagelist_fraction; gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK; +#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON +DEFINE_STATIC_KEY_TRUE(init_on_alloc); +#else +DEFINE_STATIC_KEY_FALSE(init_on_alloc); +#endif +#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON +DEFINE_STATIC_KEY_TRUE(init_on_free); +#else +DEFINE_STATIC_KEY_FALSE(init_on_free); +#endif + +static int __init early_init_on_alloc(char *buf) +{ + int ret; + bool bool_result; + + if (!buf) + return -EINVAL; + ret = kstrtobool(buf, &bool_result); + if (bool_result) + static_branch_enable(&init_on_alloc); + else + static_branch_disable(&init_on_alloc); + return ret; +} +early_param("init_on_alloc", early_init_on_alloc); + +static int __init early_init_on_free(char *buf) +{ + int ret; + bool bool_result; + + if (!buf) + return -EINVAL; + ret = kstrtobool(buf, &bool_result); + if (bool_result) + static_branch_enable(&init_on_free); + else + static_branch_disable(&init_on_free); + return ret; +} +early_param("init_on_free", early_init_on_free); /* * A cached value of the page's pageblock's migratetype, used when the page is @@ -1092,6 +1134,15 @@ static int free_tail_pages_check(struct page *head_page, struct page *page) return ret; } +static void kernel_init_free_pages(struct page *page, int numpages) +{ + int i; + + if (want_init_on_free()) + for (i = 0; i < numpages; i++) + clear_highpage(page + i); +} + static __always_inline bool free_pages_prepare(struct page *page, unsigned int order, bool check_free) { @@ -1144,6 +1195,7 @@ static __always_inline bool free_pages_prepare(struct page *page, } arch_free_page(page, order); kernel_poison_pages(page, 1 << order, 0); + kernel_init_free_pages(page, 1 << order); kernel_map_pages(page, 1 << order, 0); kasan_free_nondeferred_pages(page, order); @@ -1450,8 +1502,10 @@ meminit_pfn_in_nid(unsigned long pfn, int node, void __init memblock_free_pages(struct page *page, unsigned long pfn, unsigned int order) { - if (early_page_uninitialised(pfn)) + if (early_page_uninitialised(pfn)) { + kernel_init_free_pages(page, 1 << order); return; + } __free_pages_core(page, order); } @@ -1969,8 +2023,8 @@ static inline int check_new_page(struct page *page) static inline bool free_pages_prezeroed(void) { - return IS_ENABLED(CONFIG_PAGE_POISONING_ZERO) && - page_poisoning_enabled(); + return (IS_ENABLED(CONFIG_PAGE_POISONING_ZERO) && + page_poisoning_enabled()) || want_init_on_free(); } #ifdef CONFIG_DEBUG_VM @@ -2027,7 +2081,7 @@ static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags post_alloc_hook(page, order, gfp_flags); - if (!free_pages_prezeroed() && (gfp_flags & __GFP_ZERO)) + if (!free_pages_prezeroed() && want_init_on_alloc(gfp_flags)) for (i = 0; i < (1 << order); i++) clear_highpage(page + i); diff --git a/mm/slab.c b/mm/slab.c index 9142ee992493..fc5b3b81db60 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1891,6 +1891,14 @@ static bool set_objfreelist_slab_cache(struct kmem_cache *cachep, cachep->num = 0; + /* + * If slab auto-initialization on free is enabled, store the freelist + * off-slab, so that its contents don't end up in one of the allocated + * objects. + */ + if (unlikely(slab_want_init_on_free(cachep))) + return false; + if (cachep->ctor || flags & SLAB_TYPESAFE_BY_RCU) return false; @@ -3330,7 +3338,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, local_irq_restore(save_flags); ptr = cache_alloc_debugcheck_after(cachep, flags, ptr, caller); - if (unlikely(flags & __GFP_ZERO) && ptr) + if (unlikely(slab_want_init_on_alloc(flags, cachep)) && ptr) memset(ptr, 0, cachep->object_size); slab_post_alloc_hook(cachep, flags, 1, &ptr); @@ -3387,7 +3395,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) objp = cache_alloc_debugcheck_after(cachep, flags, objp, caller); prefetchw(objp); - if (unlikely(flags & __GFP_ZERO) && objp) + if (unlikely(slab_want_init_on_alloc(flags, cachep)) && objp) memset(objp, 0, cachep->object_size); slab_post_alloc_hook(cachep, flags, 1, &objp); @@ -3508,6 +3516,8 @@ void ___cache_free(struct kmem_cache *cachep, void *objp, struct array_cache *ac = cpu_cache_get(cachep); check_irq_off(); + if (unlikely(slab_want_init_on_free(cachep))) + memset(objp, 0, cachep->object_size); kmemleak_free_recursive(objp, cachep->flags); objp = cache_free_debugcheck(cachep, objp, caller); @@ -3595,7 +3605,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, cache_alloc_debugcheck_after_bulk(s, flags, size, p, _RET_IP_); /* Clear memory outside IRQ disabled section */ - if (unlikely(flags & __GFP_ZERO)) + if (unlikely(slab_want_init_on_alloc(flags, s))) for (i = 0; i < size; i++) memset(p[i], 0, s->object_size); diff --git a/mm/slab.h b/mm/slab.h index 43ac818b8592..24ae887359b8 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -524,4 +524,20 @@ static inline int cache_random_seq_create(struct kmem_cache *cachep, static inline void cache_random_seq_destroy(struct kmem_cache *cachep) { } #endif /* CONFIG_SLAB_FREELIST_RANDOM */ +static inline bool slab_want_init_on_alloc(gfp_t flags, struct kmem_cache *c) +{ + if (static_branch_unlikely(&init_on_alloc)) + return !(c->ctor); + else + return flags & __GFP_ZERO; +} + +static inline bool slab_want_init_on_free(struct kmem_cache *c) +{ + if (static_branch_unlikely(&init_on_free)) + return !(c->ctor); + else + return false; +} + #endif /* MM_SLAB_H */ diff --git a/mm/slob.c b/mm/slob.c index 307c2c9feb44..351d3dfee000 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -212,6 +212,19 @@ static void slob_free_pages(void *b, int order) free_pages((unsigned long)b, order); } +/* + * init_on_free=1 also implies initialization at allocation time. + * This is because newly allocated objects may contain freelist pointers + * somewhere in the middle. + */ +static inline bool slob_want_init_on_alloc(gfp_t flags, struct kmem_cache *c) +{ + if (static_branch_unlikely(&init_on_alloc) || + static_branch_unlikely(&init_on_free)) + return c ? (!c->ctor) : true; + return flags & __GFP_ZERO; +} + /* * Allocate a slob block within a given slob_page sp. */ @@ -330,8 +343,6 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node) BUG_ON(!b); spin_unlock_irqrestore(&slob_lock, flags); } - if (unlikely(gfp & __GFP_ZERO)) - memset(b, 0, size); return b; } @@ -366,6 +377,9 @@ static void slob_free(void *block, int size) return; } + if (unlikely(want_init_on_free())) + memset(block, 0, size); + if (!slob_page_free(sp)) { /* This slob page is about to become partially free. Easy! */ sp->units = units; @@ -461,6 +475,8 @@ __do_kmalloc_node(size_t size, gfp_t gfp, int node, unsigned long caller) } kmemleak_alloc(ret, size, 1, gfp); + if (unlikely(slob_want_init_on_alloc(gfp, 0))) + memset(ret, 0, size); return ret; } @@ -559,6 +575,8 @@ static void *slob_alloc_node(struct kmem_cache *c, gfp_t flags, int node) WARN_ON_ONCE(flags & __GFP_ZERO); c->ctor(b); } + if (unlikely(slob_want_init_on_alloc(flags, c))) + memset(b, 0, c->size); kmemleak_alloc_recursive(b, c->size, 1, c->flags, flags); return b; diff --git a/mm/slub.c b/mm/slub.c index d30ede89f4a6..cc091424c593 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1432,6 +1432,19 @@ static __always_inline bool slab_free_hook(struct kmem_cache *s, void *x) static inline bool slab_free_freelist_hook(struct kmem_cache *s, void **head, void **tail) { + + void *object; + void *next = *head; + void *old_tail = *tail ? *tail : *head; + + if (slab_want_init_on_free(s)) + do { + object = next; + next = get_freepointer(s, object); + memset(object, 0, s->size); + set_freepointer(s, object, next); + } while (object != old_tail); + /* * Compiler cannot detect this function can be removed if slab_free_hook() * evaluates to nothing. Thus, catch all relevant config debug options here. @@ -1441,9 +1454,7 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s, defined(CONFIG_DEBUG_OBJECTS_FREE) || \ defined(CONFIG_KASAN) - void *object; - void *next = *head; - void *old_tail = *tail ? *tail : *head; + next = *head; /* Head and tail of the reconstructed freelist */ *head = NULL; @@ -2749,8 +2760,14 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, prefetch_freepointer(s, next_object); stat(s, ALLOC_FASTPATH); } + /* + * If the object has been wiped upon free, make sure it's fully + * initialized by zeroing out freelist pointer. + */ + if (slab_want_init_on_free(s)) + *(void **)object = 0; - if (unlikely(gfpflags & __GFP_ZERO) && object) + if (unlikely(slab_want_init_on_alloc(gfpflags, s)) && object) memset(object, 0, s->object_size); slab_post_alloc_hook(s, gfpflags, 1, &object); @@ -3172,7 +3189,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, local_irq_enable(); /* Clear memory outside IRQ disabled fastpath loop */ - if (unlikely(flags & __GFP_ZERO)) { + if (unlikely(slab_want_init_on_alloc(flags, s))) { int j; for (j = 0; j < i; j++) diff --git a/net/core/sock.c b/net/core/sock.c index 067878a1e4c5..bd03e3a52f9d 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -1601,7 +1601,7 @@ static struct sock *sk_prot_alloc(struct proto *prot, gfp_t priority, sk = kmem_cache_alloc(slab, priority & ~__GFP_ZERO); if (!sk) return sk; - if (priority & __GFP_ZERO) + if (want_init_on_alloc(priority)) sk_prot_clear_nulls(sk, prot->obj_size); } else sk = kmalloc(prot->obj_size, priority); diff --git a/security/Kconfig.hardening b/security/Kconfig.hardening index 0a1d4ca314f4..4a4001f5ad25 100644 --- a/security/Kconfig.hardening +++ b/security/Kconfig.hardening @@ -159,6 +159,22 @@ config STACKLEAK_RUNTIME_DISABLE runtime to control kernel stack erasing for kernels built with CONFIG_GCC_PLUGIN_STACKLEAK. +config INIT_ON_ALLOC_DEFAULT_ON + bool "Set init_on_alloc=1 by default" + default false + help + Enable init_on_alloc=1 by default, making the kernel initialize every + page and heap allocation with zeroes. + init_on_alloc can be overridden via command line. + +config INIT_ON_FREE_DEFAULT_ON + bool "Set init_on_free=1 by default" + default false + help + Enable init_on_free=1 by default, making the kernel initialize freed + pages and slab memory with zeroes. + init_on_free can be overridden via command line. + endmenu endmenu From patchwork Wed May 8 15:37:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 10936111 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D983792A for ; Wed, 8 May 2019 15:38:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C93E920881 for ; Wed, 8 May 2019 15:38:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BC8CE28390; Wed, 8 May 2019 15:38:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F3BF920881 for ; Wed, 8 May 2019 15:38:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EB9126B02C0; Wed, 8 May 2019 11:38:20 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E1B586B02C1; Wed, 8 May 2019 11:38:20 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D306E6B02C2; Wed, 8 May 2019 11:38:20 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ot1-f69.google.com (mail-ot1-f69.google.com [209.85.210.69]) by kanga.kvack.org (Postfix) with ESMTP id A95536B02C0 for ; Wed, 8 May 2019 11:38:20 -0400 (EDT) Received: by mail-ot1-f69.google.com with SMTP id k90so11231829otk.21 for ; Wed, 08 May 2019 08:38:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=VmHYFv+F99NJ5GAkPbTyJyR0t7BgQjeUOj8lTClHhDU=; b=c8NfSDhxgCSUDBA6gAXbWOP6YzVOiur3QMDfAodUCP68mdZrbbNSmmQUwir2Vo6FlU 1YcTVJJkACsniGPgGjZb2Bs9AHtHWSriPoSALNZBdCKto1X0ZBwwJFnBeOWjWNVCv+r8 psrEdCEXWb/LpWLZYEgoF8HwsvEpMbSl47VxJQThSnNgMFU9NXpXgf+sbbEKYazJ1d4+ Ju3Zz803cVKmh5joSMqjxlvpRMFgh3M22w1be/Rxo5LS4vV0y+JXqFfFX/fcPFAjC16+ 9IfdNVqvWkn4ceusZk0nb/cO9FflnHeEuTJZwQ3zycJyiGCl35vrqfosFSf3hM4s6/yg MDdQ== X-Gm-Message-State: APjAAAU0vUH43JGD24OV4PwpRuyVUKgwDYnoFG/JcmXI3Rj7HGhofgoi 6WFkjqS+IYTVikg/Nk2l78gVkFwgOSLiygwEeZrcJI1tmgMPdJHwi92QKWOcdsFckvaHObiqcIP ureEV53JI3ZpePbP+O+yj/R4jqcgP+dOuZBAq1SoJR80MRrI9LxIFcQD0ALJyXCgbxg== X-Received: by 2002:a05:6830:1385:: with SMTP id d5mr9086498otq.163.1557329900383; Wed, 08 May 2019 08:38:20 -0700 (PDT) X-Received: by 2002:a05:6830:1385:: with SMTP id d5mr9086462otq.163.1557329899561; Wed, 08 May 2019 08:38:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557329899; cv=none; d=google.com; s=arc-20160816; b=wuKcB3mNVrZJA8qzYbiK2rdXeGTqE3pr5UaRg1VSToR8okQUfxrQEYx9iPERQX+juT gygcRiue88IIhRBLbEZNYycpkHikk3drtLiIO2icGiJEA1bIU8jLhNZ4ZFP/o88zM0rZ RfBvz0ozAjbvpVZGWO9SXULjfvzUhm/8s7QHi8x90rO4YJLc9iH+WTgRnU7SUQBQRw61 rDdhEQnb6oE6E/H6yRI8n+hEddZicI5HCHSkJlaINNDOLfApuVTIG49rUjqCkoBRyfP+ f0V2mF+K9T8IWADHE66tTFKBHdtIDJQhOe2CzG3Wr3j10ggu0UYeLK4KTyObTsauOI0X nNEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=VmHYFv+F99NJ5GAkPbTyJyR0t7BgQjeUOj8lTClHhDU=; b=UkWxXORI1lxwX+DG9xNiZogc2KC0xaMNOxBEMP4liT4tsuHLM9I19qY1cX51iKnI/O kqcnjgr1hs/9fw21XTr4kv5/ce17aDw6fP2vL5faSjfPrJsNNaRjlZ7eJnvz/EtIer37 xIdMzqqFZXaHZk4rR3rD7+Yc/pcBddgjBAU6fT+gbVOT3ncb8FSr6+0TIgJtPkJXLcML j7ZK6Krlwyq/IJfv+Xxr57nrMsbcdUHZtjWZzys9eiyDmxE4nyo6K1fmXf2g/Vr5ZlJx 3KetGGl9DQuPjjgo1azgS0Bg65Z32sBHYiZtxFRp422uu3DUrj5VFdrEDzcXVG8rvP94 y6pw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b="V6RgUx/U"; spf=pass (google.com: domain of 36_fsxaykcbq052xyb08805y.w86527eh-664fuw4.8b0@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=36_fSXAYKCBQ052xyB08805y.w86527EH-664Fuw4.8B0@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id v6sor6966575oic.136.2019.05.08.08.38.19 for (Google Transport Security); Wed, 08 May 2019 08:38:19 -0700 (PDT) Received-SPF: pass (google.com: domain of 36_fsxaykcbq052xyb08805y.w86527eh-664fuw4.8b0@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b="V6RgUx/U"; spf=pass (google.com: domain of 36_fsxaykcbq052xyb08805y.w86527eh-664fuw4.8b0@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=36_fSXAYKCBQ052xyB08805y.w86527EH-664Fuw4.8B0@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=VmHYFv+F99NJ5GAkPbTyJyR0t7BgQjeUOj8lTClHhDU=; b=V6RgUx/U1VBKqv6Y0GOzPs5293yWXA+JzDDYWYcMGH7PR6fGSpATnshwWJc65r86tm evjI4NnaHCXmyFSjPA2e4uhU8mgDmCf1052LmTGATOGwhrRMGtGcb+fDUqAJuIDBAjVw cqOtuxMN0hb/KQRV6cHlZ77qWslLhSK31gxj4DT7vtxBA0jHyPxEeDBU96dFhJrzOqW2 M5QgDxVrTjPSvp2y4EiKVuyl8XBlbHnEkWf58VXDFxeAbNhQIkVEKgdaPxc48yETlgZ0 8jPZOj4j9VauKtfpH2JNvL7Zk/Tfsvk5YzTBWk6KmvN43ytEvGQdAUf/jONOu3T56/LD qKCQ== X-Google-Smtp-Source: APXvYqx8p4G+pkm0Fz12Uy1bJ0ErL1+MdGYtfHtwmRe+Z/ufjKBFrUj79maLbYIhnXca1ZR0nPwliHvzGpM= X-Received: by 2002:a54:478a:: with SMTP id o10mr2745917oic.158.1557329899101; Wed, 08 May 2019 08:38:19 -0700 (PDT) Date: Wed, 8 May 2019 17:37:34 +0200 In-Reply-To: <20190508153736.256401-1-glider@google.com> Message-Id: <20190508153736.256401-3-glider@google.com> Mime-Version: 1.0 References: <20190508153736.256401-1-glider@google.com> X-Mailer: git-send-email 2.21.0.1020.gf2820cf01a-goog Subject: [PATCH 2/4] lib: introduce test_meminit module From: Alexander Potapenko To: akpm@linux-foundation.org, cl@linux.com, keescook@chromium.org, labbott@redhat.com Cc: linux-mm@kvack.org, linux-security-module@vger.kernel.org, kernel-hardening@lists.openwall.com, yamada.masahiro@socionext.com, jmorris@namei.org, serge@hallyn.com, ndesaulniers@google.com, kcc@google.com, dvyukov@google.com, sspatil@android.com, rdunlap@infradead.org, jannh@google.com, mark.rutland@arm.com X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add tests for heap and pagealloc initialization. These can be used to check init_on_alloc and init_on_free implementations as well as other approaches to initialization. Signed-off-by: Alexander Potapenko Cc: Kees Cook Cc: Andrew Morton Cc: Christoph Lameter Cc: Nick Desaulniers Cc: Kostya Serebryany Cc: Dmitry Vyukov Cc: Sandeep Patil Cc: Laura Abbott Cc: Jann Horn Cc: linux-mm@kvack.org Cc: linux-security-module@vger.kernel.org Cc: kernel-hardening@lists.openwall.com --- lib/Kconfig.debug | 8 ++ lib/Makefile | 1 + lib/test_meminit.c | 205 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 214 insertions(+) create mode 100644 lib/test_meminit.c diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index d5a4a4036d2f..28d20c01eb41 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2010,6 +2010,14 @@ config TEST_STACKINIT If unsure, say N. +config TEST_MEMINIT + tristate "Test level of heap/page initialization" + help + Test if the kernel is zero-initializing heap and page allocations. + This can be useful to test init_on_alloc and init_on_free features. + + If unsure, say N. + endif # RUNTIME_TESTING_MENU config MEMTEST diff --git a/lib/Makefile b/lib/Makefile index 18c2be516ab4..04d49fbb9ae7 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -90,6 +90,7 @@ obj-$(CONFIG_TEST_DEBUG_VIRTUAL) += test_debug_virtual.o obj-$(CONFIG_TEST_MEMCAT_P) += test_memcat_p.o obj-$(CONFIG_TEST_OBJAGG) += test_objagg.o obj-$(CONFIG_TEST_STACKINIT) += test_stackinit.o +obj-$(CONFIG_TEST_MEMINIT) += test_meminit.o obj-$(CONFIG_TEST_LIVEPATCH) += livepatch/ diff --git a/lib/test_meminit.c b/lib/test_meminit.c new file mode 100644 index 000000000000..6f4ed118a611 --- /dev/null +++ b/lib/test_meminit.c @@ -0,0 +1,205 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Test cases for SL[AOU]B/page initialization at alloc/free time. + */ +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#include +#include +#include +#include +#include +#include + +#define GARBAGE_INT (0x09A7BA9E) +#define GARBAGE_BYTE (0x9E) + +#define REPORT_FAILURES_IN_FN() \ + do { \ + if (failures) \ + pr_info("%s failed %d out of %d times\n", \ + __func__, failures, num_tests); \ + else \ + pr_info("all %d tests in %s passed\n", \ + num_tests, __func__); \ + } while (0) + +/* Calculate the number of uninitialized bytes in the buffer. */ +static int count_nonzero_bytes(void *ptr, size_t size) +{ + int i, ret = 0; + unsigned char *p = (unsigned char *)ptr; + + for (i = 0; i < size; i++) + if (p[i]) + ret++; + return ret; +} + +static void fill_with_garbage(void *ptr, size_t size) +{ + unsigned int *p = (unsigned int *)ptr; + int i = 0; + + while (size >= sizeof(*p)) { + p[i] = GARBAGE_INT; + i++; + size -= sizeof(*p); + } + if (size) + memset(&p[i], GARBAGE_BYTE, size); +} + +static int __init do_alloc_pages_order(int order, int *total_failures) +{ + struct page *page; + void *buf; + size_t size = PAGE_SIZE << order; + + page = alloc_pages(GFP_KERNEL, order); + buf = page_address(page); + fill_with_garbage(buf, size); + __free_pages(page, order); + + page = alloc_pages(GFP_KERNEL, order); + buf = page_address(page); + if (count_nonzero_bytes(buf, size)) + (*total_failures)++; + fill_with_garbage(buf, size); + __free_pages(page, order); + return 1; +} + +static int __init test_pages(int *total_failures) +{ + int failures = 0, num_tests = 0; + int i; + + for (i = 0; i < 10; i++) + num_tests += do_alloc_pages_order(i, &failures); + + REPORT_FAILURES_IN_FN(); + *total_failures += failures; + return num_tests; +} + +static int __init do_kmalloc_size(size_t size, int *total_failures) +{ + void *buf; + + buf = kmalloc(size, GFP_KERNEL); + fill_with_garbage(buf, size); + kfree(buf); + + buf = kmalloc(size, GFP_KERNEL); + if (count_nonzero_bytes(buf, size)) + (*total_failures)++; + fill_with_garbage(buf, size); + kfree(buf); + return 1; +} + +static int __init do_vmalloc_size(size_t size, int *total_failures) +{ + void *buf; + + buf = vmalloc(size); + fill_with_garbage(buf, size); + vfree(buf); + + buf = vmalloc(size); + if (count_nonzero_bytes(buf, size)) + (*total_failures)++; + fill_with_garbage(buf, size); + vfree(buf); + return 1; +} + +static int __init test_kvmalloc(int *total_failures) +{ + int failures = 0, num_tests = 0; + int i, size; + + for (i = 0; i < 20; i++) { + size = 1 << i; + num_tests += do_kmalloc_size(size, &failures); + num_tests += do_vmalloc_size(size, &failures); + } + + REPORT_FAILURES_IN_FN(); + *total_failures += failures; + return num_tests; +} + +#define CTOR_BYTES 4 +/* Initialize the first 4 bytes of the object. */ +void some_ctor(void *obj) +{ + memset(obj, 'A', CTOR_BYTES); +} + +static int __init do_kmem_cache_size(size_t size, bool want_ctor, + int *total_failures) +{ + struct kmem_cache *c; + void *buf; + int iter, bytes; + int fail = 0; + + c = kmem_cache_create("test_cache", size, 1, 0, + want_ctor ? some_ctor : NULL); + for (iter = 0; iter < 10; iter++) { + buf = kmem_cache_alloc(c, GFP_KERNEL); + if (!want_ctor || iter == 0) + bytes = count_nonzero_bytes(buf, size); + if (want_ctor) { + /* + * Newly initialized memory must be initialized using + * the constructor. + */ + if (iter == 0 && bytes < CTOR_BYTES) + fail = 1; + } else { + if (bytes) + fail = 1; + } + fill_with_garbage(buf, size); + kmem_cache_free(c, buf); + } + kmem_cache_destroy(c); + + *total_failures += fail; + return 1; +} + +static int __init test_kmemcache(int *total_failures) +{ + int failures = 0, num_tests = 0; + int i, size; + + for (i = 0; i < 10; i++) { + size = 4 << i; + num_tests += do_kmem_cache_size(size, false, &failures); + num_tests += do_kmem_cache_size(size, true, &failures); + } + REPORT_FAILURES_IN_FN(); + *total_failures += failures; + return num_tests; +} + +static int __init test_meminit_init(void) +{ + int failures = 0, num_tests = 0; + + num_tests += test_pages(&failures); + num_tests += test_kvmalloc(&failures); + num_tests += test_kmemcache(&failures); + + if (failures == 0) + pr_info("all %d tests passed!\n", num_tests); + else + pr_info("failures: %d out of %d\n", failures, num_tests); + + return failures ? -EINVAL : 0; +} +module_init(test_meminit_init); From patchwork Wed May 8 15:37:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 10936115 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B641C912 for ; Wed, 8 May 2019 15:38:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A65E720881 for ; Wed, 8 May 2019 15:38:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9988C28390; Wed, 8 May 2019 15:38:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DF6B220881 for ; Wed, 8 May 2019 15:38:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A990B6B02C1; Wed, 8 May 2019 11:38:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A4C786B02C2; Wed, 8 May 2019 11:38:23 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 93B416B02C3; Wed, 8 May 2019 11:38:23 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by kanga.kvack.org (Postfix) with ESMTP id 75D966B02C1 for ; Wed, 8 May 2019 11:38:23 -0400 (EDT) Received: by mail-qt1-f198.google.com with SMTP id n39so23568703qtn.0 for ; Wed, 08 May 2019 08:38:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=lo76tEV8/5tTBkkO8xlJz/8lajq8uCmmdOMOd2IZBb4=; b=tbftP7YalyeQGUxOu2hj2QZbKwd4NhJP25861rBgto+91ZujJVQHpQQTWxZRpX9KEc CxXLAbOMrfjgEG00+ANUjjotG638srzgE4Q4QweP0uyRcLjHWLyNFyiB/vrhScYISd1k s4iMWEnC4ndvLERz1q7VkLkJ3ZEViTekD/JvjKoujnTTFC/pZ3Fl7jR9tuxwJJ29ZCjS sxIAsaeQOe7vly5cEDkpSekPwyINL7/Hn2Zttcr9LTxgK0XfV4syQN3GSUybx85IaXMx ve3vNPSHTshtUQiwBskfFBzXLI0xgZwz1+Mb1mbxOe6yu6wDFaeOV9TC9HglkH3AX/eX 4Pqg== X-Gm-Message-State: APjAAAUZr5uDQfuAtfAhA5BhnVQnBj/yXb+tgqNEi22CL5sheWv0zcDA 3dryDzaAkir+szpgWw6b6yvUT9GjjaD+Sqz9sm44SoTputi8PkVN5n1BJnFo47u6PxGKA7iMwcs Kc1WP8bd/WGU/hp2JmyRZFI17X4k/uxBJhU5wbZrHZ+Vab5zNHvavUbXDkjkaoo2dUw== X-Received: by 2002:ae9:f203:: with SMTP id m3mr11373282qkg.317.1557329903235; Wed, 08 May 2019 08:38:23 -0700 (PDT) X-Received: by 2002:ae9:f203:: with SMTP id m3mr11373240qkg.317.1557329902416; Wed, 08 May 2019 08:38:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557329902; cv=none; d=google.com; s=arc-20160816; b=bgCfsoe8AjZSuj4wA/HEHAkALdTATDMqUJ+gQmHPpyg9PovsP4RXuy+/s7oY04xKoR aGvw3l+9IAe8T7IyuxNC9P+iw5vmLSQKJT5lA+mxH3jO1nGLhCHPqggQItd0FgEV0Z82 TFbn3tKMKm1TwFIIWHYa4YnGEIEtOoTblQ4ra4CrpqkUHxhycqmff0pJd27sYDnIckZM PpIWi7KOJg3UPSqJd+IfFOyzU28G4sNMdlJzNcVros6ikxAD9PsDDGUHpn72T6pGFIh9 2nBoiazrKC8PI1qnsVDf/qDBw1Ftx/GeB8yWDX2jD/K/DzxJQlCDbQ73j4qrGA5Y5N3F 6pZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=lo76tEV8/5tTBkkO8xlJz/8lajq8uCmmdOMOd2IZBb4=; b=vKgkcNZLabmgZSEP9G5bluoVzWtrqlIGKfm05DlRoXqp0yEVbzZiYa/Iu1d72sisXb 2Q7BMiBdTMKwDBIp0hEc8ro7Ars+ZiUu4cssvVbaIL0wTBWOokPi9FmMdNHoX5S17yC2 icKmMSd3/nDxUXlQmkFZMW5JSTIp3EPOQhiJBsGOV3dLYMYzoA2d5pUaVmESSUsM2Qux 9ftCVZbtXXXDEhlIXCCJLFHDjIzvAsLLfQahMwGfDG5TLI2DmR3z8v2bnaVKnq79ZGDe Df9T6zLTXfl0M9zESbhz5CxcXGthlmsbRcrUsdS6RxQgFApSmrQjyiXHqdJVLbsx+L/N +sIQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=vPEylMZX; spf=pass (google.com: domain of 37vfsxaykcbc38501e3bb381.zb985ahk-997ixz7.be3@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=37vfSXAYKCBc38501E3BB381.zB985AHK-997Ixz7.BE3@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id c3sor14254091qvt.9.2019.05.08.08.38.22 for (Google Transport Security); Wed, 08 May 2019 08:38:22 -0700 (PDT) Received-SPF: pass (google.com: domain of 37vfsxaykcbc38501e3bb381.zb985ahk-997ixz7.be3@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=vPEylMZX; spf=pass (google.com: domain of 37vfsxaykcbc38501e3bb381.zb985ahk-997ixz7.be3@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=37vfSXAYKCBc38501E3BB381.zB985AHK-997Ixz7.BE3@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=lo76tEV8/5tTBkkO8xlJz/8lajq8uCmmdOMOd2IZBb4=; b=vPEylMZXayPSZVpEaDJqWP9qm/OESynGMJRRV2MuSsH86lVqB1H02b/PiTIS+flJKP t6u5nGh7WqlGTmmiuLu9CAqTVfSYiEM6CS7PMQr8LiNxQWqIygR1z3f1vzfHvunG8MfG Nl+u+93U4iGK/9xuTGGr+SHMh4RT1GIIASx9sNQyJpVm3j1i3Pg4YdJN9WEBHKx81UNM y0n9VztOlXCZo0Mo0bd5i8gmrFPjvvepY5C/rjxdAqKOJ8m1Iw72t3OCjEbfehy4tigT H8V9L5vu7Y47nOBAZQpVNjPcmDubUS/YkZvy3vwto5ROsdazyfci1dI+FRklG39i5/QI U8zQ== X-Google-Smtp-Source: APXvYqwNgpwHjUPfZi1Dbb0SZ7eXYtog/czzOSLoWn+D6mOOwqHmlQYuA0cyyWtpdJ//wXF2GWIvL/IEdeo= X-Received: by 2002:a0c:c3d0:: with SMTP id p16mr31293733qvi.229.1557329902083; Wed, 08 May 2019 08:38:22 -0700 (PDT) Date: Wed, 8 May 2019 17:37:35 +0200 In-Reply-To: <20190508153736.256401-1-glider@google.com> Message-Id: <20190508153736.256401-4-glider@google.com> Mime-Version: 1.0 References: <20190508153736.256401-1-glider@google.com> X-Mailer: git-send-email 2.21.0.1020.gf2820cf01a-goog Subject: [PATCH 3/4] gfp: mm: introduce __GFP_NOINIT From: Alexander Potapenko To: akpm@linux-foundation.org, cl@linux.com, keescook@chromium.org, labbott@redhat.com Cc: linux-mm@kvack.org, linux-security-module@vger.kernel.org, kernel-hardening@lists.openwall.com, yamada.masahiro@socionext.com, jmorris@namei.org, serge@hallyn.com, ndesaulniers@google.com, kcc@google.com, dvyukov@google.com, sspatil@android.com, rdunlap@infradead.org, jannh@google.com, mark.rutland@arm.com X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When passed to an allocator (either pagealloc or SL[AOU]B), __GFP_NOINIT tells it to not initialize the requested memory if the init_on_alloc boot option is enabled. This can be useful in the cases newly allocated memory is going to be initialized by the caller right away. __GFP_NOINIT doesn't affect init_on_free behavior, except for SLOB, where init_on_free implies init_on_alloc. __GFP_NOINIT basically defeats the hardening against information leaks provided by init_on_alloc, so one should use it with caution. This patch also adds __GFP_NOINIT to alloc_pages() calls in SL[AOU]B. Doing so is safe, because the heap allocators initialize the pages they receive before passing memory to the callers. Slowdown for the initialization features compared to init_on_free=0, init_on_alloc=0: hackbench, init_on_free=1: +6.84% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.25% sys time (st.err 0.72%) Linux build with -j12, init_on_free=1: +8.52% wall time (st.err 0.42%) Linux build with -j12, init_on_free=1: +24.31% sys time (st.err 0.47%) Linux build with -j12, init_on_alloc=1: -0.16% wall time (st.err 0.40%) Linux build with -j12, init_on_alloc=1: +1.24% sys time (st.err 0.39%) The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error. Signed-off-by: Alexander Potapenko Cc: Andrew Morton Cc: Masahiro Yamada Cc: James Morris Cc: "Serge E. Hallyn" Cc: Nick Desaulniers Cc: Kostya Serebryany Cc: Dmitry Vyukov Cc: Kees Cook Cc: Sandeep Patil Cc: Laura Abbott Cc: Randy Dunlap Cc: Jann Horn Cc: Mark Rutland Cc: linux-mm@kvack.org Cc: linux-security-module@vger.kernel.org Cc: kernel-hardening@lists.openwall.com --- include/linux/gfp.h | 6 +++++- include/linux/mm.h | 2 +- kernel/kexec_core.c | 2 +- mm/slab.c | 2 +- mm/slob.c | 3 ++- mm/slub.c | 1 + 6 files changed, 11 insertions(+), 5 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index fdab7de7490d..66d7f5604fe2 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -44,6 +44,7 @@ struct vm_area_struct; #else #define ___GFP_NOLOCKDEP 0 #endif +#define ___GFP_NOINIT 0x1000000u /* If the above are modified, __GFP_BITS_SHIFT may need updating */ /* @@ -208,16 +209,19 @@ struct vm_area_struct; * %__GFP_COMP address compound page metadata. * * %__GFP_ZERO returns a zeroed page on success. + * + * %__GFP_NOINIT requests non-initialized memory from the underlying allocator. */ #define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN) #define __GFP_COMP ((__force gfp_t)___GFP_COMP) #define __GFP_ZERO ((__force gfp_t)___GFP_ZERO) +#define __GFP_NOINIT ((__force gfp_t)___GFP_NOINIT) /* Disable lockdep for GFP context tracking */ #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP) /* Room for N __GFP_FOO bits */ -#define __GFP_BITS_SHIFT (23 + IS_ENABLED(CONFIG_LOCKDEP)) +#define __GFP_BITS_SHIFT (25) #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1)) /** diff --git a/include/linux/mm.h b/include/linux/mm.h index ee1a1092679c..8ab152750eb4 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2618,7 +2618,7 @@ DECLARE_STATIC_KEY_FALSE(init_on_alloc); static inline bool want_init_on_alloc(gfp_t flags) { if (static_branch_unlikely(&init_on_alloc)) - return true; + return !(flags & __GFP_NOINIT); return flags & __GFP_ZERO; } diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c index f19d1a91190b..e8ed6e3c6702 100644 --- a/kernel/kexec_core.c +++ b/kernel/kexec_core.c @@ -302,7 +302,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order) { struct page *pages; - pages = alloc_pages(gfp_mask & ~__GFP_ZERO, order); + pages = alloc_pages((gfp_mask & ~__GFP_ZERO) | __GFP_NOINIT, order); if (pages) { unsigned int count, i; diff --git a/mm/slab.c b/mm/slab.c index fc5b3b81db60..f18739559825 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -1393,7 +1393,7 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, struct page *page; int nr_pages; - flags |= cachep->allocflags; + flags |= (cachep->allocflags | __GFP_NOINIT); page = __alloc_pages_node(nodeid, flags, cachep->gfporder); if (!page) { diff --git a/mm/slob.c b/mm/slob.c index 351d3dfee000..5b3c40dbd3f2 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -192,6 +192,7 @@ static void *slob_new_pages(gfp_t gfp, int order, int node) { void *page; + gfp |= __GFP_NOINIT; #ifdef CONFIG_NUMA if (node != NUMA_NO_NODE) page = __alloc_pages_node(node, gfp, order); @@ -221,7 +222,7 @@ static inline bool slob_want_init_on_alloc(gfp_t flags, struct kmem_cache *c) { if (static_branch_unlikely(&init_on_alloc) || static_branch_unlikely(&init_on_free)) - return c ? (!c->ctor) : true; + return c ? (!c->ctor) : !(flags & __GFP_NOINIT); return flags & __GFP_ZERO; } diff --git a/mm/slub.c b/mm/slub.c index cc091424c593..8b61d244fdb4 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1504,6 +1504,7 @@ static inline struct page *alloc_slab_page(struct kmem_cache *s, struct page *page; unsigned int order = oo_order(oo); + flags |= __GFP_NOINIT; if (node == NUMA_NO_NODE) page = alloc_pages(flags, order); else From patchwork Wed May 8 15:37:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 10936119 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0DF3A92A for ; Wed, 8 May 2019 15:38:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F2B3620881 for ; Wed, 8 May 2019 15:38:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E6D6828390; Wed, 8 May 2019 15:38:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 302AC20881 for ; Wed, 8 May 2019 15:38:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B35E16B02C2; Wed, 8 May 2019 11:38:26 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AE6706B02C4; Wed, 8 May 2019 11:38:26 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9DD656B02C2; Wed, 8 May 2019 11:38:26 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-oi1-f199.google.com (mail-oi1-f199.google.com [209.85.167.199]) by kanga.kvack.org (Postfix) with ESMTP id 711C56B02C2 for ; Wed, 8 May 2019 11:38:26 -0400 (EDT) Received: by mail-oi1-f199.google.com with SMTP id h186so7347990oia.13 for ; Wed, 08 May 2019 08:38:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=RIGrkpXui4ckU6KCy/63QA2I+IpeSDYCf/ua2n+WtmM=; b=k2GUu9k+JQqm45MS5oizIhHSBbSphtJfl4H+9CnMj770B+jr9NySrcdl0jwh94AbRL iVDlV5JMDMI9HAknLLc1dbqAFPDpj+ezhrB1mTcln+puTNla7XI/DM9539X0EG6PfuoE TFvSFAnSd+CW2xgk8MT0PAgFURLQtM7RXlAuCQl9GuTL2KmckJS2E26pvIU9rJv22ByL nji/TY8rzi02EX8ObWtYeV04Lc3omQ5pksSzNO6tnUP42bRgVBRguVxLnIMlmHmu97tt mTs4TSOO2URuDznMHW5wlzgWtpas/hDIiuBhPUCq0/vEzrRsIsy7l0OGgTDlNGNLeIEO 0E7w== X-Gm-Message-State: APjAAAVFcHLvdKH0oEeJzjvKzdf6VRBGqxXrIUVASsMFgbpx4QbNTATz ecNXrMYe4+jZnAhr7gJVyQtFmrnPD1t1K5Q0yd5hnMkHVir5W9Nxvux3ctNoya/PbYa39fl9kTp Iu/Xo4ZNOXdFShaKKZYhvl5/7KtTB99CWNXq5TTWMVGvXH3QKkegLMnrpVrWgo9EonQ== X-Received: by 2002:a9d:7445:: with SMTP id p5mr8388597otk.26.1557329906123; Wed, 08 May 2019 08:38:26 -0700 (PDT) X-Received: by 2002:a9d:7445:: with SMTP id p5mr8388541otk.26.1557329905408; Wed, 08 May 2019 08:38:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1557329905; cv=none; d=google.com; s=arc-20160816; b=rSVZtvKD3Dhl66ewuUaWUlOSixF4P48jv0swkEa9R6QTVPveMk/Jbii/loP3WFqzst dAs5FXowdJelSyJpbgTHId/NEhuS7f/UcsxsqkJXRlmMY0LjTCHVBbJ8hJ+uWlkk4bfs Ggt0mBPZVvET32p2iT4cktjDocPfz7NpGQ1pVOMUboPUX2ECr8qHbIqEUjrzhJFJXBbj f7h3/PWOne6HMETdP+igoYaHiYsc47VATyHyzc/HJaMR7MiGNumbZ8GZCoWX8haZ4gUE CF6oqN4eSLiwe9mNe3rNqA+xPeWKB15qKO1C4sjw5mWqaJLph7OFNJAflaAqEAu1gzcZ iFsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=RIGrkpXui4ckU6KCy/63QA2I+IpeSDYCf/ua2n+WtmM=; b=R7EDj2OeA7bn6/JqG49cfYxoexdfMm98KLs4PC10yB+oBAxHEG39BKkqi2SS/NsNR+ P9x8oZ0Q0BLr3dLoXujfdcLD3C5XA8e0C0b6FNZOWBRKvL+AQSQ/VKwPqqFr5EfZh6pa tM6mgVrxSMmqvtY3mLDEHoJaTi8vPKr6JVSUN/qhUqRbyA6erK0/8Q5zy3RKOvweoPTt 3aIxKWszDYGv08x9P/YVIfsNDR7C1gOSWYvS43X9/Lmwt6GDl0IRYTUoGRppLlcxjtGv gN6p/H/pn/dlQ8Z00NaobmgiK+z9++T1lIsTYStKvXLyP1nAcIOzNOMRRoUUP6UQhVYG zb9g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=Lh4gbz54; spf=pass (google.com: domain of 38ffsxaykcbo6b834h6ee6b4.2ecb8dkn-ccal02a.eh6@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=38ffSXAYKCBo6B834H6EE6B4.2ECB8DKN-CCAL02A.EH6@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id r5sor2426226oib.20.2019.05.08.08.38.25 for (Google Transport Security); Wed, 08 May 2019 08:38:25 -0700 (PDT) Received-SPF: pass (google.com: domain of 38ffsxaykcbo6b834h6ee6b4.2ecb8dkn-ccal02a.eh6@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=Lh4gbz54; spf=pass (google.com: domain of 38ffsxaykcbo6b834h6ee6b4.2ecb8dkn-ccal02a.eh6@flex--glider.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=38ffSXAYKCBo6B834H6EE6B4.2ECB8DKN-CCAL02A.EH6@flex--glider.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=RIGrkpXui4ckU6KCy/63QA2I+IpeSDYCf/ua2n+WtmM=; b=Lh4gbz546dcJqnxb/eDP7gIvTS/U3Sjn4/hdUClwhT9SxcWv5vnHYNv8sDeXxKxQl5 HWY/Du/t7KX50typbJC9QoSsp3/ofWI4dmRLjOyXZaZ1EeX2zhH2CcK5OEuPVJMDKYFs qQ0gkInhHzbSWl1A67524cMxGoOQDldas+NSCImyWvzkIwSVp2TQy5VUBDDcU2cxO22t z3T9NKC3D+RigZ3BhPreDEqo4+rAi9LYYDHxOwYHw8+8oINZxKZjLNucV4EuiIVqilr5 cxaqn8Ito0dpG3xXGeCQXt+frKwrMSUM3Vp0kA0TWQj6IbGFxcKq9Ub8AVafdf/vZbeO V4Jg== X-Google-Smtp-Source: APXvYqy1LM8fX/+wQzxFqgkjs9XEXQYvzWaOQ7CT1RHBIrXgmYVlrK6hD1FIqcIZUapxQZjgP2O9fyjMcmM= X-Received: by 2002:aca:4ec5:: with SMTP id c188mr2833935oib.33.1557329905089; Wed, 08 May 2019 08:38:25 -0700 (PDT) Date: Wed, 8 May 2019 17:37:36 +0200 In-Reply-To: <20190508153736.256401-1-glider@google.com> Message-Id: <20190508153736.256401-5-glider@google.com> Mime-Version: 1.0 References: <20190508153736.256401-1-glider@google.com> X-Mailer: git-send-email 2.21.0.1020.gf2820cf01a-goog Subject: [PATCH 4/4] net: apply __GFP_NOINIT to AF_UNIX sk_buff allocations From: Alexander Potapenko To: akpm@linux-foundation.org, cl@linux.com, keescook@chromium.org, labbott@redhat.com Cc: linux-mm@kvack.org, linux-security-module@vger.kernel.org, kernel-hardening@lists.openwall.com, yamada.masahiro@socionext.com, jmorris@namei.org, serge@hallyn.com, ndesaulniers@google.com, kcc@google.com, dvyukov@google.com, sspatil@android.com, rdunlap@infradead.org, jannh@google.com, mark.rutland@arm.com X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add sock_alloc_send_pskb_noinit(), which is similar to sock_alloc_send_pskb(), but allocates with __GFP_NOINIT. This helps reduce the slowdown on hackbench in the init_on_alloc mode from 6.84% to 3.45%. Slowdown for the initialization features compared to init_on_free=0, init_on_alloc=0: hackbench, init_on_free=1: +7.71% sys time (st.err 0.45%) hackbench, init_on_alloc=1: +3.45% sys time (st.err 0.86%) Linux build with -j12, init_on_free=1: +8.34% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.13% sys time (st.err 0.47%) Linux build with -j12, init_on_alloc=1: -0.04% wall time (st.err 0.46%) Linux build with -j12, init_on_alloc=1: +0.50% sys time (st.err 0.45%) The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error. Signed-off-by: Alexander Potapenko Cc: Andrew Morton Cc: Masahiro Yamada Cc: James Morris Cc: "Serge E. Hallyn" Cc: Nick Desaulniers Cc: Kostya Serebryany Cc: Dmitry Vyukov Cc: Kees Cook Cc: Sandeep Patil Cc: Laura Abbott Cc: Randy Dunlap Cc: Jann Horn Cc: Mark Rutland Cc: linux-mm@kvack.org Cc: linux-security-module@vger.kernel.org Cc: kernel-hardening@lists.openwall.com --- include/net/sock.h | 5 +++++ net/core/sock.c | 29 +++++++++++++++++++++++++---- net/unix/af_unix.c | 13 +++++++------ 3 files changed, 37 insertions(+), 10 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index 341f8bafa0cf..64bfc4fd7940 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1612,6 +1612,11 @@ struct sk_buff *sock_alloc_send_skb(struct sock *sk, unsigned long size, struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, unsigned long data_len, int noblock, int *errcode, int max_page_order); +struct sk_buff *sock_alloc_send_pskb_noinit(struct sock *sk, + unsigned long header_len, + unsigned long data_len, + int noblock, int *errcode, + int max_page_order); void *sock_kmalloc(struct sock *sk, int size, gfp_t priority); void sock_kfree_s(struct sock *sk, void *mem, int size); void sock_kzfree_s(struct sock *sk, void *mem, int size); diff --git a/net/core/sock.c b/net/core/sock.c index bd03e3a52f9d..8aabcb25fc6a 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -2187,9 +2187,11 @@ static long sock_wait_for_wmem(struct sock *sk, long timeo) * Generic send/receive buffer handlers */ -struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, - unsigned long data_len, int noblock, - int *errcode, int max_page_order) +struct sk_buff *sock_alloc_send_pskb_internal(struct sock *sk, + unsigned long header_len, + unsigned long data_len, + int noblock, int *errcode, + int max_page_order, gfp_t gfp) { struct sk_buff *skb; long timeo; @@ -2218,7 +2220,7 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, timeo = sock_wait_for_wmem(sk, timeo); } skb = alloc_skb_with_frags(header_len, data_len, max_page_order, - errcode, sk->sk_allocation); + errcode, sk->sk_allocation | gfp); if (skb) skb_set_owner_w(skb, sk); return skb; @@ -2229,8 +2231,27 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, *errcode = err; return NULL; } + +struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, + unsigned long data_len, int noblock, + int *errcode, int max_page_order) +{ + return sock_alloc_send_pskb_internal(sk, header_len, data_len, + noblock, errcode, max_page_order, /*gfp*/0); +} EXPORT_SYMBOL(sock_alloc_send_pskb); +struct sk_buff *sock_alloc_send_pskb_noinit(struct sock *sk, + unsigned long header_len, + unsigned long data_len, + int noblock, int *errcode, + int max_page_order) +{ + return sock_alloc_send_pskb_internal(sk, header_len, data_len, + noblock, errcode, max_page_order, /*gfp*/__GFP_NOINIT); +} +EXPORT_SYMBOL(sock_alloc_send_pskb_noinit); + struct sk_buff *sock_alloc_send_skb(struct sock *sk, unsigned long size, int noblock, int *errcode) { diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index ddb838a1b74c..9a45824c3c48 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -1627,9 +1627,9 @@ static int unix_dgram_sendmsg(struct socket *sock, struct msghdr *msg, BUILD_BUG_ON(SKB_MAX_ALLOC < PAGE_SIZE); } - skb = sock_alloc_send_pskb(sk, len - data_len, data_len, - msg->msg_flags & MSG_DONTWAIT, &err, - PAGE_ALLOC_COSTLY_ORDER); + skb = sock_alloc_send_pskb_noinit(sk, len - data_len, data_len, + msg->msg_flags & MSG_DONTWAIT, &err, + PAGE_ALLOC_COSTLY_ORDER); if (skb == NULL) goto out; @@ -1824,9 +1824,10 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); - skb = sock_alloc_send_pskb(sk, size - data_len, data_len, - msg->msg_flags & MSG_DONTWAIT, &err, - get_order(UNIX_SKB_FRAGS_SZ)); + skb = sock_alloc_send_pskb_noinit(sk, size - data_len, data_len, + msg->msg_flags & MSG_DONTWAIT, + &err, + get_order(UNIX_SKB_FRAGS_SZ)); if (!skb) goto out_err;