From patchwork Wed Jun 27 12:07:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris von Recklinghausen X-Patchwork-Id: 10491295 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 335C860325 for ; Wed, 27 Jun 2018 12:07:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 243A028C79 for ; Wed, 27 Jun 2018 12:07:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 185CF28C8B; Wed, 27 Jun 2018 12:07:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 83ED728C79 for ; Wed, 27 Jun 2018 12:07:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 152CB6B0010; Wed, 27 Jun 2018 08:07:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 102F66B0266; Wed, 27 Jun 2018 08:07:42 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F34426B0269; Wed, 27 Jun 2018 08:07:41 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk0-f197.google.com (mail-qk0-f197.google.com [209.85.220.197]) by kanga.kvack.org (Postfix) with ESMTP id C68496B0010 for ; Wed, 27 Jun 2018 08:07:41 -0400 (EDT) Received: by mail-qk0-f197.google.com with SMTP id 99-v6so1866042qkr.14 for ; Wed, 27 Jun 2018 05:07:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to :subject:date:message-id; bh=KT/QIIqAKEvxgCgIPkjyxvJ1KMotbZt3WR/yQ7uDdXk=; b=oGFLV+mwaDZG9qHmytSz1Ud4FA7rIjpD4KDPeSjKPqe3DPQ7BbKi+Nmy3Dptp4Yv/K Z7pojxyVUaoNAkLP/1m32NKdiqS8cSeleiq6G11VwhqQs+sGHeDd0wop3wLRIkhZDkj6 ycRiwxJGx93FRJrn6jTo/MOD16R/cyOZDTpN2Ohnz2JJVeOUNF+bBdpTDpiQTVV8YOqP eJO2DyPk+9rCbvs1Ogz4/+7j9B7N44CJvZxsjZp7E98DpJtTo770wnX81yO638Ue+M8N hhRkBJbg/4TKnsDjMfUV65auSl0+d60hY3nHAHIUDLF47Lr7xfS+DqMWUTyt0wgQ8LSq d4XA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of crecklin@redhat.com designates 66.187.233.73 as permitted sender) smtp.mailfrom=crecklin@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APt69E39gTWt6cDOlMAROUbioLa+YCkwz/FXS7e0JpDDALSoKHTAv+U3 WWKjxYnS+s0RJTCHZStGwKXsijYlCbGxmEcN9woJP5TZv+n6XqLlTfpr/ZqgBjLI1sYCyaM29Qo 98/hTj0VJTN6/fiymkTtM7TOFj9azfQ+Hdq+BEe22TUE8jeLMli0trQ38Tf9i/ZO8fw== X-Received: by 2002:ac8:5252:: with SMTP id y18-v6mr1421677qtn.11.1530101261561; Wed, 27 Jun 2018 05:07:41 -0700 (PDT) X-Google-Smtp-Source: AAOMgpfCXH1/OexqIbwyIM1Ofu7I486kYx0CExtbcesfXzATzj8GVIRcNi5+fzLRyyraKPGKHAKA X-Received: by 2002:ac8:5252:: with SMTP id y18-v6mr1421441qtn.11.1530101258361; Wed, 27 Jun 2018 05:07:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530101258; cv=none; d=google.com; s=arc-20160816; b=gK7eW6Le2NCbdCGvi41aFvE4CIQjZe5JNYa16OxWbIK2ZeI4P0LioOq0PHJCYAVPsA LV1Ty732382vszji/+ollxTi8geZLH2L7tZEDMVDhll9yQNoHteJjjVqhhqf9BHdWFNt AbAKGSGmgFe9qaHnr4W3c6o2a42gj26uIB9dYzJhFPNPO3Y6owZO7YFzXgsrCmTQaQxt k933hGo2EX9UVm029FxD8LNsMtHlkNqqT+1C2csF6eRXSHC/+mmLE2CN4RkUVQsNrJbF EqY6PFwxb1dGNtZkS71nOfNO+iwkwWs7mXuo1Ri/RNLRgSy+hysFyjzXOvajnT70Ftog jUhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:date:subject:to:from:arc-authentication-results; bh=KT/QIIqAKEvxgCgIPkjyxvJ1KMotbZt3WR/yQ7uDdXk=; b=z6w06dt05/WzPFSSTNFhrzxkqWvlfQ48YXu5lBbMuTiLQOZ1T0Y5XjwvKHUfDSi51O XsoQZgN4z27ScY/34nsNhh9BQraqU3qacrxss6N/uxUFIZ5+V9qL55SsHaapzD2PoB18 LQcNhpiYwL3FUCfVtnobs5w+bRZyGny2Q5/SXu07zyRovScvl1bFtzlYimF4Z42nnY0A zWIDdbUOhiKVLHc6bVT2BCNwMHb31/ldV5Wf4K2aPScVTJkRjzIp0HgmUBeOnnj5Lfgb LK0boxrrgS2rAOD2RizNcnPEDSWwubMCtLd4KMS66LlbJviiZnijaOwM1SpXv4OKc4P6 z0mg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of crecklin@redhat.com designates 66.187.233.73 as permitted sender) smtp.mailfrom=crecklin@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx3-rdu2.redhat.com. [66.187.233.73]) by mx.google.com with ESMTPS id l17-v6si895364qtf.21.2018.06.27.05.07.38 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 27 Jun 2018 05:07:38 -0700 (PDT) Received-SPF: pass (google.com: domain of crecklin@redhat.com designates 66.187.233.73 as permitted sender) client-ip=66.187.233.73; Authentication-Results: mx.google.com; spf=pass (google.com: domain of crecklin@redhat.com designates 66.187.233.73 as permitted sender) smtp.mailfrom=crecklin@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0CF4C77888; Wed, 27 Jun 2018 12:07:38 +0000 (UTC) Received: from crecklin.bos.com (dhcp-17-195.bos.redhat.com [10.18.17.195]) by smtp.corp.redhat.com (Postfix) with ESMTP id 80F3C2026D5B; Wed, 27 Jun 2018 12:07:37 +0000 (UTC) From: Chris von Recklinghausen To: keescook@chromium.org, labbott@redhat.com, pabeni@redhat.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel-hardening@lists.openwall.com Subject: [PATCH v3] add param that allows bootline control of hardened usercopy Date: Wed, 27 Jun 2018 08:07:35 -0400 Message-Id: <1530101255-13988-1-git-send-email-crecklin@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 27 Jun 2018 12:07:38 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 27 Jun 2018 12:07:38 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'crecklin@redhat.com' RCPT:'' X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Enabling HARDENED_USER_COPY causes measurable regressions in networking performance, up to 8% under UDP flood. I'm running an a small packet UDP flood using pktgen vs. a host b2b connected. On the receiver side the UDP packets are processed by a simple user space process that just reads and drops them: https://github.com/netoptimizer/network-testing/blob/master/src/udp_sink.c Not very useful from a functional PoV, but it helps to pin-point bottlenecks in the networking stack. When running a kernel with CONFIG_HARDENED_USERCOPY=y, I see a 5-8% regression in the receive tput, compared to the same kernel without this option enabled. With CONFIG_HARDENED_USERCOPY=y, perf shows ~6% of CPU time spent cumulatively in __check_object_size (~4%) and __virt_addr_valid (~2%). The call-chain is: __GI___libc_recvfrom entry_SYSCALL_64_after_hwframe do_syscall_64 __x64_sys_recvfrom __sys_recvfrom inet_recvmsg udp_recvmsg __check_object_size udp_recvmsg() actually calls copy_to_iter() (inlined) and the latters calls check_copy_size() (again, inlined). A generic distro may want to enable HARDENED_USER_COPY in their default kernel config, but at the same time, such distro may want to be able to avoid the performance penalties in with the default configuration and disable the stricter check on a per-boot basis. This change adds a boot parameter that conditionally disables HARDENED_USERCOPY at boot time. v2->v3: add benchmark details to commit comments Don't add new item to Documentation/admin-guide/kernel-parameters.rst rename boot param to "hardened_usercopy=" update description in Documentation/admin-guide/kernel-parameters.txt static_branch_likely -> static_branch_unlikely add __ro_after_init versions of DEFINE_STATIC_KEY_FALSE, DEFINE_STATIC_KEY_TRUE disable_huc_atboot -> enable_checks (strtobool "on" == true) v1->v2: remove CONFIG_HUC_DEFAULT_OFF default is now enabled, boot param disables move check to __check_object_size so as to not break optimization of __builtin_constant_p() include linux/atomic.h before linux/jump_label.h Signed-off-by: Chris von Recklinghausen --- .../admin-guide/kernel-parameters.txt | 11 ++++++++ include/linux/jump_label.h | 6 +++++ include/linux/thread_info.h | 5 ++++ mm/usercopy.c | 26 +++++++++++++++++++ 4 files changed, 48 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index efc7aa7a0670..560d4dc66f02 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -816,6 +816,17 @@ disable= [IPV6] See Documentation/networking/ipv6.txt. + hardened_usercopy= + [KNL] Under CONFIG_HARDENED_USERCOPY, whether + hardening is enabled for this boot. Hardened + usercopy checking is used to protect the kernel + from reading or writing beyond known memory + allocation boundaries as a proactive defense + against bounds-checking flaws in the kernel's + copy_to_user()/copy_from_user() interface. + on Perform hardened usercopy checks (default). + off Disable hardened usercopy checks. + disable_radix [PPC] Disable RADIX MMU mode on POWER9 diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index b46b541c67c4..1a0b6f17a5d6 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -299,12 +299,18 @@ struct static_key_false { #define DEFINE_STATIC_KEY_TRUE(name) \ struct static_key_true name = STATIC_KEY_TRUE_INIT +#define DEFINE_STATIC_KEY_TRUE_RO(name) \ + struct static_key_true name __ro_after_init = STATIC_KEY_TRUE_INIT + #define DECLARE_STATIC_KEY_TRUE(name) \ extern struct static_key_true name #define DEFINE_STATIC_KEY_FALSE(name) \ struct static_key_false name = STATIC_KEY_FALSE_INIT +#define DEFINE_STATIC_KEY_FALSE_RO(name) \ + struct static_key_false name __ro_after_init = STATIC_KEY_FALSE_INIT + #define DECLARE_STATIC_KEY_FALSE(name) \ extern struct static_key_false name diff --git a/include/linux/thread_info.h b/include/linux/thread_info.h index 8d8821b3689a..ab24fe2d3f87 100644 --- a/include/linux/thread_info.h +++ b/include/linux/thread_info.h @@ -109,6 +109,11 @@ static inline int arch_within_stack_frames(const void * const stack, #endif #ifdef CONFIG_HARDENED_USERCOPY +#include +#include + +DECLARE_STATIC_KEY_FALSE(bypass_usercopy_checks); + extern void __check_object_size(const void *ptr, unsigned long n, bool to_user); diff --git a/mm/usercopy.c b/mm/usercopy.c index e9e9325f7638..39f8b1409618 100644 --- a/mm/usercopy.c +++ b/mm/usercopy.c @@ -20,6 +20,8 @@ #include #include #include +#include +#include #include /* @@ -248,6 +250,9 @@ static inline void check_heap_object(const void *ptr, unsigned long n, */ void __check_object_size(const void *ptr, unsigned long n, bool to_user) { + if (static_branch_unlikely(&bypass_usercopy_checks)) + return; + /* Skip all tests if size is zero. */ if (!n) return; @@ -279,3 +284,24 @@ void __check_object_size(const void *ptr, unsigned long n, bool to_user) check_kernel_text_object((const unsigned long)ptr, n, to_user); } EXPORT_SYMBOL(__check_object_size); + +DEFINE_STATIC_KEY_FALSE_RO(bypass_usercopy_checks); +EXPORT_SYMBOL(bypass_usercopy_checks); + +static bool enable_checks __initdata = true; + +static int __init parse_hardened_usercopy(char *str) +{ + return strtobool(str, &enable_checks); +} + +__setup("hardened_usercopy=", parse_hardened_usercopy); + +static int __init set_hardened_usercopy(void) +{ + if (enable_checks == false) + static_branch_enable(&bypass_usercopy_checks); + return 1; +} + +late_initcall(set_hardened_usercopy);