From patchwork Tue Jul 3 10:00:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris von Recklinghausen X-Patchwork-Id: 10503631 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 4568C60225 for ; Tue, 3 Jul 2018 10:00:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 33131289EF for ; Tue, 3 Jul 2018 10:00:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 26D0928A2A; Tue, 3 Jul 2018 10:00:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7AE05289EF for ; Tue, 3 Jul 2018 10:00:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4002F6B0003; Tue, 3 Jul 2018 06:00:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3B1446B0005; Tue, 3 Jul 2018 06:00:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C7F96B0006; Tue, 3 Jul 2018 06:00:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk0-f198.google.com (mail-qk0-f198.google.com [209.85.220.198]) by kanga.kvack.org (Postfix) with ESMTP id F2E1E6B0003 for ; Tue, 3 Jul 2018 06:00:42 -0400 (EDT) Received: by mail-qk0-f198.google.com with SMTP id h15-v6so1579501qkj.17 for ; Tue, 03 Jul 2018 03:00:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to :subject:date:message-id; bh=Zeouxvm3ZV+7AkWK6hnLI0c5k1+cC0+ORft+Icy476s=; b=sJCEH63uIjBt0oFKrpDNyolvNxVnsJFHEm7KiHEPCBP/jkBPZlhJPMd2vFk4c0aayv /SZtuyQYeNU7pCaLaypdHbxZJij8DvbX5xWSMn3oUDUvgzrDBEGcNSWIPnWsC11lQRhJ qc/a/BnvXjTgPYj1rvYV9cIS/W7/pHr/s0SdivgDNYb95iLb8674vw7Nzqxj9X1ZpYBc s6PZeIA5nFWAckf8eSV9Iaiq+HHTG3y3NPbavFF5hy1spYjDy3aOkVUvFyEnIWL2r3Zu I0A79H+KFzOqbXmdTqmI62jTLUZdtQyl/RPLHi5Cr0zft6xBCwx5qU3VAz5Z0vkxusYJ ZgaA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of crecklin@redhat.com designates 66.187.233.73 as permitted sender) smtp.mailfrom=crecklin@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APt69E0ysgCGByVUh3LhixlQC1EY+1cfHreVDbs36ljd/4Qiz30UEdVv JUUJTZ/XqW4IyPRreOZn6N8wYZ4tgibLOIVDtJhxsAcq8Rxl2RDgz8bq8ZKjxCNVxQs7beovMLS NiUf+thwHxyGUifqegRndWIPHTNXs7chnmtbEUCxCbv/3UvEd8yPulaX/zxn8Fm7w0A== X-Received: by 2002:a37:5944:: with SMTP id n65-v6mr24715395qkb.107.1530612042752; Tue, 03 Jul 2018 03:00:42 -0700 (PDT) X-Google-Smtp-Source: AAOMgpfqXmdkLelQZAZuGmGQF6fjfRLD0yRQZSoBuMvoBP5ImwyM5soDfEoAe5idy/WMdW/xyDkZ X-Received: by 2002:a37:5944:: with SMTP id n65-v6mr24715261qkb.107.1530612040394; Tue, 03 Jul 2018 03:00:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530612040; cv=none; d=google.com; s=arc-20160816; b=vqrAJurvf8ZE/ro6PfGTz2jcDz1gi3Y7GlG0lbNlO1GtEJrYmGi8jzzrqNHoWPGZSL HQ3y0/9nevejFOpHI6o39E2k85QMQn11nqaQxX9uWrqYZPsY0mJskO7WbIhSwd153jtO w33nCBNpRsjq08qjMKgw64A5Dy67Q7JWeBk4kbMsOMShLL8WVY3xJd63owua6zNeJRMj yc1HJC6iasAjha95ad3hjgLG7V+Zoamfdks9omdcVZucr2Ohe8zLbVbjIld2KcSWl75O a10sE9HbShDVd+3TRQ0EWFSPH0RUNhywm93Hik/qNSHIMzTZHS6Ef8uOLovBVUI+R4g+ EHKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:date:subject:to:from:arc-authentication-results; bh=Zeouxvm3ZV+7AkWK6hnLI0c5k1+cC0+ORft+Icy476s=; b=0CeSKcatD47gDS2bKFrpY2RHx5+2NLFTAUfIAu33qGKzGm277BzFC/fmJRpC4Thtp/ 8xOdML7+6w14QHgQjUV2QV2F38CMNciDwVZzxqwokliD+dIyHD4OGUyY3uu0dkeMU4la jYNw8Xf1qCXc+Phs36Ih7sKhA1F/kdUKmA+jH8uExIGKSJifNs9jr7KIzTDTAGoksHKG A3oJJlM9BW3tev6utmkVEUea0A+Oh0SfuX9AypugwaCaF63uplgFlL9QOpX6mfl/hLk7 RcwIhm5mhqaBhjfndcb82l4xCJoSjf5fdrOCfalGxd2v659v1zaTRem7Qrv5tMehhODd HoFw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of crecklin@redhat.com designates 66.187.233.73 as permitted sender) smtp.mailfrom=crecklin@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx3-rdu2.redhat.com. [66.187.233.73]) by mx.google.com with ESMTPS id 42-v6si716543qvf.139.2018.07.03.03.00.40 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 03 Jul 2018 03:00:40 -0700 (PDT) Received-SPF: pass (google.com: domain of crecklin@redhat.com designates 66.187.233.73 as permitted sender) client-ip=66.187.233.73; Authentication-Results: mx.google.com; spf=pass (google.com: domain of crecklin@redhat.com designates 66.187.233.73 as permitted sender) smtp.mailfrom=crecklin@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id F2FA087A7D; Tue, 3 Jul 2018 10:00:39 +0000 (UTC) Received: from crecklin.bos.com (ovpn-120-13.rdu2.redhat.com [10.10.120.13]) by smtp.corp.redhat.com (Postfix) with ESMTP id 31106111AF37; Tue, 3 Jul 2018 10:00:39 +0000 (UTC) From: Chris von Recklinghausen To: keescook@chromium.org, labbott@redhat.com, pabeni@redhat.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel-hardening@lists.openwall.com Subject: [PATCH v6] add param that allows bootline control of hardened usercopy Date: Tue, 3 Jul 2018 06:00:37 -0400 Message-Id: <1530612037-32512-1-git-send-email-crecklin@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Tue, 03 Jul 2018 10:00:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Tue, 03 Jul 2018 10:00:40 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'crecklin@redhat.com' RCPT:'' X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Enabling HARDENED_USERCOPY causes measurable regressions in networking performance, up to 8% under UDP flood. I'm running an a small packet UDP flood using pktgen vs. a host b2b connected. On the receiver side the UDP packets are processed by a simple user space process that just reads and drops them: https://github.com/netoptimizer/network-testing/blob/master/src/udp_sink.c Not very useful from a functional PoV, but it helps to pin-point bottlenecks in the networking stack. When running a kernel with CONFIG_HARDENED_USERCOPY=y, I see a 5-8% regression in the receive tput, compared to the same kernel without this option enabled. With CONFIG_HARDENED_USERCOPY=y, perf shows ~6% of CPU time spent cumulatively in __check_object_size (~4%) and __virt_addr_valid (~2%). The call-chain is: __GI___libc_recvfrom entry_SYSCALL_64_after_hwframe do_syscall_64 __x64_sys_recvfrom __sys_recvfrom inet_recvmsg udp_recvmsg __check_object_size udp_recvmsg() actually calls copy_to_iter() (inlined) and the latters calls check_copy_size() (again, inlined). A generic distro may want to enable HARDENED_USERCOPY in their default kernel config, but at the same time, such distro may want to be able to avoid the performance penalties in with the default configuration and disable the stricter check on a per-boot basis. This change adds a boot parameter that conditionally disables HARDENED_USERCOPY at boot time. This feature is not available on platforms that don't have CONFIG_JUMP_LABEL set. v5->v6: m68k build issue was when changes were in include/linux/thread_info.h. mm/usercopy.c builds just fine without CONFIG_JUMP_LABEL or CONFIG_SMP_BROKEN. v4->v5: key off of CONFIG_JUMP_LABEL, not CONFIG_SMP_BROKEN. v3->v4: fix a couple of nits in commit comments declaration of bypass_usercopy_checks moved inside mm/usercopy.c and made static add blurb to commit comments about not enabling this functionality on platforms with CONFIG_BROKEN_ON_SMP set. v2->v3: add benchmark details to commit comments Don't add new item to Documentation/admin-guide/kernel-parameters.rst rename boot param to "hardened_usercopy=" update description in Documentation/admin-guide/kernel-parameters.txt static_branch_likely -> static_branch_unlikely add __ro_after_init versions of DEFINE_STATIC_KEY_FALSE, DEFINE_STATIC_KEY_TRUE disable_huc_atboot -> enable_checks (strtobool "on" == true) v1->v2: remove CONFIG_HUC_DEFAULT_OFF default is now enabled, boot param disables move check to __check_object_size so as to not break optimization of __builtin_constant_p() include linux/atomic.h before linux/jump_label.h Signed-off-by: Chris von Recklinghausen --- Documentation/admin-guide/kernel-parameters.txt | 11 ++++++++++ include/linux/jump_label.h | 6 ++++++ mm/usercopy.c | 27 +++++++++++++++++++++++++ 3 files changed, 44 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index efc7aa7..560d4dc 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -816,6 +816,17 @@ disable= [IPV6] See Documentation/networking/ipv6.txt. + hardened_usercopy= + [KNL] Under CONFIG_HARDENED_USERCOPY, whether + hardening is enabled for this boot. Hardened + usercopy checking is used to protect the kernel + from reading or writing beyond known memory + allocation boundaries as a proactive defense + against bounds-checking flaws in the kernel's + copy_to_user()/copy_from_user() interface. + on Perform hardened usercopy checks (default). + off Disable hardened usercopy checks. + disable_radix [PPC] Disable RADIX MMU mode on POWER9 diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index b46b541..1a0b6f1 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -299,12 +299,18 @@ struct static_key_false { #define DEFINE_STATIC_KEY_TRUE(name) \ struct static_key_true name = STATIC_KEY_TRUE_INIT +#define DEFINE_STATIC_KEY_TRUE_RO(name) \ + struct static_key_true name __ro_after_init = STATIC_KEY_TRUE_INIT + #define DECLARE_STATIC_KEY_TRUE(name) \ extern struct static_key_true name #define DEFINE_STATIC_KEY_FALSE(name) \ struct static_key_false name = STATIC_KEY_FALSE_INIT +#define DEFINE_STATIC_KEY_FALSE_RO(name) \ + struct static_key_false name __ro_after_init = STATIC_KEY_FALSE_INIT + #define DECLARE_STATIC_KEY_FALSE(name) \ extern struct static_key_false name diff --git a/mm/usercopy.c b/mm/usercopy.c index e9e9325..7c8a1e9 100644 --- a/mm/usercopy.c +++ b/mm/usercopy.c @@ -20,6 +20,8 @@ #include #include #include +#include +#include #include /* @@ -240,6 +242,8 @@ static inline void check_heap_object(const void *ptr, unsigned long n, } } +static DEFINE_STATIC_KEY_FALSE_RO(bypass_usercopy_checks); + /* * Validates that the given object is: * - not bogus address @@ -248,6 +252,9 @@ static inline void check_heap_object(const void *ptr, unsigned long n, */ void __check_object_size(const void *ptr, unsigned long n, bool to_user) { + if (static_branch_unlikely(&bypass_usercopy_checks)) + return; + /* Skip all tests if size is zero. */ if (!n) return; @@ -279,3 +286,23 @@ void __check_object_size(const void *ptr, unsigned long n, bool to_user) check_kernel_text_object((const unsigned long)ptr, n, to_user); } EXPORT_SYMBOL(__check_object_size); + +EXPORT_SYMBOL(bypass_usercopy_checks); + +static bool enable_checks __initdata = true; + +static int __init parse_hardened_usercopy(char *str) +{ + return strtobool(str, &enable_checks); +} + +__setup("hardened_usercopy=", parse_hardened_usercopy); + +static int __init set_hardened_usercopy(void) +{ + if (enable_checks == false) + static_branch_enable(&bypass_usercopy_checks); + return 1; +} + +late_initcall(set_hardened_usercopy);