From patchwork Thu Jan 17 00:32:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10767239 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1955514E5 for ; Thu, 17 Jan 2019 00:33:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 056DF2DDA1 for ; Thu, 17 Jan 2019 00:33:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EA5632DDF1; Thu, 17 Jan 2019 00:33:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 23E672DDA1 for ; Thu, 17 Jan 2019 00:33:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 29BAE8E0007; Wed, 16 Jan 2019 19:33:40 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0BF948E000C; Wed, 16 Jan 2019 19:33:39 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF6DB8E000A; Wed, 16 Jan 2019 19:33:39 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id 746B48E0005 for ; Wed, 16 Jan 2019 19:33:39 -0500 (EST) Received: by mail-pl1-f197.google.com with SMTP id a9so4960857pla.2 for ; Wed, 16 Jan 2019 16:33:39 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=pq6MSHG/FPqQxPMfNwdf/s7Eb60udNDy2cwn4wbvNFc=; b=hShz4L37Xf4hpUnN0eQmi9csTox9UnXOGDHpyF2lETw+8/dkrOT2loAVKgyC4dz+/P Rq1dpvD8wiVDrSo5APBo6f3scIx8nYzfRFIBMryKoFIis7XzCkog4ftB0M6g5aNK17F+ J4xI9lLU+FcK8EtPcAh9NwvhTvoGaDu/LzyZVEsvBjpiCPjeU/Yi/AJfoIUl+GPMUuYV u/KaOKFhGzRIJ8hCnLswthA6mCxACIP4aghKU5UYZI1u7P9Sfma0Q9ej5+aOHJ6Ucq2Z 8sEknSlZHpRDcBuSlKI8PR6F0ug7uJRwAw4CmvdlpXRM7jne8iPcqhjS5EstNyqmN5D5 0yrQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AJcUukfwrdANjzSHn8IsSdyLikV2sk2OuO/ZldkaLI8rzjk451OK/3qR +U5Y/BH847jGDk2r8S+n/n3VBdsbZ6wIhOxH35zhaY0OCXTr+9lv4H2as/b7ay7Ztad8Vpqzl2d CTzkLUgufc5ER5HuB2fAsXDAmJ5810dr1h+xEE1BQimkGYNt4uCcejBHzcbCSDD65ig== X-Received: by 2002:a63:295:: with SMTP id 143mr11060960pgc.362.1547685219088; Wed, 16 Jan 2019 16:33:39 -0800 (PST) X-Google-Smtp-Source: ALg8bN7u/i0bmTU1ec03RiGjrB3KwP1LMhjnmEfAAV/yZsTUN5TMn4g78CveZavdsrw60KBhf7Cd X-Received: by 2002:a63:295:: with SMTP id 143mr11060896pgc.362.1547685217992; Wed, 16 Jan 2019 16:33:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547685217; cv=none; d=google.com; s=arc-20160816; b=eTJv1pT2nJfRpCrJXzWptJTgwWxBnUJkLR8+/A/+0EpGP+zib37yYbFho6Nbn7Lbsz 04sTiQPW8IahG5dQavzqkmcCsj129VRJXeZBlPufwE2jVD5m9V+1d1k+MWrOhyGSvlbx D5BTIpHNn6CRJiIvi+tFKQQRZTdcC749oWIGiuNH5uShzUYPpCmKHKeRjKWge4BGBjsq FzX0TohmuWT+SFGemYc3YlKXdYycsdQIdcu9fifL5paxZ7hD6E8l4+X/94CkVcbHY7J4 zshMDgNQXrpRU456ko3tXCvoM7+u9eQfrrSiNU78sU7VexyLXIt1Quaijq1I+DUxcvk/ 0IXw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=pq6MSHG/FPqQxPMfNwdf/s7Eb60udNDy2cwn4wbvNFc=; b=yF+ENOJWkjueZ3zweBM/v70Y3zlcqyoiQFdXdRAdEFnTJhx/hRPItZ6AtnDR4cP5xy Y5fLRB89LsOTrmUBskfiRosC+WiXfgeOqMIn4zoEf3vLtPIo7Xgk5Q0rTwXuVCxusjee 4Nl9zOXeiH5vDXM0QeP1rmamHNRXKtRGP7PxaYaIbKIgkFCHU/iYqtXfjjqzshr9VaKB 8sn3rJ1HbA/e+Z/V181+yOnjL86zt0ogYSCl3Ng4QA3QjFF5r6zbHoJLoTGLqs42xAQt /OCLf+WeJ7ceIHL+htm65bLtGA953AklrFwGiSE/kD2QzRaYPb1PEyh4AO6KrTefmyyh 918A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga04.intel.com (mga04.intel.com. [192.55.52.120]) by mx.google.com with ESMTPS id q20si7678846pll.255.2019.01.16.16.33.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Jan 2019 16:33:37 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) client-ip=192.55.52.120; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2019 16:33:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,488,1539673200"; d="scan'208";a="292166010" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.79]) by orsmga005.jf.intel.com with ESMTP; 16 Jan 2019 16:33:35 -0800 From: Rick Edgecombe To: Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Borislav Petkov , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu , Rick Edgecombe Subject: [PATCH 01/17] Fix "x86/alternatives: Lockdep-enforce text_mutex in text_poke*()" Date: Wed, 16 Jan 2019 16:32:43 -0800 Message-Id: <20190117003259.23141-2-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190117003259.23141-1-rick.p.edgecombe@intel.com> References: <20190117003259.23141-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit text_mutex is currently expected to be held before text_poke() is called, but we kgdb does not take the mutex, and instead *supposedly* ensures the lock is not taken and will not be acquired by any other core while text_poke() is running. The reason for the "supposedly" comment is that it is not entirely clear that this would be the case if gdb_do_roundup is zero. This patch creates two wrapper functions, text_poke() and text_poke_kgdb() which do or do not run the lockdep assertion respectively. While we are at it, change the return code of text_poke() to something meaningful. One day, callers might actually respect it and the existing BUG_ON() when patching fails could be removed. For kgdb, the return value can actually be used. Cc: Andy Lutomirski Cc: Kees Cook Cc: Dave Hansen Cc: Masami Hiramatsu Fixes: 9222f606506c ("x86/alternatives: Lockdep-enforce text_mutex in text_poke*()") Suggested-by: Peter Zijlstra Acked-by: Jiri Kosina Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe Reviewed-by: Masami Hiramatsu --- arch/x86/include/asm/text-patching.h | 1 + arch/x86/kernel/alternative.c | 52 ++++++++++++++++++++-------- arch/x86/kernel/kgdb.c | 11 +++--- 3 files changed, 45 insertions(+), 19 deletions(-) diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h index e85ff65c43c3..f8fc8e86cf01 100644 --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -35,6 +35,7 @@ extern void *text_poke_early(void *addr, const void *opcode, size_t len); * inconsistent instruction while you patch. */ extern void *text_poke(void *addr, const void *opcode, size_t len); +extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len); extern int poke_int3_handler(struct pt_regs *regs); extern void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler); extern int after_bootmem; diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index ebeac487a20c..c6a3a10a2fd5 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -678,18 +678,7 @@ void *__init_or_module text_poke_early(void *addr, const void *opcode, return addr; } -/** - * text_poke - Update instructions on a live kernel - * @addr: address to modify - * @opcode: source of the copy - * @len: length to copy - * - * Only atomic text poke/set should be allowed when not doing early patching. - * It means the size must be writable atomically and the address must be aligned - * in a way that permits an atomic write. It also makes sure we fit on a single - * page. - */ -void *text_poke(void *addr, const void *opcode, size_t len) +static void *__text_poke(void *addr, const void *opcode, size_t len) { unsigned long flags; char *vaddr; @@ -702,8 +691,6 @@ void *text_poke(void *addr, const void *opcode, size_t len) */ BUG_ON(!after_bootmem); - lockdep_assert_held(&text_mutex); - if (!core_kernel_text((unsigned long)addr)) { pages[0] = vmalloc_to_page(addr); pages[1] = vmalloc_to_page(addr + PAGE_SIZE); @@ -732,6 +719,43 @@ void *text_poke(void *addr, const void *opcode, size_t len) return addr; } +/** + * text_poke - Update instructions on a live kernel + * @addr: address to modify + * @opcode: source of the copy + * @len: length to copy + * + * Only atomic text poke/set should be allowed when not doing early patching. + * It means the size must be writable atomically and the address must be aligned + * in a way that permits an atomic write. It also makes sure we fit on a single + * page. + */ +void *text_poke(void *addr, const void *opcode, size_t len) +{ + lockdep_assert_held(&text_mutex); + + return __text_poke(addr, opcode, len); +} + +/** + * text_poke_kgdb - Update instructions on a live kernel by kgdb + * @addr: address to modify + * @opcode: source of the copy + * @len: length to copy + * + * Only atomic text poke/set should be allowed when not doing early patching. + * It means the size must be writable atomically and the address must be aligned + * in a way that permits an atomic write. It also makes sure we fit on a single + * page. + * + * Context: should only be used by kgdb, which ensures no other core is running, + * despite the fact it does not hold the text_mutex. + */ +void *text_poke_kgdb(void *addr, const void *opcode, size_t len) +{ + return __text_poke(addr, opcode, len); +} + static void do_sync_core(void *info) { sync_core(); diff --git a/arch/x86/kernel/kgdb.c b/arch/x86/kernel/kgdb.c index 5db08425063e..1461544cba8b 100644 --- a/arch/x86/kernel/kgdb.c +++ b/arch/x86/kernel/kgdb.c @@ -758,13 +758,13 @@ int kgdb_arch_set_breakpoint(struct kgdb_bkpt *bpt) if (!err) return err; /* - * It is safe to call text_poke() because normal kernel execution + * It is safe to call text_poke_kgdb() because normal kernel execution * is stopped on all cores, so long as the text_mutex is not locked. */ if (mutex_is_locked(&text_mutex)) return -EBUSY; - text_poke((void *)bpt->bpt_addr, arch_kgdb_ops.gdb_bpt_instr, - BREAK_INSTR_SIZE); + text_poke_kgdb((void *)bpt->bpt_addr, arch_kgdb_ops.gdb_bpt_instr, + BREAK_INSTR_SIZE); err = probe_kernel_read(opc, (char *)bpt->bpt_addr, BREAK_INSTR_SIZE); if (err) return err; @@ -783,12 +783,13 @@ int kgdb_arch_remove_breakpoint(struct kgdb_bkpt *bpt) if (bpt->type != BP_POKE_BREAKPOINT) goto knl_write; /* - * It is safe to call text_poke() because normal kernel execution + * It is safe to call text_poke_kgdb() because normal kernel execution * is stopped on all cores, so long as the text_mutex is not locked. */ if (mutex_is_locked(&text_mutex)) goto knl_write; - text_poke((void *)bpt->bpt_addr, bpt->saved_instr, BREAK_INSTR_SIZE); + text_poke_kgdb((void *)bpt->bpt_addr, bpt->saved_instr, + BREAK_INSTR_SIZE); err = probe_kernel_read(opc, (char *)bpt->bpt_addr, BREAK_INSTR_SIZE); if (err || memcmp(opc, bpt->saved_instr, BREAK_INSTR_SIZE)) goto knl_write; From patchwork Thu Jan 17 00:32:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10767237 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 44E3D1390 for ; Thu, 17 Jan 2019 00:33:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 308F1284DA for ; Thu, 17 Jan 2019 00:33:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 217402FD2E; Thu, 17 Jan 2019 00:33:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A24742FD2C for ; Thu, 17 Jan 2019 00:33:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E2BE68E0008; Wed, 16 Jan 2019 19:33:39 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DE3878E0005; Wed, 16 Jan 2019 19:33:39 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 922258E0008; Wed, 16 Jan 2019 19:33:39 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id 43FF38E0003 for ; Wed, 16 Jan 2019 19:33:39 -0500 (EST) Received: by mail-pl1-f198.google.com with SMTP id l9so4939356plt.7 for ; Wed, 16 Jan 2019 16:33:39 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=riU4AUNFoiigx578aDCwA4TwOChKMPTJViD3Oo4NuoA=; b=LVaFtSOli4d4DP2h6tLIubwDMoWlOlMr24gyUgybDB3QrkPFNt5omk5eTlPn19yZGS dqEceHesJfEyqnDtfcD6jTh6kD3iB+f0bJgcwbnvgRQfkLAXWPMwCQOzKpya4GUi1w3p n6njSKa6rMJ5JirRjRFuCUTa+ajDoERjW2IQNjA0N5dznWKnGx76dfhWAdYjJt/tE3ue i2ou7a6MPmJRbwz0YGPaUdn5aEVyJum72dsN5eT64lCO+8q15bGGjLmwvOZrVdELCMO0 SSmaLfeyCOQf+ueh/9Leayt6Zsp5OWhJCZ8iRmNpO8zlO7DBYCg+Mbhj5g2mfHYtNA3y fwCQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AJcUukcnIWMHkIgMg6/49kra0VuVEJKfxxHTA7FUc7frKDsXUgu7Bzne GY2+uxR1m0gD0nNh2zumWtE1cw7uHJyRTC5kJizN6fV/VTmiJmH2IZ2+4Z4ZbHyWqFV5AO4hTwb Hgv6MdJ5ao2bx5Gaa/eq8fyvAo7qTl1bTXqTjgJFM2eKN1SSzos6MESvswE54zZ+N2w== X-Received: by 2002:a63:f201:: with SMTP id v1mr10826836pgh.232.1547685218870; Wed, 16 Jan 2019 16:33:38 -0800 (PST) X-Google-Smtp-Source: ALg8bN59uWM4vCbKk0LPCCcMKdVFzzM/V9n/hmNya9AbD4tn63K9pLVqvRGcFpfaAcybNb3XPd4J X-Received: by 2002:a63:f201:: with SMTP id v1mr10826767pgh.232.1547685217925; Wed, 16 Jan 2019 16:33:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547685217; cv=none; d=google.com; s=arc-20160816; b=FEJsGhRVe6pbvVhXF9Way3yxMh9tTbmZl37rvc78PJbkcoigE6jd6maQOkaXQCO1uE 0Nck85wD3Yfu/zts+YB/Nh6AF7mqfDdH3L+Jg30eWoeRWLs4e1zrhlg97pbkTPg3j8eR XKtDv3lwIDlT0m1E0qD+VhPGB+mGZJSQNpGN3WO9ZednVMQ4L2e7kyWmUy2TGgEMIfGU uFZU4SzSoiuH5MQULU7GTUvXxpz5yKrmB+HADq26mByfZJ90T5jRcKUeGcMZiPIY0sCb /T3xsL+dzZ5Tb/5ti1T91SXQVmVHie1AM/3YPsd/G0UuHa3E75BXhx+/SdGwOes/+bNV Ay7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=riU4AUNFoiigx578aDCwA4TwOChKMPTJViD3Oo4NuoA=; b=Z63y16PrkqJt3GWsiXRypK7naioVkAtc/DfkRMHRR7+Ng8crKjyMyXgLL5lQn369dv Bdh4LXaK3RNsQ4MG0H3sr4bPf8lyTwhdFljlNGfmoAXIJuaW3JiEtb6mwliCebbyYkmM zcszxmBvUoRgEkOG/TkwP5/bsdzJKm2dwKRLw+AGtmU3Zz7zpZ+okRgS7W+OH14l5G5s d15sHnXnZxC5Zpl0ArkbzPd3YrZ0rT51rxQmFP5JOvgRvYhOuKIHByh3HXK8jADtxYQZ 8kyRB6+PG0/NDhQnEOYJ3jVigMZt3V96wVBSZfEASz2AGNlVwLT0MH3OkgZF9mnXN0f0 j1Ew== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga04.intel.com (mga04.intel.com. [192.55.52.120]) by mx.google.com with ESMTPS id y10si7582915plt.406.2019.01.16.16.33.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Jan 2019 16:33:37 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) client-ip=192.55.52.120; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2019 16:33:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,488,1539673200"; d="scan'208";a="292166012" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.79]) by orsmga005.jf.intel.com with ESMTP; 16 Jan 2019 16:33:35 -0800 From: Rick Edgecombe To: Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Borislav Petkov , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu , Rick Edgecombe Subject: [PATCH 02/17] x86/jump_label: Use text_poke_early() during early init Date: Wed, 16 Jan 2019 16:32:44 -0800 Message-Id: <20190117003259.23141-3-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190117003259.23141-1-rick.p.edgecombe@intel.com> References: <20190117003259.23141-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit There is no apparent reason not to use text_poke_early() while we are during early-init and we do not patch code that might be on the stack (i.e., we'll return to the middle of the patched code). This appears to be the case of jump-labels, so do so. This is required for the next patches that would set a temporary mm for patching, which is initialized after some static-keys are enabled/disabled. Cc: Andy Lutomirski Cc: Kees Cook Cc: Dave Hansen Cc: Masami Hiramatsu Cc: Peter Zijlstra Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/jump_label.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c index f99bd26bd3f1..e36cfec0f35e 100644 --- a/arch/x86/kernel/jump_label.c +++ b/arch/x86/kernel/jump_label.c @@ -50,7 +50,12 @@ static void __ref __jump_label_transform(struct jump_entry *entry, jmp.offset = jump_entry_target(entry) - (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); - if (early_boot_irqs_disabled) + /* + * As long as we're UP and not yet marked RO, we can use + * text_poke_early; SYSTEM_BOOTING guarantees both, as we switch to + * SYSTEM_SCHEDULING before going either. + */ + if (system_state == SYSTEM_BOOTING) poker = text_poke_early; if (type == JUMP_LABEL_JMP) { From patchwork Thu Jan 17 00:32:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10767233 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2E61414E5 for ; Thu, 17 Jan 2019 00:33:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1ECA72FD1B for ; Thu, 17 Jan 2019 00:33:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1CE512FD2C; Thu, 17 Jan 2019 00:33:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 933C12FD35 for ; Thu, 17 Jan 2019 00:33:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1CCDA8E0002; Wed, 16 Jan 2019 19:33:39 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 155EC8E0003; Wed, 16 Jan 2019 19:33:39 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 074358E0002; Wed, 16 Jan 2019 19:33:39 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id B1AC98E0003 for ; Wed, 16 Jan 2019 19:33:38 -0500 (EST) Received: by mail-pg1-f198.google.com with SMTP id r13so5003872pgb.7 for ; Wed, 16 Jan 2019 16:33:38 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=ezRmX4cq7Q+xV7duJ1xb3mXnXLtmokQHa4CGkRg5SRM=; b=FhHkah2D464d4oOB5Q2JNFGlNwkOpIo5qvQCWUHdJ/UmtI3rXv4NaB8fDgNc72J1pb QXN82ZmLv8exKA8q157mfUotdoIkTTEM6i+haHnm1ueaWsdGge6pV8CEaicVYIUeDZ11 0Ph0mjghThbg7KEn/JClm6ETSxWzeY5P7mrMO4tRfvU5q6qwPi4FvVRO/mQw13PMEmy3 kLiLlDRunVUGs+kQwZjM5txvBZbkdoVstKzBJ7D6hGH8fmRBeitekmd7vS6WkgQMScFQ 2XNuXEEtFJNAihyEq/CdKFSIPQEkXrYrB67BY15BRrDBomt/0J7axKQAiFD00FXr/WIs WDng== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AJcUukckv2/O1wxzG566R/I6yue/97gQ6FyID+EKOnogYYruZey9jJ06 QrDNWQ+Dn1/HGdEvJpsc+sUFKUC4etpdvmNOmJ0YstpETW6ACwVnvWc5AzUo0+86Pui5F4juB0v 7a4P6ynK+CTh9O4oY7mBTPF5L8Np5wczGMX9esIG2cVB6NkwAnRLi7mJHLafEwYVmRQ== X-Received: by 2002:a63:30c8:: with SMTP id w191mr11834927pgw.120.1547685218336; Wed, 16 Jan 2019 16:33:38 -0800 (PST) X-Google-Smtp-Source: ALg8bN55lywrvWGxYvvFHt5GBMxdPQ0mwOdfFR/kwuig5Arrv6rDoFMNYa/gmPUUIqLy5eb1cOoZ X-Received: by 2002:a63:30c8:: with SMTP id w191mr11834852pgw.120.1547685217353; Wed, 16 Jan 2019 16:33:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547685217; cv=none; d=google.com; s=arc-20160816; b=ypMeh0vKX/Im3NBqD2OVIchbMh45ekMa2TU+DQQSzMpDOWtuaNEQJL539EQBphBQRH EJENobUBqnSxbCYCb0d220KA5KrJZuLUVs9A+1rNvbpKH31/u7yla0JusSNoe+NiZO0r mjONelHtzVHQB+1UYZOU09aoSQwCFdMkJH4XrjKfhlrrVeKkc9jQEMG0cK7LRwPF6ViH YeGS6xQ1dt0mtY1i1LQj21E50Gb93O+PzjkcGZxuxzqL2Ya34WP9qo1YvZ8RCp2UnvEk xCM2RBggLxt7DSyDfSKG7VP1M/sb53Q2n2mqb68yePHcFQRx5jBb96t65RlTYlwvMbI9 bB2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=ezRmX4cq7Q+xV7duJ1xb3mXnXLtmokQHa4CGkRg5SRM=; b=Pyp7NVSQrCkP6+vyCZDlflWbSRISg0HcuYjGMCKc1pXkI/S2/OuG8ylVZYJTbnLR/X 15IE362oFSIHeAq056NzyU11ZpcaxXf+RiUE+pkkt2vAZHiK8CH3AwtA2BZ6hqAshteo DEUKUkhMmefVsxuX23dyzMyxV/YBfdD+NDHUeJzCSx5GZz8mGQrrKv8HJfPkeqYPQ7NX 4adiDxYF9n5RkglRBoOwuiRmTApoF562W89aIeDGWkZel0LGyivVHo9p5U344q0chx+M ZdOehIz+rnj/Qxsj/WmLNLGegG5Bte5ekU8QrzOWiam7V4EdZI2JAB4RjlxOoeiNXS8Z ffTQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga04.intel.com (mga04.intel.com. [192.55.52.120]) by mx.google.com with ESMTPS id y10si7582915plt.406.2019.01.16.16.33.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Jan 2019 16:33:37 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) client-ip=192.55.52.120; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2019 16:33:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,488,1539673200"; d="scan'208";a="292166015" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.79]) by orsmga005.jf.intel.com with ESMTP; 16 Jan 2019 16:33:35 -0800 From: Rick Edgecombe To: Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Borislav Petkov , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Kees Cook , Dave Hansen , Nadav Amit , Rick Edgecombe Subject: [PATCH 03/17] x86/mm: temporary mm struct Date: Wed, 16 Jan 2019 16:32:45 -0800 Message-Id: <20190117003259.23141-4-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190117003259.23141-1-rick.p.edgecombe@intel.com> References: <20190117003259.23141-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Andy Lutomirski Sometimes we want to set a temporary page-table entries (PTEs) in one of the cores, without allowing other cores to use - even speculatively - these mappings. There are two benefits for doing so: (1) Security: if sensitive PTEs are set, temporary mm prevents their use in other cores. This hardens the security as it prevents exploding a dangling pointer to overwrite sensitive data using the sensitive PTE. (2) Avoiding TLB shootdowns: the PTEs do not need to be flushed in remote page-tables. To do so a temporary mm_struct can be used. Mappings which are private for this mm can be set in the userspace part of the address-space. During the whole time in which the temporary mm is loaded, interrupts must be disabled. The first use-case for temporary PTEs, which will follow, is for poking the kernel text. [ Commit message was written by Nadav ] Cc: Kees Cook Cc: Peter Zijlstra Cc: Dave Hansen Reviewed-by: Masami Hiramatsu Tested-by: Masami Hiramatsu Signed-off-by: Andy Lutomirski Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/mmu_context.h | 32 ++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 0ca50611e8ce..0141b7fa6d01 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -338,4 +338,36 @@ static inline unsigned long __get_current_cr3_fast(void) return cr3; } +typedef struct { + struct mm_struct *prev; +} temporary_mm_state_t; + +/* + * Using a temporary mm allows to set temporary mappings that are not accessible + * by other cores. Such mappings are needed to perform sensitive memory writes + * that override the kernel memory protections (e.g., W^X), without exposing the + * temporary page-table mappings that are required for these write operations to + * other cores. + * + * Context: The temporary mm needs to be used exclusively by a single core. To + * harden security IRQs must be disabled while the temporary mm is + * loaded, thereby preventing interrupt handler bugs from override the + * kernel memory protection. + */ +static inline temporary_mm_state_t use_temporary_mm(struct mm_struct *mm) +{ + temporary_mm_state_t state; + + lockdep_assert_irqs_disabled(); + state.prev = this_cpu_read(cpu_tlbstate.loaded_mm); + switch_mm_irqs_off(NULL, mm, current); + return state; +} + +static inline void unuse_temporary_mm(temporary_mm_state_t prev) +{ + lockdep_assert_irqs_disabled(); + switch_mm_irqs_off(NULL, prev.prev, current); +} + #endif /* _ASM_X86_MMU_CONTEXT_H */ From patchwork Thu Jan 17 00:32:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10767245 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C99C814E5 for ; Thu, 17 Jan 2019 00:33:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 48CEF2DDF1 for ; Thu, 17 Jan 2019 00:33:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3BF222DE41; Thu, 17 Jan 2019 00:33:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B644C2DDF1 for ; Thu, 17 Jan 2019 00:33:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 83F5D8E000A; Wed, 16 Jan 2019 19:33:40 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 68A878E000D; Wed, 16 Jan 2019 19:33:40 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2933C8E0005; Wed, 16 Jan 2019 19:33:40 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id B6F078E0007 for ; Wed, 16 Jan 2019 19:33:39 -0500 (EST) Received: by mail-pg1-f197.google.com with SMTP id o9so5002982pgv.19 for ; Wed, 16 Jan 2019 16:33:39 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=jGmZxTwCPwKrU5MJafxn96gllJHJ2dQQnbKJVURa11Y=; b=j+xoE0kkNyyIgaMVKxTBYRXMnPhWm6IJwe2FJNbY4caxTITY5dB+kiCsbbwZMHSbGL kM6Euzn53+S7Nc+mh8y6mYiFgOsa1V21Nv3yVSb1ekCLfTA9h2Hxbu5rVmteqW+VKfKn ytsMrvXCkRnVWavGUZdy6JbkRD2dGfuOHfIOFhkIH80TOG/1J0B45+qotifMU7r0v9gC VUHXh87J7hfUSmq+cl4swTP1PKVg2K5s4OY6m1JZhL6AQOHfGYZYfwq6I5zCgK36Q+si LxWxj0gOHVBJksec+iq6xLMU19YTv0QWh9xhXQEsJU2TkOIYOugzKeLUtpMSLm1GZVNb kxng== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AJcUukcpXOMsf+xerjHw+fodKuKLIhtflfpMdHqDAL/zjrhjH/ZVdPRn bf/H1dTSPfcLwq+/BwlloJ833dEwnE0rBhr4OD6qIAT0Wlc+Q/mSyqpoKOErIbA03+xwo1g52xe Sz8aCwfdYoHUI2LuO7tnxP9ZYxa/Ywgs8ZJz+64aJJNZlyjL3U4Rw9zlolceoA8qZnQ== X-Received: by 2002:a63:4948:: with SMTP id y8mr11499335pgk.32.1547685219386; Wed, 16 Jan 2019 16:33:39 -0800 (PST) X-Google-Smtp-Source: ALg8bN68OhMANrRzUMJ/heURu6SPJAUgARhtwyoKRs83qMh0IX283jYsPz1omBXg/mXDQMPj/5cP X-Received: by 2002:a63:4948:: with SMTP id y8mr11499255pgk.32.1547685218299; Wed, 16 Jan 2019 16:33:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547685218; cv=none; d=google.com; s=arc-20160816; b=PcC2GB1ZqMXqN/tzU84N+0yKtVg0vbknzBTT9L7HJW58zEhfQMveP62qYbN+zDTR0j WxbW+Q+kok0fRfKLbJj7xcQ+iVottCTebUdeTWDMtiaTpJCVa9BeNxQEKFs//pyjnLFu FzF2mr9JPV9+yDAMmv+0J+0boO1SrNADbFL6W21A5+ZZ3mECsymPtc+OPWgPXcm40ypp O6hk2UaI0L/NZp5Q41swuzWhgVZzq/vO8W0i6j/tom+j1b2CbnYlsLfwI0K6pnmyq/9n 91r2fejMsChCEkhQIOC1aI/iPqbR+xNMMqnZv5pcHU1ZnBKwT4E2cl+Cufn/Xy9cJGDn 7TBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=jGmZxTwCPwKrU5MJafxn96gllJHJ2dQQnbKJVURa11Y=; b=fjrAarnS2gaU33M9mqtGiKedKF5L9GUKeNUn+5I7BpI60ShsPA2fhzm8Xpi+GDo6Hl uWHK6IhcMg8jHxWaQd9+4QkEE8iOS51fpFqM6LKOk78H8+fK8W0v16uX+hfPrQQzl119 m4WlEvbc0lAXgGmljHwzW1FgWFsX5edxTUvjROc2X3oXNEUeos0Vc77Ta/4q6q28Z7WJ HVHaVQe8vxKOKKSHdP6WFPjQEqySx3DpJyWrR6MXMRLb4p1iuTi6cZrqKCWH0fET0GTJ S6sMDbEv9kEvVyEhssbN2x64HEqwp4ioVZ0Q+U2D7puvL0jGA/jwozy519SmOaWusugQ +QUw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga04.intel.com (mga04.intel.com. [192.55.52.120]) by mx.google.com with ESMTPS id q20si7678846pll.255.2019.01.16.16.33.38 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Jan 2019 16:33:38 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) client-ip=192.55.52.120; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2019 16:33:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,488,1539673200"; d="scan'208";a="292166019" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.79]) by orsmga005.jf.intel.com with ESMTP; 16 Jan 2019 16:33:35 -0800 From: Rick Edgecombe To: Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Borislav Petkov , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Kees Cook , Dave Hansen , Rick Edgecombe Subject: [PATCH 04/17] fork: provide a function for copying init_mm Date: Wed, 16 Jan 2019 16:32:46 -0800 Message-Id: <20190117003259.23141-5-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190117003259.23141-1-rick.p.edgecombe@intel.com> References: <20190117003259.23141-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit Provide a function for copying init_mm. This function will be later used for setting a temporary mm. Cc: Andy Lutomirski Cc: Kees Cook Cc: Peter Zijlstra Cc: Dave Hansen Reviewed-by: Masami Hiramatsu Tested-by: Masami Hiramatsu Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- include/linux/sched/task.h | 1 + kernel/fork.c | 24 ++++++++++++++++++------ 2 files changed, 19 insertions(+), 6 deletions(-) diff --git a/include/linux/sched/task.h b/include/linux/sched/task.h index 44c6f15800ff..c5a00a7b3beb 100644 --- a/include/linux/sched/task.h +++ b/include/linux/sched/task.h @@ -76,6 +76,7 @@ extern void exit_itimers(struct signal_struct *); extern long _do_fork(unsigned long, unsigned long, unsigned long, int __user *, int __user *, unsigned long); extern long do_fork(unsigned long, unsigned long, unsigned long, int __user *, int __user *); struct task_struct *fork_idle(int); +struct mm_struct *copy_init_mm(void); extern pid_t kernel_thread(int (*fn)(void *), void *arg, unsigned long flags); extern long kernel_wait4(pid_t, int __user *, int, struct rusage *); diff --git a/kernel/fork.c b/kernel/fork.c index b69248e6f0e0..d7b156c49f29 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1299,13 +1299,20 @@ void mm_release(struct task_struct *tsk, struct mm_struct *mm) complete_vfork_done(tsk); } -/* - * Allocate a new mm structure and copy contents from the - * mm structure of the passed in task structure. +/** + * dup_mm() - duplicates an existing mm structure + * @tsk: the task_struct with which the new mm will be associated. + * @oldmm: the mm to duplicate. + * + * Allocates a new mm structure and copy contents from the provided + * @oldmm structure. + * + * Return: the duplicated mm or NULL on failure. */ -static struct mm_struct *dup_mm(struct task_struct *tsk) +static struct mm_struct *dup_mm(struct task_struct *tsk, + struct mm_struct *oldmm) { - struct mm_struct *mm, *oldmm = current->mm; + struct mm_struct *mm; int err; mm = allocate_mm(); @@ -1372,7 +1379,7 @@ static int copy_mm(unsigned long clone_flags, struct task_struct *tsk) } retval = -ENOMEM; - mm = dup_mm(tsk); + mm = dup_mm(tsk, current->mm); if (!mm) goto fail_nomem; @@ -2187,6 +2194,11 @@ struct task_struct *fork_idle(int cpu) return task; } +struct mm_struct *copy_init_mm(void) +{ + return dup_mm(NULL, &init_mm); +} + /* * Ok, this is the main fork-routine. * From patchwork Thu Jan 17 00:32:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10767247 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1D78F14E5 for ; Thu, 17 Jan 2019 00:34:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BAD0F2DDDC for ; Thu, 17 Jan 2019 00:33:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AB3FE2DE41; Thu, 17 Jan 2019 00:33:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F0D8A2DDDC for ; Thu, 17 Jan 2019 00:33:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ABC0C8E0009; Wed, 16 Jan 2019 19:33:40 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A329E8E000E; Wed, 16 Jan 2019 19:33:40 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D9648E0009; Wed, 16 Jan 2019 19:33:40 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id C40788E0009 for ; Wed, 16 Jan 2019 19:33:39 -0500 (EST) Received: by mail-pl1-f199.google.com with SMTP id l9so4939369plt.7 for ; Wed, 16 Jan 2019 16:33:39 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=TldYzqiExwf3rqiOIlhUUDyqJwbGTn6e1zIomGztBUw=; b=Fk8BO0FfpllBA1sTs5Gh8A/jq4Cmd2mJta/GRVxrZA2JV5rY8lIhqoJAGpcX0yPcbj 5/BmGOhjikO/bDqfhbCTsBHVx+EhBLKe5Cyt6b2PusQWXfHkEOBoB5zx5L089HbpEK4R 9Tm2nXtuPTM5iYSfVijK9HZcijCkC5Xno32bt9EpDZDsJEdw6lHupFXI6VUz4l5wtlq6 HfooqiNKMvJ5L+evH/zHUKDRfKtKF2VlE++MYyvu91L9yK2wZ1tn7K1YbQRC2iiK22DX bE7RRpg2PGbCJTSOyKE5wgZ+ZV1q5pRDGnf0Wtwt53nSxVpiaXBiOeyGYVscn9pymjmI vqSg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AJcUukcfDoXgaihJJmISPu2pSl/gV+xflU33ryLRWe6jIe+aUCYmBRlF aDkGtThNTArU7ZvBVrQEMNh9r6P5K8nUB3UZ3h8LFxxQEvWY+usPyk4C6IWOwXGhzh23Ad9viaM jqYMt8C36l0g3Wx9tPaCWAkuWKiLswIbzIi5JgcvmTKx9xuYoF7QgdRIK7CBEkne69w== X-Received: by 2002:a17:902:6bc9:: with SMTP id m9mr12616635plt.173.1547685219433; Wed, 16 Jan 2019 16:33:39 -0800 (PST) X-Google-Smtp-Source: ALg8bN7axEiX/+vD1RpMbvCmgfQqLa5KmKsKfNDpFDl/P0WAs9HqMYOm9z65fEUb6ypIST/l3+iZ X-Received: by 2002:a17:902:6bc9:: with SMTP id m9mr12616547plt.173.1547685218369; Wed, 16 Jan 2019 16:33:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547685218; cv=none; d=google.com; s=arc-20160816; b=TievufwodFzaytaZV+FItuyfHLQuywFjMF+BU3qxRmRNbw05czw9kkBEEK02WmYoSf pwmjsY0c+aLtLyzI9wPliIbh/HPDYGVgKqlkiBK6puEwfW3qSeoIONtuLK97rLo+Bn1x sstoG19JvDPfrSQgd6iadi2gj2AIrWDY8A40SmvvCgp6itjfmYDaqrRVvgFKJlz9jc2b /JqCb9CeD2T+qjTN4Lpn/krwgD9RjyyPz+PoqakmDEUxQVM8lmlRbE0qjHwLwoQiAkaS zg9uhm8O1sxw8n0Xs9wpkBDhq87/XON6X9D4laEaoazgaBjj4W6pLLRTkC2PdB8eRk0r ddbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=TldYzqiExwf3rqiOIlhUUDyqJwbGTn6e1zIomGztBUw=; b=0YuQNeoJsIJwvcNfJviZreYL50QSwGwojDx5EURNceI79mokZwmXoLlkzrnAg5pZRF sywJNtL2gjq9H8sWjOkLUPUhmygS2tlXQYJubN0adML7AZ65YsOInWfPlVjdzVYsU+Zd E2CHWPAyLoY7QX+zCiEUlZ0JmcCxFVg2JhJvGd6IykG6LuyJHatS8knNw8VDA2PDmloX nl4D8XMNh5HSbSvU0CHqLFleI1C6Ka+B2obp0iS34laYuU3j8kXQA3NxYQkYAbSxyWnQ NAzWjgvtBIN8Dm6bRY5R9rtmJwd0lBZj041Wkh5fQUoYIcJfgY/mIlz05Qg0xzkHGJ/J dfSg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga04.intel.com (mga04.intel.com. [192.55.52.120]) by mx.google.com with ESMTPS id y10si7582915plt.406.2019.01.16.16.33.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Jan 2019 16:33:38 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) client-ip=192.55.52.120; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2019 16:33:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,488,1539673200"; d="scan'208";a="292166022" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.79]) by orsmga005.jf.intel.com with ESMTP; 16 Jan 2019 16:33:35 -0800 From: Rick Edgecombe To: Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Borislav Petkov , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Kees Cook , Dave Hansen , Rick Edgecombe Subject: [PATCH 05/17] x86/alternative: initializing temporary mm for patching Date: Wed, 16 Jan 2019 16:32:47 -0800 Message-Id: <20190117003259.23141-6-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190117003259.23141-1-rick.p.edgecombe@intel.com> References: <20190117003259.23141-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit To prevent improper use of the PTEs that are used for text patching, we want to use a temporary mm struct. We initailize it by copying the init mm. The address that will be used for patching is taken from the lower area that is usually used for the task memory. Doing so prevents the need to frequently synchronize the temporary-mm (e.g., when BPF programs are installed), since different PGDs are used for the task memory. Finally, we randomize the address of the PTEs to harden against exploits that use these PTEs. Cc: Kees Cook Cc: Peter Zijlstra Cc: Dave Hansen Reviewed-by: Masami Hiramatsu Tested-by: Masami Hiramatsu Suggested-by: Andy Lutomirski Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/pgtable.h | 3 +++ arch/x86/include/asm/text-patching.h | 2 ++ arch/x86/kernel/alternative.c | 3 +++ arch/x86/mm/init_64.c | 36 ++++++++++++++++++++++++++++ init/main.c | 3 +++ 5 files changed, 47 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 40616e805292..e8f630d9a2ed 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1021,6 +1021,9 @@ static inline void __meminit init_trampoline_default(void) /* Default trampoline pgd value */ trampoline_pgd_entry = init_top_pgt[pgd_index(__PAGE_OFFSET)]; } + +void __init poking_init(void); + # ifdef CONFIG_RANDOMIZE_MEMORY void __meminit init_trampoline(void); # else diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h index f8fc8e86cf01..a75eed841eed 100644 --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -39,5 +39,7 @@ extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len); extern int poke_int3_handler(struct pt_regs *regs); extern void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler); extern int after_bootmem; +extern __ro_after_init struct mm_struct *poking_mm; +extern __ro_after_init unsigned long poking_addr; #endif /* _ASM_X86_TEXT_PATCHING_H */ diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index c6a3a10a2fd5..57fdde308bb6 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -678,6 +678,9 @@ void *__init_or_module text_poke_early(void *addr, const void *opcode, return addr; } +__ro_after_init struct mm_struct *poking_mm; +__ro_after_init unsigned long poking_addr; + static void *__text_poke(void *addr, const void *opcode, size_t len) { unsigned long flags; diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index bccff68e3267..125c8c48aa24 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -53,6 +53,7 @@ #include #include #include +#include #include "mm_internal.h" @@ -1383,6 +1384,41 @@ unsigned long memory_block_size_bytes(void) return memory_block_size_probed; } +/* + * Initialize an mm_struct to be used during poking and a pointer to be used + * during patching. + */ +void __init poking_init(void) +{ + spinlock_t *ptl; + pte_t *ptep; + + poking_mm = copy_init_mm(); + BUG_ON(!poking_mm); + + /* + * Randomize the poking address, but make sure that the following page + * will be mapped at the same PMD. We need 2 pages, so find space for 3, + * and adjust the address if the PMD ends after the first one. + */ + poking_addr = TASK_UNMAPPED_BASE; + if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) + poking_addr += (kaslr_get_random_long("Poking") & PAGE_MASK) % + (TASK_SIZE - TASK_UNMAPPED_BASE - 3 * PAGE_SIZE); + + if (((poking_addr + PAGE_SIZE) & ~PMD_MASK) == 0) + poking_addr += PAGE_SIZE; + + /* + * We need to trigger the allocation of the page-tables that will be + * needed for poking now. Later, poking may be performed in an atomic + * section, which might cause allocation to fail. + */ + ptep = get_locked_pte(poking_mm, poking_addr, &ptl); + BUG_ON(!ptep); + pte_unmap_unlock(ptep, ptl); +} + #ifdef CONFIG_SPARSEMEM_VMEMMAP /* * Initialise the sparsemem vmemmap using huge-pages at the PMD level. diff --git a/init/main.c b/init/main.c index e2e80ca3165a..f5947ba53bb4 100644 --- a/init/main.c +++ b/init/main.c @@ -496,6 +496,8 @@ void __init __weak thread_stack_cache_init(void) void __init __weak mem_encrypt_init(void) { } +void __init __weak poking_init(void) { } + bool initcall_debug; core_param(initcall_debug, initcall_debug, bool, 0644); @@ -730,6 +732,7 @@ asmlinkage __visible void __init start_kernel(void) taskstats_init_early(); delayacct_init(); + poking_init(); check_bugs(); acpi_subsystem_init(); From patchwork Thu Jan 17 00:32:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10767255 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6C8B21390 for ; Thu, 17 Jan 2019 00:34:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8877C2DDDC for ; Thu, 17 Jan 2019 00:34:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7BA572DE41; Thu, 17 Jan 2019 00:34:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B8D2F2DDDC for ; Thu, 17 Jan 2019 00:34:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 177988E000D; Wed, 16 Jan 2019 19:33:41 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DDE448E0010; Wed, 16 Jan 2019 19:33:40 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 926928E0006; Wed, 16 Jan 2019 19:33:40 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 3B6768E0006 for ; Wed, 16 Jan 2019 19:33:40 -0500 (EST) Received: by mail-pg1-f200.google.com with SMTP id o17so5009241pgi.14 for ; Wed, 16 Jan 2019 16:33:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=rZi1tUrX4fiQHdOgLSpxZC/PkiG3Sfgl7iH9TJ92CQs=; b=KhmTAnlRQX+YfTYyD0JhcESxL3iFSyAwFVs11xMH9DlSS05vJ8BHlxseGZuZ7EKrer xLyu+Q3imOKcOI6F3B+n9pP3msynSf/sC/akTlXJ/99kYUlbIz+ffJCeiNsnddIeDhfM cubuGQi60oOlGXazBOLQODZ5JkwkXDOUCxSEkM+wxq3YhFfvvtur3TYYfbu1gqhdBfTo bk0RgkQsFn9eVIH4g6FheH6akFUc7e7P87AtUH+c7Wn+jLhLPuzVkiRjOsZrBrw+g1rw 5NUFGWOCordSS7iXNLbBUiUDht3grZat5pay9j4EqR8GdlBD/ZN5ZbArlH9yU3YdVfHR r8CA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AJcUukegiXFLgODFZUKb4ui+nG6mIX2Cxzh+wcbonDLgWcrcyQA7SCkr VJvoEM8m4VNoVRwgapD1Aiy3G3hbaI8e7g0VwY03ssSpCoU7T9XHI4xKnE5c2dRwl+RuZ2FjQE+ sMDEAVSMkDiAMnq10CrgGYeRuHN4WNxV07X7XlkqBIyV9CAmKY4F0kfBnlZlayaxMCg== X-Received: by 2002:a17:902:8d94:: with SMTP id v20mr12639480plo.194.1547685219816; Wed, 16 Jan 2019 16:33:39 -0800 (PST) X-Google-Smtp-Source: ALg8bN7dwI4pLqvDe2nARJD1mdmCi+phZtH6J3PXMDiiK9ZTDDJ3vC7fVtyUwX4t0kmbrqv6hA3y X-Received: by 2002:a17:902:8d94:: with SMTP id v20mr12639397plo.194.1547685218803; Wed, 16 Jan 2019 16:33:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547685218; cv=none; d=google.com; s=arc-20160816; b=TYuBTMLechwvIfG8shFi+QKpeOhN3YNxkJkTVE64PrcyH/ngA8zefjE9L4KylXtlUj FbPJaeyhfwTz9sESKc7TvC9hElQueFn5BUCtq47iNsvHQG+jtW2MKX50vl2ybhcFQhgy vz+TNcI4q7vqmKLvDEfvWipZQHm2076N5a4SpBqcAQ+zW/nzQd58pPJoFnfxZWCwOk5B dwN3HjpHLx0B8CoyqMtTU/knpkxwEHu8iq9yYPAktPqQPTwYZRWLvn8JYaTvk7NPFqx5 Bv6ZNDS5cuXmwfAUf7l1W+Yo3h0zqYqbTGyY+AD1YNyqHjrwVd2HQVFTOOXt9nJ8mvjb kWOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=rZi1tUrX4fiQHdOgLSpxZC/PkiG3Sfgl7iH9TJ92CQs=; b=iyAKh0oUyUqztnPaWXEbKzH3Bk73ABMyBBQmSJXlqUq/QDneJc73Uhky4/Y9Csun/B 1bKhHjIeGS8L1Q85LDMVy17HGtopISfLIHmuH5Hdq7FI6rHgUo2zLqF2XiewZyGEtj7f Z+hnndyCh3VrGQyRRGlADcQ5Qv9OZJaKursbwYXU9TQMAeM9T6RutV2sqrhPv59/dS0K FjLJhb8lJYPWRyGxkbvLsoKRHc9vO6UuISPK5qKVvBtS/UeYvAykbzdgTpdFw0CRdWmD 8SzAajgmYqdcXbjm2h7XBaFaaBvVPl2l5lVTji4WaKKteVYnJ6Jt81Lzp2rNUN+kz8Ui suEA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga04.intel.com (mga04.intel.com. [192.55.52.120]) by mx.google.com with ESMTPS id q20si7678846pll.255.2019.01.16.16.33.38 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Jan 2019 16:33:38 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) client-ip=192.55.52.120; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2019 16:33:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,488,1539673200"; d="scan'208";a="292166027" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.79]) by orsmga005.jf.intel.com with ESMTP; 16 Jan 2019 16:33:35 -0800 From: Rick Edgecombe To: Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Borislav Petkov , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu , Rick Edgecombe Subject: [PATCH 06/17] x86/alternative: use temporary mm for text poking Date: Wed, 16 Jan 2019 16:32:48 -0800 Message-Id: <20190117003259.23141-7-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190117003259.23141-1-rick.p.edgecombe@intel.com> References: <20190117003259.23141-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit text_poke() can potentially compromise the security as it sets temporary PTEs in the fixmap. These PTEs might be used to rewrite the kernel code from other cores accidentally or maliciously, if an attacker gains the ability to write onto kernel memory. Moreover, since remote TLBs are not flushed after the temporary PTEs are removed, the time-window in which the code is writable is not limited if the fixmap PTEs - maliciously or accidentally - are cached in the TLB. To address these potential security hazards, we use a temporary mm for patching the code. Finally, text_poke() is also not conservative enough when mapping pages, as it always tries to map 2 pages, even when a single one is sufficient. So try to be more conservative, and do not map more than needed. Cc: Andy Lutomirski Cc: Kees Cook Cc: Peter Zijlstra Cc: Dave Hansen Cc: Masami Hiramatsu Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/fixmap.h | 2 - arch/x86/kernel/alternative.c | 109 +++++++++++++++++++++++++++------- arch/x86/xen/mmu_pv.c | 2 - 3 files changed, 87 insertions(+), 26 deletions(-) diff --git a/arch/x86/include/asm/fixmap.h b/arch/x86/include/asm/fixmap.h index 50ba74a34a37..9da8cccdf3fb 100644 --- a/arch/x86/include/asm/fixmap.h +++ b/arch/x86/include/asm/fixmap.h @@ -103,8 +103,6 @@ enum fixed_addresses { #ifdef CONFIG_PARAVIRT FIX_PARAVIRT_BOOTMAP, #endif - FIX_TEXT_POKE1, /* reserve 2 pages for text_poke() */ - FIX_TEXT_POKE0, /* first page is last, because allocation is backward */ #ifdef CONFIG_X86_INTEL_MID FIX_LNW_VRTC, #endif diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 57fdde308bb6..8fc4685f3117 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -683,41 +684,105 @@ __ro_after_init unsigned long poking_addr; static void *__text_poke(void *addr, const void *opcode, size_t len) { + bool cross_page_boundary = offset_in_page(addr) + len > PAGE_SIZE; + temporary_mm_state_t prev; + struct page *pages[2] = {NULL}; unsigned long flags; - char *vaddr; - struct page *pages[2]; - int i; + pte_t pte, *ptep; + spinlock_t *ptl; /* - * While boot memory allocator is runnig we cannot use struct - * pages as they are not yet initialized. + * While boot memory allocator is running we cannot use struct pages as + * they are not yet initialized. */ BUG_ON(!after_bootmem); if (!core_kernel_text((unsigned long)addr)) { pages[0] = vmalloc_to_page(addr); - pages[1] = vmalloc_to_page(addr + PAGE_SIZE); + if (cross_page_boundary) + pages[1] = vmalloc_to_page(addr + PAGE_SIZE); } else { pages[0] = virt_to_page(addr); WARN_ON(!PageReserved(pages[0])); - pages[1] = virt_to_page(addr + PAGE_SIZE); + if (cross_page_boundary) + pages[1] = virt_to_page(addr + PAGE_SIZE); } - BUG_ON(!pages[0]); + BUG_ON(!pages[0] || (cross_page_boundary && !pages[1])); + local_irq_save(flags); - set_fixmap(FIX_TEXT_POKE0, page_to_phys(pages[0])); - if (pages[1]) - set_fixmap(FIX_TEXT_POKE1, page_to_phys(pages[1])); - vaddr = (char *)fix_to_virt(FIX_TEXT_POKE0); - memcpy(&vaddr[(unsigned long)addr & ~PAGE_MASK], opcode, len); - clear_fixmap(FIX_TEXT_POKE0); - if (pages[1]) - clear_fixmap(FIX_TEXT_POKE1); - local_flush_tlb(); - sync_core(); - /* Could also do a CLFLUSH here to speed up CPU recovery; but - that causes hangs on some VIA CPUs. */ - for (i = 0; i < len; i++) - BUG_ON(((char *)addr)[i] != ((char *)opcode)[i]); + + /* + * The lock is not really needed, but this allows to avoid open-coding. + */ + ptep = get_locked_pte(poking_mm, poking_addr, &ptl); + + /* + * This must not fail; preallocated in poking_init(). + */ + VM_BUG_ON(!ptep); + + pte = mk_pte(pages[0], PAGE_KERNEL); + set_pte_at(poking_mm, poking_addr, ptep, pte); + + if (cross_page_boundary) { + pte = mk_pte(pages[1], PAGE_KERNEL); + set_pte_at(poking_mm, poking_addr + PAGE_SIZE, ptep + 1, pte); + } + + /* + * Loading the temporary mm behaves as a compiler barrier, which + * guarantees that the PTE will be set at the time memcpy() is done. + */ + prev = use_temporary_mm(poking_mm); + + kasan_disable_current(); + memcpy((u8 *)poking_addr + offset_in_page(addr), opcode, len); + kasan_enable_current(); + + /* + * Ensure that the PTE is only cleared after the instructions of memcpy + * were issued by using a compiler barrier. + */ + barrier(); + + pte_clear(poking_mm, poking_addr, ptep); + + /* + * __flush_tlb_one_user() performs a redundant TLB flush when PTI is on, + * as it also flushes the corresponding "user" address spaces, which + * does not exist. + * + * Poking, however, is already very inefficient since it does not try to + * batch updates, so we ignore this problem for the time being. + * + * Since the PTEs do not exist in other kernel address-spaces, we do + * not use __flush_tlb_one_kernel(), which when PTI is on would cause + * more unwarranted TLB flushes. + * + * There is a slight anomaly here: the PTE is a supervisor-only and + * (potentially) global and we use __flush_tlb_one_user() but this + * should be fine. + */ + __flush_tlb_one_user(poking_addr); + if (cross_page_boundary) { + pte_clear(poking_mm, poking_addr + PAGE_SIZE, ptep + 1); + __flush_tlb_one_user(poking_addr + PAGE_SIZE); + } + + /* + * Loading the previous page-table hierarchy requires a serializing + * instruction that already allows the core to see the updated version. + * Xen-PV is assumed to serialize execution in a similar manner. + */ + unuse_temporary_mm(prev); + + pte_unmap_unlock(ptep, ptl); + /* + * If the text doesn't match what we just wrote; something is + * fundamentally screwy, there's nothing we can really do about that. + */ + BUG_ON(memcmp(addr, opcode, len)); + local_irq_restore(flags); return addr; } diff --git a/arch/x86/xen/mmu_pv.c b/arch/x86/xen/mmu_pv.c index 0f4fe206dcc2..82b181fcefe5 100644 --- a/arch/x86/xen/mmu_pv.c +++ b/arch/x86/xen/mmu_pv.c @@ -2319,8 +2319,6 @@ static void xen_set_fixmap(unsigned idx, phys_addr_t phys, pgprot_t prot) #elif defined(CONFIG_X86_VSYSCALL_EMULATION) case VSYSCALL_PAGE: #endif - case FIX_TEXT_POKE0: - case FIX_TEXT_POKE1: /* All local page mappings */ pte = pfn_pte(phys, prot); break; From patchwork Thu Jan 17 00:32:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10767257 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 93A8F1390 for ; Thu, 17 Jan 2019 00:34:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C25782DDDC for ; Thu, 17 Jan 2019 00:34:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B5F302DE41; Thu, 17 Jan 2019 00:34:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 451BA2DDDC for ; Thu, 17 Jan 2019 00:34:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B6968E0011; Wed, 16 Jan 2019 19:33:41 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3AA678E0005; Wed, 16 Jan 2019 19:33:41 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E9D578E0011; Wed, 16 Jan 2019 19:33:40 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id 99AEB8E000B for ; Wed, 16 Jan 2019 19:33:40 -0500 (EST) Received: by mail-pl1-f197.google.com with SMTP id j8so4969372plb.1 for ; Wed, 16 Jan 2019 16:33:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=ZRt3/09bTA5lgrgBgk9vlq5yB4N7KtlQ5lx0rCSFIjo=; b=e+9UFKPBIt6Xsx7MUHzdKBLFr5JCPZCni7N0Vxgvkmb5YEcoqjmpM40Kfnn6xxjEHd oVkOUpkkQwNv89HjnXdPfGu1j+9YhvX5Ir2VTFMuNcmYvvfH167OMPiTX5NCE2IrDKHA JMrpeGivpk5flByV57x9NnBct8QwpU2qYM4NLR9xbEfo68W/cUNqZf+Th2rtLCpeYjur tE68TTJ0Oyq0ufuC0a6KRTgQVaWRP3/Zu5UZYUx4fyxh84+tTflEO/qd4YMDcJEDNTLi pKTu6NituP/sMHsE7++bcYQi2pI6gjdiWjO+nbZU+UdtsoljhIyP/ylZe0MH2GF/Cnzv cVCQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AJcUukfuDZ3+tvXvok+UuM9IoebBbFaE4wq63FmnUAMhnvi+U7F8ks3Y 4/c51DpfQz9xpMO1EAlUArzuqLDqzoJmr8sJN0wKKLe9jbvJZJjO2qeiAIuHK7v8oRCbPx3qnT6 yuogzSJyRaH/QBixAsvPg+8KPKYKU6/xMMIFqjh8uKiV/059SH/ILd0lSHcAuPTXnbA== X-Received: by 2002:a17:902:2e01:: with SMTP id q1mr12467675plb.97.1547685220263; Wed, 16 Jan 2019 16:33:40 -0800 (PST) X-Google-Smtp-Source: ALg8bN6nDxkwO6KFb92waV3hj9/QIWOFisa3+XDBnu/y8L3qT/zeUZV5sGqSFUpQRYiG2YFGp/nQ X-Received: by 2002:a17:902:2e01:: with SMTP id q1mr12467575plb.97.1547685219078; Wed, 16 Jan 2019 16:33:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547685219; cv=none; d=google.com; s=arc-20160816; b=xcP8tEwUZvhYPFeBK3b+G4KH3ME24gzlGpsAdOTeQO8hg3/TVSEvYqJHJBFwg9IfrI QPvr3swKFwJcu9LEnYGIAfV9Byb+A7M9UdBNlb1iz6p4s2gaMn5YduvqC2LbdogbnvVd LkAlBzLaQ6pyxaKufFXvlj78fr4LnufGUxOSUmn4dbzq9KMIPSTBLvNTHeMsvENLyfVX hMhuXH7K19g1aeoJD/5iGBCoEJw/QoqlkObZOKJhggfYyczNNFiR2rb2o3EhC1mFvGPf q5FFv5Gsn/GgeS65RkwCjnNF1wwHOBWDyiMbYfyDupxH7rve0UVzvokcLJ3+tT/jnAgM iQcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=ZRt3/09bTA5lgrgBgk9vlq5yB4N7KtlQ5lx0rCSFIjo=; b=DzE/kir9C4My5PBkmkYYGLnk/7AhAz1nxZ6J+yZ7OL6pfMbsRNCjuVcdpHV1lF07U9 qvifJsqKbobnPP+98JWu6fDpIVs2YygVCm1SCT8OKofS1KPurf/Ly6R6bW8w07mQbUSw bHzs0IuWYlslli24PWSIaPK1BAd5b8rRVk/TVNBtenV6Gd2CUaqdvQFkSuxqQNlHjw25 gk9Dq+ZkjCoaGwcbi8X2Mi8syPLq40vkKjzYwW2wRGKaGwFqm0sMOX1JfWzNLJR9QztG C1w+OkVX9XF8qHUkmlwE4EA4694vCkvXSuiyaJjPdAlGivbuOBIJLlFFg6QTsY24mF9V EGIg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga04.intel.com (mga04.intel.com. [192.55.52.120]) by mx.google.com with ESMTPS id y10si7582915plt.406.2019.01.16.16.33.38 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Jan 2019 16:33:39 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) client-ip=192.55.52.120; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2019 16:33:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,488,1539673200"; d="scan'208";a="292166031" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.79]) by orsmga005.jf.intel.com with ESMTP; 16 Jan 2019 16:33:35 -0800 From: Rick Edgecombe To: Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Borislav Petkov , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Rick Edgecombe Subject: [PATCH 07/17] x86/kgdb: avoid redundant comparison of patched code Date: Wed, 16 Jan 2019 16:32:49 -0800 Message-Id: <20190117003259.23141-8-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190117003259.23141-1-rick.p.edgecombe@intel.com> References: <20190117003259.23141-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit text_poke() already ensures that the written value is the correct one and fails if that is not the case. There is no need for an additional comparison. Remove it. Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/kgdb.c | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/arch/x86/kernel/kgdb.c b/arch/x86/kernel/kgdb.c index 1461544cba8b..057af9187a04 100644 --- a/arch/x86/kernel/kgdb.c +++ b/arch/x86/kernel/kgdb.c @@ -746,7 +746,6 @@ void kgdb_arch_set_pc(struct pt_regs *regs, unsigned long ip) int kgdb_arch_set_breakpoint(struct kgdb_bkpt *bpt) { int err; - char opc[BREAK_INSTR_SIZE]; bpt->type = BP_BREAKPOINT; err = probe_kernel_read(bpt->saved_instr, (char *)bpt->bpt_addr, @@ -765,11 +764,6 @@ int kgdb_arch_set_breakpoint(struct kgdb_bkpt *bpt) return -EBUSY; text_poke_kgdb((void *)bpt->bpt_addr, arch_kgdb_ops.gdb_bpt_instr, BREAK_INSTR_SIZE); - err = probe_kernel_read(opc, (char *)bpt->bpt_addr, BREAK_INSTR_SIZE); - if (err) - return err; - if (memcmp(opc, arch_kgdb_ops.gdb_bpt_instr, BREAK_INSTR_SIZE)) - return -EINVAL; bpt->type = BP_POKE_BREAKPOINT; return err; @@ -777,9 +771,6 @@ int kgdb_arch_set_breakpoint(struct kgdb_bkpt *bpt) int kgdb_arch_remove_breakpoint(struct kgdb_bkpt *bpt) { - int err; - char opc[BREAK_INSTR_SIZE]; - if (bpt->type != BP_POKE_BREAKPOINT) goto knl_write; /* @@ -790,10 +781,7 @@ int kgdb_arch_remove_breakpoint(struct kgdb_bkpt *bpt) goto knl_write; text_poke_kgdb((void *)bpt->bpt_addr, bpt->saved_instr, BREAK_INSTR_SIZE); - err = probe_kernel_read(opc, (char *)bpt->bpt_addr, BREAK_INSTR_SIZE); - if (err || memcmp(opc, bpt->saved_instr, BREAK_INSTR_SIZE)) - goto knl_write; - return err; + return 0; knl_write: return probe_kernel_write((char *)bpt->bpt_addr, From patchwork Thu Jan 17 00:32:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10767319 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E28426C2 for ; Thu, 17 Jan 2019 00:35:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 523B62E297 for ; Thu, 17 Jan 2019 00:35:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 424A32DDDC; Thu, 17 Jan 2019 00:35:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C80E72DDDC for ; Thu, 17 Jan 2019 00:35:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B42018E0003; Wed, 16 Jan 2019 19:35:05 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B18808E0002; Wed, 16 Jan 2019 19:35:05 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A30F38E0003; Wed, 16 Jan 2019 19:35:05 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 649098E0002 for ; Wed, 16 Jan 2019 19:35:05 -0500 (EST) Received: by mail-pg1-f198.google.com with SMTP id r16so5013956pgr.15 for ; Wed, 16 Jan 2019 16:35:05 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=/2dO2C+UYfBQiXTmhGySAi6PBFtX+ameLwxuka7ZeCk=; b=lgBjG0aFI6opQekB8jrM15xzJFm2yTkMc9MQM8Ykc2ncJQKdVP23OMmxIxYoAZaz/i nSRQm3O5tyhKT9nLfhN3TY9NFdtJzJ59wmw1acgffvpjvKLKIVdkLX2buZCYZvU06icJ rFhFH6VfOb04Nt9Zoy1Fe8gkY7T2IGa0E1MCuMof92YjZpHK/JiBtUIpDBZnawsFpb5K I+uykLBX+Ug4AqIqR0YXt3vafcmCpHTwr1iXiuMqVMVplzWH2wMpchEAi+NnQD4f+pxZ NeHUdqJMM63j7rC4AYd7X1zE5bx4B9EIoVNdhdJX4WHSyIo4j5t9iwy5P4UI2duJw5ux ZTAg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AJcUukdOqXR4oy0UfI+s75ta8uHQh1InYUfsA9J8OEevixyQjldJ/N6I XkMJLpJQl6Nkv06cQEQtpZOylhukKaAWVRg+UtX5/B7ANmQ97TlYtdVrRGya2LQL775CY2mYBPa b1q6UFjewEG9OtklTwFNfc+0OuFRFVYeK1lHE0cuU/1cTzQyeaMnfrUhpe0H5C47dTQ== X-Received: by 2002:a17:902:442:: with SMTP id 60mr12227521ple.73.1547685305078; Wed, 16 Jan 2019 16:35:05 -0800 (PST) X-Google-Smtp-Source: ALg8bN6EZx2l/Jws836j7q4BB625fzzM3tH+dbw7dR1DPbmMrdp6NGme/zI/aom2uuZgoyxyemtq X-Received: by 2002:a17:902:442:: with SMTP id 60mr12221005ple.73.1547685219028; Wed, 16 Jan 2019 16:33:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547685219; cv=none; d=google.com; s=arc-20160816; b=zzmQOhH/MsnvQOEdrwjLRADQ4LF+Rs00dwTlZWi0dxvR9XxCg//1Or91QZyDcffd4z GlKi+W7qBf3gMbUC3sXsqa0OCHhp3RvqHa13SWO+uZeTAf3T+utLdBI+D35556fzmuhf ASVtqQ5Fr+osg+/9X/tQlJ6o0/4IrGCn/2n5CiYXUAqCgFGa3wrR2VfuDms5fq6s7HLH kAJhUlM89ZlD5l7F+epljHSIgICUpRdMN3rVvjvGlOQ0+BVuYW+NWbHaIlxffcT/1RRZ F7rMPdh+IS95NJmXcZs40UWWmbFTzv9lQ9CW5l5vMysdvvW2Da3vEEZqPNfkvsj+FjH3 E2cw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=/2dO2C+UYfBQiXTmhGySAi6PBFtX+ameLwxuka7ZeCk=; b=hFl+pFVDPYEqvwITszzGzznffxLfuUv+CkTgz4cgKlwuoKty1QeErUTCIdg8aPRmzm U2ZvGZoyttCn3DkCCT43NHXjVCIOU4bkHt/8UuJ2WoNXVw8q7LWVAflQLTHQAF7tIZLT klqYxIaWU/GoKozaiYLcEHtkIDxp2IpEHEsmEc81RlWR7WEKUue2f5bPEto0bd0ZXyYr 0yqSmThLNttmSVOa9Z6LB9gVgRE8uRuye53KDP+WIGsp+YuMLmSebCrpCiV2wjbQd516 hah+DIF8wwJeQmSoDyIdDpqy1xWhxHrXyDGYvmy9uVu9HrM9UL+M+GCwaUJ6uiuECwIk TO3g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga04.intel.com (mga04.intel.com. [192.55.52.120]) by mx.google.com with ESMTPS id 1si7848435plo.195.2019.01.16.16.33.38 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Jan 2019 16:33:39 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) client-ip=192.55.52.120; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2019 16:33:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,488,1539673200"; d="scan'208";a="292166034" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.79]) by orsmga005.jf.intel.com with ESMTP; 16 Jan 2019 16:33:36 -0800 From: Rick Edgecombe To: Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Borislav Petkov , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Steven Rostedt , Rick Edgecombe Subject: [PATCH 08/17] x86/ftrace: set trampoline pages as executable Date: Wed, 16 Jan 2019 16:32:50 -0800 Message-Id: <20190117003259.23141-9-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190117003259.23141-1-rick.p.edgecombe@intel.com> References: <20190117003259.23141-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit Since alloc_module() will not set the pages as executable soon, we need to do so for ftrace trampoline pages after they are allocated. For the time being, we do not change ftrace to use the text_poke() interface. As a result, ftrace breaks still breaks W^X. Cc: Steven Rostedt Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe Reviewed-by: Steven Rostedt (VMware) --- arch/x86/kernel/ftrace.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index 8257a59704ae..eb4a1937e72c 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -742,6 +742,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) unsigned long end_offset; unsigned long op_offset; unsigned long offset; + unsigned long npages; unsigned long size; unsigned long retq; unsigned long *ptr; @@ -774,6 +775,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) return 0; *tramp_size = size + RET_SIZE + sizeof(void *); + npages = DIV_ROUND_UP(*tramp_size, PAGE_SIZE); /* Copy ftrace_caller onto the trampoline memory */ ret = probe_kernel_read(trampoline, (void *)start_offset, size); @@ -818,6 +820,13 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) /* ALLOC_TRAMP flags lets us know we created it */ ops->flags |= FTRACE_OPS_FL_ALLOC_TRAMP; + /* + * Module allocation needs to be completed by making the page + * executable. The page is still writable, which is a security hazard, + * but anyhow ftrace breaks W^X completely. + */ + set_memory_x((unsigned long)trampoline, npages); + return (unsigned long)trampoline; fail: tramp_free(trampoline, *tramp_size); From patchwork Thu Jan 17 00:32:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10767271 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CF9B46C2 for ; Thu, 17 Jan 2019 00:34:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 365962DE41 for ; Thu, 17 Jan 2019 00:34:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 29DEB2E297; Thu, 17 Jan 2019 00:34:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A68E42DE41 for ; Thu, 17 Jan 2019 00:34:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BD9528E0013; Wed, 16 Jan 2019 19:33:41 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B6B658E0010; Wed, 16 Jan 2019 19:33:41 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B4BB8E0010; Wed, 16 Jan 2019 19:33:41 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id D3E1D8E000E for ; Wed, 16 Jan 2019 19:33:40 -0500 (EST) Received: by mail-pf1-f198.google.com with SMTP id m3so6004626pfj.14 for ; Wed, 16 Jan 2019 16:33:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=c13Wr+3Necp9qzFvMVo2W40U43LJE4xOuse7XO7h/k8=; b=h7Bcyyfe5yodGYMjyUblA3Eli5pVSjGQ54kAQ2PkeyfG9P9tvstnIfDqY1BALBXQo4 ERy4rLHs8t5kx6aIupYCXVZg5sCYREZGYPhMHYJjsc3JgK/ccnYBFro571Q2l/wWIc4C otShlm4MvVfHYlgFaP+EFeeLBEHL9EIhBFhBYBFPvgNiVQ47JjRGhKBYi+QmToPvtmsN XnPO2tmEgTSKt3Dzt0zXeMlGX8FgXfidtGRMMWH61+dPAOiPq3I3l/89dYAsMdX9UwDW YdB3e9icrVByTj5mQYSbCc94OperSJMAm3OcsQ6IWFigCOB4Xka/oR4aDDU9ZUsrN1Yf vGyw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AJcUukcJhsYYWRtDcTSzjXvTrIudKNnTzsBDd5saNfil9I0/drB4OIVJ Zcz7NLKEk3sfCLMzNfKBdcuY3C63KXrK/iHbKkDHrLlgiWecrhE+LWJoAdz/5wBNXS2Rydn5WL+ MzGaZUprBZ3YRmVORJWx/nafBEJ2PpQRp0FtXE+TcsFm+uH39iodJ8TT7i1c29drizA== X-Received: by 2002:a17:902:b48b:: with SMTP id y11mr12374115plr.200.1547685220513; Wed, 16 Jan 2019 16:33:40 -0800 (PST) X-Google-Smtp-Source: ALg8bN622EIZuJQkVDR+UJuKPFKeA/AUfDVMHH51rytokT9iU4mOIq7xlDXfeBTW1gVa+sC4Nmuj X-Received: by 2002:a17:902:b48b:: with SMTP id y11mr12374015plr.200.1547685219292; Wed, 16 Jan 2019 16:33:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547685219; cv=none; d=google.com; s=arc-20160816; b=cxXc6Zv5JipJedxliAHstej527F8lUGTOa8u0xpAyxLYIJCoBFYGscjhiZvadEaGZC W6C0hJULyr51NAZYnJI0s6+wnFpzp0j9ah1ocOtbhip2Hr+iTh9fbLubIz6g5gDVLfM5 S6bOfVjw0PQQzz292QkUjlprHmrTntOVcu6qae2o6HHBeOHEmq7pyoC0SfzvInoby52T u3yMYwpY6fBbyCMM0U/ncQWCwXAtm21rhdy980rhT/GauVmQINV35A3IfF1puGkSO0p9 kbbmwjcqwrz2qfxLnY7MlzZnAYmSyAmfABMYewuU29KdDi4d+C5vGnwzxYzEyECVFlaV EuIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=c13Wr+3Necp9qzFvMVo2W40U43LJE4xOuse7XO7h/k8=; b=Ch8FbCwKElVYAaKQQX/7u32mMr1K2M20yK2GUsnFoghVIu9XnkAh2Y+Me8ptJD0PMJ i/y2XNTdXoxrHmz+nx6aLzAWN+Y9wT2hHBGFGRrvMkUdrjTeWsqqfdUrkbsYwdHmOhZc tRDc2Lv6rhTlXFDpm8ES64RpCRGZNXP9SRdD7aidVKcPrFnz3yyk6wsJkE1/3L7uCAgI uvRXG8g8x1sjTAver4cfzm/oU9+TRi7487SC1Dj2ozStLVRpjiOdkn6PlejalT7Dy4aA 4cgrYHKiBwSbw24IaeatPF5CTX+kCEW24RagwYNFIUD7dpIiscKUy0pftSRtvgLfuCfB sEEA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga04.intel.com (mga04.intel.com. [192.55.52.120]) by mx.google.com with ESMTPS id q20si7678846pll.255.2019.01.16.16.33.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Jan 2019 16:33:39 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) client-ip=192.55.52.120; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2019 16:33:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,488,1539673200"; d="scan'208";a="292166037" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.79]) by orsmga005.jf.intel.com with ESMTP; 16 Jan 2019 16:33:36 -0800 From: Rick Edgecombe To: Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Borislav Petkov , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Masami Hiramatsu , Rick Edgecombe Subject: [PATCH 09/17] x86/kprobes: Instruction pages initialization enhancements Date: Wed, 16 Jan 2019 16:32:51 -0800 Message-Id: <20190117003259.23141-10-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190117003259.23141-1-rick.p.edgecombe@intel.com> References: <20190117003259.23141-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit This patch is a preparatory patch for a following patch that makes module allocated pages non-executable. The patch sets the page as executable after allocation. In the future, we may get better protection of executables. For example, by using hypercalls to request the hypervisor to protect VM executable pages from modifications using nested page-tables. This would allow us to ensure the executable has not changed between allocation and its write-protection. While at it, do some small cleanup of what appears to be unnecessary masking. Cc: Masami Hiramatsu Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe Acked-by: Masami Hiramatsu --- arch/x86/kernel/kprobes/core.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index 4ba75afba527..fac692e36833 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -431,8 +431,20 @@ void *alloc_insn_page(void) void *page; page = module_alloc(PAGE_SIZE); - if (page) - set_memory_ro((unsigned long)page & PAGE_MASK, 1); + if (page == NULL) + return NULL; + + /* + * First make the page read-only, and then only then make it executable + * to prevent it from being W+X in between. + */ + set_memory_ro((unsigned long)page, 1); + + /* + * TODO: Once additional kernel code protection mechanisms are set, ensure + * that the page was not maliciously altered and it is still zeroed. + */ + set_memory_x((unsigned long)page, 1); return page; } @@ -440,8 +452,12 @@ void *alloc_insn_page(void) /* Recover page to RW mode before releasing it */ void free_insn_page(void *page) { - set_memory_nx((unsigned long)page & PAGE_MASK, 1); - set_memory_rw((unsigned long)page & PAGE_MASK, 1); + /* + * First make the page non-executable, and then only then make it + * writable to prevent it from being W+X in between. + */ + set_memory_nx((unsigned long)page, 1); + set_memory_rw((unsigned long)page, 1); module_memfree(page); } From patchwork Thu Jan 17 00:32:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10767263 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E48501390 for ; Thu, 17 Jan 2019 00:34:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0045A2DDDC for ; Thu, 17 Jan 2019 00:34:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E7F4E2DFE3; Thu, 17 Jan 2019 00:34:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B8C792DDDC for ; Thu, 17 Jan 2019 00:34:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 70B2B8E0005; Wed, 16 Jan 2019 19:33:41 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4FE188E0012; Wed, 16 Jan 2019 19:33:41 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C19A8E000B; Wed, 16 Jan 2019 19:33:41 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id A86688E000F for ; Wed, 16 Jan 2019 19:33:40 -0500 (EST) Received: by mail-pf1-f198.google.com with SMTP id r9so6012624pfb.13 for ; Wed, 16 Jan 2019 16:33:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=D6OolJLyuKqXEHfXtl+1+6DU546MbG6uUf80skqkoRY=; b=LLBFCh4RfFROrKnEWOBEevtkS6oCrPdLoIQgZB6JhyhLn3TiuwxTI/izqO5DTyiUPq 8rUajP4O2M2FWKoshQ6aD4uLRJ/RKjZhw79SniPPztNXhyRPmd4h6QMrM9ldXTpWQq0/ vxFmITuGZTfRPh2idCPhphoNhsAZpPs4w24vgKEyUG0ODTlcuYtnW24C6Q0/qtr78ilU v09rP/xMW+s7iv/46XDfQ3GwTz1khMBXHc3/Rdt3w7SEf2ejkKYYTu59az2ymWfj2mFJ LnuAvoIc+cytlvw9f3Zf9XznqJl8i+ciMDNUUUX3RDmuQ+R9hFTtU3K8X1jy1b5ukEqr aMEw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AJcUukd50/6W4vUNk0Sl8ALueJXdQzUutTk22hXtG6hxWm4o9Fb7cD/a Rk/CFaaUROwae9QW1lENBnD0wfMdA+xRv95jHu97qzxwkhAwiR+3PbF47knN1FvsEL8c/KziiMl MAg3HBh+rIyxPcraCyeYeqdfdE4CIKpJTUiSOB+gJOUWG1dDBDCl/VBzPd/FEkDBgHg== X-Received: by 2002:a63:24c2:: with SMTP id k185mr11234150pgk.406.1547685220263; Wed, 16 Jan 2019 16:33:40 -0800 (PST) X-Google-Smtp-Source: ALg8bN4/iilhC7MC4N8px4zMMznFwEJsuo0aRWZ7VoqUemNjBaQUqZfr23GOYNwn6DzhMWIeU+O1 X-Received: by 2002:a63:24c2:: with SMTP id k185mr11234081pgk.406.1547685219287; Wed, 16 Jan 2019 16:33:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547685219; cv=none; d=google.com; s=arc-20160816; b=wz7VobxeqisUrVmOXSluWIMwsZ4vUrbI7E/P7fYbPL/12tvQFxiTAlB4zCmDSkwyVi sm47e4Ny0n8kJk/NsclSGmKcA/h4ML7Ue+fI50Rt4lnN4SqWlKyiGVR//Dm6ksV47Ww7 GIVYXRs5zOQ8ltKtOcM75ow4PYz8oUhzj6f61ZB5M0T89KsF7jjgZ5lFAlr7AiPhBFIK EBa30ocoHusXOWPapgSVlVVwJi/sEpnCXEOvsXn6C2mivEtOjeYx0NqifGevy5oF/OPu PTvaEGq2hTdhWT6JkpV/EXJ87zDMqMmD2icLL2swgNqbWJpODRDI5xhIOaAIr01/VW+k 6viQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=D6OolJLyuKqXEHfXtl+1+6DU546MbG6uUf80skqkoRY=; b=m/YPWB45mn36Mcj8hwvfqj2xntVxUn3bkeV0NKMOpxEQGHU88rMq97wt9d2togiYpr PqaN3vfK82sw90Qb96MiioWOLSscrJDNSlFg4bay5TN/9gNlB3OI2S1yukvkD9/XdDLw 7KbTgeary6xgUAZzAx4MU5Yk+Fy0hPYPO9yluF5FNoy4MJXeIua+9UZQoHdVqAWCsW93 6wMEKx8bhOwsKIddROKnhQpnUemL8FL09/rlAj1JQqS658/7lad422oYIXuJyhjzGJcO 4dyjp4oEpqjjogvdd0EK5HL3N2i+Yk5A8aJx42QilSTgW+FkngdYFiFIcI1M7EWpkFZD mvTA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga04.intel.com (mga04.intel.com. [192.55.52.120]) by mx.google.com with ESMTPS id 1si7848435plo.195.2019.01.16.16.33.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Jan 2019 16:33:39 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) client-ip=192.55.52.120; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2019 16:33:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,488,1539673200"; d="scan'208";a="292166041" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.79]) by orsmga005.jf.intel.com with ESMTP; 16 Jan 2019 16:33:36 -0800 From: Rick Edgecombe To: Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Borislav Petkov , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu , Rick Edgecombe Subject: [PATCH 10/17] x86: avoid W^X being broken during modules loading Date: Wed, 16 Jan 2019 16:32:52 -0800 Message-Id: <20190117003259.23141-11-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190117003259.23141-1-rick.p.edgecombe@intel.com> References: <20190117003259.23141-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit When modules and BPF filters are loaded, there is a time window in which some memory is both writable and executable. An attacker that has already found another vulnerability (e.g., a dangling pointer) might be able to exploit this behavior to overwrite kernel code. This patch prevents having writable executable PTEs in this stage. In addition, avoiding having R+X mappings can also slightly simplify the patching of modules code on initialization (e.g., by alternatives and static-key), as would be done in the next patch. This was actually the main motivation for this patch. To avoid having W+X mappings, set them initially as RW (NX) and after they are set as RO set them as X as well. Setting them as executable is done as a separate step to avoid one core in which the old PTE is cached (hence writable), and another which sees the updated PTE (executable), which would break the W^X protection. Cc: Kees Cook Cc: Peter Zijlstra Cc: Dave Hansen Cc: Masami Hiramatsu Suggested-by: Thomas Gleixner Suggested-by: Andy Lutomirski Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/alternative.c | 28 +++++++++++++++++++++------- arch/x86/kernel/module.c | 2 +- include/linux/filter.h | 4 ++-- kernel/module.c | 5 +++++ 4 files changed, 29 insertions(+), 10 deletions(-) diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 8fc4685f3117..18415e3b6000 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -667,15 +667,29 @@ void __init alternative_instructions(void) * handlers seeing an inconsistent instruction while you patch. */ void *__init_or_module text_poke_early(void *addr, const void *opcode, - size_t len) + size_t len) { unsigned long flags; - local_irq_save(flags); - memcpy(addr, opcode, len); - local_irq_restore(flags); - sync_core(); - /* Could also do a CLFLUSH here to speed up CPU recovery; but - that causes hangs on some VIA CPUs. */ + + if (static_cpu_has(X86_FEATURE_NX) && + is_module_text_address((unsigned long)addr)) { + /* + * Modules text is marked initially as non-executable, so the + * code cannot be running and speculative code-fetches are + * prevented. We can just change the code. + */ + memcpy(addr, opcode, len); + } else { + local_irq_save(flags); + memcpy(addr, opcode, len); + local_irq_restore(flags); + sync_core(); + + /* + * Could also do a CLFLUSH here to speed up CPU recovery; but + * that causes hangs on some VIA CPUs. + */ + } return addr; } diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c index b052e883dd8c..cfa3106faee4 100644 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -87,7 +87,7 @@ void *module_alloc(unsigned long size) p = __vmalloc_node_range(size, MODULE_ALIGN, MODULES_VADDR + get_module_load_offset(), MODULES_END, GFP_KERNEL, - PAGE_KERNEL_EXEC, 0, NUMA_NO_NODE, + PAGE_KERNEL, 0, NUMA_NO_NODE, __builtin_return_address(0)); if (p && (kasan_module_alloc(p, size) < 0)) { vfree(p); diff --git a/include/linux/filter.h b/include/linux/filter.h index ad106d845b22..f18cd317faf8 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -483,7 +483,7 @@ struct bpf_prog { u16 pages; /* Number of allocated pages */ u16 jited:1, /* Is our filter JIT'ed? */ jit_requested:1,/* archs need to JIT the prog */ - undo_set_mem:1, /* Passed set_memory_ro() checkpoint */ + undo_set_mem:1, /* Passed set_memory_ro() checkpoint */ gpl_compatible:1, /* Is filter GPL compatible? */ cb_access:1, /* Is control block accessed? */ dst_needed:1, /* Do we need dst entry? */ @@ -681,7 +681,6 @@ bpf_ctx_narrow_access_ok(u32 off, u32 size, u32 size_default) static inline void bpf_prog_lock_ro(struct bpf_prog *fp) { - fp->undo_set_mem = 1; set_memory_ro((unsigned long)fp, fp->pages); } @@ -694,6 +693,7 @@ static inline void bpf_prog_unlock_ro(struct bpf_prog *fp) static inline void bpf_jit_binary_lock_ro(struct bpf_binary_header *hdr) { set_memory_ro((unsigned long)hdr, hdr->pages); + set_memory_x((unsigned long)hdr, hdr->pages); } static inline void bpf_jit_binary_unlock_ro(struct bpf_binary_header *hdr) diff --git a/kernel/module.c b/kernel/module.c index 2ad1b5239910..ae1b77da6a20 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -1950,8 +1950,13 @@ void module_enable_ro(const struct module *mod, bool after_init) return; frob_text(&mod->core_layout, set_memory_ro); + frob_text(&mod->core_layout, set_memory_x); + frob_rodata(&mod->core_layout, set_memory_ro); + frob_text(&mod->init_layout, set_memory_ro); + frob_text(&mod->init_layout, set_memory_x); + frob_rodata(&mod->init_layout, set_memory_ro); if (after_init) From patchwork Thu Jan 17 00:32:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10767287 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 10B286C2 for ; Thu, 17 Jan 2019 00:34:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 167702E1E1 for ; Thu, 17 Jan 2019 00:34:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 089E22E297; Thu, 17 Jan 2019 00:34:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 808FD2E2B7 for ; Thu, 17 Jan 2019 00:34:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F2AC38E000B; Wed, 16 Jan 2019 19:33:41 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E27808E0015; Wed, 16 Jan 2019 19:33:41 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 995908E0012; Wed, 16 Jan 2019 19:33:41 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id 496498E000E for ; Wed, 16 Jan 2019 19:33:41 -0500 (EST) Received: by mail-pg1-f197.google.com with SMTP id d71so5039695pgc.1 for ; Wed, 16 Jan 2019 16:33:41 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=1oCH1vqwofeA+CR+BzYGyrdjwP4k7rQNDYFA/zzkTDg=; b=H2yBR3BxIbEpSpY27XDBkLr6TGAthIe6B3ynoebJs046LM3Q0iQQGrYLG5wOXHUEVy 4O6AsQoQ0LBTVh9ZWA41CI44hb6kanuDLmHj+5A5YuZHi9piUbNNNTDT5W86+dWYzroi l3U+OIBxJbHalav9J8Y1PMiJkQn90MK+tGG8eCIOQjedX0KdW6fXaAZY5Hk/FPBNvcSk OSSzYRgAV18mQ++VNB8YAOwc36Qt3Q4r3spSYl7ruVwzU0j4c6r31acEIHdC4hqH2iml bZFsYGmBtHcrhgxTgxznJARZHGpYxJ3YGhkP8WqmMVlqd6KZze59XdrblZFZubZj688Q I6Rw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AJcUukcWfSNKwMoNynFXPrzfIBnMd2d7XsVlkWLi2N7+zINYLZwW6PEb tUHi8GyqjR5vhvluB+P3ZZ6/AOtOlsektBtzDHdU/iHSihO7qGRdHGgDKgFJT7iXdhZ8RM4Huz8 geTOJpYaC7iL2HYSTQjFl2Qr7l0jO22TtuoyD4fuet6Czz5v7tlmSvYmjBbXlTvHabQ== X-Received: by 2002:a63:f201:: with SMTP id v1mr10826948pgh.232.1547685220470; Wed, 16 Jan 2019 16:33:40 -0800 (PST) X-Google-Smtp-Source: ALg8bN422vP6ZRjAdeXzcxhYUK/FBHmouT/KJ38p6lX95WIwqK9bMjzBtwRZtWhv2FSWEBT11NFZ X-Received: by 2002:a63:f201:: with SMTP id v1mr10826896pgh.232.1547685219711; Wed, 16 Jan 2019 16:33:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547685219; cv=none; d=google.com; s=arc-20160816; b=nxiOiIy1+bdxyV47dmUhRb4pk8JS32B/vX315fCzeFfQSyKbVjgDoxm8aks0g9gmgt JmgyB9KAKhzfVzIwapav97wseWimhBQV+v5a7uEq6dZwemm30i98Luia//qmkaeDFQmS EjMwcVTf9ueEMbZ0VNuLUrrMOIhZ5ZJN3O2xmZAmusWJds6WbTcnaEk0poMxyKv919x7 GwTXT9CbFvXS1m3+JTl95LSfgnV7oqdqZ/yimprhojK8UOmQufxOJUkGHpU1aY+KzvxG SfIusefqDZoGsQJGj8vOVNOQ/Ete3AFFRQZReDvxKmZ7baSWPr1SGXImvDjulGsMOcOG K+LQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=1oCH1vqwofeA+CR+BzYGyrdjwP4k7rQNDYFA/zzkTDg=; b=X3dG/oPhpcKoB1oa2/VKcOcQvR1wcILZZzwwlAp0jrEVAmDooFfVY3uQbCIyo0tNT1 OFeK6RO9dVXslQl6pBswfFL90oDboAakLXW0SV6Eggd8G10rMB4zw9KPrwA/iuN5x2VF 7cBUDvMS7WUQbyoOT4Gr8qQqGeSIYWocn6aiS4C/ZApNSi5hSLNZfMKkFjCyQoHCRwKW /sjolCPmaEWiRuXLmFzZ7fparKOiEkoIWjzlrREHSoRJngsZDqUFInfDkt3Bhx0ZFfm9 AkgedoM0xVfQF+FK5GM7Uzu8/zDJqyrNQ4c5djv2sqRUbF883QWPRcA6rm6oWafhErvI kxgQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga04.intel.com (mga04.intel.com. [192.55.52.120]) by mx.google.com with ESMTPS id y10si7582915plt.406.2019.01.16.16.33.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Jan 2019 16:33:39 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) client-ip=192.55.52.120; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2019 16:33:37 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,488,1539673200"; d="scan'208";a="292166043" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.79]) by orsmga005.jf.intel.com with ESMTP; 16 Jan 2019 16:33:36 -0800 From: Rick Edgecombe To: Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Borislav Petkov , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu , Rick Edgecombe Subject: [PATCH 11/17] x86/jump-label: remove support for custom poker Date: Wed, 16 Jan 2019 16:32:53 -0800 Message-Id: <20190117003259.23141-12-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190117003259.23141-1-rick.p.edgecombe@intel.com> References: <20190117003259.23141-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit There are only two types of poking: early and breakpoint based. The use of a function pointer to perform poking complicates the code and is probably inefficient due to the use of indirect branches. Cc: Andy Lutomirski Cc: Kees Cook Cc: Peter Zijlstra Cc: Dave Hansen Cc: Masami Hiramatsu Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/jump_label.c | 24 ++++++++---------------- 1 file changed, 8 insertions(+), 16 deletions(-) diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c index e36cfec0f35e..427facef8aff 100644 --- a/arch/x86/kernel/jump_label.c +++ b/arch/x86/kernel/jump_label.c @@ -37,7 +37,6 @@ static void bug_at(unsigned char *ip, int line) static void __ref __jump_label_transform(struct jump_entry *entry, enum jump_label_type type, - void *(*poker)(void *, const void *, size_t), int init) { union jump_code_union jmp; @@ -50,14 +49,6 @@ static void __ref __jump_label_transform(struct jump_entry *entry, jmp.offset = jump_entry_target(entry) - (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); - /* - * As long as we're UP and not yet marked RO, we can use - * text_poke_early; SYSTEM_BOOTING guarantees both, as we switch to - * SYSTEM_SCHEDULING before going either. - */ - if (system_state == SYSTEM_BOOTING) - poker = text_poke_early; - if (type == JUMP_LABEL_JMP) { if (init) { expect = default_nop; line = __LINE__; @@ -80,16 +71,17 @@ static void __ref __jump_label_transform(struct jump_entry *entry, bug_at((void *)jump_entry_code(entry), line); /* - * Make text_poke_bp() a default fallback poker. + * As long as we're UP and not yet marked RO, we can use + * text_poke_early; SYSTEM_BOOTING guarantees both, as we switch to + * SYSTEM_SCHEDULING before going either. * * At the time the change is being done, just ignore whether we * are doing nop -> jump or jump -> nop transition, and assume * always nop being the 'currently valid' instruction - * */ - if (poker) { - (*poker)((void *)jump_entry_code(entry), code, - JUMP_LABEL_NOP_SIZE); + if (init || system_state == SYSTEM_BOOTING) { + text_poke_early((void *)jump_entry_code(entry), code, + JUMP_LABEL_NOP_SIZE); return; } @@ -101,7 +93,7 @@ void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) { mutex_lock(&text_mutex); - __jump_label_transform(entry, type, NULL, 0); + __jump_label_transform(entry, type, 0); mutex_unlock(&text_mutex); } @@ -131,5 +123,5 @@ __init_or_module void arch_jump_label_transform_static(struct jump_entry *entry, jlstate = JL_STATE_NO_UPDATE; } if (jlstate == JL_STATE_UPDATE) - __jump_label_transform(entry, type, text_poke_early, 1); + __jump_label_transform(entry, type, 1); } From patchwork Thu Jan 17 00:32:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10767267 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 099A114E5 for ; Thu, 17 Jan 2019 00:34:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4A2E32DDDC for ; Thu, 17 Jan 2019 00:34:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3E5CF2DE41; Thu, 17 Jan 2019 00:34:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A4FD72DFD8 for ; Thu, 17 Jan 2019 00:34:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B4878E000E; Wed, 16 Jan 2019 19:33:41 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 86B538E0013; Wed, 16 Jan 2019 19:33:41 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2C71F8E000F; Wed, 16 Jan 2019 19:33:41 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id BDD468E0005 for ; Wed, 16 Jan 2019 19:33:40 -0500 (EST) Received: by mail-pl1-f198.google.com with SMTP id g13so4943423plo.10 for ; Wed, 16 Jan 2019 16:33:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=se2UgweWDUv6UvpNReMwED+7UCj9/063SvA/PxiU/I8=; b=WQtE6AM3Toywkf09rgFDv0K0ZwnRA7iub0hxN4+yZ+z071R5Qo57SDqasPYtsp2grK WiCBdgIIxvzPx1O8T8cjhwvM+QXveoIxojkLlkIWchAe5oWXuKmB1AN4iCiN/SMnyvwb KQu/waI2QpIUND1PL6lTne88rV0FNh8VgJElpOAD7IbVgDYq76u9KK0OrcXq5YYkfAaO s26iJD2iWQVnkKK1tzCiqSTjI9BgPn1Zvq69Vk2p/uM1BFZTmKEptf7/NVPrkHzI10jC h7/zszqmf+Q5xSNFBS9ygkRXnY2Kv2rcdly4p4KgH6tSbN+it3jOwN6AlBTw5q4Fle4g w9Eg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AJcUukcFNK/FBA0ph+aVAWkglUs2WEV1IyMrfUbywUgwG3wqf6SOCRZ7 spiFbNHubPNKcloAwMiJ3CLknnUR/6mfbs885pyNRXJol2wy6C6SPIbdrp8YrlqpB8NtdXNdcGH X2fo4VwzMC1Xx91tj63rAPDcN4u64fwaUVdqwNEvVeEG8jLJoNzj9gTXKG0b/Pkuq4A== X-Received: by 2002:a63:111c:: with SMTP id g28mr11295251pgl.85.1547685220430; Wed, 16 Jan 2019 16:33:40 -0800 (PST) X-Google-Smtp-Source: ALg8bN4U65x3pdhZBLz+CiW0+YLQUKLOc7jD+dCwV8vssPUcndp8eqNiFkOsVubo1aKLiBrRrTyn X-Received: by 2002:a63:111c:: with SMTP id g28mr11295192pgl.85.1547685219609; Wed, 16 Jan 2019 16:33:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547685219; cv=none; d=google.com; s=arc-20160816; b=uULQ3t2lkzGBJyPGCgKiU+9oI1wWM6IPnmNDFFI+hoOnKg+KXXk6g3Nri4ZbwICwaJ WETjd+BXRHuIZ550oADAfqnVL0+eC6z1MVM/cpA2VQnJZLX+EL9dHB+0mYrfIXE+BFzm ZP7HDSEMt1sStOEmIKbmyfJkBScui+uRTXi3z6snABNMlXIfsdEJdA28ErOksnRtsFho BluUTWdi6e/8jSLjoXE2ZiMwGEJ1hfq+G8Ldhtnxw43Aay1OqXuwhTzd5YzJOirIACdR xvNH4Dplv9yXkcu8wheBFtjSKJA/bEIsOBZeotaPkWEOy4409ghw4U/9/xfLrDDO5eAs ChAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=se2UgweWDUv6UvpNReMwED+7UCj9/063SvA/PxiU/I8=; b=V1uiTZqEh+OJ/WBlYcSd+Syx/5EV1O1JqpMNPNgYc0mjoM0mqwbYzszL3iZ+OZ+pDy nh/a0eRUZ2ZJFwQYDMi3Vz3nJXvWIGyfGXmnbJy6ic6gLP3p8rorYQVql4Az9uS9tYVW gNkxN79jce4s0Nu8UYhMYyThb4zLpOa+2xp5rtAyOfZwpmm6YrA/plsnrEAaJIeWBS/k WdGoFjdyLYRww6vM/lpvgBM8IIcHosHsw8vTSYhr/6DUHS7z0tSj+GQL529YMO65zqoB HIkyUwdkbliem4EeT1wrlp3iynh30lBYcd45pc88DahUJtzYmlurMIb48nYgC9wFaWP4 hO4A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga04.intel.com (mga04.intel.com. [192.55.52.120]) by mx.google.com with ESMTPS id 1si7848435plo.195.2019.01.16.16.33.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Jan 2019 16:33:39 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) client-ip=192.55.52.120; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2019 16:33:37 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,488,1539673200"; d="scan'208";a="292166048" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.79]) by orsmga005.jf.intel.com with ESMTP; 16 Jan 2019 16:33:36 -0800 From: Rick Edgecombe To: Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Borislav Petkov , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu , Rick Edgecombe Subject: [PATCH 12/17] x86/alternative: Remove the return value of text_poke_*() Date: Wed, 16 Jan 2019 16:32:54 -0800 Message-Id: <20190117003259.23141-13-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190117003259.23141-1-rick.p.edgecombe@intel.com> References: <20190117003259.23141-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit The return value of text_poke_early() and text_poke_bp() is useless. Remove it. Cc: Andy Lutomirski Cc: Kees Cook Cc: Peter Zijlstra Cc: Dave Hansen Cc: Masami Hiramatsu Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/text-patching.h | 4 ++-- arch/x86/kernel/alternative.c | 11 ++++------- 2 files changed, 6 insertions(+), 9 deletions(-) diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h index a75eed841eed..c90678fd391a 100644 --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -18,7 +18,7 @@ static inline void apply_paravirt(struct paravirt_patch_site *start, #define __parainstructions_end NULL #endif -extern void *text_poke_early(void *addr, const void *opcode, size_t len); +extern void text_poke_early(void *addr, const void *opcode, size_t len); /* * Clear and restore the kernel write-protection flag on the local CPU. @@ -37,7 +37,7 @@ extern void *text_poke_early(void *addr, const void *opcode, size_t len); extern void *text_poke(void *addr, const void *opcode, size_t len); extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len); extern int poke_int3_handler(struct pt_regs *regs); -extern void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler); +extern void text_poke_bp(void *addr, const void *opcode, size_t len, void *handler); extern int after_bootmem; extern __ro_after_init struct mm_struct *poking_mm; extern __ro_after_init unsigned long poking_addr; diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 18415e3b6000..2740ad2c6f21 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -264,7 +264,7 @@ static void __init_or_module add_nops(void *insns, unsigned int len) extern struct alt_instr __alt_instructions[], __alt_instructions_end[]; extern s32 __smp_locks[], __smp_locks_end[]; -void *text_poke_early(void *addr, const void *opcode, size_t len); +void text_poke_early(void *addr, const void *opcode, size_t len); /* * Are we looking at a near JMP with a 1 or 4-byte displacement. @@ -666,8 +666,8 @@ void __init alternative_instructions(void) * instructions. And on the local CPU you need to be protected again NMI or MCE * handlers seeing an inconsistent instruction while you patch. */ -void *__init_or_module text_poke_early(void *addr, const void *opcode, - size_t len) +void __init_or_module text_poke_early(void *addr, const void *opcode, + size_t len) { unsigned long flags; @@ -690,7 +690,6 @@ void *__init_or_module text_poke_early(void *addr, const void *opcode, * that causes hangs on some VIA CPUs. */ } - return addr; } __ro_after_init struct mm_struct *poking_mm; @@ -893,7 +892,7 @@ int poke_int3_handler(struct pt_regs *regs) * replacing opcode * - sync cores */ -void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler) +void text_poke_bp(void *addr, const void *opcode, size_t len, void *handler) { unsigned char int3 = 0xcc; @@ -935,7 +934,5 @@ void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler) * the writing of the new instruction. */ bp_patching_in_progress = false; - - return addr; } From patchwork Thu Jan 17 00:32:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10767235 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F1BFC14E5 for ; Thu, 17 Jan 2019 00:33:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E0BCA2FD0E for ; Thu, 17 Jan 2019 00:33:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D4F5D2FD2B; Thu, 17 Jan 2019 00:33:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 46EE82FD0E for ; Thu, 17 Jan 2019 00:33:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 930C78E0003; Wed, 16 Jan 2019 19:33:39 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 85D368E0007; Wed, 16 Jan 2019 19:33:39 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7261E8E0006; Wed, 16 Jan 2019 19:33:39 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id 306708E0005 for ; Wed, 16 Jan 2019 19:33:39 -0500 (EST) Received: by mail-pl1-f197.google.com with SMTP id g12so4907677pll.22 for ; Wed, 16 Jan 2019 16:33:39 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=dIBVlpwSnsc3TOZe2Cceb+JTk5oYVFqlObcXRp5oqJs=; b=t//AXpJfktTaD97UarZ9y0G4q+xLeO2WPWWDzx1o9we+JrQxWW8Hrwc6DP7YGos3e3 e60IbkvrgqRWDsGnnqfIbMvIj3DtK6vExP4VZY0oQbgBp8oz6oo+j4c2h3CZwCc0y3RH +81mCcJRDDlHJAZUCyqd153jrO+5PHkYB0jlFpYDnvjWQPN1p1fJBalfnqJrBZ5Z+obU 1wJDZXi4WGjJv+ssQojNwb6QjxoDulqKuxsQpxEH5DTGClOTyNOWTw1tVGYuuDWj3CRL c0q70mba01Vf1qnZ0MaK8HKTwh8QY6GkM2gXMvJqxmAX3B7IqyPBgR+Dy0sG9kBFeYyK h4AA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AJcUukfRJHugwLguVQE4F0g+q5oAYm4HR1SYmdyrxERizv3xKT6WIfPy R3HXo3QS61wVyO2IShBN8gEOMO2lyQMCITMwpo1bLcAEgaaaJMFd1tJ8EZ35k0DUT6SKTOJt74f guXUVn329G/hRtStPzmEzj1HcUMNGVWhbyfWWf3oHUGK3U+zkSERb13bs4YohRahFiw== X-Received: by 2002:a17:902:584:: with SMTP id f4mr13147858plf.28.1547685218848; Wed, 16 Jan 2019 16:33:38 -0800 (PST) X-Google-Smtp-Source: ALg8bN64rnhVDG+YSOk6T9qkjX6WGIQ6BLPqKZ2tGMsyWWzqZveriTar+uNEVTTFuG0erjI+TKBF X-Received: by 2002:a17:902:584:: with SMTP id f4mr13147751plf.28.1547685217721; Wed, 16 Jan 2019 16:33:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547685217; cv=none; d=google.com; s=arc-20160816; b=uR3yDYT5oIBnuBZL/BztZVes53yoHzx8Gx1CMGtRsktIhFPab4kbp0EQqzNi5UvYe3 1DRi7yWKaeSfwhIduQs2THoXVop1GOM6dhha/FTT5uw9Mgg8tP5XjkJ6dYstxCzBhdHD LktLw1mgKHVkQ8t5ArmnkJTd/wogiRp5lN+pmbpO0mLoyjogoldKAb6N7BWCCUwsAFTk +AdsLSUL1dKBy7jn8omtA/ZFohSh21l/dwnv0lfwNytZiYoPHO3aldo42qoDY2MT88sr ryTTXPcoqR6/VAJiCc3qeHdNpD1xmve/VdwZqAmCboI0a0d/0rAqWC8S4EXaI4YeuObI i9WA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=dIBVlpwSnsc3TOZe2Cceb+JTk5oYVFqlObcXRp5oqJs=; b=xb+cdRa2pmomo9FP165AEK0s4hXPIVMU8mUh2fbPvkvVaE8xShGJd1dk0I4MDYvDY1 jNDeKnef4U/bflLtEXcQACwjW3Lu2OOWN352HikDHUr3bwfy3Yz0vJ2s/nSPRS/ClGTy BRpA5AdVrj2YvHU0BeLtq0I2VdWU6/W4WA6tClwVoeoTVvhxNvUd+/DPi14uirEyTFoH Q7u0fyhdFIDG65hQZJdiQV3fkpq4BZvFEEUuAKWFguDB8j363G4Imqo81zJJtBik8W56 4cmDIpWMTlBQgA+n7vXjbur0SxtQvYNjDsk3qs7ULeAH+xAQQpNhEXIVDi2vn2kDblhO P4KA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga05.intel.com (mga05.intel.com. [192.55.52.43]) by mx.google.com with ESMTPS id m3si7883258pld.331.2019.01.16.16.33.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Jan 2019 16:33:37 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.43 as permitted sender) client-ip=192.55.52.43; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2019 16:33:37 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,488,1539673200"; d="scan'208";a="292166050" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.79]) by orsmga005.jf.intel.com with ESMTP; 16 Jan 2019 16:33:36 -0800 From: Rick Edgecombe To: Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Borislav Petkov , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe Subject: [PATCH 13/17] Add set_alias_ function and x86 implementation Date: Wed, 16 Jan 2019 16:32:55 -0800 Message-Id: <20190117003259.23141-14-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190117003259.23141-1-rick.p.edgecombe@intel.com> References: <20190117003259.23141-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This adds two new functions set_alias_default_noflush and set_alias_nv_noflush for setting the alias mapping for the page to its default valid permissions and to an invalid state that cannot be cached in a TLB, respectively. These functions to not flush the TLB. Note, __kernel_map_pages does something similar but flushes the TLB and doesn't reset the permission bits to default on all architectures. There is also an ARCH config ARCH_HAS_SET_ALIAS for specifying whether these have an actual implementation or a default empty one. Cc: Dave Hansen Cc: Andy Lutomirski Cc: Peter Zijlstra Signed-off-by: Rick Edgecombe --- arch/Kconfig | 4 ++++ arch/x86/Kconfig | 1 + arch/x86/include/asm/set_memory.h | 3 +++ arch/x86/mm/pageattr.c | 14 +++++++++++--- include/linux/set_memory.h | 10 ++++++++++ 5 files changed, 29 insertions(+), 3 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 4cfb6de48f79..4ef9db190f2d 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -249,6 +249,10 @@ config ARCH_HAS_FORTIFY_SOURCE config ARCH_HAS_SET_MEMORY bool +# Select if arch has all set_alias_nv/default() functions +config ARCH_HAS_SET_ALIAS + bool + # Select if arch init_task must go in the __init_task_data section config ARCH_TASK_STRUCT_ON_STACK bool diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 15af091611e2..14ad28769256 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -66,6 +66,7 @@ config X86 select ARCH_HAS_UACCESS_FLUSHCACHE if X86_64 select ARCH_HAS_UACCESS_MCSAFE if X86_64 && X86_MCE select ARCH_HAS_SET_MEMORY + select ARCH_HAS_SET_ALIAS select ARCH_HAS_STRICT_KERNEL_RWX select ARCH_HAS_STRICT_MODULE_RWX select ARCH_HAS_SYNC_CORE_BEFORE_USERMODE diff --git a/arch/x86/include/asm/set_memory.h b/arch/x86/include/asm/set_memory.h index 07a25753e85c..2ef4e4222df1 100644 --- a/arch/x86/include/asm/set_memory.h +++ b/arch/x86/include/asm/set_memory.h @@ -85,6 +85,9 @@ int set_pages_nx(struct page *page, int numpages); int set_pages_ro(struct page *page, int numpages); int set_pages_rw(struct page *page, int numpages); +int set_alias_nv_noflush(struct page *page); +int set_alias_default_noflush(struct page *page); + extern int kernel_set_to_readonly; void set_kernel_text_rw(void); void set_kernel_text_ro(void); diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c index 4f8972311a77..3a51915a1410 100644 --- a/arch/x86/mm/pageattr.c +++ b/arch/x86/mm/pageattr.c @@ -2209,8 +2209,6 @@ int set_pages_rw(struct page *page, int numpages) return set_memory_rw(addr, numpages); } -#ifdef CONFIG_DEBUG_PAGEALLOC - static int __set_pages_p(struct page *page, int numpages) { unsigned long tempaddr = (unsigned long) page_address(page); @@ -2249,6 +2247,17 @@ static int __set_pages_np(struct page *page, int numpages) return __change_page_attr_set_clr(&cpa, 0); } +int set_alias_nv_noflush(struct page *page) +{ + return __set_pages_np(page, 1); +} + +int set_alias_default_noflush(struct page *page) +{ + return __set_pages_p(page, 1); +} + +#ifdef CONFIG_DEBUG_PAGEALLOC void __kernel_map_pages(struct page *page, int numpages, int enable) { if (PageHighMem(page)) @@ -2282,7 +2291,6 @@ void __kernel_map_pages(struct page *page, int numpages, int enable) } #ifdef CONFIG_HIBERNATION - bool kernel_page_present(struct page *page) { unsigned int level; diff --git a/include/linux/set_memory.h b/include/linux/set_memory.h index 2a986d282a97..d19481ac6a8f 100644 --- a/include/linux/set_memory.h +++ b/include/linux/set_memory.h @@ -10,6 +10,16 @@ #ifdef CONFIG_ARCH_HAS_SET_MEMORY #include +#ifndef CONFIG_ARCH_HAS_SET_ALIAS +static inline int set_alias_nv_noflush(struct page *page) +{ + return 0; +} +static inline int set_alias_default_noflush(struct page *page) +{ + return 0; +} +#endif #else static inline int set_memory_ro(unsigned long addr, int numpages) { return 0; } static inline int set_memory_rw(unsigned long addr, int numpages) { return 0; } From patchwork Thu Jan 17 00:32:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10767291 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 487986C2 for ; Thu, 17 Jan 2019 00:34:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CCA692DE41 for ; Thu, 17 Jan 2019 00:34:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C018C2E297; Thu, 17 Jan 2019 00:34:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3E4402DE41 for ; Thu, 17 Jan 2019 00:34:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2715E8E000F; Wed, 16 Jan 2019 19:33:42 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1CBB98E0012; Wed, 16 Jan 2019 19:33:42 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E05858E0014; Wed, 16 Jan 2019 19:33:41 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id 6440B8E000B for ; Wed, 16 Jan 2019 19:33:41 -0500 (EST) Received: by mail-pl1-f197.google.com with SMTP id 12so4923704plb.18 for ; Wed, 16 Jan 2019 16:33:41 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=YA6cxWpgPEbPNtR1Fs7WK/CCwe2Q1Rjy/WvJom+wYlI=; b=qxCoi9tqxJjC92Cmt17a3PWWXRWf+X1Gcj+mKx5UMXcMPI0n+vFgiPwfZrvscrHYd1 M89ARCL+nqsEKOaEf415o+6lBRfB85j1AqI6AmPLUuY8HkmefqUlbVXGePaM9KrRhib8 +eJfNadBQBwdZZNjrmMub5Xd8gDF319mKwchn9wP75bow0/iLymEBAsLARbVz/xrzBwU nvbjLH3fIz6F+3OJMYy5LEvGhSSRZj/p/770Qool6hX6hEWBG9CItOgLypdAMc2JgyIF Y+grEmEoytLV/0GnP0C0pfFwupiDJ1iB64rubI9wgG/hep70bKfoD5BHDZ0bfWpuWrNX tdcw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AJcUukdPPqSsa/ZvNeeAzAEh2zLS+uMrELhXRMFxa47rDHR1p0Gso9KW oJ5wh/1AwTxsNYvNQERZt6xVEMRuWazvg/SWnC4y9taj20Aa2znqROa21BP7RPM/VDk/AiUukyn ouTBmjqTdLBRSq0aXFx/KPGUaJ6XzZiRggXQWisI0FUrRL03yxhefqTxuqTmEp+uLHQ== X-Received: by 2002:a63:e5c:: with SMTP id 28mr4500137pgo.369.1547685221044; Wed, 16 Jan 2019 16:33:41 -0800 (PST) X-Google-Smtp-Source: ALg8bN7badaV97gr6P+HUlIPp+qWQ2JQLH1+iR1MemqP5bp63iK4P78MdrO8Yo7efP5cEdmCYo3H X-Received: by 2002:a63:e5c:: with SMTP id 28mr4500051pgo.369.1547685219798; Wed, 16 Jan 2019 16:33:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547685219; cv=none; d=google.com; s=arc-20160816; b=UiCZXJR6FbwzOrsvA7a8Xx33Bq7rhne3iwnZ/f+VBj6b3ukHOPb+G4STtEu27ypRW5 FftX+qhq+7kzsDIC2Qt/vSuFiJnZZkIjAknfxKMA0JqC3iqGGWbIA5qAjqA7fzeooTrj MY51I7eigZizuxT1W6UQ5dFpQ2lqCQN3DH3InZXipMLqu30YrJ8Ci5tO3ayw3A18B3ly zL03WfJYzz21QOd31EzYovGb1+8Zf0Y+bxcXdxUMET0FTH6MVeUgenGMwE/ReX9yTIys VV10HDMPT9BGeyqsZ0ijdm2cvlnLTS6mO7AN6Wgqw/hbl2BaGx8SfXqmlRD1piXPmpXi 3JYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=YA6cxWpgPEbPNtR1Fs7WK/CCwe2Q1Rjy/WvJom+wYlI=; b=uDnD1PxD3Cgr0au7CwNzjugfAb3IpwRFN3h92HaCffoUQWZ6myYnv91L0KoTBbRrJU V4+YJLUs+xHdX1+z2OzLCtk+6ufsmCPLEnvl33yi5z2FPCG+Bh4OJ2v98z6X9mPXdFYo EMSRQIGJdMnWmAla4rdoaTuMtzQSU/QcRYfC9PUzjRPAgkloR/JKyfabaTdTua2Odg0G bA9GzdKJiRYzBVE+tYqtPAPAjYzqYOW5HOAANJ7FsawwA7ZWY8Ao/H3n+gW6KQA5RKI9 Gdj7ZGi3W6qGSNmDgR0TmUbC5BTONtPcfs03yd5HwkrDHgvUSrdgDdKHdRs2q8WQCVLE uNRw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga04.intel.com (mga04.intel.com. [192.55.52.120]) by mx.google.com with ESMTPS id q20si7678846pll.255.2019.01.16.16.33.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Jan 2019 16:33:39 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) client-ip=192.55.52.120; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2019 16:33:37 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,488,1539673200"; d="scan'208";a="292166053" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.79]) by orsmga005.jf.intel.com with ESMTP; 16 Jan 2019 16:33:36 -0800 From: Rick Edgecombe To: Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Borislav Petkov , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , "Rafael J. Wysocki" , Pavel Machek Subject: [PATCH 14/17] mm: Make hibernate handle unmapped pages Date: Wed, 16 Jan 2019 16:32:56 -0800 Message-Id: <20190117003259.23141-15-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190117003259.23141-1-rick.p.edgecombe@intel.com> References: <20190117003259.23141-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP For architectures with CONFIG_ARCH_HAS_SET_ALIAS, pages can be unmapped briefly on the directmap, even when CONFIG_DEBUG_PAGEALLOC is not configured. So this changes kernel_map_pages and kernel_page_present to be defined when CONFIG_ARCH_HAS_SET_ALIAS is defined as well. It also changes places (page_alloc.c) where those functions are assumed to only be implemented when CONFIG_DEBUG_PAGEALLOC is defined. So now when CONFIG_ARCH_HAS_SET_ALIAS=y, hibernate will handle not present page when saving. Previously this was already done when CONFIG_DEBUG_PAGEALLOC was configured. It does not appear to have a big hibernating performance impact. Before: [ 4.670938] PM: Wrote 171996 kbytes in 0.21 seconds (819.02 MB/s) After: [ 4.504714] PM: Wrote 178932 kbytes in 0.22 seconds (813.32 MB/s) Cc: Dave Hansen Cc: Andy Lutomirski Cc: Peter Zijlstra Cc: "Rafael J. Wysocki" Cc: Pavel Machek Signed-off-by: Rick Edgecombe Acked-by: Pavel Machek --- arch/x86/mm/pageattr.c | 4 ---- include/linux/mm.h | 18 ++++++------------ mm/page_alloc.c | 6 ++++-- 3 files changed, 10 insertions(+), 18 deletions(-) diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c index 3a51915a1410..717bdc188aab 100644 --- a/arch/x86/mm/pageattr.c +++ b/arch/x86/mm/pageattr.c @@ -2257,7 +2257,6 @@ int set_alias_default_noflush(struct page *page) return __set_pages_p(page, 1); } -#ifdef CONFIG_DEBUG_PAGEALLOC void __kernel_map_pages(struct page *page, int numpages, int enable) { if (PageHighMem(page)) @@ -2302,11 +2301,8 @@ bool kernel_page_present(struct page *page) pte = lookup_address((unsigned long)page_address(page), &level); return (pte_val(*pte) & _PAGE_PRESENT); } - #endif /* CONFIG_HIBERNATION */ -#endif /* CONFIG_DEBUG_PAGEALLOC */ - int __init kernel_map_pages_in_pgd(pgd_t *pgd, u64 pfn, unsigned long address, unsigned numpages, unsigned long page_flags) { diff --git a/include/linux/mm.h b/include/linux/mm.h index 80bb6408fe73..b362a280a919 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2642,37 +2642,31 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { } #endif -#ifdef CONFIG_DEBUG_PAGEALLOC extern bool _debug_pagealloc_enabled; -extern void __kernel_map_pages(struct page *page, int numpages, int enable); static inline bool debug_pagealloc_enabled(void) { - return _debug_pagealloc_enabled; + return IS_ENABLED(CONFIG_DEBUG_PAGEALLOC) && _debug_pagealloc_enabled; } +#if defined(CONFIG_DEBUG_PAGEALLOC) || defined(CONFIG_ARCH_HAS_SET_ALIAS) +extern void __kernel_map_pages(struct page *page, int numpages, int enable); + static inline void kernel_map_pages(struct page *page, int numpages, int enable) { - if (!debug_pagealloc_enabled()) - return; - __kernel_map_pages(page, numpages, enable); } #ifdef CONFIG_HIBERNATION extern bool kernel_page_present(struct page *page); #endif /* CONFIG_HIBERNATION */ -#else /* CONFIG_DEBUG_PAGEALLOC */ +#else /* CONFIG_DEBUG_PAGEALLOC || CONFIG_ARCH_HAS_SET_ALIAS */ static inline void kernel_map_pages(struct page *page, int numpages, int enable) {} #ifdef CONFIG_HIBERNATION static inline bool kernel_page_present(struct page *page) { return true; } #endif /* CONFIG_HIBERNATION */ -static inline bool debug_pagealloc_enabled(void) -{ - return false; -} -#endif /* CONFIG_DEBUG_PAGEALLOC */ +#endif /* CONFIG_DEBUG_PAGEALLOC || CONFIG_ARCH_HAS_SET_ALIAS */ #ifdef __HAVE_ARCH_GATE_AREA extern struct vm_area_struct *get_gate_vma(struct mm_struct *mm); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d295c9bc01a8..c10a0d484aa6 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1074,7 +1074,8 @@ static __always_inline bool free_pages_prepare(struct page *page, } arch_free_page(page, order); kernel_poison_pages(page, 1 << order, 0); - kernel_map_pages(page, 1 << order, 0); + if (debug_pagealloc_enabled()) + kernel_map_pages(page, 1 << order, 0); kasan_free_nondeferred_pages(page, order); return true; @@ -1944,7 +1945,8 @@ inline void post_alloc_hook(struct page *page, unsigned int order, set_page_refcounted(page); arch_alloc_page(page, order); - kernel_map_pages(page, 1 << order, 1); + if (debug_pagealloc_enabled()) + kernel_map_pages(page, 1 << order, 1); kernel_poison_pages(page, 1 << order, 1); kasan_alloc_pages(page, order); set_page_owner(page, order, gfp_flags); From patchwork Thu Jan 17 00:32:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10767253 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 98F3D1390 for ; Thu, 17 Jan 2019 00:34:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 914772DDF1 for ; Thu, 17 Jan 2019 00:34:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 845D42DE41; Thu, 17 Jan 2019 00:34:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A40F72DDF1 for ; Thu, 17 Jan 2019 00:34:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E2D478E0006; Wed, 16 Jan 2019 19:33:40 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B8F0F8E000D; Wed, 16 Jan 2019 19:33:40 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8366F8E0005; Wed, 16 Jan 2019 19:33:40 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id EEEEB8E000B for ; Wed, 16 Jan 2019 19:33:39 -0500 (EST) Received: by mail-pl1-f197.google.com with SMTP id o23so4967216pll.0 for ; Wed, 16 Jan 2019 16:33:39 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=nf6gT8j6oop4bMuQY1yD+Y/EdcpsAShE0ow9e3de4XM=; b=bMaKOYYgGnvOKebEtNpHMvxlX2zGvxb+DhIwUcukW42a/+ZhsS5VgdlZav5Oyhv40H 8eSrbQxyFjzbrRSg1B1BjpHVgWskpa2HdR178tMxAnadO6i+QXpY0+VQCPiTIbD1Swud ArGlNXASgi0fpshUYP2a3/N+EwSfvFjtQX0yXFShQlkprbHEx0Zyh9XUh2gZqopA/n6y VLxcZgIG3Vp5K/w90GlAaLbGidPbTHU2Q3NcT/OibcZrGH/WtlAEo0qh1gq6h4L2HwGJ K8mabmw1c7mvzpAJwQw2rE4Hc33TDhitpgejHAufSTaoy77lOtyy1SH4/87rBGoQTJ5k g3Kg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AJcUukfU1U8B+iPCSPNx2ApFckke7T4YiZ12hUmVdWyLSfWSFMw/WB91 /jROHXQruEFAHsMs/NqIuWdtJuMRK/NK+TkGEHlx+19DHDg78IoFaqby25ZyfhsNE30gbjt9xxw EthFvtUoYqsVjTunIQ+TikIh1HsX9RKNnkmz/cp42weGZyJ6kAIWoh/6xViUV3Da9aw== X-Received: by 2002:a63:62c4:: with SMTP id w187mr10671235pgb.230.1547685219547; Wed, 16 Jan 2019 16:33:39 -0800 (PST) X-Google-Smtp-Source: ALg8bN6olsK7bGCqVs1+sP/Q2sWo9u2WP/Xo9Q/1yMdxq9hScXT4s9c4NavrttS4/gjjnZkkQcNI X-Received: by 2002:a63:62c4:: with SMTP id w187mr10671128pgb.230.1547685217989; Wed, 16 Jan 2019 16:33:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547685217; cv=none; d=google.com; s=arc-20160816; b=HywNfoFbG0OgRTzxcRkjDiWJDkG9qRTbgaZSVmO9rrWMmnTCRdQRpDdgWzxaVzqVU7 FLAzGVE2/FQvVff0b885Yn+RUIu2So+i0tKjv4lj9Q4/8PrFi3sU0FngVpUzytplZ7PH YBk9Zam0LQT+XWb+FiLCQ9yMm5JejPYnpsq68ZzGRim2k/V1+1lXrdhztQ2JjUSM1Zyd /NtucmrULu6agDFQYBRs/C9ArwqB5M+j05NUQKhLT8WZcGQ5qlyVjhTKFxuhZpizlOD+ 4qBHgxVA4FISZJWzLJTO09FLfG7/nx32BOnCJ1c9hibMmmOYo8rJ7Y70Laqn8vTNslIS FjfA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=nf6gT8j6oop4bMuQY1yD+Y/EdcpsAShE0ow9e3de4XM=; b=wu6Wuxzl0D9vUL2VtgJYgw9b8KHZkfs5xWXCOM5Z45VhJmijKKEwqgTQAroVeFdNNx LC1asawq/6wxN9XdSz8JcjRRcpF1Gm4peIn46w+UCv+dC3Q4jWhaJ4EDedegAgaQ+ApF BpZZxWVHjJj6JVKMt1Z4+rEQ+XMbmGulTUJFKCpQWpg0oDHUE7bjk1oHo7Fg6iCKHgtf fFqi47iWMpCjN9naHEt0bZkePIkdlJBpuXT1CxwSZ5f3SVD2GWnpfhNbvdj2xAkNGCyU I6NNljG2pyzBmzUJKryzrmQfIptFANEd4/LsvY1Pj9NOqhMKfVyun/jltEScKtkewvMu AkGA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga05.intel.com (mga05.intel.com. [192.55.52.43]) by mx.google.com with ESMTPS id m3si7883258pld.331.2019.01.16.16.33.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Jan 2019 16:33:37 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.43 as permitted sender) client-ip=192.55.52.43; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2019 16:33:37 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,488,1539673200"; d="scan'208";a="292166056" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.79]) by orsmga005.jf.intel.com with ESMTP; 16 Jan 2019 16:33:36 -0800 From: Rick Edgecombe To: Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Borislav Petkov , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe Subject: [PATCH 15/17] vmalloc: New flags for safe vfree on special perms Date: Wed, 16 Jan 2019 16:32:57 -0800 Message-Id: <20190117003259.23141-16-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190117003259.23141-1-rick.p.edgecombe@intel.com> References: <20190117003259.23141-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This adds a new flags VM_HAS_SPECIAL_PERMS, for enabling vfree operations to immediately clear executable TLB entries to freed pages, and handle freeing memory with special permissions. It also takes care of reseting the direct map permissions for the pages being unmapped. So this flag is useful for any kind of memory with elevated permissions, or where there can be related permissions changes on the directmap. Today this is RO+X and RO memory. Although this enables directly vfreeing RO memory now, RO memory cannot be freed in an interrupt because the allocation itself is used as a node on deferred free list. So when RO memory needs to be freed in an interrupt the code doing the vfree needs to have its own work queue, as was the case before the deferred vfree list handling was added. Today there is only one case where this happens. For architectures with set_alias_ implementations this whole operation can be done with one TLB flush when centralized like this. For others with directmap permissions, currently only arm64, a backup method using set_memory functions is used to reset the directmap. When arm64 adds set_alias_ functions, this backup can be removed. When the TLB is flushed to both remove TLB entries for the vmalloc range mapping and the direct map permissions, the lazy purge operation could be done to try to save a TLB flush later. However today vm_unmap_aliases could flush a TLB range that does not include the directmap. So a helper is added with extra parameters that can allow both the vmalloc address and the direct mapping to be flushed during this operation. The behavior of the normal vm_unmap_aliases function is unchanged. Suggested-by: Dave Hansen Suggested-by: Andy Lutomirski Suggested-by: Will Deacon Signed-off-by: Rick Edgecombe --- include/linux/vmalloc.h | 13 +++++ mm/vmalloc.c | 122 +++++++++++++++++++++++++++++++++------- 2 files changed, 116 insertions(+), 19 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 398e9c95cd61..9f643f917360 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -21,6 +21,11 @@ struct notifier_block; /* in notifier.h */ #define VM_UNINITIALIZED 0x00000020 /* vm_struct is not fully initialized */ #define VM_NO_GUARD 0x00000040 /* don't add guard page */ #define VM_KASAN 0x00000080 /* has allocated kasan shadow memory */ +/* + * Memory with VM_HAS_SPECIAL_PERMS cannot be freed in an interrupt or with + * vfree_atomic. + */ +#define VM_HAS_SPECIAL_PERMS 0x00000200 /* Reset directmap and flush TLB on unmap */ /* bits [20..32] reserved for arch specific ioremap internals */ /* @@ -135,6 +140,14 @@ extern struct vm_struct *__get_vm_area_caller(unsigned long size, extern struct vm_struct *remove_vm_area(const void *addr); extern struct vm_struct *find_vm_area(const void *addr); +static inline void set_vm_special(void *addr) +{ + struct vm_struct *vm = find_vm_area(addr); + + if (vm) + vm->flags |= VM_HAS_SPECIAL_PERMS; +} + extern int map_vm_area(struct vm_struct *area, pgprot_t prot, struct page **pages); #ifdef CONFIG_MMU diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 871e41c55e23..d459b5b9649b 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -1055,24 +1056,11 @@ static void vb_free(const void *addr, unsigned long size) spin_unlock(&vb->lock); } -/** - * vm_unmap_aliases - unmap outstanding lazy aliases in the vmap layer - * - * The vmap/vmalloc layer lazily flushes kernel virtual mappings primarily - * to amortize TLB flushing overheads. What this means is that any page you - * have now, may, in a former life, have been mapped into kernel virtual - * address by the vmap layer and so there might be some CPUs with TLB entries - * still referencing that page (additional to the regular 1:1 kernel mapping). - * - * vm_unmap_aliases flushes all such lazy mappings. After it returns, we can - * be sure that none of the pages we have control over will have any aliases - * from the vmap layer. - */ -void vm_unmap_aliases(void) +static void _vm_unmap_aliases(unsigned long start, unsigned long end, + int must_flush) { - unsigned long start = ULONG_MAX, end = 0; int cpu; - int flush = 0; + int flush = must_flush; if (unlikely(!vmap_initialized)) return; @@ -1109,6 +1097,27 @@ void vm_unmap_aliases(void) flush_tlb_kernel_range(start, end); mutex_unlock(&vmap_purge_lock); } + +/** + * vm_unmap_aliases - unmap outstanding lazy aliases in the vmap layer + * + * The vmap/vmalloc layer lazily flushes kernel virtual mappings primarily + * to amortize TLB flushing overheads. What this means is that any page you + * have now, may, in a former life, have been mapped into kernel virtual + * address by the vmap layer and so there might be some CPUs with TLB entries + * still referencing that page (additional to the regular 1:1 kernel mapping). + * + * vm_unmap_aliases flushes all such lazy mappings. After it returns, we can + * be sure that none of the pages we have control over will have any aliases + * from the vmap layer. + */ +void vm_unmap_aliases(void) +{ + unsigned long start = ULONG_MAX, end = 0; + int must_flush = 0; + + _vm_unmap_aliases(start, end, must_flush); +} EXPORT_SYMBOL_GPL(vm_unmap_aliases); /** @@ -1494,6 +1503,79 @@ struct vm_struct *remove_vm_area(const void *addr) return NULL; } +static inline void set_area_alias(const struct vm_struct *area, + int (*set_alias)(struct page *page)) +{ + int i; + + for (i = 0; i < area->nr_pages; i++) { + unsigned long addr = + (unsigned long)page_address(area->pages[i]); + + if (addr) + set_alias(area->pages[i]); + } +} + +/* This handles removing and resetting vm mappings related to the vm_struct. */ +static void vm_remove_mappings(struct vm_struct *area, int deallocate_pages) +{ + unsigned long addr = (unsigned long)area->addr; + unsigned long start = ULONG_MAX, end = 0; + int special = area->flags & VM_HAS_SPECIAL_PERMS; + int i; + + /* + * The below block can be removed when all architectures that have + * direct map permissions also have set_alias_ implementations. This is + * to do resetting on the directmap for any special permissions (today + * only X), without leaving a RW+X window. + */ + if (special && !IS_ENABLED(CONFIG_ARCH_HAS_SET_ALIAS)) { + set_memory_nx(addr, area->nr_pages); + set_memory_rw(addr, area->nr_pages); + } + + remove_vm_area(area->addr); + + /* If this is not special memory, we can skip the below. */ + if (!special) + return; + + /* + * If we are not deallocating pages, we can just do the flush of the VM + * area and return. + */ + if (!deallocate_pages) { + vm_unmap_aliases(); + return; + } + + /* + * If we are here, we need to flush the vm mapping and reset the direct + * map. + * First find the start and end range of the direct mappings to make + * sure the vm_unmap_aliases flush includes the direct map. + */ + for (i = 0; i < area->nr_pages; i++) { + unsigned long addr = + (unsigned long)page_address(area->pages[i]); + if (addr) { + start = min(addr, start); + end = max(addr, end); + } + } + + /* + * First we set direct map to something not valid so that it won't be + * cached if there are any accesses after the TLB flush, then we flush + * the TLB, and reset the directmap permissions to the default. + */ + set_area_alias(area, set_alias_nv_noflush); + _vm_unmap_aliases(start, end, 1); + set_area_alias(area, set_alias_default_noflush); +} + static void __vunmap(const void *addr, int deallocate_pages) { struct vm_struct *area; @@ -1515,7 +1597,8 @@ static void __vunmap(const void *addr, int deallocate_pages) debug_check_no_locks_freed(area->addr, get_vm_area_size(area)); debug_check_no_obj_freed(area->addr, get_vm_area_size(area)); - remove_vm_area(addr); + vm_remove_mappings(area, deallocate_pages); + if (deallocate_pages) { int i; @@ -1925,8 +2008,9 @@ EXPORT_SYMBOL(vzalloc_node); void *vmalloc_exec(unsigned long size) { - return __vmalloc_node(size, 1, GFP_KERNEL, PAGE_KERNEL_EXEC, - NUMA_NO_NODE, __builtin_return_address(0)); + return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, + GFP_KERNEL, PAGE_KERNEL_EXEC, VM_HAS_SPECIAL_PERMS, + NUMA_NO_NODE, __builtin_return_address(0)); } #if defined(CONFIG_64BIT) && defined(CONFIG_ZONE_DMA32) From patchwork Thu Jan 17 00:32:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10767293 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EE454186E for ; Thu, 17 Jan 2019 00:34:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 37AD12DDDC for ; Thu, 17 Jan 2019 00:34:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2B4872E2B7; Thu, 17 Jan 2019 00:34:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0D03B2DDDC for ; Thu, 17 Jan 2019 00:34:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 59B868E0010; Wed, 16 Jan 2019 19:33:42 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 37DBA8E0014; Wed, 16 Jan 2019 19:33:42 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1AA658E0010; Wed, 16 Jan 2019 19:33:42 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 88F8D8E000F for ; Wed, 16 Jan 2019 19:33:41 -0500 (EST) Received: by mail-pg1-f200.google.com with SMTP id i124so5030652pgc.2 for ; Wed, 16 Jan 2019 16:33:41 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=xW6CRRJwi1KYibaizGiM8dXCI+UcQIffrtO8Ze96gxM=; b=A+HgKHv7YDlPr3LAlPcyzaI9otG/btGhH7yt0wPgQgf70pZNjxi1Xam1eAzYGZIq9B T5hq69RbPaiCHtQNGz80JoPFAs/Im3pvXK7bNT74MBEAkKztcGELD7KW0Fy0wkFuVFZ/ 80IaEHFe3BOATL+1bWHBodvKuOKjLaLRjxbnojkBEeTAve8z1DNnLJQm4bkCgncaeBuw TH8V1U5SKhhX7WWOjDghoSjKe5BJbckQW5MVbYA7dBU4ihB8BrWz3f7oKVfaYKOVKdhA SaXoyhtmpB5ULSTBiFUAGbf46FGn5tW58kAecE7ciUMlv0hZij+h0+AnwF8I4ihFyd0P 2rOg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AJcUukeD74P5bE5WTERtrYhDQD0pu9UurPSokSgURRbwBbTYuyyl9kVm /G64kIoVWIlN0sOd6RiW7C5Q1/Yt/4hI1PAU/27FMyXTu7jgggujdNRtqvhOXfDT3oCXDMX2VrT wRUNmUGoCe+3bzGLZ/G17vPVYX60ywOexjxWTEVZEZWEPb4x3VfKOQp1skff9CJN3Ug== X-Received: by 2002:a63:8ac4:: with SMTP id y187mr11552978pgd.446.1547685221134; Wed, 16 Jan 2019 16:33:41 -0800 (PST) X-Google-Smtp-Source: ALg8bN4zGzDWvK82v6tlCeDtDOHy+Lm93Nli8j3sTAUQTxDtNKofmEeeqFM9cQ8XvtpwQ4UtZaft X-Received: by 2002:a63:8ac4:: with SMTP id y187mr11552901pgd.446.1547685220064; Wed, 16 Jan 2019 16:33:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547685220; cv=none; d=google.com; s=arc-20160816; b=o8bp1QD92mCPa0+q/PCMTHz7V4Be6M7Ugp3eyRRut/Smej1pKQ8tovibSE2ItyowHI KkIZluJ5GxzVcGtrRFsK+4MGiEWAZ6NVX+NMJZbd7DbLSEE3AJ1aw1o+qKdg5MWH6wl2 66r8aee5sXmaqj4zlm9XSM2g7R9Psz5BtS82rs0uIhaX2bWwrnaLLTKqI7pyS/dQa8ke ZZGq1un1KHqfld6NBEgUQL6FzgcdecnOPVW6bt6Bwh/I2mZtZlmpBxjXTRsGr90Ej7en y+DtQhGZs4OhUVyj1DrxYo6N8D8n/J/Twf3vWwrda0fMYXph9heMlN8o5Qcl+O/oUDRd Gu/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=xW6CRRJwi1KYibaizGiM8dXCI+UcQIffrtO8Ze96gxM=; b=u+R0X+2AroLnWQhhglTQL0lgFNqyzoGQHQ7PeLJJ/2LzW5Y39sI3CtqYhB2KMHPU6i nvGPC49GehzrfdBZwpVkyc7RuQFRahGwUC6h5IRCrdIuR7vkt/seUr50n35eZahVabSg DtFicE+nFehZugQoonyqgyv3JGqONMqJv3frr3URJ9daQcWX7+5aMEP5TsvzzdeJx8nn NEomDiWDz28lcwCmxFgMydLkdWooEuEDyvTuMPDx2MsKFbQ+AOPdFvYAtLSvZVQxeE2m jj3k/eJFmwRjL58CTOSR17G606ogRRf9HXRcN3bMj1oynQrZxYLbXfeFwI1BpDHXcyql p/wg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga04.intel.com (mga04.intel.com. [192.55.52.120]) by mx.google.com with ESMTPS id 1si7848435plo.195.2019.01.16.16.33.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Jan 2019 16:33:40 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) client-ip=192.55.52.120; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2019 16:33:37 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,488,1539673200"; d="scan'208";a="292166060" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.79]) by orsmga005.jf.intel.com with ESMTP; 16 Jan 2019 16:33:36 -0800 From: Rick Edgecombe To: Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Borislav Petkov , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Rusty Russell , Masami Hiramatsu , Daniel Borkmann , Alexei Starovoitov , Jessica Yu , Steven Rostedt , "Paul E . McKenney" Subject: [PATCH 16/17] Plug in new special vfree flag Date: Wed, 16 Jan 2019 16:32:58 -0800 Message-Id: <20190117003259.23141-17-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190117003259.23141-1-rick.p.edgecombe@intel.com> References: <20190117003259.23141-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add new flag for handling freeing of special permissioned memory in vmalloc and remove places where memory was set RW before freeing which is no longer needed. In kprobes, bpf and ftrace this just adds the flag, and removes the now unneeded set_memory_ calls before calling vfree. In modules, the freeing of init sections is moved to a work queue, since freeing of RO memory is not supported in an interrupt by vmalloc. Instead of call_rcu, it now uses synchronize_rcu() in the work queue. Cc: Rusty Russell Cc: Masami Hiramatsu Cc: Daniel Borkmann Cc: Alexei Starovoitov Cc: Jessica Yu Cc: Steven Rostedt Cc: Paul E. McKenney Signed-off-by: Rick Edgecombe Acked-by: Steven Rostedt (VMware) --- arch/x86/kernel/ftrace.c | 6 +-- arch/x86/kernel/kprobes/core.c | 7 +--- include/linux/filter.h | 16 ++----- kernel/bpf/core.c | 1 - kernel/module.c | 77 +++++++++++++++++----------------- 5 files changed, 45 insertions(+), 62 deletions(-) diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index eb4a1937e72c..47597e028346 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -692,10 +692,6 @@ static inline void *alloc_tramp(unsigned long size) } static inline void tramp_free(void *tramp, int size) { - int npages = PAGE_ALIGN(size) >> PAGE_SHIFT; - - set_memory_nx((unsigned long)tramp, npages); - set_memory_rw((unsigned long)tramp, npages); module_memfree(tramp); } #else @@ -820,6 +816,8 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) /* ALLOC_TRAMP flags lets us know we created it */ ops->flags |= FTRACE_OPS_FL_ALLOC_TRAMP; + set_vm_special(trampoline); + /* * Module allocation needs to be completed by making the page * executable. The page is still writable, which is a security hazard, diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index fac692e36833..f2fab35bcb82 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -434,6 +434,7 @@ void *alloc_insn_page(void) if (page == NULL) return NULL; + set_vm_special(page); /* * First make the page read-only, and then only then make it executable * to prevent it from being W+X in between. @@ -452,12 +453,6 @@ void *alloc_insn_page(void) /* Recover page to RW mode before releasing it */ void free_insn_page(void *page) { - /* - * First make the page non-executable, and then only then make it - * writable to prevent it from being W+X in between. - */ - set_memory_nx((unsigned long)page, 1); - set_memory_rw((unsigned long)page, 1); module_memfree(page); } diff --git a/include/linux/filter.h b/include/linux/filter.h index f18cd317faf8..0abe812e7b75 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -20,6 +20,7 @@ #include #include #include +#include #include @@ -483,7 +484,6 @@ struct bpf_prog { u16 pages; /* Number of allocated pages */ u16 jited:1, /* Is our filter JIT'ed? */ jit_requested:1,/* archs need to JIT the prog */ - undo_set_mem:1, /* Passed set_memory_ro() checkpoint */ gpl_compatible:1, /* Is filter GPL compatible? */ cb_access:1, /* Is control block accessed? */ dst_needed:1, /* Do we need dst entry? */ @@ -681,26 +681,17 @@ bpf_ctx_narrow_access_ok(u32 off, u32 size, u32 size_default) static inline void bpf_prog_lock_ro(struct bpf_prog *fp) { + set_vm_special(fp); set_memory_ro((unsigned long)fp, fp->pages); } -static inline void bpf_prog_unlock_ro(struct bpf_prog *fp) -{ - if (fp->undo_set_mem) - set_memory_rw((unsigned long)fp, fp->pages); -} - static inline void bpf_jit_binary_lock_ro(struct bpf_binary_header *hdr) { + set_vm_special(hdr); set_memory_ro((unsigned long)hdr, hdr->pages); set_memory_x((unsigned long)hdr, hdr->pages); } -static inline void bpf_jit_binary_unlock_ro(struct bpf_binary_header *hdr) -{ - set_memory_rw((unsigned long)hdr, hdr->pages); -} - static inline struct bpf_binary_header * bpf_jit_binary_hdr(const struct bpf_prog *fp) { @@ -735,7 +726,6 @@ void __bpf_prog_free(struct bpf_prog *fp); static inline void bpf_prog_unlock_free(struct bpf_prog *fp) { - bpf_prog_unlock_ro(fp); __bpf_prog_free(fp); } diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index f908b9356025..a1a4d6f4253c 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -804,7 +804,6 @@ void __weak bpf_jit_free(struct bpf_prog *fp) if (fp->jited) { struct bpf_binary_header *hdr = bpf_jit_binary_hdr(fp); - bpf_jit_binary_unlock_ro(hdr); bpf_jit_binary_free(hdr); WARN_ON_ONCE(!bpf_prog_kallsyms_verify_off(fp)); diff --git a/kernel/module.c b/kernel/module.c index ae1b77da6a20..1af5c8e19086 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -98,6 +98,10 @@ DEFINE_MUTEX(module_mutex); EXPORT_SYMBOL_GPL(module_mutex); static LIST_HEAD(modules); +/* Work queue for freeing init sections in success case */ +static struct work_struct init_free_wq; +static struct llist_head init_free_list; + #ifdef CONFIG_MODULES_TREE_LOOKUP /* @@ -1949,6 +1953,8 @@ void module_enable_ro(const struct module *mod, bool after_init) if (!rodata_enabled) return; + set_vm_special(mod->core_layout.base); + set_vm_special(mod->init_layout.base); frob_text(&mod->core_layout, set_memory_ro); frob_text(&mod->core_layout, set_memory_x); @@ -1972,15 +1978,6 @@ static void module_enable_nx(const struct module *mod) frob_writable_data(&mod->init_layout, set_memory_nx); } -static void module_disable_nx(const struct module *mod) -{ - frob_rodata(&mod->core_layout, set_memory_x); - frob_ro_after_init(&mod->core_layout, set_memory_x); - frob_writable_data(&mod->core_layout, set_memory_x); - frob_rodata(&mod->init_layout, set_memory_x); - frob_writable_data(&mod->init_layout, set_memory_x); -} - /* Iterate through all modules and set each module's text as RW */ void set_all_modules_text_rw(void) { @@ -2024,23 +2021,8 @@ void set_all_modules_text_ro(void) } mutex_unlock(&module_mutex); } - -static void disable_ro_nx(const struct module_layout *layout) -{ - if (rodata_enabled) { - frob_text(layout, set_memory_rw); - frob_rodata(layout, set_memory_rw); - frob_ro_after_init(layout, set_memory_rw); - } - frob_rodata(layout, set_memory_x); - frob_ro_after_init(layout, set_memory_x); - frob_writable_data(layout, set_memory_x); -} - #else -static void disable_ro_nx(const struct module_layout *layout) { } static void module_enable_nx(const struct module *mod) { } -static void module_disable_nx(const struct module *mod) { } #endif #ifdef CONFIG_LIVEPATCH @@ -2120,6 +2102,11 @@ static void free_module_elf(struct module *mod) void __weak module_memfree(void *module_region) { + /* + * This memory may be RO, and freeing RO memory in an interrupt is not + * supported by vmalloc. + */ + WARN_ON(in_interrupt()); vfree(module_region); } @@ -2171,7 +2158,6 @@ static void free_module(struct module *mod) mutex_unlock(&module_mutex); /* This may be empty, but that's OK */ - disable_ro_nx(&mod->init_layout); module_arch_freeing_init(mod); module_memfree(mod->init_layout.base); kfree(mod->args); @@ -2181,7 +2167,6 @@ static void free_module(struct module *mod) lockdep_free_key_range(mod->core_layout.base, mod->core_layout.size); /* Finally, free the core (containing the module structure) */ - disable_ro_nx(&mod->core_layout); module_memfree(mod->core_layout.base); } @@ -3424,17 +3409,34 @@ static void do_mod_ctors(struct module *mod) /* For freeing module_init on success, in case kallsyms traversing */ struct mod_initfree { - struct rcu_head rcu; + struct llist_node node; void *module_init; }; -static void do_free_init(struct rcu_head *head) +static void do_free_init(struct work_struct *w) { - struct mod_initfree *m = container_of(head, struct mod_initfree, rcu); - module_memfree(m->module_init); - kfree(m); + struct llist_node *pos, *n, *list; + struct mod_initfree *initfree; + + list = llist_del_all(&init_free_list); + + synchronize_rcu(); + + llist_for_each_safe(pos, n, list) { + initfree = container_of(pos, struct mod_initfree, node); + module_memfree(initfree->module_init); + kfree(initfree); + } } +static int __init modules_wq_init(void) +{ + INIT_WORK(&init_free_wq, do_free_init); + init_llist_head(&init_free_list); + return 0; +} +module_init(modules_wq_init); + /* * This is where the real work happens. * @@ -3511,7 +3513,6 @@ static noinline int do_init_module(struct module *mod) #endif module_enable_ro(mod, true); mod_tree_remove_init(mod); - disable_ro_nx(&mod->init_layout); module_arch_freeing_init(mod); mod->init_layout.base = NULL; mod->init_layout.size = 0; @@ -3522,14 +3523,18 @@ static noinline int do_init_module(struct module *mod) * We want to free module_init, but be aware that kallsyms may be * walking this with preempt disabled. In all the failure paths, we * call synchronize_rcu(), but we don't want to slow down the success - * path, so use actual RCU here. + * path. We can't do module_memfree in an interrupt, so we do the work + * and call synchronize_rcu() in a work queue. + * * Note that module_alloc() on most architectures creates W+X page * mappings which won't be cleaned up until do_free_init() runs. Any * code such as mark_rodata_ro() which depends on those mappings to * be cleaned up needs to sync with the queued work - ie * rcu_barrier() */ - call_rcu(&freeinit->rcu, do_free_init); + if (llist_add(&freeinit->node, &init_free_list)) + schedule_work(&init_free_wq); + mutex_unlock(&module_mutex); wake_up_all(&module_wq); @@ -3826,10 +3831,6 @@ static int load_module(struct load_info *info, const char __user *uargs, module_bug_cleanup(mod); mutex_unlock(&module_mutex); - /* we can't deallocate the module until we clear memory protection */ - module_disable_ro(mod); - module_disable_nx(mod); - ddebug_cleanup: ftrace_release_mod(mod); dynamic_debug_remove(mod, info->debug); From patchwork Thu Jan 17 00:32:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10767241 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 341F514E5 for ; Thu, 17 Jan 2019 00:33:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3A5AA2DDF1 for ; Thu, 17 Jan 2019 00:33:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2D4722DFD8; Thu, 17 Jan 2019 00:33:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AD0BF2DDF1 for ; Thu, 17 Jan 2019 00:33:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4DD528E000C; Wed, 16 Jan 2019 19:33:40 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 418328E000A; Wed, 16 Jan 2019 19:33:40 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E28108E0006; Wed, 16 Jan 2019 19:33:39 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 91A338E0006 for ; Wed, 16 Jan 2019 19:33:39 -0500 (EST) Received: by mail-pf1-f197.google.com with SMTP id 68so6023689pfr.6 for ; Wed, 16 Jan 2019 16:33:39 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Dpr92+KbHY9cTHM1Ui4q2NtOJyMHY6aShd4aXoAfyzo=; b=k8BtYelLB3+4FYbABOlpFg6jkLKcFoopafO2m+58jIirmKrTCXwJeYT/V8X9buV6I8 xLr1WB8pEI6QPO3tbzMxGhf4trPM6W9rtlLt4OWKJWTjekpf3EtCEV4872UldH7W1jYb bR6szyh/71nsaslAPCfjAE1M6+97gBpAfVEpt5FvJoUTHJsqUyWkqgNvA7o7u8kiWzen TOUgn0Qpya9UgZalpmxbecACRY8iu/c7sE7vLiA3WEr71w6RXZleFcQj6K8ZOICwsRVZ MkRHuprfvQiNj1g0Hs8YBUic5Z7HVdBhg6ZzuK33QcmrhBjoJhoXdu7o/Ciunmc1CXeC 0PrA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AJcUukfDQjmP/OIWUmopqX7aQmejzWNXRDtNS/CB6Z7Azd26qP06R/2W w7NLSGknyu3Ms+ZllABF4J833dfmqtR7obVF4AC5VgDhvLnMYeL9iLDtQyE8IXPFRWebgoAQ8b8 CTcJIdL0vRy9P0+9Lcnk5IiojDu65QCV54xG7Rrm7ZeBn6CltLVl/XdW5lpL3Qe0CDw== X-Received: by 2002:a63:77ce:: with SMTP id s197mr11374241pgc.89.1547685219228; Wed, 16 Jan 2019 16:33:39 -0800 (PST) X-Google-Smtp-Source: ALg8bN6nnawnKgpMWamhf71b73pzBgdP3JqdKu6AO4s88QvIonDfrtnz9vC9E2d7+bXsSlOINpso X-Received: by 2002:a63:77ce:: with SMTP id s197mr11374173pgc.89.1547685218327; Wed, 16 Jan 2019 16:33:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547685218; cv=none; d=google.com; s=arc-20160816; b=ADodZWoIJFRAVd/gtPWLQWE40BlYla7m1gPgu+PF/WWXiVuEgkjnHpnqxq5HbvB1uo N3RXe3acRM9DlbT5r1S4SkfxVVfxkzr3IAeUDgmk6eDWgmjuZI4fYCCYrtS18KotE1ov oar/YRq7TxD4KUaLi5wgfeI8kDRMDzmosX9M7sO7nwCm1sKtBg4SRAs9v6c+cUZFuks3 Ic6Ax54WDox6EmTAG3oj3whtECMoe34mWZq4ODZGjL9UkF8XWLrJF+tD1IxUrKx7mtyV 1x5cscgOqJnY5SWyF5/txwUVABUKmuGaqYUeL0uK+HqwxrKR8LE4wR9Z/asbLGjKAuGr hd+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=Dpr92+KbHY9cTHM1Ui4q2NtOJyMHY6aShd4aXoAfyzo=; b=07qZmHPHkSTiXlkMNkDP3a98U3TsGaVRTkwur82/75iJDYVXwenAU1lIb3lqOgaj0j e2wzIAzT+39iX/cJRr62+dA/kOZK8j2wIhsVyrXpqjLvUVOy5X/1TMsfV7clgAbAsPll FkeySiQfpPtJ5aPSFaM2I+JbPSo9ew++QGhKRpldTRdUdmOH4mvr3vHrkvDf+m2vMlZi dPfoHVQdakO5OKvaeV5xGJiMYYBUfDTzO9CiUFMdEcZVXF9lVJBfOad7ekxKvze+36zT L3BpEyRR/ro1KibhgJLmYbGj8NVpvVYaZoekHXvCxdrmRGbCAuWLWq5BOxS6U/dZdEVU mM1w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga05.intel.com (mga05.intel.com. [192.55.52.43]) by mx.google.com with ESMTPS id m3si7883258pld.331.2019.01.16.16.33.38 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Jan 2019 16:33:38 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.43 as permitted sender) client-ip=192.55.52.43; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Jan 2019 16:33:37 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,488,1539673200"; d="scan'208";a="292166062" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.79]) by orsmga005.jf.intel.com with ESMTP; 16 Jan 2019 16:33:36 -0800 From: Rick Edgecombe To: Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Borislav Petkov , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Nadav Amit , Rick Edgecombe Subject: [PATCH 17/17] module: Prevent module removal racing with text_poke() Date: Wed, 16 Jan 2019 16:32:59 -0800 Message-Id: <20190117003259.23141-18-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190117003259.23141-1-rick.p.edgecombe@intel.com> References: <20190117003259.23141-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit It seems dangerous to allow code modifications to take place concurrently with module unloading. So take the text_mutex while the memory of the module is freed. Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- kernel/module.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/kernel/module.c b/kernel/module.c index 1af5c8e19086..90cfc4988d98 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -64,6 +64,7 @@ #include #include #include +#include #include #include "module-internal.h" @@ -2157,6 +2158,9 @@ static void free_module(struct module *mod) synchronize_rcu(); mutex_unlock(&module_mutex); + /* Protect against patching of the module while it is being removed */ + mutex_lock(&text_mutex); + /* This may be empty, but that's OK */ module_arch_freeing_init(mod); module_memfree(mod->init_layout.base); @@ -2168,6 +2172,7 @@ static void free_module(struct module *mod) /* Finally, free the core (containing the module structure) */ module_memfree(mod->core_layout.base); + mutex_unlock(&text_mutex); } void *__symbol_get(const char *symbol)