From patchwork Fri Apr 26 23:22:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10919921 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4C909912 for ; Sat, 27 Apr 2019 06:43:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3B56328E6B for ; Sat, 27 Apr 2019 06:43:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2EA1828E7B; Sat, 27 Apr 2019 06:43:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5862C28E6B for ; Sat, 27 Apr 2019 06:43:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ACF576B0005; Sat, 27 Apr 2019 02:43:06 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A827C6B0006; Sat, 27 Apr 2019 02:43:06 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8FBC16B0008; Sat, 27 Apr 2019 02:43:06 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id 5294A6B0005 for ; Sat, 27 Apr 2019 02:43:06 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id z12so3498299pgs.4 for ; Fri, 26 Apr 2019 23:43:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=2uQh+p4niM1XazCyX0W8SClLNopGSLFeOTKZKD97XRE=; b=MLfmDTCFjAK0unw/9p9YdFGERYFscN6J1WTT/q/Ub0fI1DD0hpfwmdGplxSf8W5KSG JtpXLcge0XNcGpXligs4OD0NT6bIt6fKMLKR8VVZT2ImVQEOeq9cR4/o6Elg+V1jqcqs QFmqVUho1WzTiTiW8ph7+/wVAIsk6u+I9R8ZLp4Q2454QlRl/XSGcDw339cpANFuYZx0 ZDWN9SoEBrwo24AvKI6sY6GFX1aKl8GR0sEiGz+ts/YP60ec5Wm8u0cEQuFyUlV1WiWy k8vI9yN9flXOYrSPfYDA1XXfVZci6+MHrKfJOEc++uXBfBw3cw9fvfZb3kdQNQQ56hIQ VLyg== X-Gm-Message-State: APjAAAXcZlFAh8vME5fKirS0dM36AfNBkjdPr5cY6MldIrW0hOY7xSji M5KGjFJJ5Kdfkobx+8SCh4qgJUSLvQtU9hpkhIy0F+o+npcrkdeYy4o/JpKNy5nLqcE0fazci9H TbWIdE0G2uGfArfOjqKP1ZhW0j/7r02h1sEQd4LyAHK1nU4GwRu7UhKscocksCNYAvw== X-Received: by 2002:a62:f24e:: with SMTP id y14mr51417190pfl.209.1556347385979; Fri, 26 Apr 2019 23:43:05 -0700 (PDT) X-Received: by 2002:a62:f24e:: with SMTP id y14mr51417122pfl.209.1556347384892; Fri, 26 Apr 2019 23:43:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347384; cv=none; d=google.com; s=arc-20160816; b=Umtte7N4qPDCYJsU8OPEKA+shf32FsUK3pUYgiKqrFBiqg0ek+Hh/0G4vV9lbKxJcF PuytvHWgGKs9ZDi+ToKj6ARoZgs01/sQdfbh1iF+RpubUPFa12XXwzBSFQKK/BCRshjj ZcZMxu9jh4VWgqdsCQHzZpIlc3Eec2LD5vmXcLRK8Yy8fCeo25PFXEa6CeGC3Zug03fn GDwTJyBCmb6PfF3vDvnE8x4pDc2t2KWExh6ph40Sj0pAqu4KHMNRMKjhvOyQ24M1XLbo q67h6ZsyWGvcLT8TkXK2AFDcFwCAIlOf6BLUPbm5KG2i0eexIbffdxVSzRfYE6/nPy54 xWDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=2uQh+p4niM1XazCyX0W8SClLNopGSLFeOTKZKD97XRE=; b=NUcHWn08ZyDLl+V59p7WFA3jWtgWJbSzLu06pkXiNP8U5MXaqX1zGLGhsBwYsbnHyJ 6s52i8PS4r2CaDb/+GI8KO+hcCcayzEBMmnD8n3qLG844+7/8WgTu4oCUoSVlyKvCMtI 12GmR5eVWEMQrvKrNMNIlkvIIUHNTl9W2cfm1cps4VQrzZpwbSwGYHb26JX/pZCUTGy7 ROXFk6ZAhKufbV/m4m9m/Y9ptohU3LvJeN+UVlt4Mcud8SHDz8cCM33+VGd1u1Ul6S74 whjdtHLJPkfvhKgpujB453uBpHIy8dbOiCQ6S6M2NCLK4zi7Y9vAawcuNHx7m3v52yij BFJQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=jRSkWPKF; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id p66sor30592729pfp.35.2019.04.26.23.43.04 for (Google Transport Security); Fri, 26 Apr 2019 23:43:04 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=jRSkWPKF; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=2uQh+p4niM1XazCyX0W8SClLNopGSLFeOTKZKD97XRE=; b=jRSkWPKFy+aX307nFA9nwPtBSyUtI3Rx2wzXkDoawmNR9EBxnEgbgOuHqpJlUzURZT sV7k69m+oa16kaqF4hrYU8VlNGgLedRPtIQMrWtFfI1lYATZNKlBhFfYsdwu71vQpo8l 4hsC4E1OYjLOCvcQc0ZNw7818nwUnGdICNM7KZF/Pj8oVL04L5HnrLC4hGuv9gcmfZac QEF3y8KkPlAdKQa7pk8VGNHlHNoii/EZVaGEr6IiXqB8Q9AYGmnR5/+s61lipbxEVDy4 WVZil8tiOMGEBeIN6IOofRQtzUr/x76R5xEihGyGfY+KVuW7UGM6aZN17saLTTNxIC81 Y9Xg== X-Google-Smtp-Source: APXvYqxfDuRpBbYxO71iQgRCpp9YwcxdysZGYp8xW7ZBvNhCbiRnBK6qmFG1IHuGhuou8D2zduF20Q== X-Received: by 2002:a62:26c1:: with SMTP id m184mr12194274pfm.102.1556347384381; Fri, 26 Apr 2019 23:43:04 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:03 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu Subject: [PATCH v6 01/24] Fix "x86/alternatives: Lockdep-enforce text_mutex in text_poke*()" Date: Fri, 26 Apr 2019 16:22:40 -0700 Message-Id: <20190426232303.28381-2-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit text_mutex is currently expected to be held before text_poke() is called, but kgdb does not take the mutex, and instead *supposedly* ensures the lock is not taken and will not be acquired by any other core while text_poke() is running. The reason for the "supposedly" comment is that it is not entirely clear that this would be the case if gdb_do_roundup is zero. Create two wrapper functions, text_poke() and text_poke_kgdb(), which do or do not run the lockdep assertion respectively. While we are at it, change the return code of text_poke() to something meaningful. One day, callers might actually respect it and the existing BUG_ON() when patching fails could be removed. For kgdb, the return value can actually be used. Cc: Andy Lutomirski Cc: Kees Cook Cc: Dave Hansen Cc: Masami Hiramatsu Fixes: 9222f606506c ("x86/alternatives: Lockdep-enforce text_mutex in text_poke*()") Suggested-by: Peter Zijlstra Acked-by: Jiri Kosina Acked-by: Peter Zijlstra (Intel) Reviewed-by: Masami Hiramatsu Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/text-patching.h | 1 + arch/x86/kernel/alternative.c | 52 ++++++++++++++++++++-------- arch/x86/kernel/kgdb.c | 11 +++--- 3 files changed, 45 insertions(+), 19 deletions(-) diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h index e85ff65c43c3..f8fc8e86cf01 100644 --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -35,6 +35,7 @@ extern void *text_poke_early(void *addr, const void *opcode, size_t len); * inconsistent instruction while you patch. */ extern void *text_poke(void *addr, const void *opcode, size_t len); +extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len); extern int poke_int3_handler(struct pt_regs *regs); extern void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler); extern int after_bootmem; diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 9a79c7808f9c..0a814d73547a 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -679,18 +679,7 @@ void *__init_or_module text_poke_early(void *addr, const void *opcode, return addr; } -/** - * text_poke - Update instructions on a live kernel - * @addr: address to modify - * @opcode: source of the copy - * @len: length to copy - * - * Only atomic text poke/set should be allowed when not doing early patching. - * It means the size must be writable atomically and the address must be aligned - * in a way that permits an atomic write. It also makes sure we fit on a single - * page. - */ -void *text_poke(void *addr, const void *opcode, size_t len) +static void *__text_poke(void *addr, const void *opcode, size_t len) { unsigned long flags; char *vaddr; @@ -703,8 +692,6 @@ void *text_poke(void *addr, const void *opcode, size_t len) */ BUG_ON(!after_bootmem); - lockdep_assert_held(&text_mutex); - if (!core_kernel_text((unsigned long)addr)) { pages[0] = vmalloc_to_page(addr); pages[1] = vmalloc_to_page(addr + PAGE_SIZE); @@ -733,6 +720,43 @@ void *text_poke(void *addr, const void *opcode, size_t len) return addr; } +/** + * text_poke - Update instructions on a live kernel + * @addr: address to modify + * @opcode: source of the copy + * @len: length to copy + * + * Only atomic text poke/set should be allowed when not doing early patching. + * It means the size must be writable atomically and the address must be aligned + * in a way that permits an atomic write. It also makes sure we fit on a single + * page. + */ +void *text_poke(void *addr, const void *opcode, size_t len) +{ + lockdep_assert_held(&text_mutex); + + return __text_poke(addr, opcode, len); +} + +/** + * text_poke_kgdb - Update instructions on a live kernel by kgdb + * @addr: address to modify + * @opcode: source of the copy + * @len: length to copy + * + * Only atomic text poke/set should be allowed when not doing early patching. + * It means the size must be writable atomically and the address must be aligned + * in a way that permits an atomic write. It also makes sure we fit on a single + * page. + * + * Context: should only be used by kgdb, which ensures no other core is running, + * despite the fact it does not hold the text_mutex. + */ +void *text_poke_kgdb(void *addr, const void *opcode, size_t len) +{ + return __text_poke(addr, opcode, len); +} + static void do_sync_core(void *info) { sync_core(); diff --git a/arch/x86/kernel/kgdb.c b/arch/x86/kernel/kgdb.c index 4ff6b4cdb941..2b203ee5b879 100644 --- a/arch/x86/kernel/kgdb.c +++ b/arch/x86/kernel/kgdb.c @@ -759,13 +759,13 @@ int kgdb_arch_set_breakpoint(struct kgdb_bkpt *bpt) if (!err) return err; /* - * It is safe to call text_poke() because normal kernel execution + * It is safe to call text_poke_kgdb() because normal kernel execution * is stopped on all cores, so long as the text_mutex is not locked. */ if (mutex_is_locked(&text_mutex)) return -EBUSY; - text_poke((void *)bpt->bpt_addr, arch_kgdb_ops.gdb_bpt_instr, - BREAK_INSTR_SIZE); + text_poke_kgdb((void *)bpt->bpt_addr, arch_kgdb_ops.gdb_bpt_instr, + BREAK_INSTR_SIZE); err = probe_kernel_read(opc, (char *)bpt->bpt_addr, BREAK_INSTR_SIZE); if (err) return err; @@ -784,12 +784,13 @@ int kgdb_arch_remove_breakpoint(struct kgdb_bkpt *bpt) if (bpt->type != BP_POKE_BREAKPOINT) goto knl_write; /* - * It is safe to call text_poke() because normal kernel execution + * It is safe to call text_poke_kgdb() because normal kernel execution * is stopped on all cores, so long as the text_mutex is not locked. */ if (mutex_is_locked(&text_mutex)) goto knl_write; - text_poke((void *)bpt->bpt_addr, bpt->saved_instr, BREAK_INSTR_SIZE); + text_poke_kgdb((void *)bpt->bpt_addr, bpt->saved_instr, + BREAK_INSTR_SIZE); err = probe_kernel_read(opc, (char *)bpt->bpt_addr, BREAK_INSTR_SIZE); if (err || memcmp(opc, bpt->saved_instr, BREAK_INSTR_SIZE)) goto knl_write; From patchwork Fri Apr 26 23:22:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10919923 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7E54E14B6 for ; Sat, 27 Apr 2019 06:43:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6E2A628E6B for ; Sat, 27 Apr 2019 06:43:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 61FD628E81; Sat, 27 Apr 2019 06:43:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CD1AF28E6B for ; Sat, 27 Apr 2019 06:43:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EF5176B0006; Sat, 27 Apr 2019 02:43:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E78AD6B0008; Sat, 27 Apr 2019 02:43:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CFA036B000A; Sat, 27 Apr 2019 02:43:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id 942EF6B0006 for ; Sat, 27 Apr 2019 02:43:07 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id g1so3591398pfo.2 for ; Fri, 26 Apr 2019 23:43:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=Y4ax2hOd4Buhijx/oNSuXPhWpslJbmlh2LJYha9eHFk=; b=N5q14PX9BktHF4nZedzHHUz3/RKa2zmtAW5o6Et1n1Q41ERhGU6oEM8NzfJBo4++dG imdMQZgpj37d5CJeoZQJqnZsLnqVpchJYuma06StwVtHoTPOIBeYYM0XQlaMn2VDTwCw VFjbC4NJxAWJzl0gkq0Ed+QriFgVy0lJDe4le1O6trvfK53aSIy8L+mdE91SfbCeh6Ai xOv+Fx85NIcD4Zse7lTH816lk44j3XzLGuvgez7E2HRF1nX26g6/aDnqjCcOn3CRXrpp D3rFFwszlcomIFqwC1QzTuDcfyoDUi+kqtAeCl9FQ2NuQlzbsDsFsXPCSvFBZYK1EyXd uK2g== X-Gm-Message-State: APjAAAX1riSoE+YbJcfNTJ1wRY/u2zG6yNHIZEnr3HAqM/Yhcp3Qz49N LJaj+wzb0HA0EDHgKmFWaXXQzdMvAZJFgnjsGCipELir42GLhIr0VhN84PtFsg7ytauGq+KES9p U6Pwiv+kdhauMrfe7tszPUJRnL8dG2WHapvdJZUp+VAmpC7G3KbYi9GPfJrqbbCFjww== X-Received: by 2002:a65:6658:: with SMTP id z24mr489349pgv.323.1556347387229; Fri, 26 Apr 2019 23:43:07 -0700 (PDT) X-Received: by 2002:a65:6658:: with SMTP id z24mr489293pgv.323.1556347386188; Fri, 26 Apr 2019 23:43:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347386; cv=none; d=google.com; s=arc-20160816; b=UdnzW8bZxdeNv03V4Yr4/6QA8yTvSihYkd+SXXA1GpK/auJPs6dzrGP66DAiC+Odm6 9KtD+1c7vn3hNLGI4xUi2ES5ljdyp5tBNSfb0En6lxp3KemONqypZ7QCW5OdYeCfTqOc gEgV+vjaf9eTD5Pw4I/Lbgs97NokqvWQ03TE8b+3KIJpKA9QuHhZEDdJtarJho/ygnaD juulEsRHXZGc9oDImuUO7c8/aP34VQ+DLhBU3ZQmKTumn4svCP9U0/MfrdbUthWswDJ2 hb+jF84PkfJNdXpBbQzYGg1rIHBFaZFALV73c3h9c1ekqbcBeyH/3v0mlk8WnMHUghCQ rkVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Y4ax2hOd4Buhijx/oNSuXPhWpslJbmlh2LJYha9eHFk=; b=c3IArCYnsGDmhGBQVgOnl+FdDYifyI3DTZ/KPGaDRe75dmZt0sQ85A6ts5wqXRQ3wd mFFkgpx/RmnyPOpWOwftSEvjd4bNMPViiR7KyeZV+QwMTfUVY57evss/sZGvs1ojdMYZ CscID4v2EzhX+fAQuEKzHCCgtIoqyuu16yXuM6hR9OYV8UbCsp0fYsv8i7s6FvHF1AN6 fhQkzD9LDOwrQgZPeyICtvVk+1nKFyh3p4qfiV87Ept27WR52ny0nMQw/wNPuJVl5VnL rlhAOqKnSfj3Lxn0sVTZLNEH6YGQ2zNX6cFGruUiNMtfslZ/bCOyxdkKe4px+K+Aircv 4C/A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=RYKIYvXm; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id r19sor1216747pgg.34.2019.04.26.23.43.06 for (Google Transport Security); Fri, 26 Apr 2019 23:43:06 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=RYKIYvXm; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Y4ax2hOd4Buhijx/oNSuXPhWpslJbmlh2LJYha9eHFk=; b=RYKIYvXmL5tJ4u/bU0iLk60YO+Ox59zCcKmrYcUn0L0mc7KUA8ZtJ2R4XNYjQk2a8B DcQM399thIjNSjXHuZQ6k7afM/SMh1l1QYkf2D0hRQ+1xwvUSHZnrIJ4UJtkR2NL1yLe l1RYwwd0hroNUCsKi5IV5ZZNm8YzK8J7h42F1sPN4gG9Y1ehTjsd+VC718e5LdQep+gS k/OhJCSCkHRbcX+8zk8NtqDpMNtIV6Cqdz0zcbSsIjK/yzdPAJzadIJkFTsIuXX0ujKs CagOndVrfBrJnKlvTuPVqrDmZeIo9JxoUR6B70xI9bSBAuYLVmprrE1fKIlJc4O0C9hS uf2w== X-Google-Smtp-Source: APXvYqzYil+3Ftso1TWpSV8095HGNQL/URTfcLD+HpGX4LAA48WZkM07V5HZ0EJt2GuXBL8qYa+EnA== X-Received: by 2002:a63:b0b:: with SMTP id 11mr12676888pgl.445.1556347385725; Fri, 26 Apr 2019 23:43:05 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:05 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu Subject: [PATCH v6 02/24] x86/jump_label: Use text_poke_early() during early init Date: Fri, 26 Apr 2019 16:22:41 -0700 Message-Id: <20190426232303.28381-3-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit There is no apparent reason not to use text_poke_early() during early-init, since no patching of code that might be on the stack is done and only a single core is running. This is required for the next patches that would set a temporary mm for text poking, and this mm is only initialized after some static-keys are enabled/disabled. Cc: Andy Lutomirski Cc: Kees Cook Cc: Dave Hansen Cc: Masami Hiramatsu Acked-by: Peter Zijlstra (Intel) Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/jump_label.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c index f99bd26bd3f1..e7d8c636b228 100644 --- a/arch/x86/kernel/jump_label.c +++ b/arch/x86/kernel/jump_label.c @@ -50,7 +50,12 @@ static void __ref __jump_label_transform(struct jump_entry *entry, jmp.offset = jump_entry_target(entry) - (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); - if (early_boot_irqs_disabled) + /* + * As long as only a single processor is running and the code is still + * not marked as RO, text_poke_early() can be used; Checking that + * system_state is SYSTEM_BOOTING guarantees it. + */ + if (system_state == SYSTEM_BOOTING) poker = text_poke_early; if (type == JUMP_LABEL_JMP) { From patchwork Fri Apr 26 23:22:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10919927 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D192D912 for ; Sat, 27 Apr 2019 06:43:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C2DCF28E6B for ; Sat, 27 Apr 2019 06:43:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B68B328E7B; Sat, 27 Apr 2019 06:43:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4405D28E6B for ; Sat, 27 Apr 2019 06:43:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8B02D6B0008; Sat, 27 Apr 2019 02:43:09 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 839566B000A; Sat, 27 Apr 2019 02:43:09 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D9BC6B000C; Sat, 27 Apr 2019 02:43:09 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 334C46B0008 for ; Sat, 27 Apr 2019 02:43:09 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id q18so3232235pll.16 for ; Fri, 26 Apr 2019 23:43:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=fM02CdP3EHsEznyJ/zI3DVkvywjW2Y0W5QE/m/kGqS8=; b=U7LrD0V4LLPfC4D1y/s4x8yNq4LWaCFm1uHiqPTh3c2581DJn/fZK43RzE/vGnJLtl z4meupkFyfSOJ876g/d3TBeA2a4vXlIQILFl1rYRqzaBXP7k+aAoNcpwT5nM97RzT+m+ I2sn0IxR0Dm+fJz7b5va0W0V9bQS573wJjmR5J7fBiSL02ohBK/zDZav3PcUj99HJLn/ 9qnYpfa/vPOeHFC6xrrRYzbWrBcqVKtXuuuPK4D74HMn82w0TmQ/MeuWunraSZUVistk XHnTclndsjnPeWo5wm1t4lCvxfyKeNil6vXrmHu67KzH24imqZTJEXcm+nf1/KkOExKH +Xzg== X-Gm-Message-State: APjAAAW4oTX52RIzbYgfWQ+e2oZQRUmx3lYtWo+Mmt5mfOWi5Mb0Ldxc RTqc6EXOKE3akt7Sy6UU6EtsNKMwGeJgNdrO+ywYHRVhRD7aCnHuIvofff8Zb1zfhLmezPZWeX6 klo7WnCeWe0o4SbQ7RFotoRpE36PkvOLGgZT7GqKBJFHIMz5Vwed1C4tUgypl5oC6aA== X-Received: by 2002:aa7:9294:: with SMTP id j20mr52918429pfa.64.1556347388739; Fri, 26 Apr 2019 23:43:08 -0700 (PDT) X-Received: by 2002:aa7:9294:: with SMTP id j20mr52918363pfa.64.1556347387552; Fri, 26 Apr 2019 23:43:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347387; cv=none; d=google.com; s=arc-20160816; b=xZTjTc0yp8T7e616wot5vcfXXjuZlueYvvf2aixdfpx8EccjXAyqXknCQJhii3YI5l D3yOsdem/k0JDpLlm6YiWHTRo3OpvklOWS9ntfBbxkkgx0Y8b3LekTD7J2bTJHvePhja iy6V7kK2M2CqPUBQUVuXTfRIv3TSBF3k3NM8syPfG/7bo03bh0vRdMFG2qF8qi27T92/ QvV1gb2AhP1I4OHaIC9ktIRUsots5g2hwqq/ZsZprdWm1o6nvGJ0nsPgohHftCEv8WNu 2i12vkkNWxbiyfp1NaeU1kWfF7yuggW/PBpq84J2DfH28LgxWfoJdTi2nQf8SDGxyYz+ vO/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=fM02CdP3EHsEznyJ/zI3DVkvywjW2Y0W5QE/m/kGqS8=; b=kTESvyOURJ8vHfH8CU1xB8n+UIJCTlb4BdJD3XwWVylUgmGYWp2K3iPSFwzI8QXRaP B4qifCQI5uAzayqQVTfln/qeD6iHbV9uIp0pWNnpsfEdKwIdNrGr8CbQcA/22vO1CPz8 6q+rhIloAqctvyKjaqKL75bxaG9oBhFPNZdRGgCZHFkaKD6n/oOU0BwEVkRAdjbolkeq 4QOqZLTGs5TKSU90I3QvP0Kj5DXTU+C/gfq1Uq3gDImxq7T8jf2mUMKEzQ+bVU4HNNLP e7MWMxUxBCDtUkpo4SCBnw2DwfXsjWuxJUddCMwTntV7cDIL3FU7ShdC0hbKTKaQPXDf agww== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=Bic4fgDh; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id v6sor26413438plp.3.2019.04.26.23.43.07 for (Google Transport Security); Fri, 26 Apr 2019 23:43:07 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=Bic4fgDh; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=fM02CdP3EHsEznyJ/zI3DVkvywjW2Y0W5QE/m/kGqS8=; b=Bic4fgDhw/zNUsH9AvCevjRRuqcBr7SeEwPi/mNBJHeVn3HWTtTQy0lqVBwB+uwJON 033sEQUJVf4JsE8w16M+5haGmPR56gXJit7dknF2eDmFepylyOdGJlykH0b1NW+PihEa WnVuUa5XLxRLLr06QvgHjZbZbm6umZ2Z7Tk21XFYdu0XsSy3LVhSR76ovrEOyDS4RqXk ethcFURK2XAu1qKbwE8JJm/EyYlwhoXg4rc/DRAR47BR1Lx07vr+RFt4HZ8bItbsDTBV RxWJwnwgvwuNGmSEvPkHWRxMq59R7mUfHELo2gZLY6QtoYxwGmeesmwvk7AesQMYnBzh N2AQ== X-Google-Smtp-Source: APXvYqxN6NLybNs8lwtcaE7rd9moWK47+PtSS3OruWZmc0YNonCQSY0zJUJKCwxGbV5SIYzeiwOfdw== X-Received: by 2002:a17:902:d83:: with SMTP id 3mr52111624plv.125.1556347387028; Fri, 26 Apr 2019 23:43:07 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:06 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Kees Cook , Dave Hansen , Nadav Amit Subject: [PATCH v6 03/24] x86/mm: Introduce temporary mm structs Date: Fri, 26 Apr 2019 16:22:42 -0700 Message-Id: <20190426232303.28381-4-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Andy Lutomirski Using a dedicated page-table for temporary PTEs prevents other cores from using - even speculatively - these PTEs, thereby providing two benefits: (1) Security hardening: an attacker that gains kernel memory writing abilities cannot easily overwrite sensitive data. (2) Avoiding TLB shootdowns: the PTEs do not need to be flushed in remote page-tables. To do so a temporary mm_struct can be used. Mappings which are private for this mm can be set in the userspace part of the address-space. During the whole time in which the temporary mm is loaded, interrupts must be disabled. The first use-case for temporary mm struct, which will follow, is for poking the kernel text. [ Commit message was written by Nadav Amit ] Cc: Kees Cook Cc: Dave Hansen Acked-by: Peter Zijlstra (Intel) Reviewed-by: Masami Hiramatsu Tested-by: Masami Hiramatsu Signed-off-by: Andy Lutomirski Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/mmu_context.h | 33 ++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 19d18fae6ec6..24dc3b810970 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -356,4 +356,37 @@ static inline unsigned long __get_current_cr3_fast(void) return cr3; } +typedef struct { + struct mm_struct *mm; +} temp_mm_state_t; + +/* + * Using a temporary mm allows to set temporary mappings that are not accessible + * by other CPUs. Such mappings are needed to perform sensitive memory writes + * that override the kernel memory protections (e.g., W^X), without exposing the + * temporary page-table mappings that are required for these write operations to + * other CPUs. Using a temporary mm also allows to avoid TLB shootdowns when the + * mapping is torn down. + * + * Context: The temporary mm needs to be used exclusively by a single core. To + * harden security IRQs must be disabled while the temporary mm is + * loaded, thereby preventing interrupt handler bugs from overriding + * the kernel memory protection. + */ +static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm) +{ + temp_mm_state_t temp_state; + + lockdep_assert_irqs_disabled(); + temp_state.mm = this_cpu_read(cpu_tlbstate.loaded_mm); + switch_mm_irqs_off(NULL, mm, current); + return temp_state; +} + +static inline void unuse_temporary_mm(temp_mm_state_t prev_state) +{ + lockdep_assert_irqs_disabled(); + switch_mm_irqs_off(NULL, prev_state.mm, current); +} + #endif /* _ASM_X86_MMU_CONTEXT_H */ From patchwork Fri Apr 26 23:22:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10919929 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B57D9912 for ; Sat, 27 Apr 2019 06:43:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A504C28E6B for ; Sat, 27 Apr 2019 06:43:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 98A7A28E7B; Sat, 27 Apr 2019 06:43:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0F84E28E6B for ; Sat, 27 Apr 2019 06:43:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DE1B06B000A; Sat, 27 Apr 2019 02:43:10 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D6BFD6B000C; Sat, 27 Apr 2019 02:43:10 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BBAEB6B000D; Sat, 27 Apr 2019 02:43:10 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id 80AAE6B000A for ; Sat, 27 Apr 2019 02:43:10 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id d1so3471562pgk.21 for ; Fri, 26 Apr 2019 23:43:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=wqvLZPneRcLgxY6Wjfk6EEQBPauaGcOxn9fq0yjJ9xE=; b=iZd05zzNwB0Qflq7/jXw8BqypFhR2t19rGxqnoEAffhHcUlGvIIVf7PizbiQNfvitY RSHoPYmVdjRWh61T39cAOtdSOY9kMgu6L2zln4pvAEWF41JFMThTNAfHA3HuuE+zYy2t 2Sn8DygHF7K2moorh2gtsECY7IdpA0gfow0fNNt9JrqGOUXDzc+jkWIRypHkrahEjdYL vyV83mumOvK5PwFZjfEuAGTQ7SY8SqZeW/0gWS1iIqmJfOdEs9RKC8yPwj58MXM0IAjt HzEDFOB8R1LExdHjZvi5LqdWg7TzDFwAvVsYkdQEjkSKq+dkxiJw3TaHoDHZOKTl6Qb+ y8qA== X-Gm-Message-State: APjAAAWSe1kYUAuyTWU7AIlyNWegohj9FXegFmlbncE70GCfho3WRtFN toKS9ia0bSUN3kZEcBeh+8CAbJeTai7Wtbux9mrxTLoL6nb0P0u0JrRtGdsCigxG86elM4q/kLC wtTw5dFuSi5K34lXkLpgVnYafXPdcpd5izcYLDQFpn3Dq+AR6cLi+8pVA5M77RyhKRw== X-Received: by 2002:a63:e850:: with SMTP id a16mr48266792pgk.195.1556347390210; Fri, 26 Apr 2019 23:43:10 -0700 (PDT) X-Received: by 2002:a63:e850:: with SMTP id a16mr48266736pgk.195.1556347388908; Fri, 26 Apr 2019 23:43:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347388; cv=none; d=google.com; s=arc-20160816; b=lBNV+3IljuuybbAh36WqvAwoLhFjTattTmXJ6zaMvqLdvWTMnQH+SefTYrCWqPfqb8 JpeEpkMYHvTjYAcB1pCpc1JHLdFdp4Pco0Hk1xVOyE6BHmWMYo4rrtz7yfUoHzIQv7Hp 3qtTga4J357Hs/6rcQ5yd9t4qnj7bZ4UTK34UP4cGIKD7qZMzTgqXwOpS3q+i3mcunku 17yIRC4NnGJyhpuD+ENKAMlpVNJCWuB4ZOcrxeC8NFPkGAAC27W+G4OXspiKFzcdAmwY Wnj+fqyzLoC6Jz7ME6C6KPhNG7vYGsOYQ0OCMj4iCU+cIAhWICIUb218fheDphMwRT9e dwog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=wqvLZPneRcLgxY6Wjfk6EEQBPauaGcOxn9fq0yjJ9xE=; b=BcdyXQ50p8M5+U2zfIVgP09UPFapVK2UXa9+OrjUKBMmhDENuTYnt44qmrLeEjl7I6 xm2bl/YDRfQBM0Rax3rkoRFnLpLhGdQmVwkj29zG5S1Dmp1C02eR6qNIKjxe2EWgiuG9 GNQS6hK9y25XlfQ6ReoX+Ujclpa1Sq2iattiIy5mPw1AKhHEidi8zbPjGp3cZnwxDban GTctkUnEXBVE3sSzQNtKCKMjFteMNAFeV7FpmgFqeWTMYmTrTteRYGUot7cHphUwerY7 v43lyWkt6D/CuAZEJEvjI5zySKn6o7Vq/wC3xFfsCdDMeBHEdeQekG1bCNm4KNAeKgBd XuHQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=vdYahecr; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id z18sor27018249plo.58.2019.04.26.23.43.08 for (Google Transport Security); Fri, 26 Apr 2019 23:43:08 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=vdYahecr; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=wqvLZPneRcLgxY6Wjfk6EEQBPauaGcOxn9fq0yjJ9xE=; b=vdYahecrwl2Sp5UawSR3og0bnv/oKfDwV/Dqodp6SG75rtt3oI1BxkHqScjMRc/B0U ndpm0CSM99X5DI5nXyZ0qilWlZ5y+YmffvE0rzTsPOh9Lg0wVC5gzIbXO3T2rI/43jKw dSv/fQEMrgklgF9DfRp0UTkxOOmbTA2q76DVrniyNWh8V594RE3EBRYYfPR9C72DIbVe QvUfF6WjCix52aEKKarcvG6J8JGMJ6kiAHPNxAecwgyTiM6P6ST8UlxjffLYXWC/UN5g Hhv9thRtl94b6Emad3qIhmqwC571N8N4kkx/zbp3oEc24WAmffQk4ZXiiSSJrBGsI7VY gu0w== X-Google-Smtp-Source: APXvYqyiprnnWU4rXtsHhhC4I5UqVMx78ccNfm1X35wazpAaZSJma1EHcKW1CFUxAhCYo5lEBNcLEA== X-Received: by 2002:a17:902:aa83:: with SMTP id d3mr8074837plr.108.1556347388411; Fri, 26 Apr 2019 23:43:08 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:07 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Nadav Amit Subject: [PATCH v6 04/24] x86/mm: Save debug registers when loading a temporary mm Date: Fri, 26 Apr 2019 16:22:43 -0700 Message-Id: <20190426232303.28381-5-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit Prevent user watchpoints from mistakenly firing while the temporary mm is being used. As the addresses of the temporary mm might overlap those of the user-process, this is necessary to prevent wrong signals or worse things from happening. Cc: Andy Lutomirski Acked-by: Peter Zijlstra (Intel) Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/mmu_context.h | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 24dc3b810970..93dff1963337 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -13,6 +13,7 @@ #include #include #include +#include extern atomic64_t last_mm_ctx_id; @@ -380,6 +381,21 @@ static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm) lockdep_assert_irqs_disabled(); temp_state.mm = this_cpu_read(cpu_tlbstate.loaded_mm); switch_mm_irqs_off(NULL, mm, current); + + /* + * If breakpoints are enabled, disable them while the temporary mm is + * used. Userspace might set up watchpoints on addresses that are used + * in the temporary mm, which would lead to wrong signals being sent or + * crashes. + * + * Note that breakpoints are not disabled selectively, which also causes + * kernel breakpoints (e.g., perf's) to be disabled. This might be + * undesirable, but still seems reasonable as the code that runs in the + * temporary mm should be short. + */ + if (hw_breakpoint_active()) + hw_breakpoint_disable(); + return temp_state; } @@ -387,6 +403,13 @@ static inline void unuse_temporary_mm(temp_mm_state_t prev_state) { lockdep_assert_irqs_disabled(); switch_mm_irqs_off(NULL, prev_state.mm, current); + + /* + * Restore the breakpoints if they were disabled before the temporary mm + * was loaded. + */ + if (hw_breakpoint_active()) + hw_breakpoint_restore(); } #endif /* _ASM_X86_MMU_CONTEXT_H */ From patchwork Fri Apr 26 23:22:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10919933 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AC18C14B6 for ; Sat, 27 Apr 2019 06:43:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9BC0928E6B for ; Sat, 27 Apr 2019 06:43:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8F61B28E7B; Sat, 27 Apr 2019 06:43:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 108F728E6B for ; Sat, 27 Apr 2019 06:43:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 703FB6B000C; Sat, 27 Apr 2019 02:43:12 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 63AB96B000D; Sat, 27 Apr 2019 02:43:12 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4DE5A6B000E; Sat, 27 Apr 2019 02:43:12 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 130EB6B000C for ; Sat, 27 Apr 2019 02:43:12 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id m9so3485820pge.7 for ; Fri, 26 Apr 2019 23:43:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=gMfxobg7i1AP5PQ14JM+oGjrmklNXXx9IUFM/PW54zo=; b=TSNETXkcIL5LN33ffyTWb2kuml3Xwsq6BVz1JxEDnv39mk2UXrfLp8WI1VgRLrPXe8 hcHy+I/Vrk/JrcL8roCx0VmBYao+dchnGXaLWiysyJ+x+p4p+JWqccp3FHoO6vyAjDy4 MAfHwQTsVcOLpblYjTZscxfkGktAsTMUgtHc3x13ZXPBtbFgMpLJr+nO6w/+Tkbt2nFQ iF+WBh7tEnOO+KShlqZ/FGdO6pWqbR6krijQP+hT2y5kM3Uj087FxxQGtV40hx3Ai/NC ix9OT/j7pbM0BBn0KJCY3U1EuOhv+4+DQ8S419roeqFjA/Dmz6D/ip5DxbDSmC7LRPmg nYXQ== X-Gm-Message-State: APjAAAWQI9yRPgg2tMSQag3/Z8Y2Oi9XrBhjpoTzqNtEyUyM5Egcz5zz u0R55tmqKXN/0hvL+vad0sKmZlKhD9BwMwsS714pSBBvsdQj25t1WMBRjWDq9VvwqNkKrvLt/aK SRYobXgRclGEmiMy2vSjweA/E8eHQirazgv8DJS0Vao64i2zAuJkkc0t4kBDvtwN77w== X-Received: by 2002:a17:902:778b:: with SMTP id o11mr27413285pll.333.1556347391745; Fri, 26 Apr 2019 23:43:11 -0700 (PDT) X-Received: by 2002:a17:902:778b:: with SMTP id o11mr27413245pll.333.1556347390559; Fri, 26 Apr 2019 23:43:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347390; cv=none; d=google.com; s=arc-20160816; b=c1iSuTMqRJdyenF6UV4EavVoZQZ9orKGTVW1MFAPX38VkLNmG7CyE5Ngb0wxfCKDEG mSqKSdr1NZcP0Y1t5vxdUIs5oUfwIwrVnELg9B9cNP8GiDFWcDbXYC4/RFQdd0jIl1kd qJ/ucBqbxZp4pZR8Ou2ls8ypkekepGRUj/xSn2vuZhgySGG1yKOim9WEMYQNFx/so3Zm QxMUJnnoRfes+9ObrSs5O6CBATfJhuu6ir5x37iuEIaXHCuJ7+dnTXm4vO4kx0emZEnb yFVmxft/vO0iyHAPIwYj5lL6+nlqhuNVfV+EbOwwcD+OwvHKKLCk14NxB0U7mLA3aEdt PPqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=gMfxobg7i1AP5PQ14JM+oGjrmklNXXx9IUFM/PW54zo=; b=Buv84Mfe0upw8tJNe3rVLCaKbMvamHxgGDnnsgSISga6Vyzn0noHBjhvXmm7PHjOyb MuPTxQHNX330qo0ydPA3JR0IHlFgqxMa4S/pB2HkfaqUEhNkkuT3GklUMg7lVeNk8QZu KEdZcEFnbMDtu5iNIEt9EPsTR4KV6AVWryLkmbXnf8Ye09fdRZa23ymUJhL0S0XPfeYu lsRqhv8JmSgwEAyqrzDCbaBxEgbStfcYh8QCFJsK/7gO9Kw9kbPYtByyNrWdFhXOkVY0 tLXlPSfISVvxzLyz2qi6Rr0E2t9Phqb6XKbLd7PBlUHuQ2+8jvzpeFjo0IdUEz0Rhqb6 gOiQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=XKJ4fAeB; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id s6sor18562958plq.62.2019.04.26.23.43.10 for (Google Transport Security); Fri, 26 Apr 2019 23:43:10 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=XKJ4fAeB; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=gMfxobg7i1AP5PQ14JM+oGjrmklNXXx9IUFM/PW54zo=; b=XKJ4fAeBY8aGdbJDn84dW83HzgSvQACPuhXWJHvF1CWt6rAKSo0+tUCXqeDEYn8KGx u3zXII+RSuYhdLmlbo5/0feDTgEEs6/tOmzti0w5Pevv/8Nua7dmelg2Hg1YTOM0HPNp IuB1XxBtPV5C1LDrt4HWOamCYTjBpu9Uc9bqXu6wh2x58Sr/Usj1f74P/vpY9jMFSCFS THBbi+fo5P15SNYnnZLc+khNDUp7StdyPifYfNcqKpXcU/HGBdU106YbtBqZ2gBNSvKt YTZSQ/QqPm8CwczIhg2BokKOhThemqJ36d1wwJl1ZB93G6rLPD545DygLOMnvs67gNXF w1Rg== X-Google-Smtp-Source: APXvYqwk9lLXQyc9bNgKLLm6YtZMcI1PVJ4KR2rM1C+cN3nT9zotDQcbIXGbwEvAmUupbSpfYScE8w== X-Received: by 2002:a17:902:a7:: with SMTP id a36mr50206141pla.111.1556347390037; Fri, 26 Apr 2019 23:43:10 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:09 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Nadav Amit , Arnaldo Carvalho de Melo Subject: [PATCH v6 05/24] uprobes: Initialize uprobes earlier Date: Fri, 26 Apr 2019 16:22:44 -0700 Message-Id: <20190426232303.28381-6-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit In order to have a separate address space for text poking, we need to duplicate init_mm early during start_kernel(). This, however, introduces a problem since uprobes functions are called from dup_mmap(), but uprobes is still not initialized in this early stage. Since uprobes initialization is necassary for fork, and since all the dependant initialization has been done when fork is initialized (percpu and vmalloc), move uprobes initialization to fork_init(). It does not seem uprobes introduces any security problem for the poking_mm. Crash and burn if uprobes initialization fails, similarly to other early initializations. Change the init_probes() name to probes_init() to match other early initialization functions name convention. Cc: Arnaldo Carvalho de Melo Reported-by: kernel test robot Signed-off-by: Nadav Amit --- include/linux/uprobes.h | 5 +++++ kernel/events/uprobes.c | 8 +++----- kernel/fork.c | 1 + 3 files changed, 9 insertions(+), 5 deletions(-) diff --git a/include/linux/uprobes.h b/include/linux/uprobes.h index 103a48a48872..12bf0b68ed92 100644 --- a/include/linux/uprobes.h +++ b/include/linux/uprobes.h @@ -115,6 +115,7 @@ struct uprobes_state { struct xol_area *xol_area; }; +extern void __init uprobes_init(void); extern int set_swbp(struct arch_uprobe *aup, struct mm_struct *mm, unsigned long vaddr); extern int set_orig_insn(struct arch_uprobe *aup, struct mm_struct *mm, unsigned long vaddr); extern bool is_swbp_insn(uprobe_opcode_t *insn); @@ -154,6 +155,10 @@ extern void arch_uprobe_copy_ixol(struct page *page, unsigned long vaddr, struct uprobes_state { }; +static inline void uprobes_init(void) +{ +} + #define uprobe_get_trap_addr(regs) instruction_pointer(regs) static inline int diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index c5cde87329c7..e6a0d6be87e3 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -2294,16 +2294,14 @@ static struct notifier_block uprobe_exception_nb = { .priority = INT_MAX-1, /* notified after kprobes, kgdb */ }; -static int __init init_uprobes(void) +void __init uprobes_init(void) { int i; for (i = 0; i < UPROBES_HASH_SZ; i++) mutex_init(&uprobes_mmap_mutex[i]); - if (percpu_init_rwsem(&dup_mmap_sem)) - return -ENOMEM; + BUG_ON(percpu_init_rwsem(&dup_mmap_sem)); - return register_die_notifier(&uprobe_exception_nb); + BUG_ON(register_die_notifier(&uprobe_exception_nb)); } -__initcall(init_uprobes); diff --git a/kernel/fork.c b/kernel/fork.c index 9dcd18aa210b..44fba5e5e916 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -815,6 +815,7 @@ void __init fork_init(void) #endif lockdep_init_task(&init_task); + uprobes_init(); } int __weak arch_dup_task_struct(struct task_struct *dst, From patchwork Fri Apr 26 23:22:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10919935 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 460B4912 for ; Sat, 27 Apr 2019 06:43:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 35BB828E6B for ; Sat, 27 Apr 2019 06:43:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2A09628E7B; Sat, 27 Apr 2019 06:43:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A039D28E6B for ; Sat, 27 Apr 2019 06:43:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 491896B000D; Sat, 27 Apr 2019 02:43:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 417C96B000E; Sat, 27 Apr 2019 02:43:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 292916B0010; Sat, 27 Apr 2019 02:43:14 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id E2B8F6B000D for ; Sat, 27 Apr 2019 02:43:13 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id a141so1877347pfa.13 for ; Fri, 26 Apr 2019 23:43:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=PqX9u33LMatWrF2rdzOLAo8PTOZ3S7r/OUOEdwJc/wY=; b=bBjkvi6cSBcHFF5q+MuxOePWH2MW63t4/XLoRmYru1jIgsfZQVGUTaNY1jDqNOsU6g iTbzRixBnbBp3b3TcKRcu/eSYSifXfUp6hNo5O4AxOW+bt7ufqwANbKfk0q1mONTWcCh Dk4HfBFS51SwaaZnB04SLed2g1e5ggKNJgI5DmlgBN3AeOxfZGX1+DkyAa7lzR+kM1at XHTbbnU9nZetoY5C+jPsy80QVeT5uTnPLys76i0Q0+wcoRhDqxuWDp07DLI41yfYoepN Mdzwv1iWQPS8EVrjLTEa5DPdKHUzD9ZdRYfsRT61DX4nMfo6+zxB2AVU/Vw6XxPK1lsW JSwg== X-Gm-Message-State: APjAAAXHXvKvhkLTWqjO0Q+ftv51Or3JP50ibnAdQjhuRyPa6v5PF/1F h1zZd0ocMlnW7/OlaLeThCMuUiCVvQScco3kUdpuHHECi+iZTrD3dKcAIeCjX8E1a6KJI6D9AT0 PK7G4d533njqJ5B6X4frrlfA+MXcAjllwU9P6n2dYvs688ThzkSpCgBZQ8MVy4QaNRg== X-Received: by 2002:a63:6942:: with SMTP id e63mr47115634pgc.102.1556347393603; Fri, 26 Apr 2019 23:43:13 -0700 (PDT) X-Received: by 2002:a63:6942:: with SMTP id e63mr47115588pgc.102.1556347392539; Fri, 26 Apr 2019 23:43:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347392; cv=none; d=google.com; s=arc-20160816; b=ipZyNPV9hKHE0z7ebIX9hFgbbr9XiKI6+EcdUan70OKGyauFy8a6QVhDoaxNahplHh jSQ9VcF2/mTEbTGYgs3pcurYIi2l5ge6/tnSV5bwkN8jpORtrD+n6tRsVkO/mM8OJ8e6 36tzo/1IMN6/XUZoFqNTPC8Qkblbmolz/jAVawpZzyS9FTzJypCzXGtDl2vYl6fLyZ48 8eBNKw+1v+Tx5gAfGxareByxKeojAc2NdSqKhYQyr8M7+/dYKdow98J4CKOx+N9cobXe dZ20mfGF3lMg+v7aVvn2Ryrg7DKObnmdCPViYlk6Gb/cF6Fk3T0UKTtVfQKQaNJpVJrT YupQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=PqX9u33LMatWrF2rdzOLAo8PTOZ3S7r/OUOEdwJc/wY=; b=ExWfzxX1dKCiqpLrVO7azJWQPHLfBdDHy4WNjDx/N1ehL9E1jM2IfPEp5fCLt2HqDA /gRFDdgtXVua3+LAgP1DWj2K9QpM6spRV1+LfFsddEWDxt7I/li0Y/5LWXXWQD04H43I Dyq36t11RN7dk9TGTZx1FnyJHvEtJlzldj/hYlPFJ1M6y0PkacdWhW5j5Wg1sr2kBoy9 s6K3bF7yZ/+MFgmzXRuvMdg2bsugnfnV66im5shdLejyEvIjfrkLUrlXEFf1ZEGbL9co aZOa4Ro+tKul2gO3kudDGT1Kc+4ipeRlZBTqf3yQpA6j7J0XZ3Bcp0Elma92+cj/ZV2Z psyw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=dLUUcPuc; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id c12sor30517841pfr.33.2019.04.26.23.43.12 for (Google Transport Security); Fri, 26 Apr 2019 23:43:12 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=dLUUcPuc; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=PqX9u33LMatWrF2rdzOLAo8PTOZ3S7r/OUOEdwJc/wY=; b=dLUUcPucbWSjI0KF0GYdfa3//vc9Trf9GNL1Jxgol/lAzL/Qif+nwieAhan1enOtj9 IEvF0toGPmi5evSU5LGqWY70kCiO3p7KKseN+96U+aK7PFiGt1LmM5YwgAU+lA6dRruQ txGFF06hrpSSsexZDBeyJTD8iZpafmkscb7dw3FJPuVKduR/83ccqIn9U64Bhl7mXai2 QvPXBsrBhxIENhDIjXt5l8fioHpUH4RBLdAlGLQjejsyatNlX3PJPjeZX8AdLYgJMv+s txk3ENcm6pNvFPMHYkZ9djMndR8b2wvyhfIPdTbE4Wq/3iIH/ChGjSPLA1uzJUN9rbuN 8qpw== X-Google-Smtp-Source: APXvYqwtiOYLXN6z+omvr9q5XmanCyE26RTxB/9RLJMeynpzcv4zb2Q6alFcE2kD+WU/ILJ0eVx92Q== X-Received: by 2002:aa7:920b:: with SMTP id 11mr49825084pfo.3.1556347392003; Fri, 26 Apr 2019 23:43:12 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:11 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Nadav Amit , Kees Cook , Dave Hansen Subject: [PATCH v6 06/24] fork: Provide a function for copying init_mm Date: Fri, 26 Apr 2019 16:22:45 -0700 Message-Id: <20190426232303.28381-7-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit Provide a function for copying init_mm. This function will be later used for setting a temporary mm. Cc: Andy Lutomirski Cc: Kees Cook Cc: Dave Hansen Acked-by: Peter Zijlstra (Intel) Reviewed-by: Masami Hiramatsu Tested-by: Masami Hiramatsu Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- include/linux/sched/task.h | 1 + kernel/fork.c | 24 ++++++++++++++++++------ 2 files changed, 19 insertions(+), 6 deletions(-) diff --git a/include/linux/sched/task.h b/include/linux/sched/task.h index 2e97a2227045..f1227f2c38a4 100644 --- a/include/linux/sched/task.h +++ b/include/linux/sched/task.h @@ -76,6 +76,7 @@ extern void exit_itimers(struct signal_struct *); extern long _do_fork(unsigned long, unsigned long, unsigned long, int __user *, int __user *, unsigned long); extern long do_fork(unsigned long, unsigned long, unsigned long, int __user *, int __user *); struct task_struct *fork_idle(int); +struct mm_struct *copy_init_mm(void); extern pid_t kernel_thread(int (*fn)(void *), void *arg, unsigned long flags); extern long kernel_wait4(pid_t, int __user *, int, struct rusage *); diff --git a/kernel/fork.c b/kernel/fork.c index 44fba5e5e916..fbe9dfcd8680 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1299,13 +1299,20 @@ void mm_release(struct task_struct *tsk, struct mm_struct *mm) complete_vfork_done(tsk); } -/* - * Allocate a new mm structure and copy contents from the - * mm structure of the passed in task structure. +/** + * dup_mm() - duplicates an existing mm structure + * @tsk: the task_struct with which the new mm will be associated. + * @oldmm: the mm to duplicate. + * + * Allocates a new mm structure and duplicates the provided @oldmm structure + * content into it. + * + * Return: the duplicated mm or NULL on failure. */ -static struct mm_struct *dup_mm(struct task_struct *tsk) +static struct mm_struct *dup_mm(struct task_struct *tsk, + struct mm_struct *oldmm) { - struct mm_struct *mm, *oldmm = current->mm; + struct mm_struct *mm; int err; mm = allocate_mm(); @@ -1372,7 +1379,7 @@ static int copy_mm(unsigned long clone_flags, struct task_struct *tsk) } retval = -ENOMEM; - mm = dup_mm(tsk); + mm = dup_mm(tsk, current->mm); if (!mm) goto fail_nomem; @@ -2187,6 +2194,11 @@ struct task_struct *fork_idle(int cpu) return task; } +struct mm_struct *copy_init_mm(void) +{ + return dup_mm(NULL, &init_mm); +} + /* * Ok, this is the main fork-routine. * From patchwork Fri Apr 26 23:22:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10919937 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EBB60912 for ; Sat, 27 Apr 2019 06:43:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D8A3928E6B for ; Sat, 27 Apr 2019 06:43:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CA85F28E7B; Sat, 27 Apr 2019 06:43:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9AE0A28E6B for ; Sat, 27 Apr 2019 06:43:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 68BBA6B000E; Sat, 27 Apr 2019 02:43:16 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 63D0C6B0010; Sat, 27 Apr 2019 02:43:16 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 48D5F6B0266; Sat, 27 Apr 2019 02:43:16 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 0C5B56B000E for ; Sat, 27 Apr 2019 02:43:16 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id d12so3580654pfn.9 for ; Fri, 26 Apr 2019 23:43:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=ZPFv6m+iH8EeSMp9wFrys9PGKYZJjpGjv1/CHTAfS5M=; b=BHJYMsMs3M2sxMYG7Prjx4we/kxygG/FYXCGYY/pXehpJaGr+vc55mkZC/4tzAf9PJ 86zFBeQ0xqDfNjpKC2PVZfCJ7G3tnkR4iSu86KuqPOG7eUcv8f2RDoVtgCxSnzAVAlR4 3EjRbHcBXuxCDX++5d/VADWIjk8A2rdqpGirlOQaxgURTdCgSryB1EIm/15fxRFjzdB/ H/hIzwsWXiew3R5PuUgJCMBKiPdLXqKjKhN6asNMLYr0ovZir/T5OdrPg3miZn5VBwBK Y7YOSfHrTvmTDkM3Z80KIVE8nlCSlU9HvDEF5TRMx9JuWGOO3863VuXx1de6fw/rLRpL J1Fw== X-Gm-Message-State: APjAAAUewK5f0S9VB5iqGam9e4Fr8eEJJm026SQeCXIeALSwl1OZOtVm 9jJtC9UgtuyefF+9SmyxJl/8MbusOuHqDM2H7EbD23ZFloDPle/DE1ZY5f7uO5ibiXO3WQVQqAP RLIfapepDYZJtVVnyKSPDjgW92aPujVHPv1At5eRMPlrIRWCjeuWszlOs5r39UJkHTw== X-Received: by 2002:a17:902:6b44:: with SMTP id g4mr3053811plt.157.1556347395696; Fri, 26 Apr 2019 23:43:15 -0700 (PDT) X-Received: by 2002:a17:902:6b44:: with SMTP id g4mr3053756plt.157.1556347394392; Fri, 26 Apr 2019 23:43:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347394; cv=none; d=google.com; s=arc-20160816; b=GIpS4Cgqujum671yS28fl2Nd5KKSwfhOMU9Z44SrhcnADu4zAzeDbpMcUBsmr6AdVj RwhhGhBwZbl8vo/r8ipKZASsHtaplj4HFWFtzrlUZjalwDV78rR5oTFq3OcKcrVbvvqh xvoB6t9IUs2zAixwP0xrQ4gO8VoUuJfzmOWkETAQOdgaqJQFYvc0ctNEKhaPpsHxaP94 02canKPYn3NkdSY+Oy6vLTfdI4p0Y4FbDpAKAP03aQIL5mTKV8WHFCaWyxC4uRkqsaou eEbA45AVtMzTu1KzuIp5NT0QdgcWkyXg82USzZ8UFugRMerkxaQ3u8WlkZFGCONCRMGm TCaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ZPFv6m+iH8EeSMp9wFrys9PGKYZJjpGjv1/CHTAfS5M=; b=Tyehmbd9C1t5+b79AZshSFn49C97qSmltkmw/1rWBCF9yjYkJn/NROgedA1/Zid1tN P0A15PKcnU18FfygDUwXX7h2K9K77Rdy2KUUHuVfgsZQf5uVDvkkDf5/zHypswQTITjv hyc6CONW0GfUFtgspXdLINAm9v2FdpQx2yFjLKxt7Zaboca5rWSDPL37PREtE7ISDOnm +ZFaRa7ZJ0ZmpWhywEkJv7Pr5SnXmHNuHZWujhnWeM007RUlEaaKb9CSVIkFGaXjNFY3 zl55KSkMDqetQUNvXTPRgQlfFqLhczLyz/gIcpJwk5QVMKbk7GjyqAOi70Z5POtfWLdw NEhw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="p4m/oBqS"; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id c20sor7757717pgk.50.2019.04.26.23.43.14 for (Google Transport Security); Fri, 26 Apr 2019 23:43:14 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="p4m/oBqS"; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ZPFv6m+iH8EeSMp9wFrys9PGKYZJjpGjv1/CHTAfS5M=; b=p4m/oBqSG6KLPf+AqtV3OdSRGsnlwAcOGjTsQ1oMvTgRjsaBLxL2oqlvHbnUOD7DhC llMjaQPUwxdsPmzYP1Xk4zg6iskJI0V3bcjSumFgJeyn32g+AQyeuwbnaKNcunnSGV+X jFrW4YEgTD0CVzhuajD3RiuuFodFK5pQf/h/OASbcijhEUNJeR4GOVIaxGkMApARhJuz heZkZxY0ibC3WGtyGA9X5OpN8Qanr6vPnIs32DNHYf97D5Hp43Bu1zzc+18KCpllyi2H mACL4axXMqmHb8f85O6m0qGesKiJyaJ8E2Q7anmXlyeLMnsiSJBU8Ye4Tm8bsUaDuaL0 //Aw== X-Google-Smtp-Source: APXvYqyVtXk5iHrU+w85KghleSi1h9IdIHLthG9HVX8BbnAS3s4DC3QPi8s9GkBRV20LWDs74Eyk7w== X-Received: by 2002:a63:1359:: with SMTP id 25mr46901248pgt.92.1556347393845; Fri, 26 Apr 2019 23:43:13 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:13 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Nadav Amit , Kees Cook , Dave Hansen Subject: [PATCH v6 07/24] x86/alternative: Initialize temporary mm for patching Date: Fri, 26 Apr 2019 16:22:46 -0700 Message-Id: <20190426232303.28381-8-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit To prevent improper use of the PTEs that are used for text patching, the next patches will use a temporary mm struct. Initailize it by copying the init mm. The address that will be used for patching is taken from the lower area that is usually used for the task memory. Doing so prevents the need to frequently synchronize the temporary-mm (e.g., when BPF programs are installed), since different PGDs are used for the task memory. Finally, randomize the address of the PTEs to harden against exploits that use these PTEs. Cc: Kees Cook Cc: Dave Hansen Acked-by: Peter Zijlstra (Intel) Reviewed-by: Masami Hiramatsu Tested-by: Masami Hiramatsu Suggested-by: Andy Lutomirski Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/pgtable.h | 3 +++ arch/x86/include/asm/text-patching.h | 2 ++ arch/x86/kernel/alternative.c | 3 +++ arch/x86/mm/init.c | 37 ++++++++++++++++++++++++++++ init/main.c | 3 +++ 5 files changed, 48 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 5cfbbb6d458d..6b6bfdfe83aa 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1038,6 +1038,9 @@ static inline void __meminit init_trampoline_default(void) /* Default trampoline pgd value */ trampoline_pgd_entry = init_top_pgt[pgd_index(__PAGE_OFFSET)]; } + +void __init poking_init(void); + # ifdef CONFIG_RANDOMIZE_MEMORY void __meminit init_trampoline(void); # else diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h index f8fc8e86cf01..a75eed841eed 100644 --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -39,5 +39,7 @@ extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len); extern int poke_int3_handler(struct pt_regs *regs); extern void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler); extern int after_bootmem; +extern __ro_after_init struct mm_struct *poking_mm; +extern __ro_after_init unsigned long poking_addr; #endif /* _ASM_X86_TEXT_PATCHING_H */ diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 0a814d73547a..11d5c710a94f 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -679,6 +679,9 @@ void *__init_or_module text_poke_early(void *addr, const void *opcode, return addr; } +__ro_after_init struct mm_struct *poking_mm; +__ro_after_init unsigned long poking_addr; + static void *__text_poke(void *addr, const void *opcode, size_t len) { unsigned long flags; diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index f905a2371080..c25bb00955db 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -22,6 +22,7 @@ #include #include #include +#include /* * We need to define the tracepoints somewhere, and tlb.c @@ -700,6 +701,42 @@ void __init init_mem_mapping(void) early_memtest(0, max_pfn_mapped << PAGE_SHIFT); } +/* + * Initialize an mm_struct to be used during poking and a pointer to be used + * during patching. + */ +void __init poking_init(void) +{ + spinlock_t *ptl; + pte_t *ptep; + + pr_err("%s\n", __func__); + poking_mm = copy_init_mm(); + BUG_ON(!poking_mm); + + /* + * Randomize the poking address, but make sure that the following page + * will be mapped at the same PMD. We need 2 pages, so find space for 3, + * and adjust the address if the PMD ends after the first one. + */ + poking_addr = TASK_UNMAPPED_BASE; + if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) + poking_addr += (kaslr_get_random_long("Poking") & PAGE_MASK) % + (TASK_SIZE - TASK_UNMAPPED_BASE - 3 * PAGE_SIZE); + + if (((poking_addr + PAGE_SIZE) & ~PMD_MASK) == 0) + poking_addr += PAGE_SIZE; + + /* + * We need to trigger the allocation of the page-tables that will be + * needed for poking now. Later, poking may be performed in an atomic + * section, which might cause allocation to fail. + */ + ptep = get_locked_pte(poking_mm, poking_addr, &ptl); + BUG_ON(!ptep); + pte_unmap_unlock(ptep, ptl); +} + /* * devmem_is_allowed() checks to see if /dev/mem access to a certain address * is valid. The argument is a physical page number. diff --git a/init/main.c b/init/main.c index 598e278b46f7..949eed8015ec 100644 --- a/init/main.c +++ b/init/main.c @@ -504,6 +504,8 @@ void __init __weak thread_stack_cache_init(void) void __init __weak mem_encrypt_init(void) { } +void __init __weak poking_init(void) { } + bool initcall_debug; core_param(initcall_debug, initcall_debug, bool, 0644); @@ -737,6 +739,7 @@ asmlinkage __visible void __init start_kernel(void) taskstats_init_early(); delayacct_init(); + poking_init(); check_bugs(); acpi_subsystem_init(); From patchwork Fri Apr 26 23:22:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10919939 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3FA1414B6 for ; Sat, 27 Apr 2019 06:43:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2E61828E6B for ; Sat, 27 Apr 2019 06:43:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1F91828E7C; Sat, 27 Apr 2019 06:43:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DB85928E6B for ; Sat, 27 Apr 2019 06:43:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 292AC6B0010; Sat, 27 Apr 2019 02:43:18 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 21C316B0266; Sat, 27 Apr 2019 02:43:18 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0BEBB6B0269; Sat, 27 Apr 2019 02:43:18 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id C192D6B0010 for ; Sat, 27 Apr 2019 02:43:17 -0400 (EDT) Received: by mail-pl1-f197.google.com with SMTP id ba11so3227654plb.21 for ; Fri, 26 Apr 2019 23:43:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=uR9dojmxOmmxTLzrNkqbIEwKeNq3pUquy4Aauxui5YA=; b=L0N/t/29dEdme8VzRCpPp9aOjpPnCp2cxAQSz0+rQ9PJXbta2fwiHmcPfmnYF2nlK6 LPtb27weecs9fLqrZYvRQtGJZiK/ENUoj/UJRwnqCTobikJzxycsS1+Euoh9rsEAukPN bkTJ9tviIVgvtImAXDJePlSH/JToYygi3/jB42uaXlFTwq10peBrmcRRhbF02NQu2pMm oYUUtUAqimf9QKGasNNqcqPc620mZmrUTE4FKrhdGcYBN0WzKXwpQkI/pCGTYpn5DfRu Iow3k7yQw9LvNBRHf+rOKwhpsKXQ3LaKfr8g8LU20xg/5QnK18ghhyQagyak/RcNDzhn 9kkg== X-Gm-Message-State: APjAAAVwyhAhlfJaDvoSWH4WsYwcBpy12kwUGFeTyv+Nrnj7qG4mCXAv Xf/skCDptkz73v+NrptAeFiNqzqjAQKJZF7jZHW+dIAV4FrbpRIcqAZQZYpaN12OQ4+GGT6k+zU 7hLUl+FrfjMGbXW53QnCg6ykHSbE1lySOJw1+dPWfc4X8HMgOPpL4qc3qy5RmcflErg== X-Received: by 2002:a62:1d94:: with SMTP id d142mr51482403pfd.83.1556347397436; Fri, 26 Apr 2019 23:43:17 -0700 (PDT) X-Received: by 2002:a62:1d94:: with SMTP id d142mr51482330pfd.83.1556347396062; Fri, 26 Apr 2019 23:43:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347396; cv=none; d=google.com; s=arc-20160816; b=h56eidIlWO4zGiSPT0yWEQP041PcqWHl+ewUdL6CcINCposHt97HrOXfKSl17udiCx GbrXwPe+VM3aubqVX4rg3l2cjRX9ParPTZd3v1DZOxxaEXcqmjkHvRuqQurriJtY5WYH IkROLL5VKJY2NPSLsLVASeofiJttELmamx1OK3Mv90ml1aT/fMultU1xsbC6N1oE8gxc IfyeBLi2fhybh/6OOlmFKiHN6To+REX21eMOuKyBXlIv4HKDZdzcS6YmWmaRC8rvGdQL jt/Q7VqbqO6aYgTcfHg6CxMk2ZMFdYizzQ5L7AfshCWxQ2ipU2hrff+WGbk9iMf1nxOs c5Gg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=uR9dojmxOmmxTLzrNkqbIEwKeNq3pUquy4Aauxui5YA=; b=M26/jhWTZf1zc2FNmFfEDyZhya+b/L6IsgPl/SbSIatqA1jwiia9AzV/iN0e8pCGIM 0anOlJCAqlSR8kXVJU2MKydmI6uqJiFuiBNQ/1JUxgaIsBUbFB/B85dIZSMMY4VgU0TC GxddYb9Mp+5d8dhbzNzKXt3LH2kxM8N2M+U8yIobwzbT69LTUBDLYcz2fG+BDXQ3VKEW t6PlX/H7TtQ9CsBVlY9HrLFfvT25bSkXxMJmd+F+2T+vuWKs8C9V5owpvDjuqUH9c+aR KAwmOwLI3VUWjWQkLqzwM9RkwnU9r1dNJ4tGwWycU8qxD7RJr3oEnaymfkGFyw5EkhGx eDkw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=oILNkBTM; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id 1sor27121342plz.0.2019.04.26.23.43.15 for (Google Transport Security); Fri, 26 Apr 2019 23:43:16 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=oILNkBTM; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=uR9dojmxOmmxTLzrNkqbIEwKeNq3pUquy4Aauxui5YA=; b=oILNkBTMPI8hbOfV9CTJU+m4eHw0hAUfxbTTNr2/RxOTWPpKbRKo8diZaBeA/Rmbm3 h5rZrB3HG0y3SmWymgT3Ka+NNLrVVEWGy5GTcbQKRQEA3wiSNrIdF0YIzNUEW260wZEH KE+3wzmKUfVn5aWbi1XY/cRblwX7PCPz65YSoU43s/26wSMoHptHFPQTS+iXil5iGEw+ MKuU4t2I8rSFdXKCm6I3uOxZEQ1YYZq5IGQ+JuIwxQpw5oc2ihHafO/Cd4i7/tLG7zpn Edmk3MnDF5/iWexA4JDZ+DyyFjQlaqdQ8ZangVzJbGAkvZg6LY9bd008qQlPu4OkFzMW +mVw== X-Google-Smtp-Source: APXvYqxGAw4qwJwGwZPBWfL5T3kXq42Nxulr2Eo5xhNgNYA2PVHpEfWTNqR9D9nfEhSjkOIDkP0Vjg== X-Received: by 2002:a17:902:e683:: with SMTP id cn3mr6142181plb.115.1556347395492; Fri, 26 Apr 2019 23:43:15 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:14 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu Subject: [PATCH v6 08/24] x86/alternative: Use temporary mm for text poking Date: Fri, 26 Apr 2019 16:22:47 -0700 Message-Id: <20190426232303.28381-9-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit text_poke() can potentially compromise security as it sets temporary PTEs in the fixmap. These PTEs might be used to rewrite the kernel code from other cores accidentally or maliciously, if an attacker gains the ability to write onto kernel memory. Moreover, since remote TLBs are not flushed after the temporary PTEs are removed, the time-window in which the code is writable is not limited if the fixmap PTEs - maliciously or accidentally - are cached in the TLB. To address these potential security hazards, use a temporary mm for patching the code. Finally, text_poke() is also not conservative enough when mapping pages, as it always tries to map 2 pages, even when a single one is sufficient. So try to be more conservative, and do not map more than needed. Cc: Andy Lutomirski Cc: Kees Cook Cc: Dave Hansen Cc: Masami Hiramatsu Acked-by: Peter Zijlstra (Intel) Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/fixmap.h | 2 - arch/x86/kernel/alternative.c | 108 +++++++++++++++++++++++++++------- arch/x86/xen/mmu_pv.c | 2 - 3 files changed, 86 insertions(+), 26 deletions(-) diff --git a/arch/x86/include/asm/fixmap.h b/arch/x86/include/asm/fixmap.h index 50ba74a34a37..9da8cccdf3fb 100644 --- a/arch/x86/include/asm/fixmap.h +++ b/arch/x86/include/asm/fixmap.h @@ -103,8 +103,6 @@ enum fixed_addresses { #ifdef CONFIG_PARAVIRT FIX_PARAVIRT_BOOTMAP, #endif - FIX_TEXT_POKE1, /* reserve 2 pages for text_poke() */ - FIX_TEXT_POKE0, /* first page is last, because allocation is backward */ #ifdef CONFIG_X86_INTEL_MID FIX_LNW_VRTC, #endif diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 11d5c710a94f..599203876c32 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -684,41 +685,104 @@ __ro_after_init unsigned long poking_addr; static void *__text_poke(void *addr, const void *opcode, size_t len) { + bool cross_page_boundary = offset_in_page(addr) + len > PAGE_SIZE; + struct page *pages[2] = {NULL}; + temp_mm_state_t prev; unsigned long flags; - char *vaddr; - struct page *pages[2]; - int i; + pte_t pte, *ptep; + spinlock_t *ptl; + pgprot_t pgprot; /* - * While boot memory allocator is runnig we cannot use struct - * pages as they are not yet initialized. + * While boot memory allocator is running we cannot use struct pages as + * they are not yet initialized. There is no way to recover. */ BUG_ON(!after_bootmem); if (!core_kernel_text((unsigned long)addr)) { pages[0] = vmalloc_to_page(addr); - pages[1] = vmalloc_to_page(addr + PAGE_SIZE); + if (cross_page_boundary) + pages[1] = vmalloc_to_page(addr + PAGE_SIZE); } else { pages[0] = virt_to_page(addr); WARN_ON(!PageReserved(pages[0])); - pages[1] = virt_to_page(addr + PAGE_SIZE); + if (cross_page_boundary) + pages[1] = virt_to_page(addr + PAGE_SIZE); } - BUG_ON(!pages[0]); + /* + * If something went wrong, crash and burn since recovery paths are not + * implemented. + */ + BUG_ON(!pages[0] || (cross_page_boundary && !pages[1])); + local_irq_save(flags); - set_fixmap(FIX_TEXT_POKE0, page_to_phys(pages[0])); - if (pages[1]) - set_fixmap(FIX_TEXT_POKE1, page_to_phys(pages[1])); - vaddr = (char *)fix_to_virt(FIX_TEXT_POKE0); - memcpy(&vaddr[(unsigned long)addr & ~PAGE_MASK], opcode, len); - clear_fixmap(FIX_TEXT_POKE0); - if (pages[1]) - clear_fixmap(FIX_TEXT_POKE1); - local_flush_tlb(); - sync_core(); - /* Could also do a CLFLUSH here to speed up CPU recovery; but - that causes hangs on some VIA CPUs. */ - for (i = 0; i < len; i++) - BUG_ON(((char *)addr)[i] != ((char *)opcode)[i]); + + /* + * Map the page without the global bit, as TLB flushing is done with + * flush_tlb_mm_range(), which is intended for non-global PTEs. + */ + pgprot = __pgprot(pgprot_val(PAGE_KERNEL) & ~_PAGE_GLOBAL); + + /* + * The lock is not really needed, but this allows to avoid open-coding. + */ + ptep = get_locked_pte(poking_mm, poking_addr, &ptl); + + /* + * This must not fail; preallocated in poking_init(). + */ + VM_BUG_ON(!ptep); + + pte = mk_pte(pages[0], pgprot); + set_pte_at(poking_mm, poking_addr, ptep, pte); + + if (cross_page_boundary) { + pte = mk_pte(pages[1], pgprot); + set_pte_at(poking_mm, poking_addr + PAGE_SIZE, ptep + 1, pte); + } + + /* + * Loading the temporary mm behaves as a compiler barrier, which + * guarantees that the PTE will be set at the time memcpy() is done. + */ + prev = use_temporary_mm(poking_mm); + + kasan_disable_current(); + memcpy((u8 *)poking_addr + offset_in_page(addr), opcode, len); + kasan_enable_current(); + + /* + * Ensure that the PTE is only cleared after the instructions of memcpy + * were issued by using a compiler barrier. + */ + barrier(); + + pte_clear(poking_mm, poking_addr, ptep); + if (cross_page_boundary) + pte_clear(poking_mm, poking_addr + PAGE_SIZE, ptep + 1); + + /* + * Loading the previous page-table hierarchy requires a serializing + * instruction that already allows the core to see the updated version. + * Xen-PV is assumed to serialize execution in a similar manner. + */ + unuse_temporary_mm(prev); + + /* + * Flushing the TLB might involve IPIs, which would require enabled + * IRQs, but not if the mm is not used, as it is in this point. + */ + flush_tlb_mm_range(poking_mm, poking_addr, poking_addr + + (cross_page_boundary ? 2 : 1) * PAGE_SIZE, + PAGE_SHIFT, false); + + /* + * If the text does not match what we just wrote then something is + * fundamentally screwy; there's nothing we can really do about that. + */ + BUG_ON(memcmp(addr, opcode, len)); + + pte_unmap_unlock(ptep, ptl); local_irq_restore(flags); return addr; } diff --git a/arch/x86/xen/mmu_pv.c b/arch/x86/xen/mmu_pv.c index a21e1734fc1f..beb44e22afdf 100644 --- a/arch/x86/xen/mmu_pv.c +++ b/arch/x86/xen/mmu_pv.c @@ -2318,8 +2318,6 @@ static void xen_set_fixmap(unsigned idx, phys_addr_t phys, pgprot_t prot) #elif defined(CONFIG_X86_VSYSCALL_EMULATION) case VSYSCALL_PAGE: #endif - case FIX_TEXT_POKE0: - case FIX_TEXT_POKE1: /* All local page mappings */ pte = pfn_pte(phys, prot); break; From patchwork Fri Apr 26 23:22:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10919943 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EE7A6912 for ; Sat, 27 Apr 2019 06:43:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DE31328E6B for ; Sat, 27 Apr 2019 06:43:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D27BF28E7B; Sat, 27 Apr 2019 06:43:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 52AFE28E6B for ; Sat, 27 Apr 2019 06:43:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C0936B0266; Sat, 27 Apr 2019 02:43:19 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id F1EDF6B0269; Sat, 27 Apr 2019 02:43:18 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CCA876B026A; Sat, 27 Apr 2019 02:43:18 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 90D316B0266 for ; Sat, 27 Apr 2019 02:43:18 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id x9so3264968pln.0 for ; Fri, 26 Apr 2019 23:43:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=3mFI7V2IpbwOhuCcp+g9CO2dE2meFXsvyyAeb18NWyo=; b=HzkdLnvf/drcKvzh+rublwBArOc4U6uhMsNOYCjiAKme2ONCaFUVGphjmajMVsiKZf xYo2a7lUj++qoFyyZmwRKvlEV2tM2ACNfcqIfOQQ8sN7CTe9w8N8KP+z2AY3+HPLRx7l x19Vu1Xe5EANbtrVZrCotOCuhI90gpjL4yQfnzzW9fvH3VuTMowW/T02gwAREQ6HfcuY oGrz1DnbgOCexWpAxy2XKLNP/QSOHUMdiP/CgiUsIT4KuoAAhCvbC+v75tPYJ4vYQLZ+ Q3ZpbaHU3Z0AxxyOyVXUzLH7U/9UkBbOicXFg066hc9Kzk4N+WhSnLJcPnGPk017m9ib rG0Q== X-Gm-Message-State: APjAAAWCnJ/JQNTmC7Sj17xU8bKBY6JmToTPf4ms9JsuPdzV+/RFJhwM JJkZLT9AFiSN81zjxY+rBPJzX4IrSTTwh3oIbUwK0Xtty85qGxZm5hae3bJN5zoGICwjtPa527Z IBe0QgnkVRIJlR3PuvDeoW2K9yn+YmRypTW961lWcqUTRhIRIH83QYSxur5z09dDfXw== X-Received: by 2002:a63:8e4b:: with SMTP id k72mr4659946pge.428.1556347398278; Fri, 26 Apr 2019 23:43:18 -0700 (PDT) X-Received: by 2002:a63:8e4b:: with SMTP id k72mr4659902pge.428.1556347397318; Fri, 26 Apr 2019 23:43:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347397; cv=none; d=google.com; s=arc-20160816; b=F9t8ZXxqXwBV5LOfJk/tPreS6QWkNeIaRSN3kNIs43qiwyPo7NWbS4au3VCWJhygXV etq4mJnKPShE5QFvCfATnzPMZ7Mg7/Y/RW55c11AWegsDc+EKVpvES2qpznRYySjfhci xSHfItVhqNVtNJZlFW48AkggropsHtVScvp8E4ZOjzvsQYOpZe1l8hh/M8JfeXPRPk/V 9KftNNL8A69CTlJXeN+BmCOFzD9ut7n5LYw3aCKBXZpBSmZz3Ix1obVbvrgPQ4f2Z4/Y wVZS5GETf9LytGMdLecK0Er/Hzz9bhYCqDN/bEfn4SyfIQCg0RXWsPuLDdyLYCYTmo7V BE5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=3mFI7V2IpbwOhuCcp+g9CO2dE2meFXsvyyAeb18NWyo=; b=LBbDbUOWffF+8ioon5rkettulVH0oLvH/GFr8r+xiC6g+f1CAckL+GH6EHM8++aIzn NePPrvKKtxM6Gpe2z7B8unWeTui2ThQ8j49c9VmhlTmbjzKlL7xFRsSJS71eWs8exC0p Mev1rVoWPL7FRoEw3K7Qxd2htuBOl/QZPfKuLHrhz1UL5CEKlHR21GoWSbJnQg7uAURn z2EDVDZvXq3BGsreeLeViqe5Rm81viuEJtRmo30f0nvhQF/G+heBx5Vfnn/tDMY2m2Kx DgyJn2dz0Zxmz+UqS/KrjYg//4bd/VxziXbGqopE+W3vA06H8oXQt7EmbWpJfw9P49vp IpUQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=JlRIHgbr; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id w10sor26281580plz.54.2019.04.26.23.43.17 for (Google Transport Security); Fri, 26 Apr 2019 23:43:17 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=JlRIHgbr; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=3mFI7V2IpbwOhuCcp+g9CO2dE2meFXsvyyAeb18NWyo=; b=JlRIHgbr6HB0GM/dfrBCsDnNLxTkOKzTB7qYCbAx4tDSjiPWdWErWZcgNMKz3646h4 C8cKj9bZkmi6cPWikKj5uJAC7SrSg/qfrmI03cXhBRcOJHEMan3BYUSQ1ScXOsNa4u3Z IDQ4DfzWzmEsKN3rKTGM5uTvN695pUVNWWV4a8LMMvLzAhr7fWiHf2k7ZmkakgE11lMg wbrOSfmIb0CTjB/azGiS0/j+OjbriGytqXhtU6Bd27x2yAPLaWXYIt792EgmySvUAGOQ 9VF/satnq8MCbRYyAQp+FRH27QsFCLaAp8PdnQobxJM8asT3n363jT4JX9FD5F/9ebkE 0Uqg== X-Google-Smtp-Source: APXvYqx2WhBZnJIn8mqH1b7ixt4GF/6k4yp110M4lMdf7eXq6Mz7eqmVNfTsfdPSyk5JMfubTGyIEQ== X-Received: by 2002:a17:902:2927:: with SMTP id g36mr48053244plb.6.1556347396829; Fri, 26 Apr 2019 23:43:16 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:16 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Nadav Amit Subject: [PATCH v6 09/24] x86/kgdb: Avoid redundant comparison of patched code Date: Fri, 26 Apr 2019 16:22:48 -0700 Message-Id: <20190426232303.28381-10-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit text_poke() already ensures that the written value is the correct one and fails if that is not the case. There is no need for an additional comparison. Remove it. Acked-by: Peter Zijlstra (Intel) Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/kgdb.c | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/arch/x86/kernel/kgdb.c b/arch/x86/kernel/kgdb.c index 2b203ee5b879..13b13311b792 100644 --- a/arch/x86/kernel/kgdb.c +++ b/arch/x86/kernel/kgdb.c @@ -747,7 +747,6 @@ void kgdb_arch_set_pc(struct pt_regs *regs, unsigned long ip) int kgdb_arch_set_breakpoint(struct kgdb_bkpt *bpt) { int err; - char opc[BREAK_INSTR_SIZE]; bpt->type = BP_BREAKPOINT; err = probe_kernel_read(bpt->saved_instr, (char *)bpt->bpt_addr, @@ -766,11 +765,6 @@ int kgdb_arch_set_breakpoint(struct kgdb_bkpt *bpt) return -EBUSY; text_poke_kgdb((void *)bpt->bpt_addr, arch_kgdb_ops.gdb_bpt_instr, BREAK_INSTR_SIZE); - err = probe_kernel_read(opc, (char *)bpt->bpt_addr, BREAK_INSTR_SIZE); - if (err) - return err; - if (memcmp(opc, arch_kgdb_ops.gdb_bpt_instr, BREAK_INSTR_SIZE)) - return -EINVAL; bpt->type = BP_POKE_BREAKPOINT; return err; @@ -778,9 +772,6 @@ int kgdb_arch_set_breakpoint(struct kgdb_bkpt *bpt) int kgdb_arch_remove_breakpoint(struct kgdb_bkpt *bpt) { - int err; - char opc[BREAK_INSTR_SIZE]; - if (bpt->type != BP_POKE_BREAKPOINT) goto knl_write; /* @@ -791,10 +782,7 @@ int kgdb_arch_remove_breakpoint(struct kgdb_bkpt *bpt) goto knl_write; text_poke_kgdb((void *)bpt->bpt_addr, bpt->saved_instr, BREAK_INSTR_SIZE); - err = probe_kernel_read(opc, (char *)bpt->bpt_addr, BREAK_INSTR_SIZE); - if (err || memcmp(opc, bpt->saved_instr, BREAK_INSTR_SIZE)) - goto knl_write; - return err; + return 0; knl_write: return probe_kernel_write((char *)bpt->bpt_addr, From patchwork Fri Apr 26 23:22:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10919947 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3BF5C14B6 for ; Sat, 27 Apr 2019 06:43:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2BF9928E6B for ; Sat, 27 Apr 2019 06:43:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1FCB028E7B; Sat, 27 Apr 2019 06:43:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 844E328E6B for ; Sat, 27 Apr 2019 06:43:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B21826B0269; Sat, 27 Apr 2019 02:43:21 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A8F516B026A; Sat, 27 Apr 2019 02:43:21 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 79C4D6B026B; Sat, 27 Apr 2019 02:43:21 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id 411916B0269 for ; Sat, 27 Apr 2019 02:43:21 -0400 (EDT) Received: by mail-pl1-f198.google.com with SMTP id g11so1496916plt.23 for ; Fri, 26 Apr 2019 23:43:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=CLKl6NZzJuOK2yM4IgvyVy0XzBWA5gDBiZYSHfLNQ00=; b=JnmOGbdL3+J3Qesk+Y6hrJ7kweaJUGVcmzcHlfKVJc8WNs3uOc9FbPKvX7YNOITkwv Qi27FVEZ/YoDXAOqrMnXEYZiIQkr6jsam4ncPQMiPantPippyyk8iz0DSvg5h1FBUB5I R3WO+seNraaKOEMLPucCYmv4I+THwj4flaLyK4jpOA1bKmns7OQ0abt1R62gD6ECp0FO K4E1zS0V/no6AqMhgjlHJAgP/VM8ZLm7qn1Zntu3lE/H1aNF4BduLNu4Pm9jQaZ62zcw GF7g2rd35eKYQazjRZyvodNZxXPp7O2gGu9qox1CQC+HNCZYOE8guyaWwsHhF4mLA4BN GN6w== X-Gm-Message-State: APjAAAXM3LCdeMdyxQvjMUT6cd04Kgnxf71knCc2AAZRFEjOwynmseSs VxXDhER9vVbd5f+ynJfLvE7ubDnGy/D9dbCkxtSyUpRMlc+0/6VuBsE27VyM/dVX2OldsJIUhRK mY+FdxPc3+a8RRv71sB481lggmlNCH3+Vb/adKFqIvGFE/4+D2LgGgDhLsjg8geyttA== X-Received: by 2002:a62:5885:: with SMTP id m127mr17170993pfb.33.1556347400951; Fri, 26 Apr 2019 23:43:20 -0700 (PDT) X-Received: by 2002:a62:5885:: with SMTP id m127mr17170945pfb.33.1556347399835; Fri, 26 Apr 2019 23:43:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347399; cv=none; d=google.com; s=arc-20160816; b=JDcfSYt+6l4FenuIYUYIJdGl2b4LTAcu8w+S89zS83LGpzqS/G7Sdlo8H9WMyAvNVc bfPrEmCteuP9Gre9FXwEC2l842GSwZW4Bo1b3/EzGday57N47Y20k2WG4CwQ7N3J8Su9 ALOwYbV5QOGxhM4O7i4OOL+LHjofPDBhWHDfkR35vCuBwzk1zwJRXLrpZaSVjaCYlQBJ bIbMkm3MLACiURnbV/OYKyjk8BetYRqlBZnsYE/0KhX/Cbu0Xr0LgIAWG9+cuo0nEfaT qhyym7ckK4XQVfxdrI3TFqHBxxdfH3EkAVDpqJsHapy1S31lCjZCeRs358VYBvHBzMux 3PjA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CLKl6NZzJuOK2yM4IgvyVy0XzBWA5gDBiZYSHfLNQ00=; b=P3JtT3xiyYui5F/rOPDBlsv8QcHh7KdtKvViuFRiwSzVIVnycF95xocAJW8RqkvU9Z QD3iOLWNEy7smAKaSMfB/tXEVrtwFGVnUi0Ay43oHAxXl/9mnDeCsMxfTIVRKFlwjLOc FHHAfwj3M6Ct3kIlxEBv0DJsmiXEvoEVxIgJkVrzUjVxNA1oieWXytjxEx+AEwWsvFvO 0HQGxCUGqRdpUMt8L+XBJ/Wqz7BOFOy5J0YF43WDFKXVoZhSvUqGI5aryp2DZkE2ewys F/xzfIB7CRef9G6uFWKwRvsHXBZhgqDqXmHiG5dzgDos29y+3BZHL6T1xTrTe1xYoxTH rCtA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=LWiWMfB1; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id c14sor30826845pfc.71.2019.04.26.23.43.19 for (Google Transport Security); Fri, 26 Apr 2019 23:43:19 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=LWiWMfB1; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=CLKl6NZzJuOK2yM4IgvyVy0XzBWA5gDBiZYSHfLNQ00=; b=LWiWMfB1NCPlUWHzx7KNU9QCQPe3RdgA8qCNPze1WHQjmMq2EnzDsMlPceyPd+MHEm QMnqsza7hwGkI6ZFGgTOeRjeOEfXCrRJYup1t+tZNnEBvqMdc01pTHEj2bJKvz7BRo7K E/q3JXY24fjwRSyZU/Tw4XNKZ1nfyh4euDeX0+jZD8/F9QgFlj4BNpu6QQplyXiEhS74 PVZ8Bi1kj7P4bpKFaIDD7kzaPwPpjBstpFpZumGDsgFZsa4oA3jDKygoO8TIjMEYq1h+ 8j/kDmbTP0+sW4oBzq9jDl4H2CBcWV2hkh9XtxTKhVDjbhF8cy7tl1t7BO8Iros2slYv iP/w== X-Google-Smtp-Source: APXvYqx8IDeCvv7F4Tl+g2zi6Qtjw6+CBIajqUbAMvc7uvXy5z6A8pYUXx0y0vJXO744BzKTEFFO2g== X-Received: by 2002:aa7:91d6:: with SMTP id z22mr42068121pfa.242.1556347399283; Fri, 26 Apr 2019 23:43:19 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:18 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Nadav Amit Subject: [PATCH v6 10/24] x86/ftrace: Set trampoline pages as executable Date: Fri, 26 Apr 2019 16:22:49 -0700 Message-Id: <20190426232303.28381-11-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit Since alloc_module() will not set the pages as executable soon, set ftrace trampoline pages as executable after they are allocated. For the time being, do not change ftrace to use the text_poke() interface. As a result, ftrace still breaks W^X. Reviewed-by: Steven Rostedt (VMware) Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/ftrace.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index ef49517f6bb2..53ba1aa3a01f 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -730,6 +730,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) unsigned long end_offset; unsigned long op_offset; unsigned long offset; + unsigned long npages; unsigned long size; unsigned long retq; unsigned long *ptr; @@ -762,6 +763,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) return 0; *tramp_size = size + RET_SIZE + sizeof(void *); + npages = DIV_ROUND_UP(*tramp_size, PAGE_SIZE); /* Copy ftrace_caller onto the trampoline memory */ ret = probe_kernel_read(trampoline, (void *)start_offset, size); @@ -806,6 +808,12 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) /* ALLOC_TRAMP flags lets us know we created it */ ops->flags |= FTRACE_OPS_FL_ALLOC_TRAMP; + /* + * Module allocation needs to be completed by making the page + * executable. The page is still writable, which is a security hazard, + * but anyhow ftrace breaks W^X completely. + */ + set_memory_x((unsigned long)trampoline, npages); return (unsigned long)trampoline; fail: tramp_free(trampoline, *tramp_size); From patchwork Fri Apr 26 23:22:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10919953 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D23E914DB for ; Sat, 27 Apr 2019 06:43:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C122128E6B for ; Sat, 27 Apr 2019 06:43:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B525328E7B; Sat, 27 Apr 2019 06:43:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3862D28E6B for ; Sat, 27 Apr 2019 06:43:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 332076B026A; Sat, 27 Apr 2019 02:43:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 21EDE6B026B; Sat, 27 Apr 2019 02:43:23 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 04C7C6B026C; Sat, 27 Apr 2019 02:43:22 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-it1-f200.google.com (mail-it1-f200.google.com [209.85.166.200]) by kanga.kvack.org (Postfix) with ESMTP id D704D6B026A for ; Sat, 27 Apr 2019 02:43:22 -0400 (EDT) Received: by mail-it1-f200.google.com with SMTP id p23so4901523itc.7 for ; Fri, 26 Apr 2019 23:43:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=yNv4uuWY37+QDIo+8eJ5Cs1RQKFygWuhu/mUpTFc+50=; b=r+fuLtQaqCYnkGPj7mv+XlxKjEEp6a9cXsQXvS7FNd2wf59zsQjVovhLQ2W32L2XoD HUo44LRRWFk3B0H4PGkd1jmBnXDVFEo0bairyCpS5+1WwMwcMguXyPx2QQr1B/F81npE 0y5FpBjbulxyMJYhl0fgnXkX0yq9JXbPOg7WP+jiCx8x7Rmq+Z89MJoSDGFqCzadJC/p lWHGXHsKLqomzUNh/kf/hMupiJ0W988gSU6B441uoU9da0e8nkStDwBSPJhRkK6C3ls3 kfuOCzKKZnws/XqHe7d0sGo6MdnnrLxbU0FRIHO8jNsQ5ZXch5Gro5LFaAC73sshtGNH aRTg== X-Gm-Message-State: APjAAAXwGlxWWh9cCAht/FQP9Kvx1rRteoLF9HNFjJ8K6Nn1Sa7fZO2O oDmWkrXy1OdGGDUxRF4XTSyvewy9oXUg6OwRV0tIfOBptni8I4+ipUKkEbjemdPMvA7cGL0LKRJ CFOtUoIXj89LXVR+z8HYMN7IajcEqKy1+4YJs5YPx4BTOeGgTMYl3wBfwv6T5iyjXvg== X-Received: by 2002:a24:f8c7:: with SMTP id a190mr10805151ith.72.1556347402565; Fri, 26 Apr 2019 23:43:22 -0700 (PDT) X-Received: by 2002:a24:f8c7:: with SMTP id a190mr10805121ith.72.1556347401264; Fri, 26 Apr 2019 23:43:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347401; cv=none; d=google.com; s=arc-20160816; b=wlN4H8Q48VuEb2TmuE5Mbr0lGmh+bGAJI/OuP9ignJhPRHzEPIMsg2aBIe1MX1LWML RKceN9IcrSLyJROyrLomulNpbOBRgKX7LysQbC9bfwkumVeP1D8LyQvXmFAI3AHK0ddt bbRWb9rYz86zej7GBmS9CbOn99BvnBTpFpkBPQkNYGpVUZi6we7DWxhz8etUTW9S+xAB n9qreGioHp9/j6XcbLWrsQzKeltYuNWBjgkokOF4hvzZMpOC+jsbqVxD+j07HgsKKY10 0W0MUfBd4bHLwfooSv+/+YvOL/r/lw76pXGtMR2rcCURvp8zETSkNKEoWjJXR+0bTpE2 XwYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=yNv4uuWY37+QDIo+8eJ5Cs1RQKFygWuhu/mUpTFc+50=; b=E2CkCn+DAMmsWvLzV5GP1rwODgXOTt1Muy81wbkCLDjuWlI41g/J65q8zZqWYVv9bI dZCkR049ikc4t4jOFnhmaDkoRFhTnpa/9ZEEUIAomkJNs2uv91KQYomOmF6RJum+3k6P AfB7pky09B9kQcP5oXcV7in1C7RYOxSfbm9dUczGdpwkEADXrDrHxVmOOWVcvRrWKCwZ cQQ1Wle/1W+IR3xiSYVF6TkHt5slW1uenknOSqQW4tcyWRHjVFXcvRmXRPdF18OLigy9 zGfuUcY+qyQHt7CbrnA1Gh89gcNYPm3EiujhMYhDVnD9eAT6ko+FG8O3mU+A+DOb6O4S ivNA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=qfcv0jVy; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id s135sor9184478itb.9.2019.04.26.23.43.21 for (Google Transport Security); Fri, 26 Apr 2019 23:43:21 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=qfcv0jVy; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=yNv4uuWY37+QDIo+8eJ5Cs1RQKFygWuhu/mUpTFc+50=; b=qfcv0jVylRwgUSdFp0yVo6y3SoCAOjol9XbR7W5+lwraaEg2QoXyt+VrAjGp9wzdAn BOm9G9cC4mrsWbrKkWfeWua0CLcUhNFX2UYmr160CAGJNRPhpKunjycWCJnpjaUptQXK FymoezD52jYoCa9vw1d8DrTtNcDpUD3gesH2ZoYEwsgq0bZE6tTukM9zPOscjJPRRiHd 0d0EQKT1ASnMLtYxkgiq4MVCWAUZf/kjKC/u6ibFbPlbEhDhs/jBfSTvPAadE2T4MBPs yoHMRLVUDfJJ0O31Q2uS3OPiszz2lAVHUHfzcDpTk8KxpsBn4aqr0vTCALbdk7a/EPYd 2uWQ== X-Google-Smtp-Source: APXvYqxCgMCnuubydaHXfJIDthz2u46RkKoNrhpM9xg9spe5Zm1Eq7YBRg8tloTlDMLwvkBDrQMPvQ== X-Received: by 2002:a63:7f0b:: with SMTP id a11mr44847434pgd.234.1556347400564; Fri, 26 Apr 2019 23:43:20 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:20 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Nadav Amit Subject: [PATCH v6 11/24] x86/kprobes: Set instruction page as executable Date: Fri, 26 Apr 2019 16:22:50 -0700 Message-Id: <20190426232303.28381-12-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit Set the page as executable after allocation. This patch is a preparatory patch for a following patch that makes module allocated pages non-executable. While at it, do some small cleanup of what appears to be unnecessary masking. Acked-by: Masami Hiramatsu Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/kprobes/core.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index a034cb808e7e..1591852d3ac4 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -431,8 +431,20 @@ void *alloc_insn_page(void) void *page; page = module_alloc(PAGE_SIZE); - if (page) - set_memory_ro((unsigned long)page & PAGE_MASK, 1); + if (!page) + return NULL; + + /* + * First make the page read-only, and only then make it executable to + * prevent it from being W+X in between. + */ + set_memory_ro((unsigned long)page, 1); + + /* + * TODO: Once additional kernel code protection mechanisms are set, ensure + * that the page was not maliciously altered and it is still zeroed. + */ + set_memory_x((unsigned long)page, 1); return page; } @@ -440,8 +452,12 @@ void *alloc_insn_page(void) /* Recover page to RW mode before releasing it */ void free_insn_page(void *page) { - set_memory_nx((unsigned long)page & PAGE_MASK, 1); - set_memory_rw((unsigned long)page & PAGE_MASK, 1); + /* + * First make the page non-executable, and only then make it writable to + * prevent it from being W+X in between. + */ + set_memory_nx((unsigned long)page, 1); + set_memory_rw((unsigned long)page, 1); module_memfree(page); } From patchwork Fri Apr 26 23:22:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10919955 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 08BF214B6 for ; Sat, 27 Apr 2019 06:43:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EA22728E6B for ; Sat, 27 Apr 2019 06:43:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DD3BF28E7B; Sat, 27 Apr 2019 06:43:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4161728E6B for ; Sat, 27 Apr 2019 06:43:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 94E256B026B; Sat, 27 Apr 2019 02:43:24 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8D4306B026C; Sat, 27 Apr 2019 02:43:24 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7065C6B026D; Sat, 27 Apr 2019 02:43:24 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 2B9D26B026B for ; Sat, 27 Apr 2019 02:43:24 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id s26so3555564pfm.18 for ; Fri, 26 Apr 2019 23:43:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=0/++kCIdz9NinK/+MFp07AkuSYwP6lr30tD8Xi+P1Ns=; b=qhyygjg4RJM24OYfFTjZaupfDJ+o1dMgUGdKMjMGKQr689R0n0L+WfVpqhRmdMlGLv cPZzYkeJVW7fs92kZT75crDHq4YAiC+CUW3WGUr9Dpx4NibZcCk4K3oK6G/jc1qSwQAT GO6C5e3n8HII/YlPX1OMAcB9ARgGNcCLPjrEe2CK9nObtWz1ZGoQ+FY4R+fP9BfaHRBt oTpaOjze1QXROVyetRLzDFMcGX4jamLzoxmL1+GzbhuR/tgQIPIwWd3Cy9nLH6OSkJo6 FBJjqKmgJFT8/0Z6ZT1LrPN5QwOZ6l7pAdHPmzklRwY+NF7BVrvwy6b513uWAgVJRaTv 6Tcw== X-Gm-Message-State: APjAAAUhnrNZ1iH0UtWnWdM6PZfXGSjnhtuBfOVy7K/KW+3k0jzcPXsy wgefLRNzeKRQZj0fFry3PBlJJlKmDfSkx0pFBUzj41L0w75bRwt/qbWDVNIHsDh41r2ZGCvMLUn 3KjE1grjrahbGOBrY9b199dbNFWDBMa9h1VqIcSLYyaMzdlg/vEvqbcrkupkh9pJ+mg== X-Received: by 2002:a63:161d:: with SMTP id w29mr25050122pgl.395.1556347403860; Fri, 26 Apr 2019 23:43:23 -0700 (PDT) X-Received: by 2002:a63:161d:: with SMTP id w29mr25050068pgl.395.1556347402553; Fri, 26 Apr 2019 23:43:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347402; cv=none; d=google.com; s=arc-20160816; b=tqLYQojMRIuwoE5/MfWjzprmB1YaYNB6D9wxdw7snuM/Q9FTOC2cumw5yrw6mDt0l8 BqT0EL9KcJK/gM7Tm7OSpGKSixkv+zxVCfEUlh7YZSrXFyla5ngFo6eC/1Nxxt9OHLv5 6Q2qPUQXuiLWnKQzNILot+pc9UhUt6sxvFydUJQLx0p38+H3W0l7Zi6FCofTXNLyhO7d OlCFsD3Z4gKO31IS5YJFro3hDYbryN62CE0wj7qMcYKa9XUdfk3GfIVkcb0+sqImYjGe 293wODY3mZZkloMSXNhbjRmNer03ie6hwkkXKVBh1N9NC6L80s9v9PRqk62dzZwPtM+g GwLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0/++kCIdz9NinK/+MFp07AkuSYwP6lr30tD8Xi+P1Ns=; b=lpOcdu/Sao0aRCKoqpZwt8PsiwuubAHW/311VYK/ovflClcxI0dm7ySEjP1mX3D3/Q Zh31jMM3OEvs2Icj4jUni5O2nzi4GGJ4YryQKqrYNF5yX1PeVQ7F+YvvHmj/0M1OZNqw rBHjiUS3vBaWCDZg1LglPUaIkpivycF3bjXMxhWP/dDA9Sai89DxMqx3BOwWZPFcD22m ZczMzUxtijG5xwfUCc6kc8IXsDHGSRWtNaGg0ew50fgGagCkGvG9JzDvQYUCADS1uYWX uCuNplc6BjSiMrhZpEv1NAlkTBsDYLQTfBrjX08cweOH4i2jRBc9Uu0jhGsVM1dchOCF JVkw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=c8S+0Eig; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id g95sor26343843plb.5.2019.04.26.23.43.22 for (Google Transport Security); Fri, 26 Apr 2019 23:43:22 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=c8S+0Eig; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=0/++kCIdz9NinK/+MFp07AkuSYwP6lr30tD8Xi+P1Ns=; b=c8S+0EigCS6E8XvKa358lLe6LCVfzSFVuNJOVAHrtcVTOWv87dH8gGM88UuaQDW1Ql IEkRJXiL08aosg7wFKjOifGHvdFqqFOyIVAPTy32+RTxwh4obVo1hvSc3VrfhbIWeRKu ZJA1hj0UwfXTRnrJeU5WqlBmd3LvGogakdG6r5O5HCHdIZ01YxMYWq3GzL7qFIqoHj+M ImxYzSZVWsC/5XsbT15MbduZoaGGNML2WLcc4snNS8OhI4V0R2fmhxvtsFVBQg8hBw68 V9bB/2b3yhVaonMZdGNuWp1CujPDm/DoQRIcyXEO1I8gk21/7x6PmT++18rtH9HSkyQc PS4g== X-Google-Smtp-Source: APXvYqwD3aMW9+fxSDptS7zjHtrUL6axD8ahdhvdTbAECEHQ3uOfUvnD7wIP6ssynsnm5EaQ/VhgYw== X-Received: by 2002:a17:902:20c6:: with SMTP id v6mr48012719plg.276.1556347402023; Fri, 26 Apr 2019 23:43:22 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:21 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu , Jessica Yu Subject: [PATCH v6 12/24] x86/module: Avoid breaking W^X while loading modules Date: Fri, 26 Apr 2019 16:22:51 -0700 Message-Id: <20190426232303.28381-13-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit When modules and BPF filters are loaded, there is a time window in which some memory is both writable and executable. An attacker that has already found another vulnerability (e.g., a dangling pointer) might be able to exploit this behavior to overwrite kernel code. Prevent having writable executable PTEs in this stage. In addition, avoiding having W+X mappings can also slightly simplify the patching of modules code on initialization (e.g., by alternatives and static-key), as would be done in the next patch. This was actually the main motivation for this patch. To avoid having W+X mappings, set them initially as RW (NX) and after they are set as RO set them as X as well. Setting them as executable is done as a separate step to avoid one core in which the old PTE is cached (hence writable), and another which sees the updated PTE (executable), which would break the W^X protection. Cc: Kees Cook Cc: Peter Zijlstra Cc: Dave Hansen Cc: Masami Hiramatsu Cc: Jessica Yu Suggested-by: Thomas Gleixner Suggested-by: Andy Lutomirski Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/alternative.c | 28 +++++++++++++++++++++------- arch/x86/kernel/module.c | 2 +- include/linux/filter.h | 1 + kernel/module.c | 5 +++++ 4 files changed, 28 insertions(+), 8 deletions(-) diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 599203876c32..3d2b6b6fb20c 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -668,15 +668,29 @@ void __init alternative_instructions(void) * handlers seeing an inconsistent instruction while you patch. */ void *__init_or_module text_poke_early(void *addr, const void *opcode, - size_t len) + size_t len) { unsigned long flags; - local_irq_save(flags); - memcpy(addr, opcode, len); - local_irq_restore(flags); - sync_core(); - /* Could also do a CLFLUSH here to speed up CPU recovery; but - that causes hangs on some VIA CPUs. */ + + if (boot_cpu_has(X86_FEATURE_NX) && + is_module_text_address((unsigned long)addr)) { + /* + * Modules text is marked initially as non-executable, so the + * code cannot be running and speculative code-fetches are + * prevented. Just change the code. + */ + memcpy(addr, opcode, len); + } else { + local_irq_save(flags); + memcpy(addr, opcode, len); + local_irq_restore(flags); + sync_core(); + + /* + * Could also do a CLFLUSH here to speed up CPU recovery; but + * that causes hangs on some VIA CPUs. + */ + } return addr; } diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c index b052e883dd8c..cfa3106faee4 100644 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -87,7 +87,7 @@ void *module_alloc(unsigned long size) p = __vmalloc_node_range(size, MODULE_ALIGN, MODULES_VADDR + get_module_load_offset(), MODULES_END, GFP_KERNEL, - PAGE_KERNEL_EXEC, 0, NUMA_NO_NODE, + PAGE_KERNEL, 0, NUMA_NO_NODE, __builtin_return_address(0)); if (p && (kasan_module_alloc(p, size) < 0)) { vfree(p); diff --git a/include/linux/filter.h b/include/linux/filter.h index 6074aa064b54..14ec3bdad9a9 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -746,6 +746,7 @@ static inline void bpf_prog_unlock_ro(struct bpf_prog *fp) static inline void bpf_jit_binary_lock_ro(struct bpf_binary_header *hdr) { set_memory_ro((unsigned long)hdr, hdr->pages); + set_memory_x((unsigned long)hdr, hdr->pages); } static inline void bpf_jit_binary_unlock_ro(struct bpf_binary_header *hdr) diff --git a/kernel/module.c b/kernel/module.c index 0b9aa8ab89f0..2b2845ae983e 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -1950,8 +1950,13 @@ void module_enable_ro(const struct module *mod, bool after_init) return; frob_text(&mod->core_layout, set_memory_ro); + frob_text(&mod->core_layout, set_memory_x); + frob_rodata(&mod->core_layout, set_memory_ro); + frob_text(&mod->init_layout, set_memory_ro); + frob_text(&mod->init_layout, set_memory_x); + frob_rodata(&mod->init_layout, set_memory_ro); if (after_init) From patchwork Fri Apr 26 23:22:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10919963 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2E06B14DB for ; Sat, 27 Apr 2019 06:43:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1D12E28E6B for ; Sat, 27 Apr 2019 06:43:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1060528E7B; Sat, 27 Apr 2019 06:43:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 775DA28E6B for ; Sat, 27 Apr 2019 06:43:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C82D76B026D; Sat, 27 Apr 2019 02:43:25 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C09976B026E; Sat, 27 Apr 2019 02:43:25 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A5B8D6B026F; Sat, 27 Apr 2019 02:43:25 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id 597056B026D for ; Sat, 27 Apr 2019 02:43:25 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id r13so3486702pga.13 for ; Fri, 26 Apr 2019 23:43:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=g3nO4xFM3yb0X+OIqTEtEp6LbffMj61RqStKCNeSSiM=; b=jUBCRYiFnxYIhnlMJpjoZXVPqYJsutcG5NDECHOcUsINkWIqoREs54xNPjxgumcX00 TFPBeBgcVeMyuDk0pqW5GJZezew+r6I2/sA9ydC0pTLBytKRfIgi3OGt3fuHjaTcUBbY OKkp6d4EE/lQfB3g2Of2ZBTWFAamcMs1qxY/OWWcy1yPESVEEIVt+AHmeHh/HWu6NCFm 9Dej5RYDUovR1fbAO3Q1UfHBX3xzOGkUge5tNNUngGyJowVel/Nb6flTgL8BDZX2Pi6i /LBD5QBSs8EtbfSKsj5Fpuk4/b1OseDc+U1NxM5JVLzOJ42IP9o8vMFsDwYpae+OD0IY OPjA== X-Gm-Message-State: APjAAAV9sX2EjUPq8tTHBKLn1jjIobfewqCpeABHO8hZtCCV5m2UazIh yUtYdV+elfAfuZ14LqtPzrrCBWE5YTI9rWaiKhdF93kBq7dQ7dGtqSlnqPBJpMkD8IjE8Xv+GHJ FNZk+x/8TG2lA8IT6X9PoeYtXx4yl9fvpqMOdjrepQDZtVyvgIPEYRF0X4vkaw4Oc4Q== X-Received: by 2002:a62:1b8a:: with SMTP id b132mr50797154pfb.19.1556347405039; Fri, 26 Apr 2019 23:43:25 -0700 (PDT) X-Received: by 2002:a62:1b8a:: with SMTP id b132mr50797082pfb.19.1556347403859; Fri, 26 Apr 2019 23:43:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347403; cv=none; d=google.com; s=arc-20160816; b=Mn2TQVsrI6Kf5b2JZ4I9J1dXfECi48Mdxr1pe947lMy3+iPKdEH7HbF3HZGE0wYFpr qPzK92dvNsn5qj8uDaLlZsV/GwMCqR6hXfvTX5/7QgfMmmttKAYj5QtwdPO1cHqd2zB7 UVUjv+jrOSVCqcao9Ki3QZvx6xz+G1bda+oIxWGrpWs0kMX/6LiU7ZdylyuWln3GS1rb CbvSTuso8n8FfyJqu4MHpWRhns/SCeOKXdzzBIAD+zOCVxLQD9ftux4XM2edwvvikTa0 dZ7XMcnM1561Voz8QHGnTweo8gzNA82wAPFzKLbOyNMit0yhNbArUjsh406PM7KQVPh/ 4nLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=g3nO4xFM3yb0X+OIqTEtEp6LbffMj61RqStKCNeSSiM=; b=MiJuegQw9Apl1GGFIWx6wtyjVq6jYmgLlVZ98s2Zo3OezsS7wBnuBvhO0/a7UYmIyb Ux/Hhyb10Gk65o6KU8MYmADq0+Yow/BVt5CpMT/YGp7McBpx7NVFXkd8oBIeWJ0qZnTg qTydfhhfx7mR+2kaUjq4xWiYLDF1WFXBU2ff1+cNyON+eGKS7Fe7vd4iN+kZOwRDwkQu KCUhIjKoCFmU08Ya2XuKtHjrBDa/UXxJxW/zCZItb7IHO9pojLXN2IQKqxFzGxxxn9aS foDMwot0mpLyII96OBJsLn5M7t2czy0QMTPuxCTWGsFxMrKzxCApbburBw1FAYeI7wNh SFOA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=IhMsROzO; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id j7sor9239414pfa.29.2019.04.26.23.43.23 for (Google Transport Security); Fri, 26 Apr 2019 23:43:23 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=IhMsROzO; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=g3nO4xFM3yb0X+OIqTEtEp6LbffMj61RqStKCNeSSiM=; b=IhMsROzOX/uvsuQybijdR85Kqaxq4yWzjUoagIOqk1LVY6OLTEXQbjBmXHD5WxbAzP cmCoo1zMlycyEW/HfwZCgn/X+0dMFPFHyqqtEUKq6AZF6jWZrRVIUAiE8jxYjY0M9MT3 Ik7ZKhQeM7kg+3gB06XcTjx2xBZpPGhS55GH7o8s674YL85Zg5fEINa05kjxM9iG+Kfo SEIwp8JxZ7vs+rhoqI+mg8MouUQf2Zul6cu4rspQPkk8FxQa/9RXPXX4OwkD3jy0+Z/+ SBrns4/qvGYQvjrl9OPoH7DBVBfA0arFpsKvUBwFWEn4L2+Z7qiKOGh0M4x1w7s8Fe9M Y7uQ== X-Google-Smtp-Source: APXvYqyXozv8RJ3un5KRfku6J9oOZngaXmf1Y0t3bUJMwzL1TSTEhjmbFCgyEeDgaxJqumt2ZQ+7aA== X-Received: by 2002:a62:5fc7:: with SMTP id t190mr50793424pfb.191.1556347403359; Fri, 26 Apr 2019 23:43:23 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:22 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu Subject: [PATCH v6 13/24] x86/jump-label: Remove support for custom poker Date: Fri, 26 Apr 2019 16:22:52 -0700 Message-Id: <20190426232303.28381-14-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit There are only two types of poking: early and breakpoint based. The use of a function pointer to perform poking complicates the code and is probably inefficient due to the use of indirect branches. Cc: Andy Lutomirski Cc: Kees Cook Cc: Dave Hansen Cc: Masami Hiramatsu Acked-by: Peter Zijlstra (Intel) Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/jump_label.c | 26 ++++++++++---------------- 1 file changed, 10 insertions(+), 16 deletions(-) diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c index e7d8c636b228..e631c358f7f4 100644 --- a/arch/x86/kernel/jump_label.c +++ b/arch/x86/kernel/jump_label.c @@ -37,7 +37,6 @@ static void bug_at(unsigned char *ip, int line) static void __ref __jump_label_transform(struct jump_entry *entry, enum jump_label_type type, - void *(*poker)(void *, const void *, size_t), int init) { union jump_code_union jmp; @@ -50,14 +49,6 @@ static void __ref __jump_label_transform(struct jump_entry *entry, jmp.offset = jump_entry_target(entry) - (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); - /* - * As long as only a single processor is running and the code is still - * not marked as RO, text_poke_early() can be used; Checking that - * system_state is SYSTEM_BOOTING guarantees it. - */ - if (system_state == SYSTEM_BOOTING) - poker = text_poke_early; - if (type == JUMP_LABEL_JMP) { if (init) { expect = default_nop; line = __LINE__; @@ -80,16 +71,19 @@ static void __ref __jump_label_transform(struct jump_entry *entry, bug_at((void *)jump_entry_code(entry), line); /* - * Make text_poke_bp() a default fallback poker. + * As long as only a single processor is running and the code is still + * not marked as RO, text_poke_early() can be used; Checking that + * system_state is SYSTEM_BOOTING guarantees it. It will be set to + * SYSTEM_SCHEDULING before other cores are awaken and before the + * code is write-protected. * * At the time the change is being done, just ignore whether we * are doing nop -> jump or jump -> nop transition, and assume * always nop being the 'currently valid' instruction - * */ - if (poker) { - (*poker)((void *)jump_entry_code(entry), code, - JUMP_LABEL_NOP_SIZE); + if (init || system_state == SYSTEM_BOOTING) { + text_poke_early((void *)jump_entry_code(entry), code, + JUMP_LABEL_NOP_SIZE); return; } @@ -101,7 +95,7 @@ void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) { mutex_lock(&text_mutex); - __jump_label_transform(entry, type, NULL, 0); + __jump_label_transform(entry, type, 0); mutex_unlock(&text_mutex); } @@ -131,5 +125,5 @@ __init_or_module void arch_jump_label_transform_static(struct jump_entry *entry, jlstate = JL_STATE_NO_UPDATE; } if (jlstate == JL_STATE_UPDATE) - __jump_label_transform(entry, type, text_poke_early, 1); + __jump_label_transform(entry, type, 1); } From patchwork Fri Apr 26 23:22:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10919965 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7E1B214B6 for ; Sat, 27 Apr 2019 06:43:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6DE2A28E6B for ; Sat, 27 Apr 2019 06:43:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 61BD428E7B; Sat, 27 Apr 2019 06:43:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CCDCD28E6B for ; Sat, 27 Apr 2019 06:43:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 26D4C6B026E; Sat, 27 Apr 2019 02:43:27 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1F6336B026F; Sat, 27 Apr 2019 02:43:27 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C97E6B0270; Sat, 27 Apr 2019 02:43:26 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id BD3BF6B026E for ; Sat, 27 Apr 2019 02:43:26 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id v9so3485392pgg.8 for ; Fri, 26 Apr 2019 23:43:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=RkWBxdSztj8rR+UGwRQ8u46DAnXrPhDoGJsRNKYizGU=; b=bmWTaZ90OrJcMeQdhb/axqnoYxiC25pLO+eERU26KvVnwaTmFrPb4CodLmE2KjY09M 1np2OFnnRUr4kPNUM4WLmh3R1Q9HzRE6GxYFy1T0mHgq9wmbzYDplTTMqGja9VGhyqRS Tjva+yZShtggvdeJAM6L4pFoZggUJfFANzeZbV6AgkQVbv4stxLSzY8W5ti40IKmlW2t R25uFrl8DX83f86KLwc4LWaqW1W97a1PQ0wmBEsEgX6fw6w2XyVsx/X/afjezrXjuolV hqToZQqGvQjh3gGuK841necUs7Idvi5uC8asba/nH+f7e0HAb8DYKEh7EfKq5sryhkDk UL+Q== X-Gm-Message-State: APjAAAVB09xDKnMkvYEJfgqBO6fHULapcdjmekplSBcSI0AAixTOUEpT LXz2URvyn1mpKku2xup6zxzOeKQsruoP3PrVimrpfCcGagi2BdbAPKDYh/W+BWoI97yWXexwk6H /9Bm83HVzjQ0epapbeDS86HopSM/4Y6Vk6b69oJ2HIs8F67ZzqBqdZShsaTPMXEHM9w== X-Received: by 2002:a17:902:1024:: with SMTP id b33mr49035262pla.46.1556347406438; Fri, 26 Apr 2019 23:43:26 -0700 (PDT) X-Received: by 2002:a17:902:1024:: with SMTP id b33mr49035210pla.46.1556347405369; Fri, 26 Apr 2019 23:43:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347405; cv=none; d=google.com; s=arc-20160816; b=vHJwbTjoFEepOle0F0Ssxz7nwWzuZFuT2yLHuVbv9TLcpIHs31cqgjbNn9PZHr19+E DJFwnQtWnTZkLypbzafONZtV4dFOXTruOW+w/ui3q7/l3RJ3hFsGd9Kaf1/HSG8bBNd4 UmK5V8BYwkBLE+vCOkdS9qtPvAw9bARnbtASJCT7QRRDd2FrlpeeF8mkQGnXQo2SJFjJ uONpKMa2jWLJBMEkaWDMYvWC2LZMUEdr+qHHS2GY5b7ckdXPmjj4jiu+YS1gwfL3kBo9 SqlqbhlHmp+L43IbdHm1bUzNIR/+FeDF6dIPMavsX3zSBu1kBe6qCm8kMthC+RUvlEIf jE6A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=RkWBxdSztj8rR+UGwRQ8u46DAnXrPhDoGJsRNKYizGU=; b=WkaTzNjnJ2vUoXU/BZ23J7b+ojRWmCJ5RsXbfR/hiVjqYLoBQ/B/i2AkQlccnDLGJS bfEYajikSCLm0aMcNtLDQh5/NXnVaO+2kWPQABLIoJTnCoKpzwue3w9CPVuJPNYIhOWB kQIjcpS0yvC6dZt8LROn6Fcc0eaFaY5UReKTE4Vjd+lIGJ3muf9p6rf5ZOjuQG5htAQ6 qIuvJN0GQGt1XBPccB8T10ufemCw4XWSEMVWTmhJwScBoYM1zFkSt3zWMaiEMC2/6AyX Mmb0Hjxo3H7s/dxMS6mp0InBjZJXYYSHXdiIZknGl6UB/2yjSi1sr7guPRPlPDqTK7pg 8pmg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="sZ/qLuZJ"; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id f24sor30579014pfn.22.2019.04.26.23.43.25 for (Google Transport Security); Fri, 26 Apr 2019 23:43:25 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="sZ/qLuZJ"; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RkWBxdSztj8rR+UGwRQ8u46DAnXrPhDoGJsRNKYizGU=; b=sZ/qLuZJUAyo8qbKkeTMfaDU5l+q5915X9t7GRxTKJHbpjNKTHl8pciE33HsZyKxMj FHDGynPK6j9EMARDmUngJuJvUrJzSdZ856tWm7M07XqPW/SWzM6ZNX5zl88lma+UJQTP glz3jYPedES7CBs45n1TgBVlpwj8P8UlUX7TYH4J9CMC1SwMSQqAcgs/6v2TrFDCMTCc 9+WmCz+Xk4CqkdBn0hJ5YGF1/DTIcg/k9CSmxM2My4drnNNp3ITpwY4FI+yvE/59hq2C LbPH5HAGU5SG85yKGAzKgHwuQvpQ2iezoip/gl/Zm0v6Wm0PDoU46D9nD3GQpDAmdtBS P0cQ== X-Google-Smtp-Source: APXvYqymwtMNRBHdVQWTPWr7iZe7bQ6p95NnoQCCbzB15xIf0gFL50JmokfCH/M9gUZzJVqeF2oaxg== X-Received: by 2002:aa7:81d0:: with SMTP id c16mr50378631pfn.132.1556347404841; Fri, 26 Apr 2019 23:43:24 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:24 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu Subject: [PATCH v6 14/24] x86/alternative: Remove the return value of text_poke_*() Date: Fri, 26 Apr 2019 16:22:53 -0700 Message-Id: <20190426232303.28381-15-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit The return value of text_poke_early() and text_poke_bp() is useless. Remove it. Cc: Andy Lutomirski Cc: Kees Cook Cc: Dave Hansen Cc: Masami Hiramatsu Acked-by: Peter Zijlstra (Intel) Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/text-patching.h | 4 ++-- arch/x86/kernel/alternative.c | 11 ++++------- 2 files changed, 6 insertions(+), 9 deletions(-) diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h index a75eed841eed..c90678fd391a 100644 --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -18,7 +18,7 @@ static inline void apply_paravirt(struct paravirt_patch_site *start, #define __parainstructions_end NULL #endif -extern void *text_poke_early(void *addr, const void *opcode, size_t len); +extern void text_poke_early(void *addr, const void *opcode, size_t len); /* * Clear and restore the kernel write-protection flag on the local CPU. @@ -37,7 +37,7 @@ extern void *text_poke_early(void *addr, const void *opcode, size_t len); extern void *text_poke(void *addr, const void *opcode, size_t len); extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len); extern int poke_int3_handler(struct pt_regs *regs); -extern void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler); +extern void text_poke_bp(void *addr, const void *opcode, size_t len, void *handler); extern int after_bootmem; extern __ro_after_init struct mm_struct *poking_mm; extern __ro_after_init unsigned long poking_addr; diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 3d2b6b6fb20c..18f959975ea0 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -265,7 +265,7 @@ static void __init_or_module add_nops(void *insns, unsigned int len) extern struct alt_instr __alt_instructions[], __alt_instructions_end[]; extern s32 __smp_locks[], __smp_locks_end[]; -void *text_poke_early(void *addr, const void *opcode, size_t len); +void text_poke_early(void *addr, const void *opcode, size_t len); /* * Are we looking at a near JMP with a 1 or 4-byte displacement. @@ -667,8 +667,8 @@ void __init alternative_instructions(void) * instructions. And on the local CPU you need to be protected again NMI or MCE * handlers seeing an inconsistent instruction while you patch. */ -void *__init_or_module text_poke_early(void *addr, const void *opcode, - size_t len) +void __init_or_module text_poke_early(void *addr, const void *opcode, + size_t len) { unsigned long flags; @@ -691,7 +691,6 @@ void *__init_or_module text_poke_early(void *addr, const void *opcode, * that causes hangs on some VIA CPUs. */ } - return addr; } __ro_after_init struct mm_struct *poking_mm; @@ -893,7 +892,7 @@ NOKPROBE_SYMBOL(poke_int3_handler); * replacing opcode * - sync cores */ -void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler) +void text_poke_bp(void *addr, const void *opcode, size_t len, void *handler) { unsigned char int3 = 0xcc; @@ -935,7 +934,5 @@ void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler) * the writing of the new instruction. */ bp_patching_in_progress = false; - - return addr; } From patchwork Fri Apr 26 23:22:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10919971 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7F9D5912 for ; Sat, 27 Apr 2019 06:43:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6E88E28E6B for ; Sat, 27 Apr 2019 06:43:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 629A628E7B; Sat, 27 Apr 2019 06:43:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C1E2B28E6B for ; Sat, 27 Apr 2019 06:43:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 644CC6B0270; Sat, 27 Apr 2019 02:43:30 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4E0D66B0271; Sat, 27 Apr 2019 02:43:30 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 30D0C6B0272; Sat, 27 Apr 2019 02:43:30 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id DFF1B6B0270 for ; Sat, 27 Apr 2019 02:43:29 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id z12so3498634pgs.4 for ; Fri, 26 Apr 2019 23:43:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=QhBGq/JNavwlurWIVFo7GWw2xI+63VIDOiZrYQv9auY=; b=pienI7LaBzMaV5GLUlXgqOtQiWkv3Zo4UpuTta6yCLch5N0tKXiR6DOmuuNDXfM9eN 7f1FPgAsAjDIMN6VoMI6gQuEhTMdyRwZG6tY5ZZZ9OZjH0rRK0tGnCYOFjPozhloEZvz TQfMG+E4jzbq6JvwzzAiw7uLg6n0RUfv3euBzUSwhP60v62jV66ZpqDrGtr24tUNYU0L H54nLpR+mcW+459YrmOt//btm50UrfjEg9oFsfXwrDlnvVZbthmXouiOZr+OEnWiBa1P SgusRytoiR7oqjwn6/dAl/etWiknLlYPyqcUPdwj8QORVKNCg55zprOK/YLlGADq4AZl CDzQ== X-Gm-Message-State: APjAAAVtEy/fFbPM6J5CQLZkt/tnn+rgGFHH74ZRvBEkrcapoD50uIAp qlPGSkAD8fd6UjYP9KMnRg7SlBRWi4Zad3JcCzqw+H2YL+teDdsfiyV2Q5B/qEVzbmsDa3GOckw xmZTUJUz/5MKrckW15oOI5Pk/x7ATQSZMoknXfHUufQMRDgOLhde14TGUVlfj9viBWA== X-Received: by 2002:a62:62c3:: with SMTP id w186mr50088083pfb.73.1556347409542; Fri, 26 Apr 2019 23:43:29 -0700 (PDT) X-Received: by 2002:a62:62c3:: with SMTP id w186mr50088010pfb.73.1556347408302; Fri, 26 Apr 2019 23:43:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347408; cv=none; d=google.com; s=arc-20160816; b=CzP3FY1YgF3cdnK+e1oN74kRyYMHgMGZi+CIcvMS5g/GdFkFmC5DkTZGEw6qZwrrbA Z7YQYmEN9wMRILnW73u3q/HoNFmJ7QVonQHr+NDMKNl0qxqro5UwTBwzVdvg9l4JFGU9 mWOM8QJztI43Ubf1/LEGvSJfK1QUrO3CnkloRgU9JXHS4vEbtMK09RmAaUX/+VHl1s7e VqQ2VvvCkuNlqAqF9MQr7GjfCYXKfKXRKj9zprrJ8BYXLUywsc/70tNSzPhitR9y5hLl dMnVfvwVz3u/rXxI5GCUEJIEIRX7kGBtEoVLbe2rUE1OS8eazwCRquHzwWNHOESxbPHU 7LWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=QhBGq/JNavwlurWIVFo7GWw2xI+63VIDOiZrYQv9auY=; b=qkaITdu6k6H2O/d57sGhSRtUqqximCbAomhuAfEHXjzOjD92Ev2RTK3yziCb2OedN9 jdm2WLcTWIv2I7Z/u+QeC99YIOLX4E8CCJqY4NBMqglzaQ7iCcB82L8uh80goHXLPJSE SiE19sHbP13DmjfDdbAoiaAuZot/dJPTvhgl+s4o7WZmrEXI7+KwrR3/zd6IE61AG7xs QN5+9MV0uAYrGzNQXhPUOtikmNrMTot3qBoctxSxblKIvBAlV+Fv2j1mDbggIgCDJgK7 OVkYhK72vrgU5pcJQaK8r57B7e2e5r9N4PVolarR+EoLgdOqb5Zh2RRU3KXEgnXlIWBj p2ww== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=kQVzfXvt; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id n11sor2804967pgh.79.2019.04.26.23.43.28 for (Google Transport Security); Fri, 26 Apr 2019 23:43:28 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=kQVzfXvt; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=QhBGq/JNavwlurWIVFo7GWw2xI+63VIDOiZrYQv9auY=; b=kQVzfXvt7w7FyD24D6OW7cca/iq64cXIdYieWuyKwNk+RlpS1Vrvzmoacb3ff7XKNz uSrIDJVnl0StI02S4mDmT9/orsTdFCLqxairNBVJHpFQOlVF+P545I79O3XIMap7MmNx B+eyRkbN1Pn1P8cvmN/nIG6h/QsEeRGGk823Xen9FH/1q+TdS2le7wdgc7mk51khO50a FM2ov6bJpTRs8rPyT3sOey+TdqaV/ShHovsU9ChNEC3Nj8Fx8yLDbxMu7UkHtsd0HXsr xXsGi8x4xGLugp1QRQm3EmYH1Hg5T8pmM4cIqX1ra4UyE2ICcBAtQsttpAyF860DV5oa hYYA== X-Google-Smtp-Source: APXvYqzGKQmON9BzHH89UmuJkvJzJbwysA6AfztkU3vJ+XbGjoeBhsjpEhkWgnx/yazqn7g52/ieFg== X-Received: by 2002:a63:5466:: with SMTP id e38mr48658719pgm.340.1556347407789; Fri, 26 Apr 2019 23:43:27 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:27 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , "Rafael J. Wysocki" , Pavel Machek Subject: [PATCH v6 16/24] mm: Make hibernate handle unmapped pages Date: Fri, 26 Apr 2019 16:22:55 -0700 Message-Id: <20190426232303.28381-17-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Rick Edgecombe Make hibernate handle unmapped pages on the direct map when CONFIG_ARCH_HAS_SET_ALIAS is set. These functions allow for setting pages to invalid configurations, so now hibernate should check if the pages have valid mappings and handle if they are unmapped when doing a hibernate save operation. Previously this checking was already done when CONFIG_DEBUG_PAGEALLOC was configured. It does not appear to have a big hibernating performance impact. The speed of the saving operation before this change was measured as 819.02 MB/s, and after was measured at 813.32 MB/s. Before: [ 4.670938] PM: Wrote 171996 kbytes in 0.21 seconds (819.02 MB/s) After: [ 4.504714] PM: Wrote 178932 kbytes in 0.22 seconds (813.32 MB/s) Cc: Dave Hansen Cc: Andy Lutomirski Cc: Peter Zijlstra Cc: "Rafael J. Wysocki" Cc: Pavel Machek Cc: Borislav Petkov Acked-by: Pavel Machek Signed-off-by: Rick Edgecombe --- arch/x86/mm/pageattr.c | 4 ---- include/linux/mm.h | 18 ++++++------------ kernel/power/snapshot.c | 5 +++-- mm/page_alloc.c | 7 +++++-- 4 files changed, 14 insertions(+), 20 deletions(-) diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c index 3574550192c6..daf4d645e537 100644 --- a/arch/x86/mm/pageattr.c +++ b/arch/x86/mm/pageattr.c @@ -2257,7 +2257,6 @@ int set_direct_map_default_noflush(struct page *page) return __set_pages_p(page, 1); } -#ifdef CONFIG_DEBUG_PAGEALLOC void __kernel_map_pages(struct page *page, int numpages, int enable) { if (PageHighMem(page)) @@ -2302,11 +2301,8 @@ bool kernel_page_present(struct page *page) pte = lookup_address((unsigned long)page_address(page), &level); return (pte_val(*pte) & _PAGE_PRESENT); } - #endif /* CONFIG_HIBERNATION */ -#endif /* CONFIG_DEBUG_PAGEALLOC */ - int __init kernel_map_pages_in_pgd(pgd_t *pgd, u64 pfn, unsigned long address, unsigned numpages, unsigned long page_flags) { diff --git a/include/linux/mm.h b/include/linux/mm.h index 6b10c21630f5..083d7b4863ed 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2610,37 +2610,31 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { } #endif -#ifdef CONFIG_DEBUG_PAGEALLOC extern bool _debug_pagealloc_enabled; -extern void __kernel_map_pages(struct page *page, int numpages, int enable); static inline bool debug_pagealloc_enabled(void) { - return _debug_pagealloc_enabled; + return IS_ENABLED(CONFIG_DEBUG_PAGEALLOC) && _debug_pagealloc_enabled; } +#if defined(CONFIG_DEBUG_PAGEALLOC) || defined(CONFIG_ARCH_HAS_SET_DIRECT_MAP) +extern void __kernel_map_pages(struct page *page, int numpages, int enable); + static inline void kernel_map_pages(struct page *page, int numpages, int enable) { - if (!debug_pagealloc_enabled()) - return; - __kernel_map_pages(page, numpages, enable); } #ifdef CONFIG_HIBERNATION extern bool kernel_page_present(struct page *page); #endif /* CONFIG_HIBERNATION */ -#else /* CONFIG_DEBUG_PAGEALLOC */ +#else /* CONFIG_DEBUG_PAGEALLOC || CONFIG_ARCH_HAS_SET_DIRECT_MAP */ static inline void kernel_map_pages(struct page *page, int numpages, int enable) {} #ifdef CONFIG_HIBERNATION static inline bool kernel_page_present(struct page *page) { return true; } #endif /* CONFIG_HIBERNATION */ -static inline bool debug_pagealloc_enabled(void) -{ - return false; -} -#endif /* CONFIG_DEBUG_PAGEALLOC */ +#endif /* CONFIG_DEBUG_PAGEALLOC || CONFIG_ARCH_HAS_SET_DIRECT_MAP */ #ifdef __HAVE_ARCH_GATE_AREA extern struct vm_area_struct *get_gate_vma(struct mm_struct *mm); diff --git a/kernel/power/snapshot.c b/kernel/power/snapshot.c index f08a1e4ee1d4..bc9558ab1e5b 100644 --- a/kernel/power/snapshot.c +++ b/kernel/power/snapshot.c @@ -1342,8 +1342,9 @@ static inline void do_copy_page(long *dst, long *src) * safe_copy_page - Copy a page in a safe way. * * Check if the page we are going to copy is marked as present in the kernel - * page tables (this always is the case if CONFIG_DEBUG_PAGEALLOC is not set - * and in that case kernel_page_present() always returns 'true'). + * page tables. This always is the case if CONFIG_DEBUG_PAGEALLOC or + * CONFIG_ARCH_HAS_SET_DIRECT_MAP is not set. In that case kernel_page_present() + * always returns 'true'. */ static void safe_copy_page(void *dst, struct page *s_page) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d96ca5bc555b..34a70681a4af 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1131,7 +1131,9 @@ static __always_inline bool free_pages_prepare(struct page *page, } arch_free_page(page, order); kernel_poison_pages(page, 1 << order, 0); - kernel_map_pages(page, 1 << order, 0); + if (debug_pagealloc_enabled()) + kernel_map_pages(page, 1 << order, 0); + kasan_free_nondeferred_pages(page, order); return true; @@ -2001,7 +2003,8 @@ inline void post_alloc_hook(struct page *page, unsigned int order, set_page_refcounted(page); arch_alloc_page(page, order); - kernel_map_pages(page, 1 << order, 1); + if (debug_pagealloc_enabled()) + kernel_map_pages(page, 1 << order, 1); kasan_alloc_pages(page, order); kernel_poison_pages(page, 1 << order, 1); set_page_owner(page, order, gfp_flags); From patchwork Fri Apr 26 23:22:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10919977 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8D6C1912 for ; Sat, 27 Apr 2019 06:44:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7C08B28E6B for ; Sat, 27 Apr 2019 06:44:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 701F328E7B; Sat, 27 Apr 2019 06:44:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8DA1D28E6B for ; Sat, 27 Apr 2019 06:44:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F3EB46B0271; Sat, 27 Apr 2019 02:43:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EED2A6B0273; Sat, 27 Apr 2019 02:43:31 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB5AC6B0274; Sat, 27 Apr 2019 02:43:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id 9527B6B0271 for ; Sat, 27 Apr 2019 02:43:31 -0400 (EDT) Received: by mail-pl1-f198.google.com with SMTP id v5so3257884plo.4 for ; Fri, 26 Apr 2019 23:43:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=71dbHy9m/CIhCtlniLjnIfT11ydtbrFJWfPgh/aDBts=; b=BlqKcFT59zfaHX9GuIRcXdnk43dVDeSFeO5GTOkSrnyM9yQwjWSM48Y4c1nzbzJsRa FuL09xzV4W5ezvw6nqtEN6abSQk3T+usYePNdbYQqIjxCkpxDgzhFQxLzf6wfawmQIlV 7Rh4ivMc2/heWMbrJ5UrDTeOO/W0RlRe61v1wbO7Sxj36GPKNMOCsmThqpV5HFOiPacD kNkXXPFkboVh+qHqZm+gD+i5rjirqzJM0FVRX53Jad5gkBq/q66QNa2MHGn57SBakcxQ 8TGY2R/AwSWB+9TCcLArguWVsu/FkWRdGJNNIU2fdImGFAM+mGBwZOqDShHsh9EKT+Hh X4wQ== X-Gm-Message-State: APjAAAVvOmlPDBVMIFGb+ormVtOcfIINdzhkwLE5v+u70zTJpKSZ1vt5 tQELOcWbaoaVsBhsOSI8w1Ud8t6hYQW6DdW1lTURSTI3JM3J533ZKduHu9gb672HDVvsYGnvjcM FAoRtBr2BsrQ9jYR+tx5SLlCJCpsNY0vuVktg7DhSNfN82Er5mJ4B1gRb3peogPo0HA== X-Received: by 2002:a17:902:b181:: with SMTP id s1mr27791594plr.9.1556347411237; Fri, 26 Apr 2019 23:43:31 -0700 (PDT) X-Received: by 2002:a17:902:b181:: with SMTP id s1mr27791508plr.9.1556347409586; Fri, 26 Apr 2019 23:43:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347409; cv=none; d=google.com; s=arc-20160816; b=RHtlZ8OLT2dPiym622/0aHhO+AK4D//Bw5bBPTyXQSStUro5rJk8rmT4VuhnTJ4DLP MIdDo+T3q4gbFmqx0OU3S6VrP8LRJ/en8I0rPI+sUacE6aJ1MIjtyMiPaMRgc1IkK3Gn 57F5Aa2VXwTSztYGwTgz+UQwErVa05URHRgwWIS571TntvKhPLDX8qTpyg9/wNNc5RK8 3Q3A0Z4ERePhEv777qbvu5veqn01owr4/plbA8pVulF/BQQW5TdrHSjtNlJptoYZLdqE 0UpfZQH7GqCmrSSnkpSD/atZaW06ni2b7zH/PyjnLKsljRCy5dLYlPNHibiRydKECPJc Gh7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=71dbHy9m/CIhCtlniLjnIfT11ydtbrFJWfPgh/aDBts=; b=k4BLapIC+GqP8NKUYY90CLUg1gzH2vWFy+onjq71ZCqi3HBVEx04JBPrPRaYpQGz/f PYQ9OArydjvd3573BCuqDzoAzA9ce6DWlcJMwiHMXc9KDnCNfltGinEtBlJcYs7TA6A4 lfcEYu/R5lVKE3Vjp0c2ab341YH+kgka6uSYb5zbTbRvumwcjb5M9LFzBNMWOCqY5bcj VVYzm0o/6c8cLKLLR3n838tCrB0texd0qN4gK+mT21DdeR9OV+N8fmEpY4Lw14Rg2WT+ cH77L7l6TUhVGaYuVgqft6tBPNXa7Pyli7QVahxiZ/DDjBZVfFWHqCxB+aE/64Yoq8vU aluw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=qvNLDfV5; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id a9sor30460757pfj.3.2019.04.26.23.43.29 for (Google Transport Security); Fri, 26 Apr 2019 23:43:29 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=qvNLDfV5; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=71dbHy9m/CIhCtlniLjnIfT11ydtbrFJWfPgh/aDBts=; b=qvNLDfV5t6OCXjpXqE2h8qCP9bZ/sJ406eIEMibouYzKfppgqnMlAuCIygJ2OFcd9+ MMTkVSXzO0aOctNe/CiP8Zw9kIpjzutOKkzuNHcJG3fCAdytpDd18QMDAb6Knim/ohRs tBDO6V0x85WHUL2w71XgQxWYSTrcZtRKjF0HFFS4Dzn7r6bnhd8gs0bFJ2ayDhn0qXIt XGqaYfEBSbyjlozxlv7tlJsqu8eyvkVV9qOY42YDUV4gMlVY70x2mouNblsQZ7uqnSpE 5UOlPM0wci38r9uZAhlJb4I1NObgCcrLUdbvFFf4CBRT8+QPP08ffFZ0CF9w8nTK5/X2 J8hg== X-Google-Smtp-Source: APXvYqyURRPLLrg0ynk67t/jHVJBTNh21u6U/GXsGcvBn1nAU+eeDbHBRYN9HLkqe6vwNFju/6m6NQ== X-Received: by 2002:a65:500d:: with SMTP id f13mr8345688pgo.250.1556347409072; Fri, 26 Apr 2019 23:43:29 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:28 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe Subject: [PATCH v6 17/24] vmalloc: Add flag for free of special permsissions Date: Fri, 26 Apr 2019 16:22:56 -0700 Message-Id: <20190426232303.28381-18-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Rick Edgecombe Add a new flag VM_FLUSH_RESET_PERMS, for enabling vfree operations to immediately clear executable TLB entries before freeing pages, and handle resetting permissions on the directmap. This flag is useful for any kind of memory with elevated permissions, or where there can be related permissions changes on the directmap. Today this is RO+X and RO memory. Although this enables directly vfreeing non-writeable memory now, non-writable memory cannot be freed in an interrupt because the allocation itself is used as a node on deferred free list. So when RO memory needs to be freed in an interrupt the code doing the vfree needs to have its own work queue, as was the case before the deferred vfree list was added to vmalloc. For architectures with set_direct_map_ implementations this whole operation can be done with one TLB flush when centralized like this. For others with directmap permissions, currently only arm64, a backup method using set_memory functions is used to reset the directmap. When arm64 adds set_direct_map_ functions, this backup can be removed. When the TLB is flushed to both remove TLB entries for the vmalloc range mapping and the direct map permissions, the lazy purge operation could be done to try to save a TLB flush later. However today vm_unmap_aliases could flush a TLB range that does not include the directmap. So a helper is added with extra parameters that can allow both the vmalloc address and the direct mapping to be flushed during this operation. The behavior of the normal vm_unmap_aliases function is unchanged. Cc: Borislav Petkov Suggested-by: Dave Hansen Suggested-by: Andy Lutomirski Suggested-by: Will Deacon Signed-off-by: Rick Edgecombe --- include/linux/vmalloc.h | 15 ++++++ mm/vmalloc.c | 113 +++++++++++++++++++++++++++++++++------- 2 files changed, 109 insertions(+), 19 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 398e9c95cd61..c6eebb839552 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -21,6 +21,11 @@ struct notifier_block; /* in notifier.h */ #define VM_UNINITIALIZED 0x00000020 /* vm_struct is not fully initialized */ #define VM_NO_GUARD 0x00000040 /* don't add guard page */ #define VM_KASAN 0x00000080 /* has allocated kasan shadow memory */ +/* + * Memory with VM_FLUSH_RESET_PERMS cannot be freed in an interrupt or with + * vfree_atomic(). + */ +#define VM_FLUSH_RESET_PERMS 0x00000100 /* Reset direct map and flush TLB on unmap */ /* bits [20..32] reserved for arch specific ioremap internals */ /* @@ -142,6 +147,13 @@ extern int map_kernel_range_noflush(unsigned long start, unsigned long size, pgprot_t prot, struct page **pages); extern void unmap_kernel_range_noflush(unsigned long addr, unsigned long size); extern void unmap_kernel_range(unsigned long addr, unsigned long size); +static inline void set_vm_flush_reset_perms(void *addr) +{ + struct vm_struct *vm = find_vm_area(addr); + + if (vm) + vm->flags |= VM_FLUSH_RESET_PERMS; +} #else static inline int map_kernel_range_noflush(unsigned long start, unsigned long size, @@ -157,6 +169,9 @@ static inline void unmap_kernel_range(unsigned long addr, unsigned long size) { } +static inline void set_vm_flush_reset_perms(void *addr) +{ +} #endif /* Allocate/destroy a 'vmalloc' VM area. */ diff --git a/mm/vmalloc.c b/mm/vmalloc.c index e86ba6e74b50..e5e9e1fcac01 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -1059,24 +1060,9 @@ static void vb_free(const void *addr, unsigned long size) spin_unlock(&vb->lock); } -/** - * vm_unmap_aliases - unmap outstanding lazy aliases in the vmap layer - * - * The vmap/vmalloc layer lazily flushes kernel virtual mappings primarily - * to amortize TLB flushing overheads. What this means is that any page you - * have now, may, in a former life, have been mapped into kernel virtual - * address by the vmap layer and so there might be some CPUs with TLB entries - * still referencing that page (additional to the regular 1:1 kernel mapping). - * - * vm_unmap_aliases flushes all such lazy mappings. After it returns, we can - * be sure that none of the pages we have control over will have any aliases - * from the vmap layer. - */ -void vm_unmap_aliases(void) +static void _vm_unmap_aliases(unsigned long start, unsigned long end, int flush) { - unsigned long start = ULONG_MAX, end = 0; int cpu; - int flush = 0; if (unlikely(!vmap_initialized)) return; @@ -1113,6 +1099,27 @@ void vm_unmap_aliases(void) flush_tlb_kernel_range(start, end); mutex_unlock(&vmap_purge_lock); } + +/** + * vm_unmap_aliases - unmap outstanding lazy aliases in the vmap layer + * + * The vmap/vmalloc layer lazily flushes kernel virtual mappings primarily + * to amortize TLB flushing overheads. What this means is that any page you + * have now, may, in a former life, have been mapped into kernel virtual + * address by the vmap layer and so there might be some CPUs with TLB entries + * still referencing that page (additional to the regular 1:1 kernel mapping). + * + * vm_unmap_aliases flushes all such lazy mappings. After it returns, we can + * be sure that none of the pages we have control over will have any aliases + * from the vmap layer. + */ +void vm_unmap_aliases(void) +{ + unsigned long start = ULONG_MAX, end = 0; + int flush = 0; + + _vm_unmap_aliases(start, end, flush); +} EXPORT_SYMBOL_GPL(vm_unmap_aliases); /** @@ -1505,6 +1512,72 @@ struct vm_struct *remove_vm_area(const void *addr) return NULL; } +static inline void set_area_direct_map(const struct vm_struct *area, + int (*set_direct_map)(struct page *page)) +{ + int i; + + for (i = 0; i < area->nr_pages; i++) + if (page_address(area->pages[i])) + set_direct_map(area->pages[i]); +} + +/* Handle removing and resetting vm mappings related to the vm_struct. */ +static void vm_remove_mappings(struct vm_struct *area, int deallocate_pages) +{ + unsigned long addr = (unsigned long)area->addr; + unsigned long start = ULONG_MAX, end = 0; + int flush_reset = area->flags & VM_FLUSH_RESET_PERMS; + int i; + + /* + * The below block can be removed when all architectures that have + * direct map permissions also have set_direct_map_() implementations. + * This is concerned with resetting the direct map any an vm alias with + * execute permissions, without leaving a RW+X window. + */ + if (flush_reset && !IS_ENABLED(CONFIG_ARCH_HAS_SET_DIRECT_MAP)) { + set_memory_nx(addr, area->nr_pages); + set_memory_rw(addr, area->nr_pages); + } + + remove_vm_area(area->addr); + + /* If this is not VM_FLUSH_RESET_PERMS memory, no need for the below. */ + if (!flush_reset) + return; + + /* + * If not deallocating pages, just do the flush of the VM area and + * return. + */ + if (!deallocate_pages) { + vm_unmap_aliases(); + return; + } + + /* + * If execution gets here, flush the vm mapping and reset the direct + * map. Find the start and end range of the direct mappings to make sure + * the vm_unmap_aliases() flush includes the direct map. + */ + for (i = 0; i < area->nr_pages; i++) { + if (page_address(area->pages[i])) { + start = min(addr, start); + end = max(addr, end); + } + } + + /* + * Set direct map to something invalid so that it won't be cached if + * there are any accesses after the TLB flush, then flush the TLB and + * reset the direct map permissions to the default. + */ + set_area_direct_map(area, set_direct_map_invalid_noflush); + _vm_unmap_aliases(start, end, 1); + set_area_direct_map(area, set_direct_map_default_noflush); +} + static void __vunmap(const void *addr, int deallocate_pages) { struct vm_struct *area; @@ -1526,7 +1599,8 @@ static void __vunmap(const void *addr, int deallocate_pages) debug_check_no_locks_freed(area->addr, get_vm_area_size(area)); debug_check_no_obj_freed(area->addr, get_vm_area_size(area)); - remove_vm_area(addr); + vm_remove_mappings(area, deallocate_pages); + if (deallocate_pages) { int i; @@ -1961,8 +2035,9 @@ EXPORT_SYMBOL(vzalloc_node); */ void *vmalloc_exec(unsigned long size) { - return __vmalloc_node(size, 1, GFP_KERNEL, PAGE_KERNEL_EXEC, - NUMA_NO_NODE, __builtin_return_address(0)); + return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, + GFP_KERNEL, PAGE_KERNEL_EXEC, VM_FLUSH_RESET_PERMS, + NUMA_NO_NODE, __builtin_return_address(0)); } #if defined(CONFIG_64BIT) && defined(CONFIG_ZONE_DMA32) From patchwork Fri Apr 26 23:22:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10919981 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2B0F114DB for ; Sat, 27 Apr 2019 06:44:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1865E28E78 for ; Sat, 27 Apr 2019 06:44:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0BCD628E7B; Sat, 27 Apr 2019 06:44:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 47BF828E78 for ; Sat, 27 Apr 2019 06:44:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EA1F56B0273; Sat, 27 Apr 2019 02:43:32 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E4FF26B0274; Sat, 27 Apr 2019 02:43:32 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF3D16B0275; Sat, 27 Apr 2019 02:43:32 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id 94E9F6B0273 for ; Sat, 27 Apr 2019 02:43:32 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id o1so3479123pgv.15 for ; Fri, 26 Apr 2019 23:43:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=FbRVtA2o2v3UWbAlXvOdnbyNDEqG2rBMqLF6HOTpV1U=; b=gUoRhI3QuTeuaGBO/Q4abzZtEd32oRMYMnQWIZqqKPxOLMvSVDiu/AWUR0F8XfIVv/ em2VpotUOBZbzn9Aeb/cjHCjc1kESO6h0gZ61eRF1un8syPf0vkTJ+R7uQTnCYaIckDW EOF8578BZYbcJyRpXXHoRKMrgGBWC3pMu9Xthuo3xnfva3Tczv9qbvgVUGvAwovhxuH3 0Jk7kEQvtmlEUlefMCMYkL2Wr3+wYQfNA4edwxESxVEIVaqNR7nVD216DxrO0CTPKPEn i/zWDipB29b8qNuSpU9LFOxpRAegOLAR6rmg/BsMkmwFjDiMIINE1bD36WloCw7sVHsf 7pFw== X-Gm-Message-State: APjAAAVkobb2TKJurOAb0wwoxr7GS9mMh6acT26udoqPEM2NNTc654Pa eeD/c8bB7ytYaOsSYBT7Bc0kHbpXwtUMCXjobjInbeH6nlDzg8AIjl0u1YXbdAS2YJrazEh6ANq jNWSw14aNDjkEPUMQ99LZVIWYB5vZACDGQxfmIneOnxmPkL7pkjK5CHVJES03JjH4gQ== X-Received: by 2002:a65:5304:: with SMTP id m4mr23472570pgq.281.1556347412249; Fri, 26 Apr 2019 23:43:32 -0700 (PDT) X-Received: by 2002:a65:5304:: with SMTP id m4mr23472493pgq.281.1556347410951; Fri, 26 Apr 2019 23:43:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347410; cv=none; d=google.com; s=arc-20160816; b=kZ+MIXpJdsR3eH73BQ2JbxKb5ylUgMt8kbfsfTXPMBsXYeG27wSkc1uB8Ik1wNf6wG JOb2IwghBwZB/X+3tQpjmcsEQO5kWBB1GMrqlBOXxRdSIHbli9fITZAMjXjfaipvpU2f GJ8bH4hKYQCSHscCBr8BnQ148tSOykStgYoszKxw9uukLE13VsLaG4gP8m9jzjY6NOuX I7fkaQO9o4czqh4Vdm/yhL9twraJ2j5l02xfO+qo6GkUiF3SNYDWYt7hmkWIeb1ftZnq wDdswsdzVCU9onOn41vDbf9TsOMeQGR+tatNr8EhLnpdtGoVJZJtO/tGfK6jwvW2nPot k+yA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=FbRVtA2o2v3UWbAlXvOdnbyNDEqG2rBMqLF6HOTpV1U=; b=lVgIf1v6WwLZlTrxj1oqG7RcmmSUgYnLjcK15P9vOCY14bpH2Jp663O3IcymYGi+Ea 3nPITejfQX1dYuoMm2llr2heddyRtGoSKU/3P3M3RYyC3bWINdYYaRzbb7YJ6gVsiTUg vUnkciUUmBRs2zbTaie54O1Tn5PB4Nf5KLyeU8rxKnKB9i8ltldtk7yvOgOT+2lBRGez gNboILEERw+jDv3UsNsY3rvQ8HbFTrxuJw///Z6uQiVoYdeEyv1wbbfcIScsZUTNsgQq /lz8EkXaQfsoomg3TLZmApobj/ubyJCjw0Icf6VJnM9ONMTOvXQKRNrex92tzwKSl8bf A12Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=JIPBw61l; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id a12sor2766182pgw.35.2019.04.26.23.43.30 for (Google Transport Security); Fri, 26 Apr 2019 23:43:30 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=JIPBw61l; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=FbRVtA2o2v3UWbAlXvOdnbyNDEqG2rBMqLF6HOTpV1U=; b=JIPBw61lhFhbsSeNNRCeinW8YzpjhHLS0kAFQAbk6hxmpp9hUg6J0GnYaKJNU0FHKf 3bqKVJ6C8Tvx6TJOkuOrWJna7fqoUuHizJ+qKiMIA5CS7CEShzqQKK/AW7XLnX200hVn h6Uo9w63gy2P9k2VG6r72+XF0lb0lBkXJJXjhuCTEIV+s2i43bDAgsF9azMi0JLNH7ed aF+f6J70rXkPNjBlMKsdUnSwpqHyYUKOSX8pNfq8+bcRkjXvPZVggFfZsbJYWh1FzF5C WOAREpo4S3btT9Orow8C8eatHDMaItBqA+/J+o0vl9gjb3c0Jf12Oahj1tktHyDbSwU0 U95A== X-Google-Smtp-Source: APXvYqym7OynpsRkDmNxinoK0RlpSBEGSXR11tkqmQ4UsD/W+17lELi4duQgGdmbTXOTa0GEeSPAlA== X-Received: by 2002:a63:df43:: with SMTP id h3mr48430342pgj.294.1556347410385; Fri, 26 Apr 2019 23:43:30 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:29 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Jessica Yu , Steven Rostedt Subject: [PATCH v6 18/24] modules: Use vmalloc special flag Date: Fri, 26 Apr 2019 16:22:57 -0700 Message-Id: <20190426232303.28381-19-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Rick Edgecombe Use new flag for handling freeing of special permissioned memory in vmalloc and remove places where memory was set RW before freeing which is no longer needed. Since freeing of VM_FLUSH_RESET_PERMS memory is not supported in an interrupt by vmalloc, the freeing of init sections is moved to a work queue. Instead of call_rcu it now uses synchronize_rcu() in the work queue. Lastly, there is now a WARN_ON in module_memfree since it should not be called in an interrupt with special memory as is required for VM_FLUSH_RESET_PERMS. Cc: Jessica Yu Cc: Steven Rostedt Signed-off-by: Rick Edgecombe --- kernel/module.c | 77 +++++++++++++++++++++++++------------------------ 1 file changed, 39 insertions(+), 38 deletions(-) diff --git a/kernel/module.c b/kernel/module.c index 2b2845ae983e..a9020bdd4cf6 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -98,6 +98,10 @@ DEFINE_MUTEX(module_mutex); EXPORT_SYMBOL_GPL(module_mutex); static LIST_HEAD(modules); +/* Work queue for freeing init sections in success case */ +static struct work_struct init_free_wq; +static struct llist_head init_free_list; + #ifdef CONFIG_MODULES_TREE_LOOKUP /* @@ -1949,6 +1953,8 @@ void module_enable_ro(const struct module *mod, bool after_init) if (!rodata_enabled) return; + set_vm_flush_reset_perms(mod->core_layout.base); + set_vm_flush_reset_perms(mod->init_layout.base); frob_text(&mod->core_layout, set_memory_ro); frob_text(&mod->core_layout, set_memory_x); @@ -1972,15 +1978,6 @@ static void module_enable_nx(const struct module *mod) frob_writable_data(&mod->init_layout, set_memory_nx); } -static void module_disable_nx(const struct module *mod) -{ - frob_rodata(&mod->core_layout, set_memory_x); - frob_ro_after_init(&mod->core_layout, set_memory_x); - frob_writable_data(&mod->core_layout, set_memory_x); - frob_rodata(&mod->init_layout, set_memory_x); - frob_writable_data(&mod->init_layout, set_memory_x); -} - /* Iterate through all modules and set each module's text as RW */ void set_all_modules_text_rw(void) { @@ -2024,23 +2021,8 @@ void set_all_modules_text_ro(void) } mutex_unlock(&module_mutex); } - -static void disable_ro_nx(const struct module_layout *layout) -{ - if (rodata_enabled) { - frob_text(layout, set_memory_rw); - frob_rodata(layout, set_memory_rw); - frob_ro_after_init(layout, set_memory_rw); - } - frob_rodata(layout, set_memory_x); - frob_ro_after_init(layout, set_memory_x); - frob_writable_data(layout, set_memory_x); -} - #else -static void disable_ro_nx(const struct module_layout *layout) { } static void module_enable_nx(const struct module *mod) { } -static void module_disable_nx(const struct module *mod) { } #endif #ifdef CONFIG_LIVEPATCH @@ -2120,6 +2102,11 @@ static void free_module_elf(struct module *mod) void __weak module_memfree(void *module_region) { + /* + * This memory may be RO, and freeing RO memory in an interrupt is not + * supported by vmalloc. + */ + WARN_ON(in_interrupt()); vfree(module_region); } @@ -2171,7 +2158,6 @@ static void free_module(struct module *mod) mutex_unlock(&module_mutex); /* This may be empty, but that's OK */ - disable_ro_nx(&mod->init_layout); module_arch_freeing_init(mod); module_memfree(mod->init_layout.base); kfree(mod->args); @@ -2181,7 +2167,6 @@ static void free_module(struct module *mod) lockdep_free_key_range(mod->core_layout.base, mod->core_layout.size); /* Finally, free the core (containing the module structure) */ - disable_ro_nx(&mod->core_layout); module_memfree(mod->core_layout.base); } @@ -3420,17 +3405,34 @@ static void do_mod_ctors(struct module *mod) /* For freeing module_init on success, in case kallsyms traversing */ struct mod_initfree { - struct rcu_head rcu; + struct llist_node node; void *module_init; }; -static void do_free_init(struct rcu_head *head) +static void do_free_init(struct work_struct *w) { - struct mod_initfree *m = container_of(head, struct mod_initfree, rcu); - module_memfree(m->module_init); - kfree(m); + struct llist_node *pos, *n, *list; + struct mod_initfree *initfree; + + list = llist_del_all(&init_free_list); + + synchronize_rcu(); + + llist_for_each_safe(pos, n, list) { + initfree = container_of(pos, struct mod_initfree, node); + module_memfree(initfree->module_init); + kfree(initfree); + } } +static int __init modules_wq_init(void) +{ + INIT_WORK(&init_free_wq, do_free_init); + init_llist_head(&init_free_list); + return 0; +} +module_init(modules_wq_init); + /* * This is where the real work happens. * @@ -3507,7 +3509,6 @@ static noinline int do_init_module(struct module *mod) #endif module_enable_ro(mod, true); mod_tree_remove_init(mod); - disable_ro_nx(&mod->init_layout); module_arch_freeing_init(mod); mod->init_layout.base = NULL; mod->init_layout.size = 0; @@ -3518,14 +3519,18 @@ static noinline int do_init_module(struct module *mod) * We want to free module_init, but be aware that kallsyms may be * walking this with preempt disabled. In all the failure paths, we * call synchronize_rcu(), but we don't want to slow down the success - * path, so use actual RCU here. + * path. module_memfree() cannot be called in an interrupt, so do the + * work and call synchronize_rcu() in a work queue. + * * Note that module_alloc() on most architectures creates W+X page * mappings which won't be cleaned up until do_free_init() runs. Any * code such as mark_rodata_ro() which depends on those mappings to * be cleaned up needs to sync with the queued work - ie * rcu_barrier() */ - call_rcu(&freeinit->rcu, do_free_init); + if (llist_add(&freeinit->node, &init_free_list)) + schedule_work(&init_free_wq); + mutex_unlock(&module_mutex); wake_up_all(&module_wq); @@ -3822,10 +3827,6 @@ static int load_module(struct load_info *info, const char __user *uargs, module_bug_cleanup(mod); mutex_unlock(&module_mutex); - /* we can't deallocate the module until we clear memory protection */ - module_disable_ro(mod); - module_disable_nx(mod); - ddebug_cleanup: ftrace_release_mod(mod); dynamic_debug_remove(mod, info->debug); From patchwork Fri Apr 26 23:22:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10919989 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0657E912 for ; Sat, 27 Apr 2019 06:44:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EB58D28E6B for ; Sat, 27 Apr 2019 06:44:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DF8E028E7B; Sat, 27 Apr 2019 06:44:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 71D0828E6B for ; Sat, 27 Apr 2019 06:44:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1AEA26B0274; Sat, 27 Apr 2019 02:43:34 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0E4CC6B0275; Sat, 27 Apr 2019 02:43:34 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EEC976B0276; Sat, 27 Apr 2019 02:43:33 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id B478A6B0274 for ; Sat, 27 Apr 2019 02:43:33 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id g1so3591819pfo.2 for ; Fri, 26 Apr 2019 23:43:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=03x8MyH4ONo6RL8TDAuJcYQYBUTk8Xfu5EMEl5vv8zs=; b=MtFC5o/r2uxOLrDAFj1cJ9d2DPvOQg84QrxU5uImSnRGWQ1ULXbRQVkTdnjrG8IBMR gkQUIGIzV7nl/FGBFJODmf/olA/iTTg2WAFCf5qAbgvUoszaCqoufNVbUtQjxoa3lgTR c6lQ1u9BMcYAkFwQBOvNZcU5nMnztpm2WpZ+s+2g7JB4Y2UAh3zc0XCMjimAJXDJ0qsR MrxfkQgogVhkw756H/JjSmbNub5sDQPsolha4vgydGM1MZcSjxxJgpkFbz946sKokqSN X0RUwz1G3TCve/BUtLBjpHQPqhBsPR5ofb0ysBl+mrMFtAyrMNjpSL+YR3v81VMfhyVB Of2w== X-Gm-Message-State: APjAAAWPE3qIgo09QgNL3ca9GpVc4GhAsqMWah6c94ggSduxX+mu+nG8 VBJLrXnI29+Yq2yBhNcPUIEfqvbVv3CTzHkCXUIQaRQQjgLaZlxEhXgp47xs90w0Q8xs3nhXJCM fCSj3aOE4pIrLV/00vsg5Kpr3RVcWLGsGcQRxlMyJE0Q+VVZPLwVeT/49XmtYjCJ9sA== X-Received: by 2002:aa7:9285:: with SMTP id j5mr22726313pfa.129.1556347413420; Fri, 26 Apr 2019 23:43:33 -0700 (PDT) X-Received: by 2002:aa7:9285:: with SMTP id j5mr22726238pfa.129.1556347412194; Fri, 26 Apr 2019 23:43:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347412; cv=none; d=google.com; s=arc-20160816; b=dsJEPPuFZFbW7GeJHFFkpJZhIVR07EKWCX2cFtsgCff0F4dos3Hw0iO0ipT7F4O57U uDCcuuZNFxpO7ii2s4wULRSgbpxEp1fMU/NzXfNQwfIdAlkPoy7LqQzikwAq1Yet3qGo yJHGgWE84YuTG7ibGA/vddK4YRkNL5tyqyHqUEGoA/btRTXC1qxwxoKAmrr7nePETMmn AoPMXbTgGMIwXtOyphrTqxRoCUGVOHkOpZ+GT5Jwhk053hvLywE1R2mgSS/t9h78Tnux Iimqkx3YF4eRMFCE3I3JZTmei5rWY76GH/RPJ8rI6o43MKjTJ2p7v9yzkLGVCF3yQIYE xtAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=03x8MyH4ONo6RL8TDAuJcYQYBUTk8Xfu5EMEl5vv8zs=; b=bWWEjO8N6+uZzh/Bi4DC2KZXtIoaA+fvim2bCVMwwNkNagUC6NMo1MxSxjeJVe4qRr MhoHo166YHO5E/ldtPSCyecqQSqxyBtxUWt/BWMEz1jq1MQZlkPoSznTPJVjOtmbHX5J mf2Z+vZ+2oE0jdpc10273F6eERJSpUBGP6ERXlSiblGgaZo8qR/uIpBHNOrVJeKjo+YM gyTEEPcRpN6iBZDBJsvRxVtQ7bh9wKcK1bJbLqnB5ujQcjv2GTg5e3/cI3tclYT5hN2L d5pttL6pi+t8Ijp5fk7NPcLJZLmSxJxbAduZ5kgVn+lFLYHhXdHVBi+cyJ+yukXe20oV skTg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=Q470hKxC; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id g1sor26693120plp.57.2019.04.26.23.43.32 for (Google Transport Security); Fri, 26 Apr 2019 23:43:32 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=Q470hKxC; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=03x8MyH4ONo6RL8TDAuJcYQYBUTk8Xfu5EMEl5vv8zs=; b=Q470hKxCA0zkhL+IQMRmPe5UQg42Gi+l2RRmW947taZyL//Jesf10pqg7iknajn5cM 6MahwRUBormFC9ROtXZzOCBPO1osFE8XuHCncIgLvsCHxLcNuXntyLIPTNUxNKWwnbuS YivKpxFXT45f055xbj7gqKyZeZCL6izznlo4y+QcynGiNpgPJSJn6cIDoQUEWR8q3NIV QQ28y5mhdxTCv3ZGyzW3vgvH+5eRIAil3M9uo1sFkym+2/eXjUEbYhGGuoRIn6PGQ204 szqozT+L1TEHgWXr+gjrabBNVx0fQWsxwFBogU0RCWe7qqaev2wbboj0cmVtOcpDQP9M I7SA== X-Google-Smtp-Source: APXvYqwS9mBb6vrC0DhVyMmyRT5ZRdKwezrJErmAVG/FEmt+6AsmQkMSAz9lRez80K2MggQYstDNlg== X-Received: by 2002:a17:902:bd0c:: with SMTP id p12mr17355851pls.50.1556347411683; Fri, 26 Apr 2019 23:43:31 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:31 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Daniel Borkmann , Alexei Starovoitov Subject: [PATCH v6 19/24] bpf: Use vmalloc special flag Date: Fri, 26 Apr 2019 16:22:58 -0700 Message-Id: <20190426232303.28381-20-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Rick Edgecombe Use new flag VM_FLUSH_RESET_PERMS for handling freeing of special permissioned memory in vmalloc and remove places where memory was set RW before freeing which is no longer needed. Don't track if the memory is RO anymore because it is now tracked in vmalloc. Cc: Daniel Borkmann Cc: Alexei Starovoitov Signed-off-by: Rick Edgecombe --- include/linux/filter.h | 17 +++-------------- kernel/bpf/core.c | 1 - 2 files changed, 3 insertions(+), 15 deletions(-) diff --git a/include/linux/filter.h b/include/linux/filter.h index 14ec3bdad9a9..7d3abde3f183 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -20,6 +20,7 @@ #include #include #include +#include #include @@ -503,7 +504,6 @@ struct bpf_prog { u16 pages; /* Number of allocated pages */ u16 jited:1, /* Is our filter JIT'ed? */ jit_requested:1,/* archs need to JIT the prog */ - undo_set_mem:1, /* Passed set_memory_ro() checkpoint */ gpl_compatible:1, /* Is filter GPL compatible? */ cb_access:1, /* Is control block accessed? */ dst_needed:1, /* Do we need dst entry? */ @@ -733,27 +733,17 @@ bpf_ctx_narrow_access_ok(u32 off, u32 size, u32 size_default) static inline void bpf_prog_lock_ro(struct bpf_prog *fp) { - fp->undo_set_mem = 1; + set_vm_flush_reset_perms(fp); set_memory_ro((unsigned long)fp, fp->pages); } -static inline void bpf_prog_unlock_ro(struct bpf_prog *fp) -{ - if (fp->undo_set_mem) - set_memory_rw((unsigned long)fp, fp->pages); -} - static inline void bpf_jit_binary_lock_ro(struct bpf_binary_header *hdr) { + set_vm_flush_reset_perms(hdr); set_memory_ro((unsigned long)hdr, hdr->pages); set_memory_x((unsigned long)hdr, hdr->pages); } -static inline void bpf_jit_binary_unlock_ro(struct bpf_binary_header *hdr) -{ - set_memory_rw((unsigned long)hdr, hdr->pages); -} - static inline struct bpf_binary_header * bpf_jit_binary_hdr(const struct bpf_prog *fp) { @@ -789,7 +779,6 @@ void __bpf_prog_free(struct bpf_prog *fp); static inline void bpf_prog_unlock_free(struct bpf_prog *fp) { - bpf_prog_unlock_ro(fp); __bpf_prog_free(fp); } diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index ff09d32a8a1b..c605397c79f0 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -848,7 +848,6 @@ void __weak bpf_jit_free(struct bpf_prog *fp) if (fp->jited) { struct bpf_binary_header *hdr = bpf_jit_binary_hdr(fp); - bpf_jit_binary_unlock_ro(hdr); bpf_jit_binary_free(hdr); WARN_ON_ONCE(!bpf_prog_kallsyms_verify_off(fp)); From patchwork Fri Apr 26 23:22:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10919991 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 261C5912 for ; Sat, 27 Apr 2019 06:44:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1776428E6B for ; Sat, 27 Apr 2019 06:44:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0AC4628E7B; Sat, 27 Apr 2019 06:44:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 986D028E6B for ; Sat, 27 Apr 2019 06:44:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E2F986B0275; Sat, 27 Apr 2019 02:43:35 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 37AD26B0276; Sat, 27 Apr 2019 02:43:35 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 21CF86B0277; Sat, 27 Apr 2019 02:43:35 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id DAA086B0275 for ; Sat, 27 Apr 2019 02:43:34 -0400 (EDT) Received: by mail-pf1-f199.google.com with SMTP id f7so3576051pfd.7 for ; Fri, 26 Apr 2019 23:43:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=RIE66crRSFais2XMEreovT7cuc4aWzwk2c6XjFrsy1k=; b=MfbnEr5+Io6aAcv/6wwHi8fgB607akDoT/l1A4tDl1QE9gu+jO2BP1JWbQAc1pS1Ad 3DXZsdwkAQtFznpM3LXG4Qfu7Ms3qUy4tw9ugY0vvUcgzjErisYOj7u+CFbEamYsqdis OxDh/tYmnZp5ORHy7Zgk4P8/Hmuu0H+IObvKKxJfBZ3DBpnNzmzYJ4kImI8gZ5glptp5 ZUNDNW7ZlfxePZObFjBF5vD2ds/8PEEMFa5gBYDuOoISaOg3b5cwEcwlCISrtV2Ehtvw jGA7mCnUYwOdWUfrH8+JYwT3HhHHizOOtAcKhpPrYlT7KEDlw+l7DJMcPKAo1OFhYY22 QOZg== X-Gm-Message-State: APjAAAWCleKUVAdPJLR4apulNVQYBt+GXz9fztHMsoqzZ45KL4Ul7QJe 8H0HyDlZcAuyfg3ELVg15bWm4/lC/c+PpwYzf35k6ZZ/HqPD7Guv12PDgpZeq5bk80BNhsFjEx+ N6f3EwfyK6rnsPDbQ/S0wJloMC7MnJ3bRyS6kLcmkPcknvsPrV6UOcque1PJ5ame0QA== X-Received: by 2002:a63:da4e:: with SMTP id l14mr44930396pgj.96.1556347414583; Fri, 26 Apr 2019 23:43:34 -0700 (PDT) X-Received: by 2002:a63:da4e:: with SMTP id l14mr44930347pgj.96.1556347413480; Fri, 26 Apr 2019 23:43:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347413; cv=none; d=google.com; s=arc-20160816; b=POkuA9qrg1gzRhiLi+Wkey+bx7atI4fYoWDTs7XJuOapyabq32FeKr/ztm4MUXW9BO Gu+w+xo+nW5kyPQpY7JuauoJ8UI127W6InyjgT09C7VAd4FcZ/Zn6roSYehEvluqxf5j IZGIL5eJ/jX9AsjQS0nl5Qf9L2+C0E9/tA3kh/CGgFhEhkiwR5Q6SkKW/HCnMrJ8r+6s oKfYCCYIUlYHaXV/iC77PcUrHuvTYBHueEShCex9gtIiqc92SxuQI2KW0mRI+HlxlJfC Ydpm1mv4If+v44FW15w97Qf/fItmQ6i4Il78LLQ4bMA6XymXED/iI4lZx1HJdXc3YzOw nLlQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=RIE66crRSFais2XMEreovT7cuc4aWzwk2c6XjFrsy1k=; b=F9TDqMkMosgp1YxLckGEARbcEHdLrLTZDXzG90TZAnkMYBAYjtULLsZYSmnsaiWvSc T/dAv2Pcz+t59Uw0nQR48j9zTWXLYadwxYYwsQ56ZTFBoLP21SkwAECi8z6a72s0ax2r gJQ0udD3pWMNv0KtYfe6tNuVrlHeoY1CORcvck1WptVcdArSHAfjtHzQ5I8kchFf7Xll aPIPvbjWQ5bXlqwo6+2RiJVURCeDR/6pD/556KZU/vkH/66t0ki55/jrul8aFvZQrpme OpzdaEUilbLDWoLgVKgENfXIR5kUSk7Xe3XoXPJkyJPzR9uwk1tzFdTKDddKDT8uRFth c/vw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=pYvgo9ZC; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id t10sor27052107plq.14.2019.04.26.23.43.33 for (Google Transport Security); Fri, 26 Apr 2019 23:43:33 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=pYvgo9ZC; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RIE66crRSFais2XMEreovT7cuc4aWzwk2c6XjFrsy1k=; b=pYvgo9ZCfUCHvRHlTxheTL9ZRQVp4fim5mJ1MwWcKR1lA8y1aUP/jaFmmW0gvaWJ7S zVzA0PQ2EURS5SWk/peFTu1A3/0LRw7DZYQoG4uzOziiwhPRGPxHOWrU2FLz8xJKPBlh /zuS4kFNlrBUqnzlR10WasNuwWxPh6kwUkYfWX2HNs3YXw+waJ/qMVcWCQyEr/PXxH6M ipIVs/0XiprUGYbGWdjj/stD10SwODP6rYxbEpQWhHBNTSo8/W1+lQ5k//iDxItfsPxn hO3NCZNloR7//+m6Eozd9KTmliFtML7SM01N4FLmt+HleGXZ2gYxUmzOYvSuD2G1W0Lp NyOQ== X-Google-Smtp-Source: APXvYqwLfbRywYDBLBkn2s5QjOy7Ake8UGyhjw6r9cFTnOHwE2KsORvL/Fy4JLKSQeWsUj9Okam5Jg== X-Received: by 2002:a17:902:b68e:: with SMTP id c14mr52282678pls.49.1556347412993; Fri, 26 Apr 2019 23:43:32 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:32 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe Subject: [PATCH v6 20/24] x86/ftrace: Use vmalloc special flag Date: Fri, 26 Apr 2019 16:22:59 -0700 Message-Id: <20190426232303.28381-21-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Rick Edgecombe Use new flag VM_FLUSH_RESET_PERMS for handling freeing of special permissioned memory in vmalloc and remove places where memory was set NX and RW before freeing which is no longer needed. Acked-by: Steven Rostedt (VMware) Tested-by: Steven Rostedt (VMware) Signed-off-by: Rick Edgecombe --- arch/x86/kernel/ftrace.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index 53ba1aa3a01f..0caf8122d680 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -678,12 +678,8 @@ static inline void *alloc_tramp(unsigned long size) { return module_alloc(size); } -static inline void tramp_free(void *tramp, int size) +static inline void tramp_free(void *tramp) { - int npages = PAGE_ALIGN(size) >> PAGE_SHIFT; - - set_memory_nx((unsigned long)tramp, npages); - set_memory_rw((unsigned long)tramp, npages); module_memfree(tramp); } #else @@ -692,7 +688,7 @@ static inline void *alloc_tramp(unsigned long size) { return NULL; } -static inline void tramp_free(void *tramp, int size) { } +static inline void tramp_free(void *tramp) { } #endif /* Defined as markers to the end of the ftrace default trampolines */ @@ -808,6 +804,8 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) /* ALLOC_TRAMP flags lets us know we created it */ ops->flags |= FTRACE_OPS_FL_ALLOC_TRAMP; + set_vm_flush_reset_perms(trampoline); + /* * Module allocation needs to be completed by making the page * executable. The page is still writable, which is a security hazard, @@ -816,7 +814,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) set_memory_x((unsigned long)trampoline, npages); return (unsigned long)trampoline; fail: - tramp_free(trampoline, *tramp_size); + tramp_free(trampoline); return 0; } @@ -947,7 +945,7 @@ void arch_ftrace_trampoline_free(struct ftrace_ops *ops) if (!ops || !(ops->flags & FTRACE_OPS_FL_ALLOC_TRAMP)) return; - tramp_free((void *)ops->trampoline, ops->trampoline_size); + tramp_free((void *)ops->trampoline); ops->trampoline = 0; } From patchwork Fri Apr 26 23:23:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10919999 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 616F414DB for ; Sat, 27 Apr 2019 06:44:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5234528E6B for ; Sat, 27 Apr 2019 06:44:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 46D2A28E7B; Sat, 27 Apr 2019 06:44:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EAAF228E81 for ; Sat, 27 Apr 2019 06:44:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 68A766B0276; Sat, 27 Apr 2019 02:43:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5EE546B0277; Sat, 27 Apr 2019 02:43:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 466966B0278; Sat, 27 Apr 2019 02:43:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id 0D9176B0276 for ; Sat, 27 Apr 2019 02:43:36 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id n5so3492539pgk.9 for ; Fri, 26 Apr 2019 23:43:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=PMP2Cxb893S0prGS0vCi085zHCDPTtmjtSNHdboWACs=; b=evbup2rdZ+yf94IViUC4SK7UboQfysAbBoqSewJzDhobG6A4otS3M2pun/Y7lXK7Jk H2gOH+bg+0XEwuanRjqHjtmIXRqHTT7w5kSPmQyOCbD1BunhoV3gQl4gMca4XjqcctMM oPd3nt0c2wo5tVfN1laAr0ZNVaK1QPqcFotPyIqX8r2FyXFlmtALozrfhBMNN4mnJb2l HDXR2jTuhS9rMZrqdToglwRezZgwKiZl+ikAiVeLAiTbtSxXHVUqaSHjn+Xgzg8aiehk hGmk9CJf+Z3IdlstOdpLf8AUwLDG6wNno15m3Uh1uShvI4bRrqvSGXT8/ohNPVHuvOKS q/ug== X-Gm-Message-State: APjAAAWQDypGWe7iMteFEiaEjbBuuDt3tpnwbGCFeXPuuMfMYE2hatvf nSxRv3j+4T7q5XbgKJaWQv50CBFkrAIYlUaegze6w7WegdUVfHTnrkjaeG/+lpeKlZOmJLU4X6s DJg0A//+E4FTDuGhX83TLRbtzquM7qcXgxah/mPKgheDp6iIzdmnr4gDhtXZxLka+fw== X-Received: by 2002:a17:902:2e83:: with SMTP id r3mr50238539plb.153.1556347415746; Fri, 26 Apr 2019 23:43:35 -0700 (PDT) X-Received: by 2002:a17:902:2e83:: with SMTP id r3mr50238499plb.153.1556347414772; Fri, 26 Apr 2019 23:43:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347414; cv=none; d=google.com; s=arc-20160816; b=dU3vPkLMQE78N0Pxo/acyTzHpIgXtBgKuoQBdgsvqS1GEDRuGffE/pyK8a62dZQGCZ 3iJe+1C7W25ftZfIyoAHqpkcMJAeTzUNXLbYEssFC+kGNaIV1oZ8TfMw7roY61kM6Doi WWXjnLw77qGSfrTkTeCW5F2YPoFNpj2zMoXKwGR+j+UveHZ7sHgdiB1o/w1FrcW0jX7W vaUenmou/r/96kBiZbg0JOSqShCVQGUEt5Qv8+HJzQ5cjdJU9n27gOjkMcuGAWoo0E96 W9oxrljjE47mM2mMlcWdVg8x/Fdlcvk9TD5PyUd10qe3eKp41poSF3BsweHsP3aitY+4 NksA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=PMP2Cxb893S0prGS0vCi085zHCDPTtmjtSNHdboWACs=; b=qCKm8+k9X+54X47a7xoe3p4tHkkrsa/eq5dZsyGss+CwwkZtg57lUgketz32QWkTZN fTqwdOuyA1DuXy6Ojk9W9E1bUthjYhTzJ+7xBwYgzdgH7oHt56JcmnBBuamPhWbaIrma 8eI6GivRos1eZ+MhWzSta8jc/xjx0QA8hAWEWKDSU194cR/Ib3QnKDbz8LaSHAzLsYvP BX92S8j0UsT9/wRRaAI7cdDYmxaruHre1NohLb6RuBL8b7taj2K8bwIy3eu4BSUedSXU aRorIF3sMXlZWYn3RCyQ6Gy/34rt6jHXSI2NErJR8v0IyePXXHJQHGIY0AGhYcsdypXw 5CFw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=K41kvx1t; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id j6sor26814458pll.42.2019.04.26.23.43.34 for (Google Transport Security); Fri, 26 Apr 2019 23:43:34 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=K41kvx1t; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=PMP2Cxb893S0prGS0vCi085zHCDPTtmjtSNHdboWACs=; b=K41kvx1t6SFhhfzncmYY1HeK2Igm+/jVQYVM/IJLmHXShsg5nGe8rTKsYo4rZoZPC/ 6zju36lQj+i8uBKtds/aE0HT690JXiZHzsJ4iG2ZVbGtv1+m8Rexp2qGC4WzKGij4mJY pS0wLymSfhipDrS8/jIS0cMHQRk/GDuPBjL2K/M1OnR2vxv4YrIebnQ2xexSNHS3XdIV svsj4+MI3xf9ZC2zePDJQy6e/n2bUI0eeL3ftgtEvNDNqK6/0DUPSoZt2Ss6m6/VX3rn 2cMMuSy1cr6XBfOuPS4wxZ/91yZ1FLcgw2kClZNZEQvPCdnBiT4UNc5YyXetyq58UhjO W5iw== X-Google-Smtp-Source: APXvYqzt81NaHzE4N6lgOn1CEEv66yygls114EGZnoMMrIlsI1R4+4jwMYRUYyyw4vMJxMCD0c6n8g== X-Received: by 2002:a17:902:f094:: with SMTP id go20mr50490988plb.159.1556347414288; Fri, 26 Apr 2019 23:43:34 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:33 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Masami Hiramatsu Subject: [PATCH v6 21/24] x86/kprobes: Use vmalloc special flag Date: Fri, 26 Apr 2019 16:23:00 -0700 Message-Id: <20190426232303.28381-22-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Rick Edgecombe Use new flag VM_FLUSH_RESET_PERMS for handling freeing of special permissioned memory in vmalloc and remove places where memory was set NX and RW before freeing which is no longer needed. Cc: Masami Hiramatsu Signed-off-by: Rick Edgecombe --- arch/x86/kernel/kprobes/core.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index 1591852d3ac4..136695e4434a 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -434,6 +434,7 @@ void *alloc_insn_page(void) if (!page) return NULL; + set_vm_flush_reset_perms(page); /* * First make the page read-only, and only then make it executable to * prevent it from being W+X in between. @@ -452,12 +453,6 @@ void *alloc_insn_page(void) /* Recover page to RW mode before releasing it */ void free_insn_page(void *page) { - /* - * First make the page non-executable, and only then make it writable to - * prevent it from being W+X in between. - */ - set_memory_nx((unsigned long)page, 1); - set_memory_rw((unsigned long)page, 1); module_memfree(page); } From patchwork Fri Apr 26 23:23:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10920003 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8C760912 for ; Sat, 27 Apr 2019 06:44:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7D45828E6B for ; Sat, 27 Apr 2019 06:44:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 70DAF28E78; Sat, 27 Apr 2019 06:44:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 813C128E7C for ; Sat, 27 Apr 2019 06:44:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B22CA6B0277; Sat, 27 Apr 2019 02:43:37 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AA9CC6B0278; Sat, 27 Apr 2019 02:43:37 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 948786B0279; Sat, 27 Apr 2019 02:43:37 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 537DF6B0277 for ; Sat, 27 Apr 2019 02:43:37 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id b7so3226347plb.17 for ; Fri, 26 Apr 2019 23:43:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=Mq726et3FUsm8L5SG93p+KUhd7ShOgthVM8qUN29ytY=; b=m3ukOiXvW4man29I9owjO1iSQ36kmb2yNDXYurZ8SsWOr0Aaqu/pMaVNTj/Kpm41DU 9oYujswiv+Sy//rt34oqVKRahQzGRtK0zwJ3xj4m5YjfH0t2GkF75Y4gI8vXQIrv27gt uk2XEafXe4cvaNW2y1gxmNJj071lVaIFfEDcMtuMmDS9wSz2ptUFyHS69DntH12eZKff bQ4jUuskp1BnWUx+4ZDO2a9u80YO3LAdr09fRlN7tc29OxvaKwXgxLZRfMWtJxef+rn1 KTd1Qx/WhL6YWGHsHKFutGgy5N0XT8TJp0Xq7CbGWJx5HatJAROlsfUMOQAq/7aiyYJg GBNw== X-Gm-Message-State: APjAAAWBL9GitW51NqtAriftBULOvHuQIrUcsSPdvWiFJ7yHVekIdAj+ rYQQvdnIvFzyL16E+OaDnjuA0gPSwr3DkbLN0gFqd2Fozf5f4RcyyWl5K0PuCoDRc3XmSU0rcPE FY4MiJN7pdZ+peiFxcwcsN7EEnjRT0wooKsgLaPCTyl/MQYFdwo8wvQlMXqcSzyCNvQ== X-Received: by 2002:a63:5041:: with SMTP id q1mr24951430pgl.386.1556347417027; Fri, 26 Apr 2019 23:43:37 -0700 (PDT) X-Received: by 2002:a63:5041:: with SMTP id q1mr24951391pgl.386.1556347416101; Fri, 26 Apr 2019 23:43:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347416; cv=none; d=google.com; s=arc-20160816; b=0ihllf0orVZUtxx/bIPAhjmW4iD0H/qXjOo4NgdHZsWscV+19PX4y9r6jDkR2xQ62l AaSo4n+Pnc/E3Isrn4dA/upoZY2xgUpIVl2g6yvuejE4Q25PZHpve3PDvfWs5ys5vjN4 O3tfKMEmisWLbz5JR3uP5kuaswAZnik+2sKbJ9XtayigIfvv26vON0VmCaiX1jyrQ4MT uJvsZ9J08BCPekKJWCTSxp6jmpZthvpxSXrSSI27jj6JTu+GCnVzX2JaNuTqyuDCFHlS MIUBnubDmrLxCzFuPB8i7uYVtR1fdV8MBLPCrY9faglBYWapqlpskjn7Py2C34w1zWy9 VnXw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Mq726et3FUsm8L5SG93p+KUhd7ShOgthVM8qUN29ytY=; b=pSM/sLdOozy2nMpcytew7z5BhteeEr5sCPol1GOWcKfi3JloaefMMHaCPksnRaZmSD HLWyivLcn2KGXBkFUEz3OFfzATxpACtuT9vpEVHH3k/4MgTVL501KYw9Kdc7siInhYnf oZRrSFjQiCindC1eW1HAWyd0UQrQfYnLvuHOLhR5s7ldTWD5wASQ4zuyEWz7aEzxgquW hXb4rvl9JsNwarpS39DdiZB3foJY7XQZUCG8L12HMIigqH52y/wea4RNBLpbgEVbtuyS Ce7yNt19qrTWFFC840s6ehR+Z4ojQ4y2Ap7gbrWbLnC+n6srT1yJXcrvyBYbJvojNkD3 lWUg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=ZcHDpG5o; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id d63sor28993657pgc.2.2019.04.26.23.43.36 for (Google Transport Security); Fri, 26 Apr 2019 23:43:36 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=ZcHDpG5o; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Mq726et3FUsm8L5SG93p+KUhd7ShOgthVM8qUN29ytY=; b=ZcHDpG5oI881U7Ra0/L2O5huO2cwDn2YrdsQ5hiAWkm6WdBzN67nlkJvfLDLJC5q+X UOCeP+9RydbCT4dO6iGp4lZQdm3+sXHNSQC0G0grGB6NOlnyKbwZn5NdDqZwDHIXRDud 3mNZu1J6RFiZozyuuSJ7f0wBAoiYQFhmUfj5tGkTm8jduQKjEFGn6AF04k5qQYjIWsXY 4YzLX+ZgfiYxaYFVUDUXXbJdfkkMMJKkcbeIB+VGTLlKKEI2dS4TTD29m/lBt3TdxyYa a4fp8LYLp6QEbTAIinVcBUAwlwg+TwtnzyGC3pPfhsUXM3/E24y9pQ9Ah5nWVdchzhK2 LMJw== X-Google-Smtp-Source: APXvYqz6QdLKu0nIIJYQFhptNQ/c9BRHYqidvmm2brorovTM3Ggu9E591UYxGE1lKlV0dGgfmeOtWg== X-Received: by 2002:a63:e051:: with SMTP id n17mr5092049pgj.19.1556347415619; Fri, 26 Apr 2019 23:43:35 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:35 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Nadav Amit , Masami Hiramatsu Subject: [PATCH v6 22/24] x86/alternative: Comment about module removal races Date: Fri, 26 Apr 2019 16:23:01 -0700 Message-Id: <20190426232303.28381-23-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit Add a comment to clarify that users of text_poke() must ensure that no races with module removal take place. Cc: Masami Hiramatsu Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/alternative.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 18f959975ea0..7b9b49dfc05a 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -810,6 +810,11 @@ static void *__text_poke(void *addr, const void *opcode, size_t len) * It means the size must be writable atomically and the address must be aligned * in a way that permits an atomic write. It also makes sure we fit on a single * page. + * + * Note that the caller must ensure that if the modified code is part of a + * module, the module would not be removed during poking. This can be achieved + * by registering a module notifier, and ordering module removal and patching + * trough a mutex. */ void *text_poke(void *addr, const void *opcode, size_t len) { From patchwork Fri Apr 26 23:23:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10920007 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6AF0F14DB for ; Sat, 27 Apr 2019 06:44:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5D52128E78 for ; Sat, 27 Apr 2019 06:44:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4F77F28E7C; Sat, 27 Apr 2019 06:44:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E288628E6B for ; Sat, 27 Apr 2019 06:44:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CA3E06B0278; Sat, 27 Apr 2019 02:43:38 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C54656B0279; Sat, 27 Apr 2019 02:43:38 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD4A46B027A; Sat, 27 Apr 2019 02:43:38 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 728B86B0278 for ; Sat, 27 Apr 2019 02:43:38 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id l13so3503920pgp.3 for ; Fri, 26 Apr 2019 23:43:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=p6XVxxmeId6Fyrw7m3B0ifgQzW6nhwdk5H1SAhfRKS0=; b=RTcpy+NiC7R7PpBw+ceRfAGSTppaDldn2RDmHu9JdKrcXLuTv7mw/Somu/uBJHLWjg qTczf1mEKbL88YbVt3zoztYaTkrSZjWCNcMKdz/yNZxKW79kSkl/8/RuxisgRTojd9EU dT8k8lcdrv7ak8dTrR4HUtS2nSkXy8Z3jHBfLpwqQssEchhb90wHu8JVEmCqS0mCRAMW IrKehKihLntPKPE3Uxmm00Vu9IyemujRN17smK8ScQuGX1ArXnJ+2kFdRHxkfOKM/GJc +iwx9u6qSrmMvMfuZghHSBH5q+yaqy2XJaaucnjsJTfMG6ntJ9EpK1cNngGJfvH0eju/ krGQ== X-Gm-Message-State: APjAAAWO7PKC99ofUHDBDR1pPe4kCPnjlxpWjw1ILy3F75060Y8xRp0O k/0RBj4VDTGROVJ+ukMFJ8Y+KfXAf47V9NGKLC0eImrpZp607FclC+nSAF05VpP8ZfqEedwZLkB LpnDr91Vd5EsibdCUL4Vi2f74cd33l05KflBhO6N0GXyBiblvRj0wJCkLwS00gjq28g== X-Received: by 2002:a63:5166:: with SMTP id r38mr7836282pgl.429.1556347418155; Fri, 26 Apr 2019 23:43:38 -0700 (PDT) X-Received: by 2002:a63:5166:: with SMTP id r38mr7836252pgl.429.1556347417503; Fri, 26 Apr 2019 23:43:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347417; cv=none; d=google.com; s=arc-20160816; b=KEnpMbqyOGnwZudvoXbs8vNZ0854cG1ExsVWLUyQ2QcKpcDqOVa7iJjN6/5NEWQ2N/ kJndYRwlGEIh4LH6X98oIWAhJ2erL27ji0PinbwI1Y6GpS1VKDr9lTjhNhv6/eAhHlpq 5KHgn3fic/eDAtyV/qUskwlmJxCdROCVRfJ3FReig2b/toUZ07I/o2R5n8Hqls/CDSMv h2KU/HAywlkv5wyfiI0WW0B0PF1mf1DdwPiRAesI566E+dvogSHKfRXDj5uCr/2OchLq QF8SUdsL4x/3nIPo+L12UrcGyI/wK2oPUGgbmIQoM00lGEMpLCV6EoDQhh9rBahgg5W2 PDeA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=p6XVxxmeId6Fyrw7m3B0ifgQzW6nhwdk5H1SAhfRKS0=; b=UZzoysDNTOglg2FvotNOaH+KdUGRhhtr2uG4jdQ3HJcnyAYccyCLD8kgc8W9VHyyo6 KR8e1290s0Ur+hukRIQKQ3yF9G7MZsIEpb7sfQOzZuJKWZUZndTPL2fGMJc+dOVbeafR CyemrbJosLxaFPwA9nzKhHtFUwINMuLpvwZROBoes82iCMQDZTVlo9VJmgqbdlpKrSJ/ xikrW9ZYttBEjZVydH952iBB+TJ2UZbN5PK4jqQdznl7e8MtM0whHNC+uLWIBhAWHIBO 3Rygw1pX095T3c8J2g+ForuHJeemwi8Co8RMTgCCEd/YY986PJsSrlsHqb8xPz6bQaos wKEg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=JScJif31; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id i36sor26615136plb.39.2019.04.26.23.43.37 for (Google Transport Security); Fri, 26 Apr 2019 23:43:37 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=JScJif31; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=p6XVxxmeId6Fyrw7m3B0ifgQzW6nhwdk5H1SAhfRKS0=; b=JScJif310IMh72ZohwOauSCAwdoUkuEv8bR6xdhJcKtk1RynLwO3Yk/O8uL4tZE53n NrQxpNK4N1uzjciA1sdkWWtPp8ycpHXFn7qqLzJr7tYIRLsxhiosMqxmPVmHzIChx8+N dySb7k6fhrhmrpcilrlJoW9blvaY3Q+PCSiONh6FBKvoxbL7qSMy7/IzPS7lXjYYk3R/ KYc3BtX2zUuYUpbQToVTJQ3xLcXFtZGa0gKnazTNuB9oRgES+e5Dt6lyOMWvmuEim8L6 0oOzDDk8SIoEGUBtvVZ7G2upgHOvvdwFGoH6Y4DMveFflSyOioLkC8F7etEF6M59+eVc h1Dg== X-Google-Smtp-Source: APXvYqy73yIN4FITPLecL8eRrrVFTQfYnIM8wwU5XCBZXYl3CE93cW/V+JNeu8zKe2r7aDykFyaZUg== X-Received: by 2002:a17:902:d83:: with SMTP id 3mr52113119plv.125.1556347416975; Fri, 26 Apr 2019 23:43:36 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:36 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Nadav Amit Subject: [PATCH v6 23/24] mm/tlb: Provide default nmi_uaccess_okay() Date: Fri, 26 Apr 2019 16:23:02 -0700 Message-Id: <20190426232303.28381-24-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit x86 has an nmi_uaccess_okay(), but other architectures do not. Arch-independent code might need to know whether access to user addresses is ok in an NMI context or in other code whose execution context is unknown. Specifically, this function is needed for bpf_probe_write_user(). Add a default implementation of nmi_uaccess_okay() for architectures that do not have such a function. Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/tlbflush.h | 2 ++ include/asm-generic/tlb.h | 9 +++++++++ 2 files changed, 11 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 90926e8dd1f8..dee375831962 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -274,6 +274,8 @@ static inline bool nmi_uaccess_okay(void) return true; } +#define nmi_uaccess_okay nmi_uaccess_okay + /* Initialize cr4 shadow for this CPU. */ static inline void cr4_init_shadow(void) { diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index b9edc7608d90..480e5b2a5748 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -21,6 +21,15 @@ #include #include +/* + * Blindly accessing user memory from NMI context can be dangerous + * if we're in the middle of switching the current user task or switching + * the loaded mm. + */ +#ifndef nmi_uaccess_okay +# define nmi_uaccess_okay() true +#endif + #ifdef CONFIG_MMU /* From patchwork Fri Apr 26 23:23:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10920013 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 99FC614B6 for ; Sat, 27 Apr 2019 06:44:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8AF3228E6B for ; Sat, 27 Apr 2019 06:44:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7EAA828E7B; Sat, 27 Apr 2019 06:44:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 26A6028E6B for ; Sat, 27 Apr 2019 06:44:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C8FE6B0279; Sat, 27 Apr 2019 02:43:40 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 726076B027A; Sat, 27 Apr 2019 02:43:40 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 581926B027B; Sat, 27 Apr 2019 02:43:40 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id 194356B0279 for ; Sat, 27 Apr 2019 02:43:40 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id u2so3488250pgi.10 for ; Fri, 26 Apr 2019 23:43:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=TdxUBqKvjD04gvlVW2fqTrr9rgpG7ttN99ifXgy4wJ0=; b=sUqBBWn2auYRVKdhvXMd4oCVm8tD4AA920v/2LpbNARfyEPKYUyDtdoSttd2kGggO5 4b3/B+Bgf8+v6s4tN+C1/Jkcpw9lkKNKzQOBENgTr4pzXJjduEg0/4IAqrs0xLEzGpJM MpqbbvG65OIZ/2/NecDiA86eQ1lvvf1QKr3lbet0XRQA3iEGFqe19Wobz8VnmWAHCf4+ gEuxnhjvq7Y1OJV7DVbyLo/eHNpZEO5YwGnFixaj/0HaAv6CQSo160Kmmmhz4nY0WnDR uSLI3GmsVA5E851W/AEdR2gOnnxOihn5P77Oym0Tvj21pgws0X1xc7kNFqTEvBBt0TwJ wK9A== X-Gm-Message-State: APjAAAW6XeJ+NOPrdb8PzPc4eZc3xOBHK63S+odwuhcqawWp87Fupqmn MbjN3b3apTn9PwClxnFEE96xTwNbMwNQgoWAeNK/EtgvioLxSlzlOZGgNQbXPDvnF2WMUM3i49E 8gCjBDm4KTtcaOtG3LnNQ/W+gAMz0tDF1HezPYQNnJckI+AYhRdsWnIw9kYu8XO/Gxw== X-Received: by 2002:a62:fb0a:: with SMTP id x10mr17643831pfm.179.1556347419780; Fri, 26 Apr 2019 23:43:39 -0700 (PDT) X-Received: by 2002:a62:fb0a:: with SMTP id x10mr17643787pfm.179.1556347418778; Fri, 26 Apr 2019 23:43:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556347418; cv=none; d=google.com; s=arc-20160816; b=akuT8apVCEsz6sM6iqxfM0x1aBJMnvjpk2gEqUq3B25YgWGy3dvv7jfipBQbBN680O kr5mcWREyx0+n2vbvDGXi0JIkh1xMGPTTV+Ub46gZ3tfe/OfHAy3xfGR1/HMcmDgUq+h Dp6OOgoD1ZpD8FLhfg6tLJxyY8a8Wme5O7SYW0HeeaLO07/T6VieRXRpdRgXaOI3RljA VXlpLvCGQnxpD86fTFKkAlrcMAjrFshZX9zcBpv2zrMcFupsYCB6GnrYB5zP86ER5OTp ZUpdnjCzi3TgFbC7gXSGdH8JTC/8U3b3foy6nSo0009tsivlLX7s+357bDhNPAoNsrpL 7M7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=TdxUBqKvjD04gvlVW2fqTrr9rgpG7ttN99ifXgy4wJ0=; b=Td4PmaCxQhau0PAdygVojgj5LmN8TW3m+EK4Mfa30y3++uB/M8mfyg6zjsX+yI6rMz jh/34I/74+G58nkk3k5gNrEF+VlFmtUHtmPnn2s0rXCuHzR84FemRDtNEgmyrJ+agjJY iY9a5LS8oxamueTrOVuA7X6hHvQOUQpAi8Q+6x4e2md0QdIuSa2U3XRS0sFbbmE+/Erq 25NMERT4jQxV+gBfn7+2m99QIno/ggHtzDat4HOhBst3ENiOJ4V0MqFcFu5loIq3rRN1 2j7hDoiioVtwicu5t7wb31eJqBeeZInrFtOVnYYIzYHneYphkpaXYav0RS4YssxFVC0N s/JA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="fTkRQ/Y6"; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id m32sor26803551pld.7.2019.04.26.23.43.38 for (Google Transport Security); Fri, 26 Apr 2019 23:43:38 -0700 (PDT) Received-SPF: pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="fTkRQ/Y6"; spf=pass (google.com: domain of nadav.amit@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=TdxUBqKvjD04gvlVW2fqTrr9rgpG7ttN99ifXgy4wJ0=; b=fTkRQ/Y6ekEQNK89awlPXt7h4NxMjoecNBAMoo9ta42p80ijXkGhM1UL/BQmQFyWo6 LJfDPrdOvXO5UvSlwALV9jzgVbZ3dPkOetjmwgngrd6Xv4mjhDuVn4pVdt7q+ACp200R xHDNjHe63z2eq8GDtYLwE09uCfrsylk7ZaSgnWZyDkP/bBrVBZQ5DR2PNo0INMO0nJJB A8319czkJRDzqjy/YpDHiQ5QMkoO+Q0Ds/GWZq+ooXRPBqH7Yybl6d/xzosC6sOQZDv0 Hev5YYonTqwLdIPLk/p3TqQ3UvgcH8BaYIQHch4ej87FxDr5lyTtZXPQ+va/opGvDk71 04vw== X-Google-Smtp-Source: APXvYqwkqefi+5wlFbKz/qeJ/65fu9i/qlczB3LZREQFaY/+lFjrw+S/8uXGcd/aBtDXpy5vBsJ5Xg== X-Received: by 2002:a17:902:b715:: with SMTP id d21mr50699394pls.103.1556347418251; Fri, 26 Apr 2019 23:43:38 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j22sm36460145pfn.129.2019.04.26.23.43.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Apr 2019 23:43:37 -0700 (PDT) From: nadav.amit@gmail.com To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Nadav Amit , Daniel Borkmann , Alexei Starovoitov Subject: [PATCH v6 24/24] bpf: Fail bpf_probe_write_user() while mm is switched Date: Fri, 26 Apr 2019 16:23:03 -0700 Message-Id: <20190426232303.28381-25-nadav.amit@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426232303.28381-1-nadav.amit@gmail.com> References: <20190426232303.28381-1-nadav.amit@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Nadav Amit When using a temporary mm, bpf_probe_write_user() should not be able to write to user memory, since user memory addresses may be used to map kernel memory. Detect these cases and fail bpf_probe_write_user() in such cases. Cc: Daniel Borkmann Cc: Alexei Starovoitov Reported-by: Jann Horn Suggested-by: Jann Horn Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- kernel/trace/bpf_trace.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index d64c00afceb5..94b0e37d90ef 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -14,6 +14,8 @@ #include #include +#include + #include "trace_probe.h" #include "trace.h" @@ -163,6 +165,10 @@ BPF_CALL_3(bpf_probe_write_user, void *, unsafe_ptr, const void *, src, * access_ok() should prevent writing to non-user memory, but in * some situations (nommu, temporary switch, etc) access_ok() does * not provide enough validation, hence the check on KERNEL_DS. + * + * nmi_uaccess_okay() ensures the probe is not run in an interim + * state, when the task or mm are switched. This is specifically + * required to prevent the use of temporary mm. */ if (unlikely(in_interrupt() || @@ -170,6 +176,8 @@ BPF_CALL_3(bpf_probe_write_user, void *, unsafe_ptr, const void *, src, return -EPERM; if (unlikely(uaccess_kernel())) return -EPERM; + if (unlikely(!nmi_uaccess_okay())) + return -EPERM; if (!access_ok(unsafe_ptr, size)) return -EPERM;