From patchwork Fri Apr 26 00:11:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918533 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5A9E892A for ; Fri, 26 Apr 2019 07:31:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 47F5F28DA2 for ; Fri, 26 Apr 2019 07:31:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3BF4D28DA4; Fri, 26 Apr 2019 07:31:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 80B5B28DA2 for ; Fri, 26 Apr 2019 07:31:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0DB046B0005; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 08BD86B0008; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE2EA6B000A; Fri, 26 Apr 2019 03:31:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id B66A76B0005 for ; Fri, 26 Apr 2019 03:31:47 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id d12so1664062pfn.9 for ; Fri, 26 Apr 2019 00:31:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=T5NHMQswvskeKED0v6CftpEeZQL9SAjl2YkW83FLHxE=; b=RuH0tUUvvB9gNGkFaFAhLlCYdqyqnZAPxBgqzWe4J8aD0ZGEavJM8KB7AiS2lSaC/Q xF008ZDSS1BBctRRtx1xTk/4j0tE1MdbBr4vLC58mrdmk/trG+tQhA4hDzJm14x7ISmT 60AsorKwVmGFkpigFmaQUHI+7bVMN4WWVSpdTFH3SduT5yO+K/LSHKqg3yDHc7Izzzkg Zz3BrlTC87uXZ+WYAD5733CvpIz32ME7JT52F33wDbUfB3caqPFJm8xxWMuDHsQrCznk DgaaSi2lRHucxT6bFedrpnshQgVZVExQGFEAI2Hi1b0hj8yC1oOyVFEVN5pQuJvusyQF LKUg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAXZYL3TR26DXTDBvfJd0vGH50YpBlRj+wpRnV+U2Suw5wsG2RAQ BAj2sgo5RDjvQX0aIsRf15PQqc3MWE9cyyUd7H5ourrL4D51f0n9fOaq9cL9WmpEze+OSP2Fjmy sr39DxjH7fVTH9St1NKyrRlswMwFN09CDFvwUn/4T+IYwwxQqOafl7dw0lz7asVZSmQ== X-Received: by 2002:a17:902:8ecc:: with SMTP id x12mr44655209plo.0.1556263907272; Fri, 26 Apr 2019 00:31:47 -0700 (PDT) X-Google-Smtp-Source: APXvYqyYaNbW5mb7k3eyXLIyMCr5zRv4DL+mc1yilrMp+fwRimLh/oec6QaFR7wB+0MKhlPBACcT X-Received: by 2002:a17:902:8ecc:: with SMTP id x12mr44655128plo.0.1556263906234; Fri, 26 Apr 2019 00:31:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263906; cv=none; d=google.com; s=arc-20160816; b=C8/XhZiblIDUF9ddCfhgz/R+zbb/NZhy/tGYt+4ynsvGq/4D8MxkfBxb76XuFf3KG5 7n5VuttvpefIhO6RZy3YLsWSpflGnRyDaPM3PibLXA8ZpbZPVkbqQCx5Y09ym3YjAx1K 5at3yrhMZW4Exy16LWI5SYav0CcWKZmSY3glzyomaeqYS7N/VJrZlpTrzNFQkv1PjFYW 6iq2JIZNERHskmLA6rwUFLV1N9YqY3rV9eXOjm+B5+pLbHCN7vHo038WeA8HNMcUtidi LDnJFRR+nTunWO1fwaq087JD7MMr3egv0dUFw6Qrdm36YvXEjvekh1A4KLWAS+0msJpU ZemQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=T5NHMQswvskeKED0v6CftpEeZQL9SAjl2YkW83FLHxE=; b=VKm83iuUZ+iZrrlxdt8KvuQnwNiBwut78gqeYErnARYesvdRxyoLNVD7x/REqMNY4u hNZCXfjkhKzVOSEnad5bebkA/ivIeQO8Pj5FZmrlCl3EMdZO1ltAr/RzTLAc//eaLHVa l63JTea+0BJ+Z1ZKelyV/lrqkJ8SWIqGU3TmkPMQuU69eagDPRij5+6EI9slIFW6X+tw Eo3KCkdEc/A7/JPPoQvCNBo+N1gkefRZKGs4+0Mgb6cHA6oa/9ReMZSpQoeINDwQRvTh ovd8/+PjHXk0vC0ugJyFVpcQuGWitADbWLUgeh+RyoYmP76Q8973H+m7n0XPckL6bthM tsag== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-001.vmware.com (ex13-edg-ou-001.vmware.com. [208.91.0.189]) by mx.google.com with ESMTPS id v82si25417769pfa.42.2019.04.26.00.31.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:46 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) client-ip=208.91.0.189; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-001.vmware.com (10.113.208.155) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:40 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id 631F24129A; Fri, 26 Apr 2019 00:31:45 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe , Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu Subject: [PATCH v5 01/23] Fix "x86/alternatives: Lockdep-enforce text_mutex in text_poke*()" Date: Thu, 25 Apr 2019 17:11:21 -0700 Message-ID: <20190426001143.4983-2-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-001.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP text_mutex is currently expected to be held before text_poke() is called, but kgdb does not take the mutex, and instead *supposedly* ensures the lock is not taken and will not be acquired by any other core while text_poke() is running. The reason for the "supposedly" comment is that it is not entirely clear that this would be the case if gdb_do_roundup is zero. Create two wrapper functions, text_poke() and text_poke_kgdb(), which do or do not run the lockdep assertion respectively. While we are at it, change the return code of text_poke() to something meaningful. One day, callers might actually respect it and the existing BUG_ON() when patching fails could be removed. For kgdb, the return value can actually be used. Cc: Andy Lutomirski Cc: Kees Cook Cc: Dave Hansen Cc: Masami Hiramatsu Fixes: 9222f606506c ("x86/alternatives: Lockdep-enforce text_mutex in text_poke*()") Suggested-by: Peter Zijlstra Acked-by: Jiri Kosina Acked-by: Peter Zijlstra (Intel) Reviewed-by: Masami Hiramatsu Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/text-patching.h | 1 + arch/x86/kernel/alternative.c | 52 ++++++++++++++++++++-------- arch/x86/kernel/kgdb.c | 11 +++--- 3 files changed, 45 insertions(+), 19 deletions(-) diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h index e85ff65c43c3..f8fc8e86cf01 100644 --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -35,6 +35,7 @@ extern void *text_poke_early(void *addr, const void *opcode, size_t len); * inconsistent instruction while you patch. */ extern void *text_poke(void *addr, const void *opcode, size_t len); +extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len); extern int poke_int3_handler(struct pt_regs *regs); extern void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler); extern int after_bootmem; diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 9a79c7808f9c..0a814d73547a 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -679,18 +679,7 @@ void *__init_or_module text_poke_early(void *addr, const void *opcode, return addr; } -/** - * text_poke - Update instructions on a live kernel - * @addr: address to modify - * @opcode: source of the copy - * @len: length to copy - * - * Only atomic text poke/set should be allowed when not doing early patching. - * It means the size must be writable atomically and the address must be aligned - * in a way that permits an atomic write. It also makes sure we fit on a single - * page. - */ -void *text_poke(void *addr, const void *opcode, size_t len) +static void *__text_poke(void *addr, const void *opcode, size_t len) { unsigned long flags; char *vaddr; @@ -703,8 +692,6 @@ void *text_poke(void *addr, const void *opcode, size_t len) */ BUG_ON(!after_bootmem); - lockdep_assert_held(&text_mutex); - if (!core_kernel_text((unsigned long)addr)) { pages[0] = vmalloc_to_page(addr); pages[1] = vmalloc_to_page(addr + PAGE_SIZE); @@ -733,6 +720,43 @@ void *text_poke(void *addr, const void *opcode, size_t len) return addr; } +/** + * text_poke - Update instructions on a live kernel + * @addr: address to modify + * @opcode: source of the copy + * @len: length to copy + * + * Only atomic text poke/set should be allowed when not doing early patching. + * It means the size must be writable atomically and the address must be aligned + * in a way that permits an atomic write. It also makes sure we fit on a single + * page. + */ +void *text_poke(void *addr, const void *opcode, size_t len) +{ + lockdep_assert_held(&text_mutex); + + return __text_poke(addr, opcode, len); +} + +/** + * text_poke_kgdb - Update instructions on a live kernel by kgdb + * @addr: address to modify + * @opcode: source of the copy + * @len: length to copy + * + * Only atomic text poke/set should be allowed when not doing early patching. + * It means the size must be writable atomically and the address must be aligned + * in a way that permits an atomic write. It also makes sure we fit on a single + * page. + * + * Context: should only be used by kgdb, which ensures no other core is running, + * despite the fact it does not hold the text_mutex. + */ +void *text_poke_kgdb(void *addr, const void *opcode, size_t len) +{ + return __text_poke(addr, opcode, len); +} + static void do_sync_core(void *info) { sync_core(); diff --git a/arch/x86/kernel/kgdb.c b/arch/x86/kernel/kgdb.c index 4ff6b4cdb941..2b203ee5b879 100644 --- a/arch/x86/kernel/kgdb.c +++ b/arch/x86/kernel/kgdb.c @@ -759,13 +759,13 @@ int kgdb_arch_set_breakpoint(struct kgdb_bkpt *bpt) if (!err) return err; /* - * It is safe to call text_poke() because normal kernel execution + * It is safe to call text_poke_kgdb() because normal kernel execution * is stopped on all cores, so long as the text_mutex is not locked. */ if (mutex_is_locked(&text_mutex)) return -EBUSY; - text_poke((void *)bpt->bpt_addr, arch_kgdb_ops.gdb_bpt_instr, - BREAK_INSTR_SIZE); + text_poke_kgdb((void *)bpt->bpt_addr, arch_kgdb_ops.gdb_bpt_instr, + BREAK_INSTR_SIZE); err = probe_kernel_read(opc, (char *)bpt->bpt_addr, BREAK_INSTR_SIZE); if (err) return err; @@ -784,12 +784,13 @@ int kgdb_arch_remove_breakpoint(struct kgdb_bkpt *bpt) if (bpt->type != BP_POKE_BREAKPOINT) goto knl_write; /* - * It is safe to call text_poke() because normal kernel execution + * It is safe to call text_poke_kgdb() because normal kernel execution * is stopped on all cores, so long as the text_mutex is not locked. */ if (mutex_is_locked(&text_mutex)) goto knl_write; - text_poke((void *)bpt->bpt_addr, bpt->saved_instr, BREAK_INSTR_SIZE); + text_poke_kgdb((void *)bpt->bpt_addr, bpt->saved_instr, + BREAK_INSTR_SIZE); err = probe_kernel_read(opc, (char *)bpt->bpt_addr, BREAK_INSTR_SIZE); if (err || memcmp(opc, bpt->saved_instr, BREAK_INSTR_SIZE)) goto knl_write; From patchwork Fri Apr 26 00:11:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918541 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 19FAF92A for ; Fri, 26 Apr 2019 07:31:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0945A28DA2 for ; Fri, 26 Apr 2019 07:31:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F103728DA4; Fri, 26 Apr 2019 07:31:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 913A328DA2 for ; Fri, 26 Apr 2019 07:31:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 80DC56B000D; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6A4496B0269; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 433536B0008; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id DDEF46B0006 for ; Fri, 26 Apr 2019 03:31:47 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id r13so1507213pga.13 for ; Fri, 26 Apr 2019 00:31:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=3lkOJ0gCMIK7S6QGj/0XRCLT0Jm5tS9Iw+h22EmfaMM=; b=RF+gTEAB+VRQrtl62tefIOH1flInYvfpTx4d1HY+Uzac4jpLer4NNmFkC1o2CqkXja bCwuhGs4TDi6pa01vyw3viQtS2VCsdWwMI26ahk5JqoM0BIC1tYbum5yrEGq/QDeHM/U 8nwZRAbYLNS9Qjg+bmLTV7MPe5Vk6aklojh4js9bg2goIJR2W376FPChTlMQd+BJrFwl vTp+5ezKyeP4w3a3TkF3fgUeg+jPphZ1k/PV0s+3h+MjDYbCWRiK1CIItbL2xNS3oCAx dadSNx17Oah1IFRrDstX1hfAbZJ1AEiGgPd1JoVIXSV5t7MsJcR/ozPdArKillHrhknb o9Qw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAVTtDneTQtXQ+Ut+MW34MSbxdSestzHLeMrUSErKwhp/JgknK/8 RDjrRT9UZ2usnP/E8v3zcWXxIgEwvncPuehAm/+dB3eMvOAPh8uvIH2ywkiT9F0nw6pfMclFBDl 2r/Ca9TqUD7BKhCko5Ct2j1BTy0SxP4r83dmqIe4CGg7PDphio+5ajY47ZCV4cSnizA== X-Received: by 2002:a17:902:2aa8:: with SMTP id j37mr44990346plb.164.1556263907360; Fri, 26 Apr 2019 00:31:47 -0700 (PDT) X-Google-Smtp-Source: APXvYqyMInv5r8YuhBW2bD5e/U0+U/Seo7x7gk+MQnLZSqhTkRnJ2lGyqXs1T3q5HOZbQI5OmETD X-Received: by 2002:a17:902:2aa8:: with SMTP id j37mr44990254plb.164.1556263906140; Fri, 26 Apr 2019 00:31:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263906; cv=none; d=google.com; s=arc-20160816; b=nquQRDvgApec/o6cad48B5FNBFzamJVZW9KHcz09Si7FZkDbEzSAMz+t2MQA7+quvC 3TlgtF5MldUMMcyRaWjOhLgH63JRpwqMb0gVWdmOrVtq/Mz1aAXw6E3c6ZS2AOZsCOEu q8GRlrZKXpbXtuPHDLODtmGXkne04DV/yZi9JPHncw1XFSbZPYtcR/e38KKxh1KQrU/L OlOgPjgNIFlPVSqqxGBrEJOBuQdrRFRZrwix5rAG9CfH3jESFtfjZ+PQcbUSU7fENtgV Kx5ffyp0qe7sO7gRLX1q1GbbobpmBaAu/61tILdi3q4+A2nXoA64Y430409EXeYFjBzN O1fw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=3lkOJ0gCMIK7S6QGj/0XRCLT0Jm5tS9Iw+h22EmfaMM=; b=zA6lEtmLom2z4FBlLSKr7YACFm1IPGIZBz4qSo0GyLRpcaftvhaDvof2wCGmADeVfq oW/zBR/9WI3JvcklHt3LvF85Uq++8Ydj6Gkc2zosyVa1bC0SoKFfSIjmXvExfDqAmhPx YgC/ICuQvupdRzdTys/F+esVV3Pf8Cacwe2pp7gZXhpnuGSICBy8PE0zA7hpDO/n2Tmt 0ytvKQxKTsq1A7GStxkm5ftpqyxvIQ4b+jPtfmQc1PPqwWn/xP+IfwNQmu+lshd+c22r YzAPiVt8bfVc8QWbXM+CtUNiMaywEABcQXOxE+8caiuhsG7PO49E5zNJ17BC9CwbxEHy Zguw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-002.vmware.com (ex13-edg-ou-002.vmware.com. [208.91.0.190]) by mx.google.com with ESMTPS id f9si22844507pgq.347.2019.04.26.00.31.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:46 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) client-ip=208.91.0.190; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-002.vmware.com (10.113.208.156) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:44 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id 6F14E41225; Fri, 26 Apr 2019 00:31:45 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe , Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu Subject: [PATCH v5 02/23] x86/jump_label: Use text_poke_early() during early init Date: Thu, 25 Apr 2019 17:11:22 -0700 Message-ID: <20190426001143.4983-3-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-002.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP There is no apparent reason not to use text_poke_early() during early-init, since no patching of code that might be on the stack is done and only a single core is running. This is required for the next patches that would set a temporary mm for text poking, and this mm is only initialized after some static-keys are enabled/disabled. Cc: Andy Lutomirski Cc: Kees Cook Cc: Dave Hansen Cc: Masami Hiramatsu Acked-by: Peter Zijlstra (Intel) Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/jump_label.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c index f99bd26bd3f1..e7d8c636b228 100644 --- a/arch/x86/kernel/jump_label.c +++ b/arch/x86/kernel/jump_label.c @@ -50,7 +50,12 @@ static void __ref __jump_label_transform(struct jump_entry *entry, jmp.offset = jump_entry_target(entry) - (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); - if (early_boot_irqs_disabled) + /* + * As long as only a single processor is running and the code is still + * not marked as RO, text_poke_early() can be used; Checking that + * system_state is SYSTEM_BOOTING guarantees it. + */ + if (system_state == SYSTEM_BOOTING) poker = text_poke_early; if (type == JUMP_LABEL_JMP) { From patchwork Fri Apr 26 00:11:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918555 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 84E7392A for ; Fri, 26 Apr 2019 07:32:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 70D9C28DA2 for ; Fri, 26 Apr 2019 07:32:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 61DC428DA4; Fri, 26 Apr 2019 07:32:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DE8E028DA2 for ; Fri, 26 Apr 2019 07:32:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6826D6B0269; Fri, 26 Apr 2019 03:31:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 26CE26B026A; Fri, 26 Apr 2019 03:31:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D97366B000E; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 513D46B0010 for ; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id x2so1500328pge.16 for ; Fri, 26 Apr 2019 00:31:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=fM02CdP3EHsEznyJ/zI3DVkvywjW2Y0W5QE/m/kGqS8=; b=Pb9cpKScgKGeExDhe8wM+s9431ECWwdnTjH1NEIgCeNfXY9qUBzdDy++R77b3yyHeX utvYIw+Ki50tW9lI5yNDnG5WFd0onahoQJ6SMQ15+fpmwytShHELtUiO2H+f2KRozqWH oO43qlZyNqAfXvTiWkeVmmeog4ufQhn9nnAcMh/k1WEiC2z78Erm0+4D5nZdauZ2mD4r biyNqlvbQuNWc7MU9aLJinQd21vQQkpTgseoNqmzhhsJkX03Xgv9X6ikEofcWogYVlcc A6EbskLvAZ7W7OMSEau3y7u5FshVzlqAjpNFphoac+RI78f803khifaVsXW35Al1QPUu r/mQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAV3ZNUeN/xcqLCr4QhLGUPquY1A+TleLi6FZTUJNMolWHwqt/qx CHppnbkQ3aLHZ1tj6NjSE9m4HSFeTToKxPMTnRkJq12Ne29XygSB6IGnsG3HfPmyfbUF1Wm5f66 c2M2+KETCz5F/zLcUfLWc6aPCO+SRHVHo9EbZQ0sb6xOXui5EWCrNKrJGpLybvoO1Zg== X-Received: by 2002:a62:ae0f:: with SMTP id q15mr11015424pff.238.1556263907789; Fri, 26 Apr 2019 00:31:47 -0700 (PDT) X-Google-Smtp-Source: APXvYqyuMfgiag3mPiU52KnyusRS+KjA4Lrru8Asf0XkvhgJBGAScUyigznveRJDq331OSjmjgQR X-Received: by 2002:a62:ae0f:: with SMTP id q15mr11015319pff.238.1556263906333; Fri, 26 Apr 2019 00:31:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263906; cv=none; d=google.com; s=arc-20160816; b=X7fqCsVZzXTvqnqBrreUgnZOzgt0vPhWGOHlHLpg+Ov4KXJ+hkCC6X2fmxXHoAZYyI zyaKMjjUCWbOqL+NCgkDyMFvrEkUQCy8RwNFxQq3S/kJQ/AvzptwCLS/HnO0gnDrHO3W +zqoBk+WoOcMXYfTlgsJ2w7JtSYMqysEPCN8HdzAywLRhnhJZ4FdlT5T/8ol7u/kBVnx XMtHsqYxOhz3bsEU9Y5N0IgWygoVB6W92hOKNOHNlNo17yDVbhQKRLfyrVT4f6hnAcF5 /QeXoK4Ijv6l6MlhUSK2mlrgtDZKGdzJK2RNHvVoxzTBrDAwWYn6bTkPSoUgX0oYwFNP wy8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=fM02CdP3EHsEznyJ/zI3DVkvywjW2Y0W5QE/m/kGqS8=; b=NlZqRKQ2gedf0EuJ+EDu6zGjOXPjra4SOAmMDb3v2ANvBy/hxii3DmRXaOGOd92kvn l5NUhacyPGzojWcHcHFYq8qEuuWoz7P/qmDm6cDlSayjDtKBFOCVPS/BqWLJls7pTBdc NZN4ZrBvTpusNPwhzx071jGMEZhRFQfwmSC3qfU/mCU2ogs1As4508p6QHpMdbbRrFI+ 4bVDx+whr02iPLsxSoffolXHy0f2MHg3YrNuVM0ePE8jn1ubtHRxjjbdL/z5bFo3xCAc khNdnUEIGQ4R8aNvK8KDclMzLXHCu+uXTWJtfgHl6U8qFRTmF5M1k/7Fi0L/dFCx0bhF CKjw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-002.vmware.com (ex13-edg-ou-002.vmware.com. [208.91.0.190]) by mx.google.com with ESMTPS id f9si22844507pgq.347.2019.04.26.00.31.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:46 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) client-ip=208.91.0.190; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-002.vmware.com (10.113.208.156) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:44 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id 77C0D412A1; Fri, 26 Apr 2019 00:31:45 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe , Kees Cook , Dave Hansen , Nadav Amit Subject: [PATCH v5 03/23] x86/mm: Introduce temporary mm structs Date: Thu, 25 Apr 2019 17:11:23 -0700 Message-ID: <20190426001143.4983-4-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-002.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Andy Lutomirski Using a dedicated page-table for temporary PTEs prevents other cores from using - even speculatively - these PTEs, thereby providing two benefits: (1) Security hardening: an attacker that gains kernel memory writing abilities cannot easily overwrite sensitive data. (2) Avoiding TLB shootdowns: the PTEs do not need to be flushed in remote page-tables. To do so a temporary mm_struct can be used. Mappings which are private for this mm can be set in the userspace part of the address-space. During the whole time in which the temporary mm is loaded, interrupts must be disabled. The first use-case for temporary mm struct, which will follow, is for poking the kernel text. [ Commit message was written by Nadav Amit ] Cc: Kees Cook Cc: Dave Hansen Acked-by: Peter Zijlstra (Intel) Reviewed-by: Masami Hiramatsu Tested-by: Masami Hiramatsu Signed-off-by: Andy Lutomirski Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/mmu_context.h | 33 ++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 19d18fae6ec6..24dc3b810970 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -356,4 +356,37 @@ static inline unsigned long __get_current_cr3_fast(void) return cr3; } +typedef struct { + struct mm_struct *mm; +} temp_mm_state_t; + +/* + * Using a temporary mm allows to set temporary mappings that are not accessible + * by other CPUs. Such mappings are needed to perform sensitive memory writes + * that override the kernel memory protections (e.g., W^X), without exposing the + * temporary page-table mappings that are required for these write operations to + * other CPUs. Using a temporary mm also allows to avoid TLB shootdowns when the + * mapping is torn down. + * + * Context: The temporary mm needs to be used exclusively by a single core. To + * harden security IRQs must be disabled while the temporary mm is + * loaded, thereby preventing interrupt handler bugs from overriding + * the kernel memory protection. + */ +static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm) +{ + temp_mm_state_t temp_state; + + lockdep_assert_irqs_disabled(); + temp_state.mm = this_cpu_read(cpu_tlbstate.loaded_mm); + switch_mm_irqs_off(NULL, mm, current); + return temp_state; +} + +static inline void unuse_temporary_mm(temp_mm_state_t prev_state) +{ + lockdep_assert_irqs_disabled(); + switch_mm_irqs_off(NULL, prev_state.mm, current); +} + #endif /* _ASM_X86_MMU_CONTEXT_H */ From patchwork Fri Apr 26 00:11:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918549 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8FD721575 for ; Fri, 26 Apr 2019 07:32:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7D14628DA2 for ; Fri, 26 Apr 2019 07:32:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 70C8828DA3; Fri, 26 Apr 2019 07:32:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 092E428DA5 for ; Fri, 26 Apr 2019 07:32:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 073686B000C; Fri, 26 Apr 2019 03:31:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CEA7E6B000A; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F6726B026A; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id 315BF6B000C for ; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) Received: by mail-pl1-f200.google.com with SMTP id q18so1432146pll.16 for ; Fri, 26 Apr 2019 00:31:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=lPFAvintZAuLmImKiSUYUbggO4ykcKqKJ1oG7dcZTxU=; b=HHgNfHFhAoLdAr3SKt37HnqB/9lcaHVSzqd8LzwmV/GTkwe5O+ayKnd7///CdeMOJC 1ryehLSHygFOyORDjzVO1mLe5znAcsKuxqjSMGRoZAtnVbCNL4hxAiWHXHZDQiHViZlT GDf0AYphHCga//SQCvctgBqsYvevpJXVL0pK9MagErC1cGCBlACD+O4v3ckNLhr7Gs9T VmhfH54kh/rUx7kn35yO4dkVWBqa0pIrgYC1DpWNKCcMBDzFUBpX9J5TGUt5gASpBqB8 tyBWDU60i+IM6SMjHUC/PMBiXU8e/h8VfFw5lIZkX5PeyjOFv1ycCkrIf5s7iw+5Qc+C Cprg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAWjhZoNDcAsfmVQ1+UukvoLzxDAtghtOuAtnkajEVIpDnlRCf3+ C+G5aKKw722Oa1AzXxBGxYBAZWVkNGTZIIqEaX1y6rnrNhgB8RDOlR3UCUb/rxsA4egGLe36o5k G1ZXTrEO+PVgQfymRvMiVxLC7bjwr8fQcCliniu3Jj/klGLBy26+DX8sTtoEnmRTCXQ== X-Received: by 2002:a17:902:aa5:: with SMTP id 34mr28025510plp.263.1556263907850; Fri, 26 Apr 2019 00:31:47 -0700 (PDT) X-Google-Smtp-Source: APXvYqztm1mA+e0hrvuPyBCBLNWHEqQ/QxOuHtXlrzVy7LIN1TYhiXRkOaIvmAcKI19HXKDSThy5 X-Received: by 2002:a17:902:aa5:: with SMTP id 34mr28025419plp.263.1556263906517; Fri, 26 Apr 2019 00:31:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263906; cv=none; d=google.com; s=arc-20160816; b=0jRm64a4fzGvklc8M8uxK9kFW9LGqPotcwx1LIjUKwJCgR2d/rlWZWbTdcNYNj3ZPO y+MGjDfuu2KW1MO6bnSaNveFWI+mhWeR17/7Thp9//cUyfMpo3SfhoFFR8K7VOTnM1/7 BlR+7Mfc3cN0TbBPqZPycN4x3tvXTUwM/jQt5vJV2W5iZMM829eu/KpqHiA+ptX5bGkA Naz5d+h1Nf1Rvk4hXxQGoWz+eVZN95YNBWn18kZWIzUYFR/yTworgtsOx/RxAfqs4DTA asSH9ZYd9GDauNCNuIt4fjw++lS/d+W07Iv4pG8mQrrOAaSZcemMSQZEU48OBPFJ+zLV Ceag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=lPFAvintZAuLmImKiSUYUbggO4ykcKqKJ1oG7dcZTxU=; b=Ld/w86recnOmR8GKX/BbuVn65Az4GHfbDwPd05roEBn5cYKzohWXQjccdXaHUcEdUl POTGtskxlyrYrlbgJ4wqhz9r6WnYDpoPdZZFPMzT+J9KYE18jwqhwc50aamMpIDXQx1R SJL77CdsuvbtH3vEXSt9QajWbYnVcQ82ASScK5V73CR8zNozCBkfPlpubn7DbInIgAm8 f6bONOE1vk/lN0q6+KdGMcBBUEC15TrmIOSDUryUCIp2ws6bo0aggJEDROKHuD8Pqfbh Tg6F1xrX7z/AswAH9XiAOKHlCSML5A20tV5kgTXtP8f6PjynSAmIAgdqHLzukMoMipOg jHbQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-002.vmware.com (ex13-edg-ou-002.vmware.com. [208.91.0.190]) by mx.google.com with ESMTPS id f9si22844507pgq.347.2019.04.26.00.31.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:46 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) client-ip=208.91.0.190; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-002.vmware.com (10.113.208.156) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:44 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id 8042F412A4; Fri, 26 Apr 2019 00:31:45 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe , Nadav Amit Subject: [PATCH v5 04/23] x86/mm: Save debug registers when loading a temporary mm Date: Thu, 25 Apr 2019 17:11:24 -0700 Message-ID: <20190426001143.4983-5-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-002.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Prevent user watchpoints from mistakenly firing while the temporary mm is being used. As the addresses of the temporary mm might overlap those of the user-process, this is necessary to prevent wrong signals or worse things from happening. Cc: Andy Lutomirski Acked-by: Peter Zijlstra (Intel) Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/mmu_context.h | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 24dc3b810970..93dff1963337 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -13,6 +13,7 @@ #include #include #include +#include extern atomic64_t last_mm_ctx_id; @@ -380,6 +381,21 @@ static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm) lockdep_assert_irqs_disabled(); temp_state.mm = this_cpu_read(cpu_tlbstate.loaded_mm); switch_mm_irqs_off(NULL, mm, current); + + /* + * If breakpoints are enabled, disable them while the temporary mm is + * used. Userspace might set up watchpoints on addresses that are used + * in the temporary mm, which would lead to wrong signals being sent or + * crashes. + * + * Note that breakpoints are not disabled selectively, which also causes + * kernel breakpoints (e.g., perf's) to be disabled. This might be + * undesirable, but still seems reasonable as the code that runs in the + * temporary mm should be short. + */ + if (hw_breakpoint_active()) + hw_breakpoint_disable(); + return temp_state; } @@ -387,6 +403,13 @@ static inline void unuse_temporary_mm(temp_mm_state_t prev_state) { lockdep_assert_irqs_disabled(); switch_mm_irqs_off(NULL, prev_state.mm, current); + + /* + * Restore the breakpoints if they were disabled before the temporary mm + * was loaded. + */ + if (hw_breakpoint_active()) + hw_breakpoint_restore(); } #endif /* _ASM_X86_MMU_CONTEXT_H */ From patchwork Fri Apr 26 00:11:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918551 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 934F214C0 for ; Fri, 26 Apr 2019 07:32:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8128928DA2 for ; Fri, 26 Apr 2019 07:32:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 74FEC28DA4; Fri, 26 Apr 2019 07:32:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 029C228DA2 for ; Fri, 26 Apr 2019 07:32:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3259A6B000A; Fri, 26 Apr 2019 03:31:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DFD026B0269; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB7916B000C; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 472196B000E for ; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id g1so1673794pfo.2 for ; Fri, 26 Apr 2019 00:31:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=7Md7xvkom3K5hLhPxiS37AK9ZnciGjjzxPUMmpNJC18=; b=dfAmGzxZnius6IVGxWILeFsDz7y1mRrqxIqYnmkstAfhLUZnGsqYw4pbkx6swqG2yr tXGvxgmdiYlwh/x7oOOSav/acKRxfPYevHihkQ8urFpn5WfkjlqVyMrBK7hU4RfRxmMW F2IZUokXQhavOmCVhiXD5mOJLOEJ5prhIspemcGGnttlB5bCGyiT//7NMJxXspptNe/y UYvdsTXMP2EPOJpBgwUrWzAKBBz+z5yDoDTS7yzq+Nal1zzppFrrXZKNrPWINqhKRVx2 hXBXzHECEyyQ+b0WZKZYDjqzt83wFrmTYe5h38/ShCsbp5YBh+mX/nX5c2yReAmw+dea 3erg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAXD+x1Lmr237k0v95q+/Qr8P4S9sxYHgCZK0TvT6deLRKWVQw2c UJkI1hN6Agu+tLAdONqXzOOPthQ6Rhv0hEWfFAiqwu5hL3QkazylbUKA14Zua+gIJM31CbVntDa TOLBEHo19nyc1HvCKVbo9D4rLxbNpcixl/xi+6HeHxQpw7GZGcQ7Jtx1/k0IcPT4YHw== X-Received: by 2002:a17:902:5a0a:: with SMTP id q10mr4449803pli.101.1556263907926; Fri, 26 Apr 2019 00:31:47 -0700 (PDT) X-Google-Smtp-Source: APXvYqyehdSxn3q/nvH7dd3PEarUeZL+IdOWjQTIoTRM2NDXoHrtcO6ehnODm8K9bFAb3Oc22FMR X-Received: by 2002:a17:902:5a0a:: with SMTP id q10mr4449709pli.101.1556263906766; Fri, 26 Apr 2019 00:31:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263906; cv=none; d=google.com; s=arc-20160816; b=VnXSr0jo79lSwDDC7yk+JCojLj/m/1XHBRlkIsU/CJ5R5yAmpUA9EjabbdOH4bmiIJ t3ZRDrMHfzoqMWK04Pi3/Sg+Kc9lhWPw1QU27DHMQfAA1PAoZ7Q3xOq+R/FhKiIDVMMk IpHtrqvHEhS52IA0mOgKIkaxs8i8cuogxIzpZIUtvKpvHpDAYK+3ZfzTT5XugfIas9pP EHK+YMVANZY9gsd/glHnQCIvljMHkUvDnpjv4pr2vlDUvcFVo9zU60240d2U4mRuIARC xTBA4OsvVFFf3FUinA5A5ZkX+zhf5QcTlzkAfn+eIt2vWO80rcGmfxCcKycpG3LIUIhA kpUQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=7Md7xvkom3K5hLhPxiS37AK9ZnciGjjzxPUMmpNJC18=; b=WNcoU9sr/guP2j3BE8H7jynGJUCwxvrOXjwtqol5w9viBNBURFkiCZSsvB7u1wr0xO hlp/WQC9L3+HR3YKdO3prSy1Q1M7tatW72cm15WyJnT3mYQzeDE+r7qULttB9dJcHv9T iDItOUqPtvhqW/XBVd4kdcbqiJ0TjYF2S+anD56ZnKAP+0/k9zF8bQiwkf4mjmxQtE6B 98fE5v37kRawtJ0QdL4jV4FEODmxzgA13hnLT0clAV2cQe9vUOziayPgxab+v4w2V5T/ J++EBBBNiN560XOFSLW7abJIYX8IqCLfssccUHAixFhLpoEdwFxYRrleKUIe1uoaTfym iVDQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-002.vmware.com (ex13-edg-ou-002.vmware.com. [208.91.0.190]) by mx.google.com with ESMTPS id f9si22844507pgq.347.2019.04.26.00.31.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:46 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) client-ip=208.91.0.190; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-002.vmware.com (10.113.208.156) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:44 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id 87E8C41298; Fri, 26 Apr 2019 00:31:45 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe , Nadav Amit , Kees Cook , Dave Hansen Subject: [PATCH v5 05/23] fork: Provide a function for copying init_mm Date: Thu, 25 Apr 2019 17:11:25 -0700 Message-ID: <20190426001143.4983-6-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-002.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Provide a function for copying init_mm. This function will be later used for setting a temporary mm. Cc: Andy Lutomirski Cc: Kees Cook Cc: Dave Hansen Acked-by: Peter Zijlstra (Intel) Reviewed-by: Masami Hiramatsu Tested-by: Masami Hiramatsu Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- include/linux/sched/task.h | 1 + kernel/fork.c | 24 ++++++++++++++++++------ 2 files changed, 19 insertions(+), 6 deletions(-) diff --git a/include/linux/sched/task.h b/include/linux/sched/task.h index 2e97a2227045..f1227f2c38a4 100644 --- a/include/linux/sched/task.h +++ b/include/linux/sched/task.h @@ -76,6 +76,7 @@ extern void exit_itimers(struct signal_struct *); extern long _do_fork(unsigned long, unsigned long, unsigned long, int __user *, int __user *, unsigned long); extern long do_fork(unsigned long, unsigned long, unsigned long, int __user *, int __user *); struct task_struct *fork_idle(int); +struct mm_struct *copy_init_mm(void); extern pid_t kernel_thread(int (*fn)(void *), void *arg, unsigned long flags); extern long kernel_wait4(pid_t, int __user *, int, struct rusage *); diff --git a/kernel/fork.c b/kernel/fork.c index 9dcd18aa210b..099cca8f701c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1298,13 +1298,20 @@ void mm_release(struct task_struct *tsk, struct mm_struct *mm) complete_vfork_done(tsk); } -/* - * Allocate a new mm structure and copy contents from the - * mm structure of the passed in task structure. +/** + * dup_mm() - duplicates an existing mm structure + * @tsk: the task_struct with which the new mm will be associated. + * @oldmm: the mm to duplicate. + * + * Allocates a new mm structure and duplicates the provided @oldmm structure + * content into it. + * + * Return: the duplicated mm or NULL on failure. */ -static struct mm_struct *dup_mm(struct task_struct *tsk) +static struct mm_struct *dup_mm(struct task_struct *tsk, + struct mm_struct *oldmm) { - struct mm_struct *mm, *oldmm = current->mm; + struct mm_struct *mm; int err; mm = allocate_mm(); @@ -1371,7 +1378,7 @@ static int copy_mm(unsigned long clone_flags, struct task_struct *tsk) } retval = -ENOMEM; - mm = dup_mm(tsk); + mm = dup_mm(tsk, current->mm); if (!mm) goto fail_nomem; @@ -2186,6 +2193,11 @@ struct task_struct *fork_idle(int cpu) return task; } +struct mm_struct *copy_init_mm(void) +{ + return dup_mm(NULL, &init_mm); +} + /* * Ok, this is the main fork-routine. * From patchwork Fri Apr 26 00:11:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918557 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 78CA892A for ; Fri, 26 Apr 2019 07:32:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6640628DA3 for ; Fri, 26 Apr 2019 07:32:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5ABEA28DA6; Fri, 26 Apr 2019 07:32:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B82F128DA3 for ; Fri, 26 Apr 2019 07:32:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D84F6B0010; Fri, 26 Apr 2019 03:31:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 548EC6B0266; Fri, 26 Apr 2019 03:31:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F10606B0007; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 5DF306B0266 for ; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id gn10so1419648plb.23 for ; Fri, 26 Apr 2019 00:31:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=5xl4Z+yngPU6NzI5z0NBRpJ19Qz003jNFX+DruJR6U8=; b=kx/B0zzjDsndlf3LhYFY4NAULp3WkhtqF5xb3vd6W8iReBcDAHmVlnjmQv9i0HX5Fs OzEHiLheYacVLSjB4ENyRRyFwTlQivGyOxpomKFBUwR/K2DuUL1dR8PN91PIJZaZuyX0 +BkE0j54T363t5j5QwdIZEPcuzVjGFaDdW5BK6H7NLoFJRSmlyTHKXoQ5xNVQix8MY4U 3jmiIXZlSlLoO8qyi6Be9DLB2Lz2mKRGx2mhBZhcz+eup+pZe8NEa2xbAeTHGyxF5IIN DQ6/wtTamFm7Yo/+7tej8k1WLGxObgm4k6s5NWLEfDxe2Nq48yHxvRmhmnUO38drCOHD pGTQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAXo7zNQGAgfI1HMYyoPGSdlmw+Znu9s0JcH2U3wg1oDA6h8aKa6 WrhApG27aChojrM62UTcsXSirdeCMg1w4n+895QGa1MGEhCUGUs3drwOgObIxonZ9aNL8YAmdyv knM4DAExk5YLsxRfokWLako0JxMW2AZ29bAOaxz6408GSL3qfZGstAbGsTwpAZJa4Bw== X-Received: by 2002:a63:165f:: with SMTP id 31mr42513483pgw.321.1556263908018; Fri, 26 Apr 2019 00:31:48 -0700 (PDT) X-Google-Smtp-Source: APXvYqyVOtrK1i0pipILPqMhFEQmCjpa+TbvBAnimyaEyuVMkLFQTqWJ/sOwiN6NR4rQs4T4X9tn X-Received: by 2002:a63:165f:: with SMTP id 31mr42513406pgw.321.1556263906949; Fri, 26 Apr 2019 00:31:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263906; cv=none; d=google.com; s=arc-20160816; b=gKQ4KMd60FJx/6DkYaiW08NUqYyvcdv7tDpo16TADzJNv5Lgaw1jzyjlmZgUzWgn+P 9tDFj5IEqbApACWS1PhZ1djzgKyzJNTIpa2NMPgaIN/P77orJVhf7MaQHZ5T0qrp+zB0 g73G/bPlyHNwCag5Rw88ggF8VrWcaDEzc3k4T7N4LS+rmuKIu8ION6MFpmljdkuV1O2r GO2j5+GgjbvVuxZQRBb+9d59CjBedLaoCzawESU2xTXpje12I5ovh9nlzKyN1zRzkRUe 3p3egSbzMheoW3a+Idnkz2wZHaWGVe9ak8XAdFM5PIk6oAl+02nkZxpcQqc8RACUG38s OHoQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=5xl4Z+yngPU6NzI5z0NBRpJ19Qz003jNFX+DruJR6U8=; b=XY0GAgxs/3ce+7GnFy/NnxC2zn9QE9udZzbYviqIsIEIOF2FMBHkiwK7L/0ESrxUI9 XszRATxRSCyfA5MF8djTlIkhxFce3w0F0WTw4prJnXHWNeFf3O8BBxMY4bZ5g1mqaUf8 EGhilQeZtsymo3nWMfLHfRu1h7//VW6rBUOJPBUUfBcORJQTI0Z2HyW0ozecaATO+TZm GcGW36/u3b27OloBbbcf5Qctd7hVzMPJ9ZEQG+3ufOK+1AGju1wtTg61ldTEnUiMHngV myx/sQHPH8MgE1s50MNB6DIbiMvS8TsWNfijVfqTXmnusaNt5MvPnBUX44+5/lNyXpuB 6JoQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-002.vmware.com (ex13-edg-ou-002.vmware.com. [208.91.0.190]) by mx.google.com with ESMTPS id f9si22844507pgq.347.2019.04.26.00.31.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:46 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) client-ip=208.91.0.190; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-002.vmware.com (10.113.208.156) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:44 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id 9016541299; Fri, 26 Apr 2019 00:31:45 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe , Nadav Amit , Kees Cook , Dave Hansen Subject: [PATCH v5 06/23] x86/alternative: Initialize temporary mm for patching Date: Thu, 25 Apr 2019 17:11:26 -0700 Message-ID: <20190426001143.4983-7-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-002.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP To prevent improper use of the PTEs that are used for text patching, the next patches will use a temporary mm struct. Initailize it by copying the init mm. The address that will be used for patching is taken from the lower area that is usually used for the task memory. Doing so prevents the need to frequently synchronize the temporary-mm (e.g., when BPF programs are installed), since different PGDs are used for the task memory. Finally, randomize the address of the PTEs to harden against exploits that use these PTEs. Cc: Kees Cook Cc: Dave Hansen Acked-by: Peter Zijlstra (Intel) Reviewed-by: Masami Hiramatsu Tested-by: Masami Hiramatsu Suggested-by: Andy Lutomirski Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/pgtable.h | 3 +++ arch/x86/include/asm/text-patching.h | 2 ++ arch/x86/kernel/alternative.c | 3 +++ arch/x86/mm/init_64.c | 36 ++++++++++++++++++++++++++++ init/main.c | 3 +++ 5 files changed, 47 insertions(+) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 5cfbbb6d458d..6b6bfdfe83aa 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1038,6 +1038,9 @@ static inline void __meminit init_trampoline_default(void) /* Default trampoline pgd value */ trampoline_pgd_entry = init_top_pgt[pgd_index(__PAGE_OFFSET)]; } + +void __init poking_init(void); + # ifdef CONFIG_RANDOMIZE_MEMORY void __meminit init_trampoline(void); # else diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h index f8fc8e86cf01..a75eed841eed 100644 --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -39,5 +39,7 @@ extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len); extern int poke_int3_handler(struct pt_regs *regs); extern void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler); extern int after_bootmem; +extern __ro_after_init struct mm_struct *poking_mm; +extern __ro_after_init unsigned long poking_addr; #endif /* _ASM_X86_TEXT_PATCHING_H */ diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 0a814d73547a..11d5c710a94f 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -679,6 +679,9 @@ void *__init_or_module text_poke_early(void *addr, const void *opcode, return addr; } +__ro_after_init struct mm_struct *poking_mm; +__ro_after_init unsigned long poking_addr; + static void *__text_poke(void *addr, const void *opcode, size_t len) { unsigned long flags; diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index bccff68e3267..125c8c48aa24 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -53,6 +53,7 @@ #include #include #include +#include #include "mm_internal.h" @@ -1383,6 +1384,41 @@ unsigned long memory_block_size_bytes(void) return memory_block_size_probed; } +/* + * Initialize an mm_struct to be used during poking and a pointer to be used + * during patching. + */ +void __init poking_init(void) +{ + spinlock_t *ptl; + pte_t *ptep; + + poking_mm = copy_init_mm(); + BUG_ON(!poking_mm); + + /* + * Randomize the poking address, but make sure that the following page + * will be mapped at the same PMD. We need 2 pages, so find space for 3, + * and adjust the address if the PMD ends after the first one. + */ + poking_addr = TASK_UNMAPPED_BASE; + if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) + poking_addr += (kaslr_get_random_long("Poking") & PAGE_MASK) % + (TASK_SIZE - TASK_UNMAPPED_BASE - 3 * PAGE_SIZE); + + if (((poking_addr + PAGE_SIZE) & ~PMD_MASK) == 0) + poking_addr += PAGE_SIZE; + + /* + * We need to trigger the allocation of the page-tables that will be + * needed for poking now. Later, poking may be performed in an atomic + * section, which might cause allocation to fail. + */ + ptep = get_locked_pte(poking_mm, poking_addr, &ptl); + BUG_ON(!ptep); + pte_unmap_unlock(ptep, ptl); +} + #ifdef CONFIG_SPARSEMEM_VMEMMAP /* * Initialise the sparsemem vmemmap using huge-pages at the PMD level. diff --git a/init/main.c b/init/main.c index 598e278b46f7..949eed8015ec 100644 --- a/init/main.c +++ b/init/main.c @@ -504,6 +504,8 @@ void __init __weak thread_stack_cache_init(void) void __init __weak mem_encrypt_init(void) { } +void __init __weak poking_init(void) { } + bool initcall_debug; core_param(initcall_debug, initcall_debug, bool, 0644); @@ -737,6 +739,7 @@ asmlinkage __visible void __init start_kernel(void) taskstats_init_early(); delayacct_init(); + poking_init(); check_bugs(); acpi_subsystem_init(); From patchwork Fri Apr 26 00:11:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918543 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 85FD692A for ; Fri, 26 Apr 2019 07:31:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7285528DA2 for ; Fri, 26 Apr 2019 07:31:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 65CFC28DA4; Fri, 26 Apr 2019 07:31:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A476528DA2 for ; Fri, 26 Apr 2019 07:31:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AE68B6B0008; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A01736B0007; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 55CC06B000A; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) by kanga.kvack.org (Postfix) with ESMTP id 17FDE6B000A for ; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) Received: by mail-pg1-f198.google.com with SMTP id b37so1494087pgl.19 for ; Fri, 26 Apr 2019 00:31:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=mKLi4shgLz2mH8MLU6WsCSQGo+jrfYOaZJ1RA9T3nOs=; b=k4cE6ukTyLDOBMndqUL7D4nl0kz/wwZzMqUJa7pr6lwydCdMPBFC9kbYhoCVJkzxuF MlGx+ogmVi7J3VU73GSCDGMGMvKYvu8CLYEdWa0wXMBWJlzpqcixdWhqsUB8nxJfE7fm fr4J0n6gaSVtuv05+LgU1TllbJu2mv9DSIcqbeWBoNadHCIbDfKDDJ8Z3Jw1qdr5I8/P l7ghsnUJWu+DZOJB9Z8sBRF8iPFiaWm5yzRtHAGi+JLOC3Rt8UNzYPPq3CSYUZpQ8bx3 WAIzZF7tjyHjWLNKNsRETF+gSZj1b4Nelm7nUttpVhO9fcATRalEDoeuBCdhtv/k2L2U fgkg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAU90gzHelch4L/k5B5T5FavOc1SDS/I6h8ARiIr22KIuPfk2M6H eMsL7PXOjbgtKZQsZGR4gpd1Hoeuo+v3kkEakFvYfICR/SM5ICy/+hd/hed1cANbHGbJKlammM5 tuuGmsoovU7zymUjJFa5E3C4iimSypmlI0WhT4miQ7qW/llmLYK6k12XVJWI150OM6A== X-Received: by 2002:aa7:8b4c:: with SMTP id i12mr26951793pfd.189.1556263907706; Fri, 26 Apr 2019 00:31:47 -0700 (PDT) X-Google-Smtp-Source: APXvYqw4wZNoR/QXEW54erprBufNYxe0Ofntb31pVBWevhm1DARj4rBABWq3Uzu3JoPpSEGyzluy X-Received: by 2002:aa7:8b4c:: with SMTP id i12mr26951717pfd.189.1556263906621; Fri, 26 Apr 2019 00:31:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263906; cv=none; d=google.com; s=arc-20160816; b=0V7dTZOcRqhiyIQep8n70QsAkmFC2VDXZRFaNbIVCPL5zmm7D7xfrm0U21HgiBtaEs VRHBC328uw7NzcAPK0VsPdvwT6L49cfbqnP+lqabw7PoGpoRuBF6+EGu43D8Gu9nC85K rdFgzRzqXHtIQrFLx8HM6bcpIhZ/ug0WA/6XEviiW1VnY1tHKstctcBr/enYGs+1+a5G l3xjXvbHkI/4kZIVQ7x4lBNs2MUZuS/qCrhWlXbCKtJFh66PxezmQ6w4tgCEKRWKNMIM 3Bxp3QzFhXYNWVWXL1Xgy1EX3xIr7+BGKOllRmPhptUyHPSbtudLJW4kFc4Y09oJnC1F 4FTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=mKLi4shgLz2mH8MLU6WsCSQGo+jrfYOaZJ1RA9T3nOs=; b=VrKTjYqJnq5oxB0gUpYBqqFMH0lWDxvium0sZ/kuXm+/vDSh8CRweTOxf2p/Hz9bkt hUgzqQtU7HLjBALViILeQIkBg5DasoCQrJrUkO7pZ04dsKEey3eySNSgQpaJr49mkW72 qW8xWa6vxwLRNAxPElIpnej2tiR8c4t+1McFPuO4j+GMnUxDGdm7Cxd8ySq2CBL5b8/6 OL/fT0aTdUlkK0Vsown+bY428WRld7+RIvuHUvfmixS1Rsend0xi5xGR85PeBrauCdtS INyQcrBur3XhIdT8Gc8qNrN3WdkASwgG6kX0ivHoUyMrBlajjkaqCtyB6qQRrKbGqM8L WLAQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-001.vmware.com (ex13-edg-ou-001.vmware.com. [208.91.0.189]) by mx.google.com with ESMTPS id v82si25417769pfa.42.2019.04.26.00.31.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:46 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) client-ip=208.91.0.189; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-001.vmware.com (10.113.208.155) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:40 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id 9879D412A1; Fri, 26 Apr 2019 00:31:45 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe , Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu Subject: [PATCH v5 07/23] x86/alternative: Use temporary mm for text poking Date: Thu, 25 Apr 2019 17:11:27 -0700 Message-ID: <20190426001143.4983-8-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-001.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP text_poke() can potentially compromise security as it sets temporary PTEs in the fixmap. These PTEs might be used to rewrite the kernel code from other cores accidentally or maliciously, if an attacker gains the ability to write onto kernel memory. Moreover, since remote TLBs are not flushed after the temporary PTEs are removed, the time-window in which the code is writable is not limited if the fixmap PTEs - maliciously or accidentally - are cached in the TLB. To address these potential security hazards, use a temporary mm for patching the code. Finally, text_poke() is also not conservative enough when mapping pages, as it always tries to map 2 pages, even when a single one is sufficient. So try to be more conservative, and do not map more than needed. Cc: Andy Lutomirski Cc: Kees Cook Cc: Dave Hansen Cc: Masami Hiramatsu Acked-by: Peter Zijlstra (Intel) Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/fixmap.h | 2 - arch/x86/kernel/alternative.c | 108 +++++++++++++++++++++++++++------- arch/x86/xen/mmu_pv.c | 2 - 3 files changed, 86 insertions(+), 26 deletions(-) diff --git a/arch/x86/include/asm/fixmap.h b/arch/x86/include/asm/fixmap.h index 50ba74a34a37..9da8cccdf3fb 100644 --- a/arch/x86/include/asm/fixmap.h +++ b/arch/x86/include/asm/fixmap.h @@ -103,8 +103,6 @@ enum fixed_addresses { #ifdef CONFIG_PARAVIRT FIX_PARAVIRT_BOOTMAP, #endif - FIX_TEXT_POKE1, /* reserve 2 pages for text_poke() */ - FIX_TEXT_POKE0, /* first page is last, because allocation is backward */ #ifdef CONFIG_X86_INTEL_MID FIX_LNW_VRTC, #endif diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 11d5c710a94f..599203876c32 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -684,41 +685,104 @@ __ro_after_init unsigned long poking_addr; static void *__text_poke(void *addr, const void *opcode, size_t len) { + bool cross_page_boundary = offset_in_page(addr) + len > PAGE_SIZE; + struct page *pages[2] = {NULL}; + temp_mm_state_t prev; unsigned long flags; - char *vaddr; - struct page *pages[2]; - int i; + pte_t pte, *ptep; + spinlock_t *ptl; + pgprot_t pgprot; /* - * While boot memory allocator is runnig we cannot use struct - * pages as they are not yet initialized. + * While boot memory allocator is running we cannot use struct pages as + * they are not yet initialized. There is no way to recover. */ BUG_ON(!after_bootmem); if (!core_kernel_text((unsigned long)addr)) { pages[0] = vmalloc_to_page(addr); - pages[1] = vmalloc_to_page(addr + PAGE_SIZE); + if (cross_page_boundary) + pages[1] = vmalloc_to_page(addr + PAGE_SIZE); } else { pages[0] = virt_to_page(addr); WARN_ON(!PageReserved(pages[0])); - pages[1] = virt_to_page(addr + PAGE_SIZE); + if (cross_page_boundary) + pages[1] = virt_to_page(addr + PAGE_SIZE); } - BUG_ON(!pages[0]); + /* + * If something went wrong, crash and burn since recovery paths are not + * implemented. + */ + BUG_ON(!pages[0] || (cross_page_boundary && !pages[1])); + local_irq_save(flags); - set_fixmap(FIX_TEXT_POKE0, page_to_phys(pages[0])); - if (pages[1]) - set_fixmap(FIX_TEXT_POKE1, page_to_phys(pages[1])); - vaddr = (char *)fix_to_virt(FIX_TEXT_POKE0); - memcpy(&vaddr[(unsigned long)addr & ~PAGE_MASK], opcode, len); - clear_fixmap(FIX_TEXT_POKE0); - if (pages[1]) - clear_fixmap(FIX_TEXT_POKE1); - local_flush_tlb(); - sync_core(); - /* Could also do a CLFLUSH here to speed up CPU recovery; but - that causes hangs on some VIA CPUs. */ - for (i = 0; i < len; i++) - BUG_ON(((char *)addr)[i] != ((char *)opcode)[i]); + + /* + * Map the page without the global bit, as TLB flushing is done with + * flush_tlb_mm_range(), which is intended for non-global PTEs. + */ + pgprot = __pgprot(pgprot_val(PAGE_KERNEL) & ~_PAGE_GLOBAL); + + /* + * The lock is not really needed, but this allows to avoid open-coding. + */ + ptep = get_locked_pte(poking_mm, poking_addr, &ptl); + + /* + * This must not fail; preallocated in poking_init(). + */ + VM_BUG_ON(!ptep); + + pte = mk_pte(pages[0], pgprot); + set_pte_at(poking_mm, poking_addr, ptep, pte); + + if (cross_page_boundary) { + pte = mk_pte(pages[1], pgprot); + set_pte_at(poking_mm, poking_addr + PAGE_SIZE, ptep + 1, pte); + } + + /* + * Loading the temporary mm behaves as a compiler barrier, which + * guarantees that the PTE will be set at the time memcpy() is done. + */ + prev = use_temporary_mm(poking_mm); + + kasan_disable_current(); + memcpy((u8 *)poking_addr + offset_in_page(addr), opcode, len); + kasan_enable_current(); + + /* + * Ensure that the PTE is only cleared after the instructions of memcpy + * were issued by using a compiler barrier. + */ + barrier(); + + pte_clear(poking_mm, poking_addr, ptep); + if (cross_page_boundary) + pte_clear(poking_mm, poking_addr + PAGE_SIZE, ptep + 1); + + /* + * Loading the previous page-table hierarchy requires a serializing + * instruction that already allows the core to see the updated version. + * Xen-PV is assumed to serialize execution in a similar manner. + */ + unuse_temporary_mm(prev); + + /* + * Flushing the TLB might involve IPIs, which would require enabled + * IRQs, but not if the mm is not used, as it is in this point. + */ + flush_tlb_mm_range(poking_mm, poking_addr, poking_addr + + (cross_page_boundary ? 2 : 1) * PAGE_SIZE, + PAGE_SHIFT, false); + + /* + * If the text does not match what we just wrote then something is + * fundamentally screwy; there's nothing we can really do about that. + */ + BUG_ON(memcmp(addr, opcode, len)); + + pte_unmap_unlock(ptep, ptl); local_irq_restore(flags); return addr; } diff --git a/arch/x86/xen/mmu_pv.c b/arch/x86/xen/mmu_pv.c index a21e1734fc1f..beb44e22afdf 100644 --- a/arch/x86/xen/mmu_pv.c +++ b/arch/x86/xen/mmu_pv.c @@ -2318,8 +2318,6 @@ static void xen_set_fixmap(unsigned idx, phys_addr_t phys, pgprot_t prot) #elif defined(CONFIG_X86_VSYSCALL_EMULATION) case VSYSCALL_PAGE: #endif - case FIX_TEXT_POKE0: - case FIX_TEXT_POKE1: /* All local page mappings */ pte = pfn_pte(phys, prot); break; From patchwork Fri Apr 26 00:11:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918657 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 53A2F1575 for ; Fri, 26 Apr 2019 07:33:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 445A428DA2 for ; Fri, 26 Apr 2019 07:33:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 389F628DA5; Fri, 26 Apr 2019 07:33:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B2E9428DA2 for ; Fri, 26 Apr 2019 07:33:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC7206B0287; Fri, 26 Apr 2019 03:33:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B14A46B0289; Fri, 26 Apr 2019 03:33:23 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C7EB6B028A; Fri, 26 Apr 2019 03:33:23 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 4A74F6B0289 for ; Fri, 26 Apr 2019 03:33:23 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id x9so1464936pln.0 for ; Fri, 26 Apr 2019 00:33:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=DXoaQOfmRoLFd3xwEem3ey63J4Sb1Jdcc0n/TAChKh8=; b=YTA96CvEFQPhJEGCIBmYa59yc/G627Az7+q6Ot8zz79HGMRx6d/EkKCobSdHbu0YqD FjHccP2JTmLf4pXszqFLmMq3STjtCfOpOjaJ0tk5Vjq1+5IrXpG8wKRN12sgHBTeX9qn U0zWJcUxTw4oBiv9iauHbbL94N7fo8JgClQxFn5FsD3T82ClN4mEe6VQa+jWa2d0kf11 XWFNuHPWhUNeteCa5SY4/vYZhUDUludf/3f7MsIURdVq8fxoK5IGxSN5WrFXKNm8QIAz Vaeuf8ttOfuYXA+trBgssHBz81juMRTXAVuoGuVkjj+/n2gggwBnF9vsX+nPmzoSpoKh 1KtA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAXM8IFRw7S1/rUFx4Z1Q8qbAp1c70YKqFG0PqOmCMoD2Bo2jFvH WN9LXlWYqbQTdF8B6Ti6sNxA30XQa2EJnU9aiMEwWgcZxnkQsnNfpAFuqbakvA/wRFmqeFdgoVV 6ts+3wdlMp2XYHrLqAkNWoai5VE0LhHSEdK/fb9rda5KtCTbmshiD+jt9v3vVbV+KwA== X-Received: by 2002:a17:902:2a03:: with SMTP id i3mr45985430plb.229.1556264002960; Fri, 26 Apr 2019 00:33:22 -0700 (PDT) X-Google-Smtp-Source: APXvYqwGXzypBw6OltpnQmePY/F5N9S0sarW8Uq9pJwYGJF4jFFzlBdxgAKGtccB5xOrP+oye502 X-Received: by 2002:a17:902:2a03:: with SMTP id i3mr45977889plb.229.1556263907348; Fri, 26 Apr 2019 00:31:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263907; cv=none; d=google.com; s=arc-20160816; b=1AK0nzzi5hz/aRGlqTvNv+9UtzHGGZqnD57sdIvs128VdwopN69na5aLARZWvoZWKN mMOHmKw0kRqVAHG8XYzqWQir6ReGvnaM7+QngPtXXzmPpePz7X6OxMswa299n9IHotwx yyzAEAW9PvEBhfHIPmehpAyxtvoC2s2xZFdcggtS0O4s2BLbS05gRZqo4YRHMboTD7W+ Tk0wmm8KqnqTld0l6R0TY9lKU94RroSJcLVb3aLqnCqlVOxdtBaxXt7dRpzv4qItoV2z DI25A+pqKR8qAXFcJpjSWKQPuzF8ezNJMHcQmdLaTwHNynKhqm2PZqLI9KeoDcpF3C5d nQTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=DXoaQOfmRoLFd3xwEem3ey63J4Sb1Jdcc0n/TAChKh8=; b=C5qxUg88mrN0UGQdohRg0kXe15OBYycjGH+tAyFEIYqlpFlKjol2hHOUdk68W0OsfA QIHGM45BqGPVrhpazjWZ28D+LeTJW6w68Ge3t/bKHbm9Jqw/wYgwQP0Htv+BV4BcvsL7 fkBXBmJBTKCPAMPM69BhSLZhSj+mMFH+HuTib9dehSgAUU8m7D+kxz66RKrE5KQnYNVs q/0mwQploDAHajvUBeB40KejB5RWjWnRJqizAhNz81yUqX3fpaoZe64h992iSCoduska YkUp0eE+09ILcAoZKyUAcOL1BEA8195XGMfuWbrO6H9A1PNeCbgG2963Qzt6cTgP2ffO 6f9w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-002.vmware.com (ex13-edg-ou-002.vmware.com. [208.91.0.190]) by mx.google.com with ESMTPS id f9si22844507pgq.347.2019.04.26.00.31.47 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:47 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) client-ip=208.91.0.190; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-002.vmware.com (10.113.208.156) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:44 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id A1013412A3; Fri, 26 Apr 2019 00:31:45 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe , Nadav Amit Subject: [PATCH v5 08/23] x86/kgdb: Avoid redundant comparison of patched code Date: Thu, 25 Apr 2019 17:11:28 -0700 Message-ID: <20190426001143.4983-9-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-002.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP text_poke() already ensures that the written value is the correct one and fails if that is not the case. There is no need for an additional comparison. Remove it. Acked-by: Peter Zijlstra (Intel) Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/kgdb.c | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/arch/x86/kernel/kgdb.c b/arch/x86/kernel/kgdb.c index 2b203ee5b879..13b13311b792 100644 --- a/arch/x86/kernel/kgdb.c +++ b/arch/x86/kernel/kgdb.c @@ -747,7 +747,6 @@ void kgdb_arch_set_pc(struct pt_regs *regs, unsigned long ip) int kgdb_arch_set_breakpoint(struct kgdb_bkpt *bpt) { int err; - char opc[BREAK_INSTR_SIZE]; bpt->type = BP_BREAKPOINT; err = probe_kernel_read(bpt->saved_instr, (char *)bpt->bpt_addr, @@ -766,11 +765,6 @@ int kgdb_arch_set_breakpoint(struct kgdb_bkpt *bpt) return -EBUSY; text_poke_kgdb((void *)bpt->bpt_addr, arch_kgdb_ops.gdb_bpt_instr, BREAK_INSTR_SIZE); - err = probe_kernel_read(opc, (char *)bpt->bpt_addr, BREAK_INSTR_SIZE); - if (err) - return err; - if (memcmp(opc, arch_kgdb_ops.gdb_bpt_instr, BREAK_INSTR_SIZE)) - return -EINVAL; bpt->type = BP_POKE_BREAKPOINT; return err; @@ -778,9 +772,6 @@ int kgdb_arch_set_breakpoint(struct kgdb_bkpt *bpt) int kgdb_arch_remove_breakpoint(struct kgdb_bkpt *bpt) { - int err; - char opc[BREAK_INSTR_SIZE]; - if (bpt->type != BP_POKE_BREAKPOINT) goto knl_write; /* @@ -791,10 +782,7 @@ int kgdb_arch_remove_breakpoint(struct kgdb_bkpt *bpt) goto knl_write; text_poke_kgdb((void *)bpt->bpt_addr, bpt->saved_instr, BREAK_INSTR_SIZE); - err = probe_kernel_read(opc, (char *)bpt->bpt_addr, BREAK_INSTR_SIZE); - if (err || memcmp(opc, bpt->saved_instr, BREAK_INSTR_SIZE)) - goto knl_write; - return err; + return 0; knl_write: return probe_kernel_write((char *)bpt->bpt_addr, From patchwork Fri Apr 26 00:11:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918685 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DD98692A for ; Fri, 26 Apr 2019 07:33:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CDD2728DA2 for ; Fri, 26 Apr 2019 07:33:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C228828DA4; Fri, 26 Apr 2019 07:33:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6CE4628DA2 for ; Fri, 26 Apr 2019 07:33:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 71E1D6B028E; Fri, 26 Apr 2019 03:33:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6A33F6B028F; Fri, 26 Apr 2019 03:33:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 546C36B0290; Fri, 26 Apr 2019 03:33:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id 138EE6B028E for ; Fri, 26 Apr 2019 03:33:51 -0400 (EDT) Received: by mail-pl1-f197.google.com with SMTP id s19so1455964plp.6 for ; Fri, 26 Apr 2019 00:33:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=/dQyv1Uey2RUfadDTtULdd0YvB0m8yojiZj6sZU6y84=; b=J7L32BQ9CODFxE5CNXUT9jvgawizkQTbf/U0qcRk/XUQsZH+taUBHPgIYoudFzOXW8 Or4puIEvzJuSey6kEOPl944NJyaHcjJQYyquyPGfqA9cq6tCoKPjwqIvyBkbpTAi77Kx NKtKKxIZO+af3nFvA16/PITMLExmJGouMYzZOPpENn4Q29eA0xjcesEZrQiZxKILZPH6 6LmX4+omQoTLslO0ufSY1NsVWM3SFUCNhy47o/kUFKHDyNR0hLNpZSN6EEltCfHYwXtk ZParJBj8RXooeVSFuOtlrFddIAs38VvMErvnkc9crvIU/TwIhooGuVgHG2DSscfzd1Vw eoCg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAVmqx9h/aT6DXExjnzn1oB9w1BikGmOSzGwjAAgpq2+TcCwgPbJ bPoUzEd/7ItbUDEqGNAlErDOPGAOZhbrUKbsqjI6ayNvxO9MRxdaPD/Uw1j6nIDkMPGuDDyHvG3 2LhuLt0SJH9MwcC+u/YboIbR1k5YAUAyvrT9pIbFWcgxHWxE0k7mtJuol6DMzyCnKDQ== X-Received: by 2002:aa7:9f49:: with SMTP id h9mr20191985pfr.173.1556264030747; Fri, 26 Apr 2019 00:33:50 -0700 (PDT) X-Google-Smtp-Source: APXvYqyBh4afjxHQXL1ygESEfnhx5TzwyAbxhbgQuaYE28noD0rCnWjO1VeB2q+9ITSoQ/2TxXZg X-Received: by 2002:aa7:9f49:: with SMTP id h9mr20183121pfr.173.1556263907157; Fri, 26 Apr 2019 00:31:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263907; cv=none; d=google.com; s=arc-20160816; b=pJXQVspMVXqWiELxk/FL7J5YpqkrolCbZpCBQcI3On6lFue9L2TxqMw+nIxkWy37dh ChU52bpQq8J3jW6q6l4g0nLZxsbcPzS0t2reqCxCp/UmY060nBjqE9B3C2Bry3JpTUJZ t8hOfOcF4F6dxhvd+19cTj18uAtUnbukwKCLOgvWxomEoW2LE9Et2IMn1CpzLSHhsadc zcOt5aZhp/79YaSQ2uI7aaFmagb8Tw/nNZtCLvm3jQuxTYX6L+2nPmdWb8Voy0QFk443 ZWzI3Fov8tVLei3XoC0ix5FFi0js7f2ymozmILvgiUZVWMHfZ4xaYNgd3RWfilo/NAPw 9Ujw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=/dQyv1Uey2RUfadDTtULdd0YvB0m8yojiZj6sZU6y84=; b=AiWPWp5w3Nl1CuapYsuFAg2Tp5wmNmE5Ilzv+lyzCL3Dyn/YwM4Gr2G9S8Nn0y680G yTMQN2FIZ8mctVpL/FQtle6gNI71e7QQOGPhWkkAkmhoLC72m73HW0894Uxtka0xMZW4 vmXTRSywgbrQjQuYnXcm/4601mGfuCXtey5KJ4K597Q2m1yYjtqt+NMGBC0/vfPBpY7E koIS3JDxFxIweT90vmnuR10ScBuRr8Ma/gps61J0wDAcfE68LAojPLUoZtI8ir1Kr5IC AlHYq+y1O/TNjVy4GnVvx3d6c4EU682ap4WJehWIf1JqCAyAfQDwifEEgVn81dwhlEH2 2iCQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-002.vmware.com (ex13-edg-ou-002.vmware.com. [208.91.0.190]) by mx.google.com with ESMTPS id f9si22844507pgq.347.2019.04.26.00.31.47 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:47 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) client-ip=208.91.0.190; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-002.vmware.com (10.113.208.156) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:44 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id AA61841299; Fri, 26 Apr 2019 00:31:45 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe , Nadav Amit Subject: [PATCH v5 09/23] x86/ftrace: Set trampoline pages as executable Date: Thu, 25 Apr 2019 17:11:29 -0700 Message-ID: <20190426001143.4983-10-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-002.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Since alloc_module() will not set the pages as executable soon, set ftrace trampoline pages as executable after they are allocated. For the time being, do not change ftrace to use the text_poke() interface. As a result, ftrace still breaks W^X. Reviewed-by: Steven Rostedt (VMware) Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/ftrace.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index ef49517f6bb2..53ba1aa3a01f 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -730,6 +730,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) unsigned long end_offset; unsigned long op_offset; unsigned long offset; + unsigned long npages; unsigned long size; unsigned long retq; unsigned long *ptr; @@ -762,6 +763,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) return 0; *tramp_size = size + RET_SIZE + sizeof(void *); + npages = DIV_ROUND_UP(*tramp_size, PAGE_SIZE); /* Copy ftrace_caller onto the trampoline memory */ ret = probe_kernel_read(trampoline, (void *)start_offset, size); @@ -806,6 +808,12 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) /* ALLOC_TRAMP flags lets us know we created it */ ops->flags |= FTRACE_OPS_FL_ALLOC_TRAMP; + /* + * Module allocation needs to be completed by making the page + * executable. The page is still writable, which is a security hazard, + * but anyhow ftrace breaks W^X completely. + */ + set_memory_x((unsigned long)trampoline, npages); return (unsigned long)trampoline; fail: tramp_free(trampoline, *tramp_size); From patchwork Fri Apr 26 00:11:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918571 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 719901575 for ; Fri, 26 Apr 2019 07:32:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 61E8228DA6 for ; Fri, 26 Apr 2019 07:32:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5554528DA4; Fri, 26 Apr 2019 07:32:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D80BD28DA2 for ; Fri, 26 Apr 2019 07:32:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 53FB26B026C; Fri, 26 Apr 2019 03:31:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 064986B026E; Fri, 26 Apr 2019 03:31:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D03B56B0007; Fri, 26 Apr 2019 03:31:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 27D9B6B026C for ; Fri, 26 Apr 2019 03:31:49 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id 33so1491047pgv.17 for ; Fri, 26 Apr 2019 00:31:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=MM86D2MJFg2ajXMn2L8v3GMZHyeRYJVGp1yjvaMMFQk=; b=ge19kVNuSFKbnlrZD3nKcaQrN+zI+lUtdwiepaVDq9E8yf14HNC1sDYVtCM/N0Jxi4 PAb8msrts6fxqXPWDC0oAAjVqZYJHkXh3cSEHypKOw9dQktUxc/lQSZyC+VaO6uLq6f3 CsKCsdKh8Wa1pzM9VGCdaGX0eUhAIIe+QeaULIuJrXgoqP+8m1kQolJK6FCBnHFRPhRk QAEiIRuN4Z6x9tqVBe3lalU8JBeyY6NVaB2gRd22W/X9A43d72slOIcC0fxzluqyRj7N aGjTAdaLpKoYYI/7+ehBMnBzcicsN4uJkJ4FRPTF6uU2p9HuMxdDOVc7W1bvWGpaQV8l 11tw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAUqX4CMXRznn2cHITvHJ2eFsjEeQNW2VLVw0XgpGxl9PRptWkFr 2C/YsNiJo650/rtbwfQChEGM1ekdrenafrKZYs5L/VyD3zmXKYMpkG555PK3LQ8xq9E/TOWZVKZ h6uPWZx81JehA3yDuq6Pq1mb/IiA/kmXoZKULyfUcn9Bts5FwHWIK/m3gic4HAjYnkw== X-Received: by 2002:a63:cc0d:: with SMTP id x13mr41943541pgf.280.1556263908821; Fri, 26 Apr 2019 00:31:48 -0700 (PDT) X-Google-Smtp-Source: APXvYqwzAvntBPuqxEuZGkc97mHX+Mpe3DWsl5erVfwPmpCp8yBLbB9W7eTcAYiCS1ylrxDxyoBX X-Received: by 2002:a63:cc0d:: with SMTP id x13mr41943460pgf.280.1556263907568; Fri, 26 Apr 2019 00:31:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263907; cv=none; d=google.com; s=arc-20160816; b=rGwsTjz0RPXIGxonc+0CeTt8cO6lpcCGtU438yPWhR3Mpm7atDBf/uBC42Faw+6HX3 x+RhpVOBCPhTvH/ayc0x8ZVLk9MbxyNgZQahQCDHfNPTec01CGmnoPPwDpByBpqu4daL 7dQmTV8z9aiGdF3O70LAuxRo8NRqDGNhc/qRhGe4FL07ZGVCmc+cTTcKm/ZUJ8GhdFLg 0CZoc61zpTxqrfJOuukjDiq8bgGo2yG2fKPgnvQvf3gcsC1vSYb+NmrbXCqn1pYvC0Xw J0LfBusI7jpoxMdtIdDLHNgshQsi2Hz0FCCPDT03g4iTcfG3DNEcJ2NP3kpord5ZITER CBLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=MM86D2MJFg2ajXMn2L8v3GMZHyeRYJVGp1yjvaMMFQk=; b=IjjH7rR6QKoPCNQxHAZ024O2Nrt1L73kf6PRmeFctmm3L0rwkqR3lkh/RkcPh36f1q 3xjMGCklvi2yeHBz1oxnT7t3JEKl1S3aVgwyU7Ixgfl9bngGj+WiV9642hO4833wh+OW GgV1lmaWWr6TfgZI7U4tyQO3ZJfNmlnWhYJjrXYLc7s5U5TT7rozhUwcATZemdeCOz/l CVzpX7j9b1a1N9CQ44DlRkfTTfDn5U1PekGDISmzQl/oOiV0ZFoGuFOmIWkidha6gWFa QtpvwAbC+hJ/Zmx1edh0IOp2lC0hFyzV+JEp293AMdBPBdpbHDveUTA8gTJzy9ZHFkhh lkdA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-002.vmware.com (ex13-edg-ou-002.vmware.com. [208.91.0.190]) by mx.google.com with ESMTPS id f9si22844507pgq.347.2019.04.26.00.31.47 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:47 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) client-ip=208.91.0.190; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-002.vmware.com (10.113.208.156) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:44 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id B36F4412A2; Fri, 26 Apr 2019 00:31:45 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe , Nadav Amit Subject: [PATCH v5 10/23] x86/kprobes: Set instruction page as executable Date: Thu, 25 Apr 2019 17:11:30 -0700 Message-ID: <20190426001143.4983-11-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-002.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Set the page as executable after allocation. This patch is a preparatory patch for a following patch that makes module allocated pages non-executable. While at it, do some small cleanup of what appears to be unnecessary masking. Acked-by: Masami Hiramatsu Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/kprobes/core.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index a034cb808e7e..1591852d3ac4 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -431,8 +431,20 @@ void *alloc_insn_page(void) void *page; page = module_alloc(PAGE_SIZE); - if (page) - set_memory_ro((unsigned long)page & PAGE_MASK, 1); + if (!page) + return NULL; + + /* + * First make the page read-only, and only then make it executable to + * prevent it from being W+X in between. + */ + set_memory_ro((unsigned long)page, 1); + + /* + * TODO: Once additional kernel code protection mechanisms are set, ensure + * that the page was not maliciously altered and it is still zeroed. + */ + set_memory_x((unsigned long)page, 1); return page; } @@ -440,8 +452,12 @@ void *alloc_insn_page(void) /* Recover page to RW mode before releasing it */ void free_insn_page(void *page) { - set_memory_nx((unsigned long)page & PAGE_MASK, 1); - set_memory_rw((unsigned long)page & PAGE_MASK, 1); + /* + * First make the page non-executable, and only then make it writable to + * prevent it from being W+X in between. + */ + set_memory_nx((unsigned long)page, 1); + set_memory_rw((unsigned long)page, 1); module_memfree(page); } From patchwork Fri Apr 26 00:11:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918561 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4C94592A for ; Fri, 26 Apr 2019 07:32:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3BC8C28DA3 for ; Fri, 26 Apr 2019 07:32:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2FA4B28DA6; Fri, 26 Apr 2019 07:32:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9C08528DA3 for ; Fri, 26 Apr 2019 07:32:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D03C86B026B; Fri, 26 Apr 2019 03:31:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C4FB56B026A; Fri, 26 Apr 2019 03:31:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 85FCE6B026E; Fri, 26 Apr 2019 03:31:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id D2B566B0010 for ; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id z7so1526531pgc.1 for ; Fri, 26 Apr 2019 00:31:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=n/B3+n3f5xXPTebK3I1nT05hBod2ytL0DtnQ9iVLEbk=; b=bKN5d+IFg0nIRjryPbcikns5iRjBGXZAImSfB3CtK/1gG5gIowJvq518vxEvOWQ/tH 1Xbqxzwcb4PvntOTptzcAyYbD0EWNEOrmkUJIFM2qApXv7UJdtW+w0eHHWyt4IIgaPhC h7baicCZU9Hx4fBjurLfz/8b7dd/6KhE+HGRf0zXdDTgdqg+z154/ZCQU5CHjv58Y3Y9 NcqRse5xeF1UpF4p7dwjmP0SbV5SJBN1xDwdCJObjAUUaTo5SMiuLKHVll73IGsYujbx crXpDIyz8/89AypdJ/K/35fXc6whPn7yWd9WFTwBaAi1X01wyWq30Nvb8ABfp/cLXhOA 1+hw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAXiC/mmmtvaZK14zCwBlWaSOHyYTxr/aOSbQXOzlp9AMxWqL9oM VONQ3jg3Tg6se2ppH9HfIAqsWFApswdP3OuHe4V/NQ176V/wA1Z+ShEMRmcMIE/31wXTJh6QRCp KCNsc6vNowFqFAf60n6KAnPrTXuS60LvajIv95f9ENfcQ6TfFl2Z7pry2Z1whfwQ2dw== X-Received: by 2002:a62:1c87:: with SMTP id c129mr20611441pfc.113.1556263908314; Fri, 26 Apr 2019 00:31:48 -0700 (PDT) X-Google-Smtp-Source: APXvYqy6ewiujwcoLTCfMrT68M6wGAt72XmDo0iew7rIeeELWOI8LK2GigqHCTmDco6HGCqrKIga X-Received: by 2002:a62:1c87:: with SMTP id c129mr20611339pfc.113.1556263906881; Fri, 26 Apr 2019 00:31:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263906; cv=none; d=google.com; s=arc-20160816; b=LXd0VlzG8bQ5O+BLzGKX2l/DzDqlcwEI34mdMMbwUb8hd8sNDcrYr8l+ergr303UF4 Q6yKSitGvVstyjm8WX0pbdWqAYjZ79Yazn8EYk55nMTm/jQ+1Zi0bVgVvivsNI9htU7r ogY+JOnomGPzZcBsPDaQufhUY4UdtEKHVv1gzAREBM1bYQ6KFFnc/rSpqZdxoEi1JZgN EZ8eo3F7NJw+worhwAKlBxpfSJ0XFnpcEVgLE3lNCZry5k/PzCGhjPdECO3SeMxxTTTP bCJP4gtlBJ+v8Dvj2o2cKCMaxKp19fzr7dnC5UE74C1KyUIh83VC6zgIcEzEKKC3IdSb 7Dbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=n/B3+n3f5xXPTebK3I1nT05hBod2ytL0DtnQ9iVLEbk=; b=f3VSrroOqoE+2xZ0MFXNo52mS5hzz+AMZxkHEDlnPVNPbn2i1slRTakXEfTod2K9uW 10vVjgytxAs6qhhfEFD09A01w4GVY0JC5K/6xXdCE9qLu21t/UNkA7MyZ52FTJUA4jQg Qa4VHnKjCjvPH2Q+Juygzr+iFkKF5t7XUUYmX8g4hy63UNHynUVJg5iolTfEjn43s5lk r4RKBwbZUYy2ZbVDNrBNTZ8lCzQtyOG/HaEQyF7qszoc1SNWimxUD9YMgAafcIWYz+dt p1Q4YC0UpdlJYoAFAG81N6sG1Z3zhBJbmtPM3ENz6X76sGn9STeET4DOkc88qYktlc1E NyOA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-001.vmware.com (ex13-edg-ou-001.vmware.com. [208.91.0.189]) by mx.google.com with ESMTPS id v82si25417769pfa.42.2019.04.26.00.31.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:46 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) client-ip=208.91.0.189; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-001.vmware.com (10.113.208.155) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:40 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id BC4EC412A4; Fri, 26 Apr 2019 00:31:45 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe , Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu , Jessica Yu Subject: [PATCH v5 11/23] x86/module: Avoid breaking W^X while loading modules Date: Thu, 25 Apr 2019 17:11:31 -0700 Message-ID: <20190426001143.4983-12-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-001.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When modules and BPF filters are loaded, there is a time window in which some memory is both writable and executable. An attacker that has already found another vulnerability (e.g., a dangling pointer) might be able to exploit this behavior to overwrite kernel code. Prevent having writable executable PTEs in this stage. In addition, avoiding having W+X mappings can also slightly simplify the patching of modules code on initialization (e.g., by alternatives and static-key), as would be done in the next patch. This was actually the main motivation for this patch. To avoid having W+X mappings, set them initially as RW (NX) and after they are set as RO set them as X as well. Setting them as executable is done as a separate step to avoid one core in which the old PTE is cached (hence writable), and another which sees the updated PTE (executable), which would break the W^X protection. Cc: Kees Cook Cc: Peter Zijlstra Cc: Dave Hansen Cc: Masami Hiramatsu Cc: Jessica Yu Suggested-by: Thomas Gleixner Suggested-by: Andy Lutomirski Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/alternative.c | 28 +++++++++++++++++++++------- arch/x86/kernel/module.c | 2 +- include/linux/filter.h | 1 + kernel/module.c | 5 +++++ 4 files changed, 28 insertions(+), 8 deletions(-) diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 599203876c32..3d2b6b6fb20c 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -668,15 +668,29 @@ void __init alternative_instructions(void) * handlers seeing an inconsistent instruction while you patch. */ void *__init_or_module text_poke_early(void *addr, const void *opcode, - size_t len) + size_t len) { unsigned long flags; - local_irq_save(flags); - memcpy(addr, opcode, len); - local_irq_restore(flags); - sync_core(); - /* Could also do a CLFLUSH here to speed up CPU recovery; but - that causes hangs on some VIA CPUs. */ + + if (boot_cpu_has(X86_FEATURE_NX) && + is_module_text_address((unsigned long)addr)) { + /* + * Modules text is marked initially as non-executable, so the + * code cannot be running and speculative code-fetches are + * prevented. Just change the code. + */ + memcpy(addr, opcode, len); + } else { + local_irq_save(flags); + memcpy(addr, opcode, len); + local_irq_restore(flags); + sync_core(); + + /* + * Could also do a CLFLUSH here to speed up CPU recovery; but + * that causes hangs on some VIA CPUs. + */ + } return addr; } diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c index b052e883dd8c..cfa3106faee4 100644 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -87,7 +87,7 @@ void *module_alloc(unsigned long size) p = __vmalloc_node_range(size, MODULE_ALIGN, MODULES_VADDR + get_module_load_offset(), MODULES_END, GFP_KERNEL, - PAGE_KERNEL_EXEC, 0, NUMA_NO_NODE, + PAGE_KERNEL, 0, NUMA_NO_NODE, __builtin_return_address(0)); if (p && (kasan_module_alloc(p, size) < 0)) { vfree(p); diff --git a/include/linux/filter.h b/include/linux/filter.h index 6074aa064b54..14ec3bdad9a9 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -746,6 +746,7 @@ static inline void bpf_prog_unlock_ro(struct bpf_prog *fp) static inline void bpf_jit_binary_lock_ro(struct bpf_binary_header *hdr) { set_memory_ro((unsigned long)hdr, hdr->pages); + set_memory_x((unsigned long)hdr, hdr->pages); } static inline void bpf_jit_binary_unlock_ro(struct bpf_binary_header *hdr) diff --git a/kernel/module.c b/kernel/module.c index 0b9aa8ab89f0..2b2845ae983e 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -1950,8 +1950,13 @@ void module_enable_ro(const struct module *mod, bool after_init) return; frob_text(&mod->core_layout, set_memory_ro); + frob_text(&mod->core_layout, set_memory_x); + frob_rodata(&mod->core_layout, set_memory_ro); + frob_text(&mod->init_layout, set_memory_ro); + frob_text(&mod->init_layout, set_memory_x); + frob_rodata(&mod->init_layout, set_memory_ro); if (after_init) From patchwork Fri Apr 26 00:11:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918575 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6AB8592A for ; Fri, 26 Apr 2019 07:32:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5ACD428DA2 for ; Fri, 26 Apr 2019 07:32:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4EF2028DA4; Fri, 26 Apr 2019 07:32:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CD4EA28DA2 for ; Fri, 26 Apr 2019 07:32:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 70A5E6B026E; Fri, 26 Apr 2019 03:31:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2139E6B026F; Fri, 26 Apr 2019 03:31:50 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D5D766B0270; Fri, 26 Apr 2019 03:31:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id 593596B0007 for ; Fri, 26 Apr 2019 03:31:49 -0400 (EDT) Received: by mail-pl1-f200.google.com with SMTP id s19so1452523plp.6 for ; Fri, 26 Apr 2019 00:31:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=a0R0Xf8x/FEfdOPmfJkRWPwDaNy7YthB2Sa0kPsS6+I=; b=LH9SIQfE1XyNGHJd6az+wlgJMgxULNTRAywS/mqRJPdw9ff4tRzTmh4jZdASm0ubE9 0njHGKlfwRvJz1gt5y/Uzihrqeq7+y3cp8UEZHQS1rgMm6rUsYZEmtF0diTi0hQjDIMB rn/bbeQfLf6RCIMh7VurSrQf0OTWyv2CNPoM9FxyfPVfIR9m3KXPFVEkBoJa8Q4U5Kus omDys61dKzl952EpgTC96CLKSeUqUXFKaa6DBc60KoP8esaX86+K38RBzYh8KohkMP/L +V9lymc/SbkARqDBV7fakfAJZXX9wLI6QfEit9ZrlmZwgAxqJxJyZjEreLFlY0RRo0GQ xzKg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAXzxdfSuPF1ps820PAGn976zTXdYZUVlXYvli+slz0+bZsKwlEM IZ9Sm1evus4WsBk+Oc3hcQxV5eByam34VVgpLa+5BLR61ENmyouH1t4UpFtAh0Yc0sqSAnnADr4 itZ2EHqyuIZvY+sHFzwkNUfJtxRAcjPLDxEItrTllcDl3vxnnY5NBFkrBGVMfGoVUrg== X-Received: by 2002:aa7:83ce:: with SMTP id j14mr45833842pfn.57.1556263909004; Fri, 26 Apr 2019 00:31:49 -0700 (PDT) X-Google-Smtp-Source: APXvYqxi0C/sdqEsCN+lcuy5Aac/o2B4843kAO2oEFdxaNHFiibUr3d1Z90pVdSZnc8rHeqRp74N X-Received: by 2002:aa7:83ce:: with SMTP id j14mr45833742pfn.57.1556263907760; Fri, 26 Apr 2019 00:31:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263907; cv=none; d=google.com; s=arc-20160816; b=EPlxyqScbwYqGhPHJQ8dqy7qjm9q43tk9JZnGihCmaXjEC4hrMj87YRSiC+VzYM+os /5Za1XeAW6CU/gpIJhHe8lziRc7wrS9zEOHuRRGjuGOUKoQ8H+JmkCYDX+DbCEPwqdqC ju1nrwzMxNHlx3EIV5q73Knza3F+DBA4HhIQLR4njVzIHNZGHppspfq26MfJGfP3/BwA Fz1erXAhMw1d4kSdW02di2gaD5YapfN6flxLeT064PrvBOHEuwVfac6yprGC8x4vVAIl 2VCvdoB4uE+9yfl0nfLpk0GShqfdKEIx7CI4oyi+5UEvCMJ/EDqBeKd7//E1KdM6JsrT nWKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=a0R0Xf8x/FEfdOPmfJkRWPwDaNy7YthB2Sa0kPsS6+I=; b=wtv3bcqcFPxPWBT0I/qnJu068Ea5kVVmnlx8fchkNa/B3T+VpUpTMHihrT7DL6Yxuw doEm9vSdrwzKv+SUDBhHgUE4Clh59e3C8kOX+i7cyr16G4Yq06+a0rRrwN+hJ2Y6hjT3 dzqfkczF5soe99BCNBQBSLn++DnL/dRTfAFES8nyAhbgk+xe/45KNJlNHcuE4pNFUFsm 804a+mlecr83+SVByH+rrdMH7Dh5VCzL4kiJFST0GEBWWnFr0cApafX7Qwfmbt6rIPxi e+J6rhhJk2jltitblRj5dRV0UCDRnSg1FKpIZn6j5g+uzXx3sODshGSlcqap4F5o3iwC r/lQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-002.vmware.com (ex13-edg-ou-002.vmware.com. [208.91.0.190]) by mx.google.com with ESMTPS id f9si22844507pgq.347.2019.04.26.00.31.47 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:47 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) client-ip=208.91.0.190; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-002.vmware.com (10.113.208.156) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:44 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id C60F7412A6; Fri, 26 Apr 2019 00:31:45 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe , Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu Subject: [PATCH v5 12/23] x86/jump-label: Remove support for custom poker Date: Thu, 25 Apr 2019 17:11:32 -0700 Message-ID: <20190426001143.4983-13-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-002.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP There are only two types of poking: early and breakpoint based. The use of a function pointer to perform poking complicates the code and is probably inefficient due to the use of indirect branches. Cc: Andy Lutomirski Cc: Kees Cook Cc: Dave Hansen Cc: Masami Hiramatsu Acked-by: Peter Zijlstra (Intel) Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/jump_label.c | 26 ++++++++++---------------- 1 file changed, 10 insertions(+), 16 deletions(-) diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c index e7d8c636b228..e631c358f7f4 100644 --- a/arch/x86/kernel/jump_label.c +++ b/arch/x86/kernel/jump_label.c @@ -37,7 +37,6 @@ static void bug_at(unsigned char *ip, int line) static void __ref __jump_label_transform(struct jump_entry *entry, enum jump_label_type type, - void *(*poker)(void *, const void *, size_t), int init) { union jump_code_union jmp; @@ -50,14 +49,6 @@ static void __ref __jump_label_transform(struct jump_entry *entry, jmp.offset = jump_entry_target(entry) - (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); - /* - * As long as only a single processor is running and the code is still - * not marked as RO, text_poke_early() can be used; Checking that - * system_state is SYSTEM_BOOTING guarantees it. - */ - if (system_state == SYSTEM_BOOTING) - poker = text_poke_early; - if (type == JUMP_LABEL_JMP) { if (init) { expect = default_nop; line = __LINE__; @@ -80,16 +71,19 @@ static void __ref __jump_label_transform(struct jump_entry *entry, bug_at((void *)jump_entry_code(entry), line); /* - * Make text_poke_bp() a default fallback poker. + * As long as only a single processor is running and the code is still + * not marked as RO, text_poke_early() can be used; Checking that + * system_state is SYSTEM_BOOTING guarantees it. It will be set to + * SYSTEM_SCHEDULING before other cores are awaken and before the + * code is write-protected. * * At the time the change is being done, just ignore whether we * are doing nop -> jump or jump -> nop transition, and assume * always nop being the 'currently valid' instruction - * */ - if (poker) { - (*poker)((void *)jump_entry_code(entry), code, - JUMP_LABEL_NOP_SIZE); + if (init || system_state == SYSTEM_BOOTING) { + text_poke_early((void *)jump_entry_code(entry), code, + JUMP_LABEL_NOP_SIZE); return; } @@ -101,7 +95,7 @@ void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) { mutex_lock(&text_mutex); - __jump_label_transform(entry, type, NULL, 0); + __jump_label_transform(entry, type, 0); mutex_unlock(&text_mutex); } @@ -131,5 +125,5 @@ __init_or_module void arch_jump_label_transform_static(struct jump_entry *entry, jlstate = JL_STATE_NO_UPDATE; } if (jlstate == JL_STATE_UPDATE) - __jump_label_transform(entry, type, text_poke_early, 1); + __jump_label_transform(entry, type, 1); } From patchwork Fri Apr 26 00:11:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918633 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 05D2014C0 for ; Fri, 26 Apr 2019 07:33:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EA61B28DA2 for ; Fri, 26 Apr 2019 07:33:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DDE7228DA4; Fri, 26 Apr 2019 07:33:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 61E3E28DA2 for ; Fri, 26 Apr 2019 07:33:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7692C6B000C; Fri, 26 Apr 2019 03:33:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 740516B0285; Fri, 26 Apr 2019 03:33:08 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6562A6B0286; Fri, 26 Apr 2019 03:33:08 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id 2EFE46B000C for ; Fri, 26 Apr 2019 03:33:08 -0400 (EDT) Received: by mail-pl1-f200.google.com with SMTP id b7so1431987plb.17 for ; Fri, 26 Apr 2019 00:33:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=kjKVRKltof8mlgZuDLjB7nrEkQ/mRzwiPTIy69QpyZ8=; b=QhhpdRCvnqW8KzqfXnnXaRFCqLheWTghKmfESS5Oi9WY4EM8IJqXW2tol2sZ2Z4fZn s7YvXTxsLu+j5OOKpOzBn282bSDoSkjpd0yLfowfs1wOEGawZl9o+C48hoI/9uUHFVGT Z/E4/RUpxDX1uM2VBBRgNEj3nCsrE70rzetz7dU29RcQ/FP5K8aDLg+Dq4FtS7UKleES qLZELg2TQmVO9eu7maf/4edxWL+UDQWrXC24YFBqQlDSD6N58msHngp40kIyboLhSm/b WWadevf8IWtBgT8q3ulBjiT5j9gQqg9nUVRP+0Go336oOT6wsrtfkDWiPp0UeYaKc6Ir kWww== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAVXkTQk6mGSG52CGxdxHrbXVCVDjYuKHAq2J29IJ/lPvXsZHRMB jd36SYZmCpz0/ZtZKUNAFeZPCzFBqFd/4x0pPCKlrl0UV9AdDkRqmfSpPEe45/tqa5UYzNqJgGg 6EukBBWpSYukoH0cDxaYn9t3/RvjtKaJ3wo2sIHqNl2u0PG8yXbCe8RujYXeRUfcLJQ== X-Received: by 2002:a17:902:2a6a:: with SMTP id i97mr44508949plb.273.1556263987848; Fri, 26 Apr 2019 00:33:07 -0700 (PDT) X-Google-Smtp-Source: APXvYqx6bJE/C7x3IiovG2PnRtewK1o1Z5Zz1uIH2NDiNK6PrTRU5/PIVPbPuAj8dduMOoGV4IT6 X-Received: by 2002:a17:902:2a6a:: with SMTP id i97mr44502760plb.273.1556263908144; Fri, 26 Apr 2019 00:31:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263908; cv=none; d=google.com; s=arc-20160816; b=nLcPIKbslzSHeybX6xpoONIpVczYWcNWh+pdECe0j92s+tMCKD7D6+ePni3jHG5r6b PGOhei21PTtKnqBFRykydrGTKRtm3nCJhOQD7jMqbWCHoVm5qIstBIlaQ0iwTGtDqmeU p75buFXA8t1AdvuvpcgSth1H0p9YEAIdpGCDECdYqlABdkAGBphnF4iMqM4S8c8AK68y 9nzMn/7AiiiQJjYtXri9MjSQewCPBj9eHe/lBHd80fxuFgyycIzfw+74YUW8xidAsglc cWV+cTmzRp/rAZO28mPp5kIHMEmYYJwuzLZg5fzKda9Hs4pMmjxEKBAfloi5QS9Jd05G 56Fw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=kjKVRKltof8mlgZuDLjB7nrEkQ/mRzwiPTIy69QpyZ8=; b=AsvICAMhn9XReM5X11jnBa2Vw4Cu4s6yciGPxDcGGVBXjsTLxCEhyZaYu2OND0uf3I oP7mlKCH/AipRbffvl00U87OIEj6BjJt2Xo9axQ3he12JQ3UCtj5Zn8lIjMBuPydWaax YWxsh/RKONHrN/cxDTkAbkoYDUMzA7HsLpTWTmjFTDxhMn1FnqEGMJ6++8AiCRSmI1lj StzhHGMgZ1v73r3FozovYW76OmEYAQxTicz9tM24KAs44O2qtIrNDV1+6hNtiXEn8LzG 40XN/RN6Of6IECLyaMf4bXtjjdhZPCqCIY1pO1aY2F/jOYhdNLmCX92dgMa8A5imfUoB T21w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-002.vmware.com (ex13-edg-ou-002.vmware.com. [208.91.0.190]) by mx.google.com with ESMTPS id f9si22844507pgq.347.2019.04.26.00.31.48 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:48 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) client-ip=208.91.0.190; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-002.vmware.com (10.113.208.156) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:44 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id CF76F41298; Fri, 26 Apr 2019 00:31:45 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe , Nadav Amit , Kees Cook , Dave Hansen , Masami Hiramatsu Subject: [PATCH v5 13/23] x86/alternative: Remove the return value of text_poke_*() Date: Thu, 25 Apr 2019 17:11:33 -0700 Message-ID: <20190426001143.4983-14-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-002.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The return value of text_poke_early() and text_poke_bp() is useless. Remove it. Cc: Andy Lutomirski Cc: Kees Cook Cc: Dave Hansen Cc: Masami Hiramatsu Acked-by: Peter Zijlstra (Intel) Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/text-patching.h | 4 ++-- arch/x86/kernel/alternative.c | 11 ++++------- 2 files changed, 6 insertions(+), 9 deletions(-) diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h index a75eed841eed..c90678fd391a 100644 --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -18,7 +18,7 @@ static inline void apply_paravirt(struct paravirt_patch_site *start, #define __parainstructions_end NULL #endif -extern void *text_poke_early(void *addr, const void *opcode, size_t len); +extern void text_poke_early(void *addr, const void *opcode, size_t len); /* * Clear and restore the kernel write-protection flag on the local CPU. @@ -37,7 +37,7 @@ extern void *text_poke_early(void *addr, const void *opcode, size_t len); extern void *text_poke(void *addr, const void *opcode, size_t len); extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len); extern int poke_int3_handler(struct pt_regs *regs); -extern void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler); +extern void text_poke_bp(void *addr, const void *opcode, size_t len, void *handler); extern int after_bootmem; extern __ro_after_init struct mm_struct *poking_mm; extern __ro_after_init unsigned long poking_addr; diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 3d2b6b6fb20c..18f959975ea0 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -265,7 +265,7 @@ static void __init_or_module add_nops(void *insns, unsigned int len) extern struct alt_instr __alt_instructions[], __alt_instructions_end[]; extern s32 __smp_locks[], __smp_locks_end[]; -void *text_poke_early(void *addr, const void *opcode, size_t len); +void text_poke_early(void *addr, const void *opcode, size_t len); /* * Are we looking at a near JMP with a 1 or 4-byte displacement. @@ -667,8 +667,8 @@ void __init alternative_instructions(void) * instructions. And on the local CPU you need to be protected again NMI or MCE * handlers seeing an inconsistent instruction while you patch. */ -void *__init_or_module text_poke_early(void *addr, const void *opcode, - size_t len) +void __init_or_module text_poke_early(void *addr, const void *opcode, + size_t len) { unsigned long flags; @@ -691,7 +691,6 @@ void *__init_or_module text_poke_early(void *addr, const void *opcode, * that causes hangs on some VIA CPUs. */ } - return addr; } __ro_after_init struct mm_struct *poking_mm; @@ -893,7 +892,7 @@ NOKPROBE_SYMBOL(poke_int3_handler); * replacing opcode * - sync cores */ -void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler) +void text_poke_bp(void *addr, const void *opcode, size_t len, void *handler) { unsigned char int3 = 0xcc; @@ -935,7 +934,5 @@ void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler) * the writing of the new instruction. */ bp_patching_in_progress = false; - - return addr; } From patchwork Fri Apr 26 00:11:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918579 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7986A14C0 for ; Fri, 26 Apr 2019 07:32:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6973D28DA2 for ; Fri, 26 Apr 2019 07:32:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5DF5C28DA4; Fri, 26 Apr 2019 07:32:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D249528DA2 for ; Fri, 26 Apr 2019 07:32:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C34106B026F; Fri, 26 Apr 2019 03:31:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4D3B66B0266; Fri, 26 Apr 2019 03:31:50 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D4DC6B026A; Fri, 26 Apr 2019 03:31:50 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id B9D4F6B0266 for ; Fri, 26 Apr 2019 03:31:49 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id y2so1659966pfn.13 for ; Fri, 26 Apr 2019 00:31:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=KTT3A6U4NQUMX3UyJFnXteqziiXHIb6+tR0GZna3rFw=; b=XEX0vJMTCC986W7GqL7WVQ5utVDAn7fsP6SAh3TnY7JffavQHCLsiZl4rX30UycqKy tcNpS+5VNrABRnvg8Y2ZWoeIVkw3cEv4OR/fIh6wKU8cf7Tm/PXNPN7gMKde18LWFixG bY1p3OyE6QhKnXmlziEY9aOvFbyOhLqfyROvzW0oale/Txal5Jtp1rr8hohMFgnwbk94 HQq/0mNJSEc6ZZpgXV0i7fa0vZwfpQHU+rGlRAaGuaTgEAUTNTHW3jnso5o5CMQO1LT2 Ch6XRkpl1GhXYzp99MreeizSG2fAVuZSGfsyMLs1xmRoWLZypLY6/OZg9W/AOO0eOqYv YoXQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAUqXb2ot+XW6m9hLrYbifFQlvx2QYnINfvuqfReNHYB4n+Ha6Xw jiHG06y3Q8pgAnSBAkuncz7wVztLzwW3WV47IzIfJiujBny2aGzffRLcfrzw49va0/lquWdqjBI 0Au+DedGl6y3oUoqNt7t86rLVbn0S1wqbrJQyK1H1ybkO3/gOS7PnPMqTDHPt/o1NRQ== X-Received: by 2002:a62:474a:: with SMTP id u71mr44164050pfa.87.1556263909418; Fri, 26 Apr 2019 00:31:49 -0700 (PDT) X-Google-Smtp-Source: APXvYqwRYLjLGCR7EDAjtQi5NLvJXU/YK/939ExPOMraZEJx73r+tmAKQyXAwbYytNJRKSGrKTCx X-Received: by 2002:a62:474a:: with SMTP id u71mr44163940pfa.87.1556263907940; Fri, 26 Apr 2019 00:31:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263907; cv=none; d=google.com; s=arc-20160816; b=YjSpPH0zyeJfdD/BU+DCYTGFirPSadfamceFqJ5bf8HThsSxV4yrjnm7G7nzwcPQUQ dwTQ0ieETVj4xpyaKTTd1ZZcVc2cuvPe7tYHumjN5+dGvZlZgKlbimjr1sc+2rgMzJ+c Xp5/NzVkiaK4TTs5Jqn2QLrzXFQxbsu6KPrLkHeVJDrEwXac813T65EFm9rxS3MkL7EZ P8N9PMHaT1AFj1Nu3X0AKkkeamQSGAsZnwlgcUvtsTvKfo3BmKf7fq/ERUG2TY5b112j wo0Yn2GP6NuTNs6rokrSD8oTl67N8wfK8MBGfc32ldnEjbwhKZ2KNQnwVXsnVtL8CZTY pzOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=KTT3A6U4NQUMX3UyJFnXteqziiXHIb6+tR0GZna3rFw=; b=jsFprV6g5HwGE+pB153Lm0FyehL0l7puGnSqLApDe8W8jdzF/rtk4zqzLdXTKVnvoi C5fa2jn8QEyTcoHOeJBLExXMX5TGAKutICTOi1WN+rO9E2fQiDnU7ufw9h4MYsSZxHom PwSlG97H5obe3sfVFuH4rjTVKA9uLiJpxMhWz3FHypJ6YFbKbiHKF2gjAqU7BBbdYbop Ic7IcprsTBd3tqypXI0zXtHuE16nq3gZbyn5Pxxvy9gbpAJm54RLAZHBFHOJvvfqht/O w6JEqn1MOi+1ZyYAkuhbiSzq3xEOAvhOqdKVWDzBrjv2NkcohtsBpcp09Tl90ggdtdkq h48A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-002.vmware.com (ex13-edg-ou-002.vmware.com. [208.91.0.190]) by mx.google.com with ESMTPS id f9si22844507pgq.347.2019.04.26.00.31.47 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:47 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) client-ip=208.91.0.190; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-002.vmware.com (10.113.208.156) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:44 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id D96B44129C; Fri, 26 Apr 2019 00:31:45 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe Subject: [PATCH v5 14/23] x86/mm/cpa: Add set_direct_map_ functions Date: Thu, 25 Apr 2019 17:11:34 -0700 Message-ID: <20190426001143.4983-15-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-002.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Rick Edgecombe Add two new functions set_direct_map_default_noflush() and set_direct_map_invalid_noflush() for setting the direct map alias for the page to its default valid permissions and to an invalid state that cannot be cached in a TLB, respectively. These functions do not flush the TLB. Note, __kernel_map_pages() does something similar but flushes the TLB and doesn't reset the permission bits to default on all architectures. Also add an ARCH config ARCH_HAS_SET_DIRECT_MAP for specifying whether these have an actual implementation or a default empty one. Cc: Dave Hansen Cc: Andy Lutomirski Cc: Peter Zijlstra Signed-off-by: Rick Edgecombe --- arch/Kconfig | 4 ++++ arch/x86/Kconfig | 1 + arch/x86/include/asm/set_memory.h | 3 +++ arch/x86/mm/pageattr.c | 14 +++++++++++--- include/linux/set_memory.h | 11 +++++++++++ 5 files changed, 30 insertions(+), 3 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 3ab446bd12ef..5e43fcbad4ca 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -249,6 +249,10 @@ config ARCH_HAS_FORTIFY_SOURCE config ARCH_HAS_SET_MEMORY bool +# Select if arch has all set_direct_map_invalid/default() functions +config ARCH_HAS_SET_DIRECT_MAP + bool + # Select if arch init_task must go in the __init_task_data section config ARCH_TASK_STRUCT_ON_STACK bool diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 2ec5e850b807..45d788354376 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -66,6 +66,7 @@ config X86 select ARCH_HAS_UACCESS_FLUSHCACHE if X86_64 select ARCH_HAS_UACCESS_MCSAFE if X86_64 && X86_MCE select ARCH_HAS_SET_MEMORY + select ARCH_HAS_SET_DIRECT_MAP select ARCH_HAS_STRICT_KERNEL_RWX select ARCH_HAS_STRICT_MODULE_RWX select ARCH_HAS_SYNC_CORE_BEFORE_USERMODE diff --git a/arch/x86/include/asm/set_memory.h b/arch/x86/include/asm/set_memory.h index 07a25753e85c..ae7b909dc242 100644 --- a/arch/x86/include/asm/set_memory.h +++ b/arch/x86/include/asm/set_memory.h @@ -85,6 +85,9 @@ int set_pages_nx(struct page *page, int numpages); int set_pages_ro(struct page *page, int numpages); int set_pages_rw(struct page *page, int numpages); +int set_direct_map_invalid_noflush(struct page *page); +int set_direct_map_default_noflush(struct page *page); + extern int kernel_set_to_readonly; void set_kernel_text_rw(void); void set_kernel_text_ro(void); diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c index 4c570612e24e..3574550192c6 100644 --- a/arch/x86/mm/pageattr.c +++ b/arch/x86/mm/pageattr.c @@ -2209,8 +2209,6 @@ int set_pages_rw(struct page *page, int numpages) return set_memory_rw(addr, numpages); } -#ifdef CONFIG_DEBUG_PAGEALLOC - static int __set_pages_p(struct page *page, int numpages) { unsigned long tempaddr = (unsigned long) page_address(page); @@ -2249,6 +2247,17 @@ static int __set_pages_np(struct page *page, int numpages) return __change_page_attr_set_clr(&cpa, 0); } +int set_direct_map_invalid_noflush(struct page *page) +{ + return __set_pages_np(page, 1); +} + +int set_direct_map_default_noflush(struct page *page) +{ + return __set_pages_p(page, 1); +} + +#ifdef CONFIG_DEBUG_PAGEALLOC void __kernel_map_pages(struct page *page, int numpages, int enable) { if (PageHighMem(page)) @@ -2282,7 +2291,6 @@ void __kernel_map_pages(struct page *page, int numpages, int enable) } #ifdef CONFIG_HIBERNATION - bool kernel_page_present(struct page *page) { unsigned int level; diff --git a/include/linux/set_memory.h b/include/linux/set_memory.h index 2a986d282a97..b5071497b8cb 100644 --- a/include/linux/set_memory.h +++ b/include/linux/set_memory.h @@ -17,6 +17,17 @@ static inline int set_memory_x(unsigned long addr, int numpages) { return 0; } static inline int set_memory_nx(unsigned long addr, int numpages) { return 0; } #endif +#ifndef CONFIG_ARCH_HAS_SET_DIRECT_MAP +static inline int set_direct_map_invalid_noflush(struct page *page) +{ + return 0; +} +static inline int set_direct_map_default_noflush(struct page *page) +{ + return 0; +} +#endif + #ifndef set_mce_nospec static inline int set_mce_nospec(unsigned long pfn) { From patchwork Fri Apr 26 00:11:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918693 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5DC8315E9 for ; Fri, 26 Apr 2019 07:33:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4E25D28DA2 for ; Fri, 26 Apr 2019 07:33:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4215D28DA4; Fri, 26 Apr 2019 07:33:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6F89628DA3 for ; Fri, 26 Apr 2019 07:33:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 177056B0283; Fri, 26 Apr 2019 03:33:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0FF336B028F; Fri, 26 Apr 2019 03:33:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE0186B0284; Fri, 26 Apr 2019 03:33:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id B0B016B028F for ; Fri, 26 Apr 2019 03:33:54 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id s26so1649654pfm.18 for ; Fri, 26 Apr 2019 00:33:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=QhBGq/JNavwlurWIVFo7GWw2xI+63VIDOiZrYQv9auY=; b=hv2eC5gvYKEMxpgFHEpgkRMZ40z2MfMJIvTSHBFdQ2RFbwRy3f4z6N/ruKKoB1FYPs aIGAwr7K85jFqn5F4F0TDtGbwfL7m5xQ4AorkqudxbQcuvYy1odf0KyUj1IlLvYNF1U2 vOEpy5xWP8B7qjJb+Y3xnum6yxXQZH+2ezE3wm3RQvB8az9Y+Hou1fMf1JZJVxmV66On g2maFzhlUNrzXhg9gglZ+6v5peuIC+8xzDZpTbH/xWLo96D7gpPbTJX5E2WRGxRfp0qU 60uCjqY1eAgCT6dIi3IvYf/nw7HFjaMx9EttD61jfqy8JFnNR7ZpItrCi5sclm1sQ9af s4xg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAWxEjoF4PmZ8UCDUQdCt+cX5enDYtI9BL8orc7qkaHBNdD4ssbz 2kqHAzmZWeWyoEx0itmXAJrKJBLqFTV+ewA7eXm4rQg15UeZxlvZZC6ckz4632rBIwkaRxXgkfi eijemH8NZVos4zzdvKmFITNy113hsevmAHm6DyitVsLzsBq+YkR9lsof1clmxVOqWUA== X-Received: by 2002:a17:902:4101:: with SMTP id e1mr45472959pld.25.1556264034376; Fri, 26 Apr 2019 00:33:54 -0700 (PDT) X-Google-Smtp-Source: APXvYqziGnB/Wz0fuBoOjeXlkNp2hL0a3qic3M0fgQTWdeDpqEGfxcebYdsO8/3PSSiBL9OVxlcv X-Received: by 2002:a17:902:4101:: with SMTP id e1mr45463377pld.25.1556263907080; Fri, 26 Apr 2019 00:31:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263907; cv=none; d=google.com; s=arc-20160816; b=zn9hK+uGthdJYZRQhzvqdyxqT/glgS+a2ZITWGgtjTYJ3eTGxdG3dP5idAJyXdv1q6 IFc/yPpC4gEzXHxFYC95G0JtTBKpdUguje8xqu0r6du6l9crRl+3zRLJCmSebFIJ4/z5 Uek4wozO1KaZM1ZLkcO5KIRhthuII0NxuZYYigBQ5uPsSs2M+GvzjrcA408CfZXyC+A/ WkU+c2SX/MLh8Y1TuXoP/1+iS37da5m8l3VJIOOqXau80ay+0x2oxeYOnrbGkrrUOzYX Otr0xHl18eOZ4amwtI80W3rKZZDi1trim2F6Q3HgdOc/vjEiThSkyu4t3yL3I5K486f4 rFPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=QhBGq/JNavwlurWIVFo7GWw2xI+63VIDOiZrYQv9auY=; b=vOFHJeCrnk3cL9EQ6BwMt+2qT1ykOzRsfG46UfGfAgE6VHU/39LG7pBPNj+ly40GWc h3JETjWNPyag1/10NxeF8ZptIBryjPfB3TNQrt+YlWrD6TcEOXEHL6IvYJJweikzwC4v gQxZagteve+CZBPGfFSP45Y6eBYm4V8sLli3st++wNyedgvQjpqN5F/XwE0P5YFQy+Rb faFJerDt2RcNiaLArn0Ro5Wko4CcVj7q3l4Z/BWeFd751V1m3iZ365WovXJrTHjNtOFk 1IHMgAi+bBXJpY01gPtcjHLG5nAF/savYzNDHC+zMsWfZBbmA63XoeJOsD/J2iKmmpz/ 8IHw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-001.vmware.com (ex13-edg-ou-001.vmware.com. [208.91.0.189]) by mx.google.com with ESMTPS id v82si25417769pfa.42.2019.04.26.00.31.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:47 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) client-ip=208.91.0.189; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-001.vmware.com (10.113.208.155) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:40 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id E15254129A; Fri, 26 Apr 2019 00:31:45 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe , "Rafael J. Wysocki" , Pavel Machek Subject: [PATCH v5 15/23] mm: Make hibernate handle unmapped pages Date: Thu, 25 Apr 2019 17:11:35 -0700 Message-ID: <20190426001143.4983-16-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-001.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Rick Edgecombe Make hibernate handle unmapped pages on the direct map when CONFIG_ARCH_HAS_SET_ALIAS is set. These functions allow for setting pages to invalid configurations, so now hibernate should check if the pages have valid mappings and handle if they are unmapped when doing a hibernate save operation. Previously this checking was already done when CONFIG_DEBUG_PAGEALLOC was configured. It does not appear to have a big hibernating performance impact. The speed of the saving operation before this change was measured as 819.02 MB/s, and after was measured at 813.32 MB/s. Before: [ 4.670938] PM: Wrote 171996 kbytes in 0.21 seconds (819.02 MB/s) After: [ 4.504714] PM: Wrote 178932 kbytes in 0.22 seconds (813.32 MB/s) Cc: Dave Hansen Cc: Andy Lutomirski Cc: Peter Zijlstra Cc: "Rafael J. Wysocki" Cc: Pavel Machek Cc: Borislav Petkov Acked-by: Pavel Machek Signed-off-by: Rick Edgecombe --- arch/x86/mm/pageattr.c | 4 ---- include/linux/mm.h | 18 ++++++------------ kernel/power/snapshot.c | 5 +++-- mm/page_alloc.c | 7 +++++-- 4 files changed, 14 insertions(+), 20 deletions(-) diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c index 3574550192c6..daf4d645e537 100644 --- a/arch/x86/mm/pageattr.c +++ b/arch/x86/mm/pageattr.c @@ -2257,7 +2257,6 @@ int set_direct_map_default_noflush(struct page *page) return __set_pages_p(page, 1); } -#ifdef CONFIG_DEBUG_PAGEALLOC void __kernel_map_pages(struct page *page, int numpages, int enable) { if (PageHighMem(page)) @@ -2302,11 +2301,8 @@ bool kernel_page_present(struct page *page) pte = lookup_address((unsigned long)page_address(page), &level); return (pte_val(*pte) & _PAGE_PRESENT); } - #endif /* CONFIG_HIBERNATION */ -#endif /* CONFIG_DEBUG_PAGEALLOC */ - int __init kernel_map_pages_in_pgd(pgd_t *pgd, u64 pfn, unsigned long address, unsigned numpages, unsigned long page_flags) { diff --git a/include/linux/mm.h b/include/linux/mm.h index 6b10c21630f5..083d7b4863ed 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2610,37 +2610,31 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { } #endif -#ifdef CONFIG_DEBUG_PAGEALLOC extern bool _debug_pagealloc_enabled; -extern void __kernel_map_pages(struct page *page, int numpages, int enable); static inline bool debug_pagealloc_enabled(void) { - return _debug_pagealloc_enabled; + return IS_ENABLED(CONFIG_DEBUG_PAGEALLOC) && _debug_pagealloc_enabled; } +#if defined(CONFIG_DEBUG_PAGEALLOC) || defined(CONFIG_ARCH_HAS_SET_DIRECT_MAP) +extern void __kernel_map_pages(struct page *page, int numpages, int enable); + static inline void kernel_map_pages(struct page *page, int numpages, int enable) { - if (!debug_pagealloc_enabled()) - return; - __kernel_map_pages(page, numpages, enable); } #ifdef CONFIG_HIBERNATION extern bool kernel_page_present(struct page *page); #endif /* CONFIG_HIBERNATION */ -#else /* CONFIG_DEBUG_PAGEALLOC */ +#else /* CONFIG_DEBUG_PAGEALLOC || CONFIG_ARCH_HAS_SET_DIRECT_MAP */ static inline void kernel_map_pages(struct page *page, int numpages, int enable) {} #ifdef CONFIG_HIBERNATION static inline bool kernel_page_present(struct page *page) { return true; } #endif /* CONFIG_HIBERNATION */ -static inline bool debug_pagealloc_enabled(void) -{ - return false; -} -#endif /* CONFIG_DEBUG_PAGEALLOC */ +#endif /* CONFIG_DEBUG_PAGEALLOC || CONFIG_ARCH_HAS_SET_DIRECT_MAP */ #ifdef __HAVE_ARCH_GATE_AREA extern struct vm_area_struct *get_gate_vma(struct mm_struct *mm); diff --git a/kernel/power/snapshot.c b/kernel/power/snapshot.c index f08a1e4ee1d4..bc9558ab1e5b 100644 --- a/kernel/power/snapshot.c +++ b/kernel/power/snapshot.c @@ -1342,8 +1342,9 @@ static inline void do_copy_page(long *dst, long *src) * safe_copy_page - Copy a page in a safe way. * * Check if the page we are going to copy is marked as present in the kernel - * page tables (this always is the case if CONFIG_DEBUG_PAGEALLOC is not set - * and in that case kernel_page_present() always returns 'true'). + * page tables. This always is the case if CONFIG_DEBUG_PAGEALLOC or + * CONFIG_ARCH_HAS_SET_DIRECT_MAP is not set. In that case kernel_page_present() + * always returns 'true'. */ static void safe_copy_page(void *dst, struct page *s_page) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d96ca5bc555b..34a70681a4af 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1131,7 +1131,9 @@ static __always_inline bool free_pages_prepare(struct page *page, } arch_free_page(page, order); kernel_poison_pages(page, 1 << order, 0); - kernel_map_pages(page, 1 << order, 0); + if (debug_pagealloc_enabled()) + kernel_map_pages(page, 1 << order, 0); + kasan_free_nondeferred_pages(page, order); return true; @@ -2001,7 +2003,8 @@ inline void post_alloc_hook(struct page *page, unsigned int order, set_page_refcounted(page); arch_alloc_page(page, order); - kernel_map_pages(page, 1 << order, 1); + if (debug_pagealloc_enabled()) + kernel_map_pages(page, 1 << order, 1); kasan_alloc_pages(page, order); kernel_poison_pages(page, 1 << order, 1); set_page_owner(page, order, gfp_flags); From patchwork Fri Apr 26 00:11:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918649 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2AF1D92A for ; Fri, 26 Apr 2019 07:33:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1B55328DA2 for ; Fri, 26 Apr 2019 07:33:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0E6DA28DA4; Fri, 26 Apr 2019 07:33:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 48FFD28DA2 for ; Fri, 26 Apr 2019 07:33:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1C9A86B0285; Fri, 26 Apr 2019 03:33:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 126956B0287; Fri, 26 Apr 2019 03:33:23 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EBC596B0288; Fri, 26 Apr 2019 03:33:22 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id AA1D46B0285 for ; Fri, 26 Apr 2019 03:33:22 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id j12so1507156pgl.14 for ; Fri, 26 Apr 2019 00:33:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=71dbHy9m/CIhCtlniLjnIfT11ydtbrFJWfPgh/aDBts=; b=axEStOYP4aHHe/O11wMkRHEHrYsOpaCnkk2GZma3u2c396UyrjJGAC4EYBHK9veIwG sTIDNunTiIDVc0wILAN6g+go4wRXTVrfzPqFe8O9Q6nAuFqOyIt05j3i7xPsed4OqDaX E5GDJmicifwjULKYeBJOG7WXmL3baSrr6O7hAwg+Jp65ZIA0P3t1Kf9LE7hbQCgwSegD 9Jxnat5fa7UhgRHJIo+LDpMjMIE44ePPYGiRpQB4VBd/4T/zxINssxAerMj3u3vX92qQ VW3Ehatsxg5sxY2jl+/L/AozH9HwZucPMXX24OrcBKRsKW3BMeY0p7dS6ETPCdohvxd1 yyAQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAULLAat5BWheaPxmljRIDBGMzHzsHMI01uPG4k5+quKWIBYEJHv LhjS4tmTrcDuPbh1SvNolT86vw5OF22yv4kfFgmAkdBrb+xd5OCqUMtZnz6kEIGU/WzqDOIKrXz A9AGVLdtBg3847oyEBWOcW0xjqoRKqWRFDeWWJ3BLjoK12vCE0XoPS/Rcx4q/nhyZCQ== X-Received: by 2002:a63:5720:: with SMTP id l32mr12309742pgb.438.1556264002293; Fri, 26 Apr 2019 00:33:22 -0700 (PDT) X-Google-Smtp-Source: APXvYqzIWT7+U7N+7xGiwoHbWPzvtILPXVMXgH4eQHz7vj2lJW5aST9P9CAKgsiDpqtfI0VdgL2S X-Received: by 2002:a63:5720:: with SMTP id l32mr12303026pgb.438.1556263908742; Fri, 26 Apr 2019 00:31:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263908; cv=none; d=google.com; s=arc-20160816; b=EeNIGf/KhGc9v8fmrSVFF8arfrtMzYlGDlIOteOotKuAD5k5UFS2D3fIknMCOMw8QI UsOftG8fsyhJ/wUzdcRXNgajWi+ZBXmuQi3iV/kDvsKdak6ckx7xa/vlIUVh2s6IwtJp 2HZYdmB9nn2IQhelaTXjMtLTDPNE0/2KQU/HQ6E+b7HoSQxT+L7/L7kr19gp0RYHgne1 habJBDs+hJ/oZCFeQv/y+mpZZbWSvVwEG4Z/JDKbBMCddxHRe32vrudZUAjwoo3oGy5R DHFPVjsMkf0FFiw48CibRhx+ZZ6nbw2PuwX6ddTOIKc4PXLnRm3zt6qBsB4KFIXnBJBX qAag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=71dbHy9m/CIhCtlniLjnIfT11ydtbrFJWfPgh/aDBts=; b=0NC964OC6mpBcjP7hSzNWLm1oEdGOIWdzVXoQAj8JHraGROcfCx+Gz5fSI3TsJ7WvV guwLhPUrNZwQGdry8p7NhzQWJNt3ZXCsrf0qu+mif2qBQAN54yD9A8Au076qikhAHcPw 9/7r0+XrfpOlQnwDI+Y5jUa4iEmeU6F1OLtPx8v1e64rO28P6bD5l6DS0Co9GN4VntbS Weq4+NkvP2jMf+zJ37KuY9ZGyd789Yez+PAfpgjJSuACkoxqpLTOsmrrKIGrXbfkNlvU BpRGMa2JxLwudlMNtInGZMvgJhOwSuN29kAP2Mlcd1Xmu4rHjid8TRIDR2pt7SrrZYkM mhZg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-002.vmware.com (ex13-edg-ou-002.vmware.com. [208.91.0.190]) by mx.google.com with ESMTPS id f9si22844507pgq.347.2019.04.26.00.31.48 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:48 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) client-ip=208.91.0.190; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-002.vmware.com (10.113.208.156) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:44 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id E9EC6412A3; Fri, 26 Apr 2019 00:31:45 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe Subject: [PATCH v5 16/23] vmalloc: Add flag for free of special permsissions Date: Thu, 25 Apr 2019 17:11:36 -0700 Message-ID: <20190426001143.4983-17-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-002.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Rick Edgecombe Add a new flag VM_FLUSH_RESET_PERMS, for enabling vfree operations to immediately clear executable TLB entries before freeing pages, and handle resetting permissions on the directmap. This flag is useful for any kind of memory with elevated permissions, or where there can be related permissions changes on the directmap. Today this is RO+X and RO memory. Although this enables directly vfreeing non-writeable memory now, non-writable memory cannot be freed in an interrupt because the allocation itself is used as a node on deferred free list. So when RO memory needs to be freed in an interrupt the code doing the vfree needs to have its own work queue, as was the case before the deferred vfree list was added to vmalloc. For architectures with set_direct_map_ implementations this whole operation can be done with one TLB flush when centralized like this. For others with directmap permissions, currently only arm64, a backup method using set_memory functions is used to reset the directmap. When arm64 adds set_direct_map_ functions, this backup can be removed. When the TLB is flushed to both remove TLB entries for the vmalloc range mapping and the direct map permissions, the lazy purge operation could be done to try to save a TLB flush later. However today vm_unmap_aliases could flush a TLB range that does not include the directmap. So a helper is added with extra parameters that can allow both the vmalloc address and the direct mapping to be flushed during this operation. The behavior of the normal vm_unmap_aliases function is unchanged. Cc: Borislav Petkov Suggested-by: Dave Hansen Suggested-by: Andy Lutomirski Suggested-by: Will Deacon Signed-off-by: Rick Edgecombe --- include/linux/vmalloc.h | 15 ++++++ mm/vmalloc.c | 113 +++++++++++++++++++++++++++++++++------- 2 files changed, 109 insertions(+), 19 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 398e9c95cd61..c6eebb839552 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -21,6 +21,11 @@ struct notifier_block; /* in notifier.h */ #define VM_UNINITIALIZED 0x00000020 /* vm_struct is not fully initialized */ #define VM_NO_GUARD 0x00000040 /* don't add guard page */ #define VM_KASAN 0x00000080 /* has allocated kasan shadow memory */ +/* + * Memory with VM_FLUSH_RESET_PERMS cannot be freed in an interrupt or with + * vfree_atomic(). + */ +#define VM_FLUSH_RESET_PERMS 0x00000100 /* Reset direct map and flush TLB on unmap */ /* bits [20..32] reserved for arch specific ioremap internals */ /* @@ -142,6 +147,13 @@ extern int map_kernel_range_noflush(unsigned long start, unsigned long size, pgprot_t prot, struct page **pages); extern void unmap_kernel_range_noflush(unsigned long addr, unsigned long size); extern void unmap_kernel_range(unsigned long addr, unsigned long size); +static inline void set_vm_flush_reset_perms(void *addr) +{ + struct vm_struct *vm = find_vm_area(addr); + + if (vm) + vm->flags |= VM_FLUSH_RESET_PERMS; +} #else static inline int map_kernel_range_noflush(unsigned long start, unsigned long size, @@ -157,6 +169,9 @@ static inline void unmap_kernel_range(unsigned long addr, unsigned long size) { } +static inline void set_vm_flush_reset_perms(void *addr) +{ +} #endif /* Allocate/destroy a 'vmalloc' VM area. */ diff --git a/mm/vmalloc.c b/mm/vmalloc.c index e86ba6e74b50..e5e9e1fcac01 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -1059,24 +1060,9 @@ static void vb_free(const void *addr, unsigned long size) spin_unlock(&vb->lock); } -/** - * vm_unmap_aliases - unmap outstanding lazy aliases in the vmap layer - * - * The vmap/vmalloc layer lazily flushes kernel virtual mappings primarily - * to amortize TLB flushing overheads. What this means is that any page you - * have now, may, in a former life, have been mapped into kernel virtual - * address by the vmap layer and so there might be some CPUs with TLB entries - * still referencing that page (additional to the regular 1:1 kernel mapping). - * - * vm_unmap_aliases flushes all such lazy mappings. After it returns, we can - * be sure that none of the pages we have control over will have any aliases - * from the vmap layer. - */ -void vm_unmap_aliases(void) +static void _vm_unmap_aliases(unsigned long start, unsigned long end, int flush) { - unsigned long start = ULONG_MAX, end = 0; int cpu; - int flush = 0; if (unlikely(!vmap_initialized)) return; @@ -1113,6 +1099,27 @@ void vm_unmap_aliases(void) flush_tlb_kernel_range(start, end); mutex_unlock(&vmap_purge_lock); } + +/** + * vm_unmap_aliases - unmap outstanding lazy aliases in the vmap layer + * + * The vmap/vmalloc layer lazily flushes kernel virtual mappings primarily + * to amortize TLB flushing overheads. What this means is that any page you + * have now, may, in a former life, have been mapped into kernel virtual + * address by the vmap layer and so there might be some CPUs with TLB entries + * still referencing that page (additional to the regular 1:1 kernel mapping). + * + * vm_unmap_aliases flushes all such lazy mappings. After it returns, we can + * be sure that none of the pages we have control over will have any aliases + * from the vmap layer. + */ +void vm_unmap_aliases(void) +{ + unsigned long start = ULONG_MAX, end = 0; + int flush = 0; + + _vm_unmap_aliases(start, end, flush); +} EXPORT_SYMBOL_GPL(vm_unmap_aliases); /** @@ -1505,6 +1512,72 @@ struct vm_struct *remove_vm_area(const void *addr) return NULL; } +static inline void set_area_direct_map(const struct vm_struct *area, + int (*set_direct_map)(struct page *page)) +{ + int i; + + for (i = 0; i < area->nr_pages; i++) + if (page_address(area->pages[i])) + set_direct_map(area->pages[i]); +} + +/* Handle removing and resetting vm mappings related to the vm_struct. */ +static void vm_remove_mappings(struct vm_struct *area, int deallocate_pages) +{ + unsigned long addr = (unsigned long)area->addr; + unsigned long start = ULONG_MAX, end = 0; + int flush_reset = area->flags & VM_FLUSH_RESET_PERMS; + int i; + + /* + * The below block can be removed when all architectures that have + * direct map permissions also have set_direct_map_() implementations. + * This is concerned with resetting the direct map any an vm alias with + * execute permissions, without leaving a RW+X window. + */ + if (flush_reset && !IS_ENABLED(CONFIG_ARCH_HAS_SET_DIRECT_MAP)) { + set_memory_nx(addr, area->nr_pages); + set_memory_rw(addr, area->nr_pages); + } + + remove_vm_area(area->addr); + + /* If this is not VM_FLUSH_RESET_PERMS memory, no need for the below. */ + if (!flush_reset) + return; + + /* + * If not deallocating pages, just do the flush of the VM area and + * return. + */ + if (!deallocate_pages) { + vm_unmap_aliases(); + return; + } + + /* + * If execution gets here, flush the vm mapping and reset the direct + * map. Find the start and end range of the direct mappings to make sure + * the vm_unmap_aliases() flush includes the direct map. + */ + for (i = 0; i < area->nr_pages; i++) { + if (page_address(area->pages[i])) { + start = min(addr, start); + end = max(addr, end); + } + } + + /* + * Set direct map to something invalid so that it won't be cached if + * there are any accesses after the TLB flush, then flush the TLB and + * reset the direct map permissions to the default. + */ + set_area_direct_map(area, set_direct_map_invalid_noflush); + _vm_unmap_aliases(start, end, 1); + set_area_direct_map(area, set_direct_map_default_noflush); +} + static void __vunmap(const void *addr, int deallocate_pages) { struct vm_struct *area; @@ -1526,7 +1599,8 @@ static void __vunmap(const void *addr, int deallocate_pages) debug_check_no_locks_freed(area->addr, get_vm_area_size(area)); debug_check_no_obj_freed(area->addr, get_vm_area_size(area)); - remove_vm_area(addr); + vm_remove_mappings(area, deallocate_pages); + if (deallocate_pages) { int i; @@ -1961,8 +2035,9 @@ EXPORT_SYMBOL(vzalloc_node); */ void *vmalloc_exec(unsigned long size) { - return __vmalloc_node(size, 1, GFP_KERNEL, PAGE_KERNEL_EXEC, - NUMA_NO_NODE, __builtin_return_address(0)); + return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, + GFP_KERNEL, PAGE_KERNEL_EXEC, VM_FLUSH_RESET_PERMS, + NUMA_NO_NODE, __builtin_return_address(0)); } #if defined(CONFIG_64BIT) && defined(CONFIG_ZONE_DMA32) From patchwork Fri Apr 26 00:11:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918577 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2BEFA14C0 for ; Fri, 26 Apr 2019 07:32:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1963528DA2 for ; Fri, 26 Apr 2019 07:32:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0D6B728DA4; Fri, 26 Apr 2019 07:32:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 65FEA28DA2 for ; Fri, 26 Apr 2019 07:32:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8FFCA6B026A; Fri, 26 Apr 2019 03:31:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3C0C76B0270; Fri, 26 Apr 2019 03:31:50 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 08D5A6B026D; Fri, 26 Apr 2019 03:31:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id A984F6B026F for ; Fri, 26 Apr 2019 03:31:49 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id w9so1438369plz.11 for ; Fri, 26 Apr 2019 00:31:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=FbRVtA2o2v3UWbAlXvOdnbyNDEqG2rBMqLF6HOTpV1U=; b=Z3u5wIwIBnB6ljuegUedpWsH3t7W3hBMpGduIIPtrOyRcvx7Vuy4cafuVBzU7ch+go kWbfpAiqmsl1skOC8hsUHJfTveAQv8MeRD0lm4HLXvjQkuf2sK3fXrNtpWJLkGtmg0fH 35LSIK2fgWYVNOczL3pZgTPvNOxB8rMZB/8ceNiK3BQmqzWMPC5Vhmu6lChch1e1mQQN uT0lws0vJ1oZrBYwX3z9ULvISKwQFLjiss1C21kXVLC+r/kFkM51EbrYFPqBD3di1Ihi U/36Kc3Kwli5wCS9FOCWoRptnuVkNXSvk0Wjv2c6knAx9rNkNSWdSlndMRstRrBClSlt CJ6w== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAVPWWUobIfJcTaAXO01BX64lGAD8JpR1ppPbevFETsvXnNeh3nU aTduxp2mvNO9Ja3hTnsfK6+1wZTWoGtl9OZc0k4T3ULQWoqTRP6vESYm77xHNQCFE6xFwBiuLHg Eu4pnNQXHR/H5tbJeGUEPUAlhAa1ipE5J5uX5q7jCYG9nkGueUD7m1sxG0dYnELvqNQ== X-Received: by 2002:a17:902:3183:: with SMTP id x3mr44271753plb.170.1556263909317; Fri, 26 Apr 2019 00:31:49 -0700 (PDT) X-Google-Smtp-Source: APXvYqz+olJwuJJKGfoagbm7Kix/nVhOeGf+kuu2frF5QRzp6oSNzLb9fV4Nuzlgolap9v50YJS9 X-Received: by 2002:a17:902:3183:: with SMTP id x3mr44271679plb.170.1556263908323; Fri, 26 Apr 2019 00:31:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263908; cv=none; d=google.com; s=arc-20160816; b=ds9fLNlEtNYEvRHrUVuT2jIWzODiFEFXnfTw+Msyjz+wzo5mbp8S6t/eIepgya5uIV 1BTEV1Oy8CeO9luOZaC0dNwqMYTza8POM1xNa2coMQYnP/V8T7VFY9UzpzviSgiLdndS JtyTqSHwYNqnSB5Y/UPX2AyS5dX12off0RuZJZHmYgs/55NYBGlQ7fQVge5ZD7ePzcQo JqXdsViSbX1b1N7i8S0Fr5/PykE31X/Us8Gij/Ydehv8uSYmRpxBNuZ/XWrvmFlR4VmB eYOYsG7MiE3EeIV+HB3PCDVOAWLNv+dGRXQ3bqS42mxHLHta5q7QC9WqX+xuU9Ctjly3 MZFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=FbRVtA2o2v3UWbAlXvOdnbyNDEqG2rBMqLF6HOTpV1U=; b=iwKv2GKmk46z3ivRqgWRuly/IzbrN0aEAzqGYds/xf+diZq1slSTwnw4qZqbmV7Ttq bTGnxwcnHpRBk21x62qUXzwXtmmjdlfD/DP36jBtJ2ld8dULYY/pe4ozUIFyXmvkR2N+ LH61SoqiZdmwRrtn/JnMHZ+mYCTC1du3Juob4IanVzVNCaehmGMG+KjDs+pJQuDvpZd/ N25WMK5iz1knT3NYl0xwblR/iUnTrcl93XZ/nP9EXD3iRlqFIzw89HxnXbA3zYiFz+Hi C5TJYs2kU22jHTmpdMCcRZRZOwh/WCmswfnnWhbozew7KK7ZMNMZu7OAEPJ276GDu+Km BNtg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-002.vmware.com (ex13-edg-ou-002.vmware.com. [208.91.0.190]) by mx.google.com with ESMTPS id f9si22844507pgq.347.2019.04.26.00.31.48 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:48 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) client-ip=208.91.0.190; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-002.vmware.com (10.113.208.156) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:44 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id F2B3F41225; Fri, 26 Apr 2019 00:31:45 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe , Jessica Yu , Steven Rostedt Subject: [PATCH v5 17/23] modules: Use vmalloc special flag Date: Thu, 25 Apr 2019 17:11:37 -0700 Message-ID: <20190426001143.4983-18-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-002.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Rick Edgecombe Use new flag for handling freeing of special permissioned memory in vmalloc and remove places where memory was set RW before freeing which is no longer needed. Since freeing of VM_FLUSH_RESET_PERMS memory is not supported in an interrupt by vmalloc, the freeing of init sections is moved to a work queue. Instead of call_rcu it now uses synchronize_rcu() in the work queue. Lastly, there is now a WARN_ON in module_memfree since it should not be called in an interrupt with special memory as is required for VM_FLUSH_RESET_PERMS. Cc: Jessica Yu Cc: Steven Rostedt Signed-off-by: Rick Edgecombe --- kernel/module.c | 77 +++++++++++++++++++++++++------------------------ 1 file changed, 39 insertions(+), 38 deletions(-) diff --git a/kernel/module.c b/kernel/module.c index 2b2845ae983e..a9020bdd4cf6 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -98,6 +98,10 @@ DEFINE_MUTEX(module_mutex); EXPORT_SYMBOL_GPL(module_mutex); static LIST_HEAD(modules); +/* Work queue for freeing init sections in success case */ +static struct work_struct init_free_wq; +static struct llist_head init_free_list; + #ifdef CONFIG_MODULES_TREE_LOOKUP /* @@ -1949,6 +1953,8 @@ void module_enable_ro(const struct module *mod, bool after_init) if (!rodata_enabled) return; + set_vm_flush_reset_perms(mod->core_layout.base); + set_vm_flush_reset_perms(mod->init_layout.base); frob_text(&mod->core_layout, set_memory_ro); frob_text(&mod->core_layout, set_memory_x); @@ -1972,15 +1978,6 @@ static void module_enable_nx(const struct module *mod) frob_writable_data(&mod->init_layout, set_memory_nx); } -static void module_disable_nx(const struct module *mod) -{ - frob_rodata(&mod->core_layout, set_memory_x); - frob_ro_after_init(&mod->core_layout, set_memory_x); - frob_writable_data(&mod->core_layout, set_memory_x); - frob_rodata(&mod->init_layout, set_memory_x); - frob_writable_data(&mod->init_layout, set_memory_x); -} - /* Iterate through all modules and set each module's text as RW */ void set_all_modules_text_rw(void) { @@ -2024,23 +2021,8 @@ void set_all_modules_text_ro(void) } mutex_unlock(&module_mutex); } - -static void disable_ro_nx(const struct module_layout *layout) -{ - if (rodata_enabled) { - frob_text(layout, set_memory_rw); - frob_rodata(layout, set_memory_rw); - frob_ro_after_init(layout, set_memory_rw); - } - frob_rodata(layout, set_memory_x); - frob_ro_after_init(layout, set_memory_x); - frob_writable_data(layout, set_memory_x); -} - #else -static void disable_ro_nx(const struct module_layout *layout) { } static void module_enable_nx(const struct module *mod) { } -static void module_disable_nx(const struct module *mod) { } #endif #ifdef CONFIG_LIVEPATCH @@ -2120,6 +2102,11 @@ static void free_module_elf(struct module *mod) void __weak module_memfree(void *module_region) { + /* + * This memory may be RO, and freeing RO memory in an interrupt is not + * supported by vmalloc. + */ + WARN_ON(in_interrupt()); vfree(module_region); } @@ -2171,7 +2158,6 @@ static void free_module(struct module *mod) mutex_unlock(&module_mutex); /* This may be empty, but that's OK */ - disable_ro_nx(&mod->init_layout); module_arch_freeing_init(mod); module_memfree(mod->init_layout.base); kfree(mod->args); @@ -2181,7 +2167,6 @@ static void free_module(struct module *mod) lockdep_free_key_range(mod->core_layout.base, mod->core_layout.size); /* Finally, free the core (containing the module structure) */ - disable_ro_nx(&mod->core_layout); module_memfree(mod->core_layout.base); } @@ -3420,17 +3405,34 @@ static void do_mod_ctors(struct module *mod) /* For freeing module_init on success, in case kallsyms traversing */ struct mod_initfree { - struct rcu_head rcu; + struct llist_node node; void *module_init; }; -static void do_free_init(struct rcu_head *head) +static void do_free_init(struct work_struct *w) { - struct mod_initfree *m = container_of(head, struct mod_initfree, rcu); - module_memfree(m->module_init); - kfree(m); + struct llist_node *pos, *n, *list; + struct mod_initfree *initfree; + + list = llist_del_all(&init_free_list); + + synchronize_rcu(); + + llist_for_each_safe(pos, n, list) { + initfree = container_of(pos, struct mod_initfree, node); + module_memfree(initfree->module_init); + kfree(initfree); + } } +static int __init modules_wq_init(void) +{ + INIT_WORK(&init_free_wq, do_free_init); + init_llist_head(&init_free_list); + return 0; +} +module_init(modules_wq_init); + /* * This is where the real work happens. * @@ -3507,7 +3509,6 @@ static noinline int do_init_module(struct module *mod) #endif module_enable_ro(mod, true); mod_tree_remove_init(mod); - disable_ro_nx(&mod->init_layout); module_arch_freeing_init(mod); mod->init_layout.base = NULL; mod->init_layout.size = 0; @@ -3518,14 +3519,18 @@ static noinline int do_init_module(struct module *mod) * We want to free module_init, but be aware that kallsyms may be * walking this with preempt disabled. In all the failure paths, we * call synchronize_rcu(), but we don't want to slow down the success - * path, so use actual RCU here. + * path. module_memfree() cannot be called in an interrupt, so do the + * work and call synchronize_rcu() in a work queue. + * * Note that module_alloc() on most architectures creates W+X page * mappings which won't be cleaned up until do_free_init() runs. Any * code such as mark_rodata_ro() which depends on those mappings to * be cleaned up needs to sync with the queued work - ie * rcu_barrier() */ - call_rcu(&freeinit->rcu, do_free_init); + if (llist_add(&freeinit->node, &init_free_list)) + schedule_work(&init_free_wq); + mutex_unlock(&module_mutex); wake_up_all(&module_wq); @@ -3822,10 +3827,6 @@ static int load_module(struct load_info *info, const char __user *uargs, module_bug_cleanup(mod); mutex_unlock(&module_mutex); - /* we can't deallocate the module until we clear memory protection */ - module_disable_ro(mod); - module_disable_nx(mod); - ddebug_cleanup: ftrace_release_mod(mod); dynamic_debug_remove(mod, info->debug); From patchwork Fri Apr 26 00:11:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918581 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9C21692A for ; Fri, 26 Apr 2019 07:32:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8A65E28DA3 for ; Fri, 26 Apr 2019 07:32:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7E25B28DA4; Fri, 26 Apr 2019 07:32:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0968228DA2 for ; Fri, 26 Apr 2019 07:32:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 069B06B0266; Fri, 26 Apr 2019 03:31:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B18916B0274; Fri, 26 Apr 2019 03:31:50 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6A9626B0272; Fri, 26 Apr 2019 03:31:50 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 1CCFF6B0007 for ; Fri, 26 Apr 2019 03:31:50 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id f7so1491694pgi.20 for ; Fri, 26 Apr 2019 00:31:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=03x8MyH4ONo6RL8TDAuJcYQYBUTk8Xfu5EMEl5vv8zs=; b=KwQvd75x1sDx1bT4KwvVFnZGiftKeP7Ao+LoyDmfuYl/2OySHRzRyto9t7frjzLBGi t3sK1JjMrbsOn1xnEKBirDN9FQDfq7e7/crlbqvCt5pRw5sdwtAW51JBzzvIb7I2H+oE wd1RklG1YA1H7NZQXnt4m8eDyQ27+Ogk1ULDeQVEC11GVrpZtEk4wUslO//f5PGfBEfn SPhG613zVgcXaVf7kCRO0xTlVD6iUwDgzQszhCJeSqMww2nbQWp0wvQnE4XE/qc0X3wn 5QDkQTd3FFPHG6rKrdZ/0Z7v7UGK8VN00ad58st7XGKpdM6phhEWzWySNSAmGw+Be96J urGA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAV0ziiGBHjdgtMXyp6Wo1s9mLrKIfyh49hfXlRhq+jEioR9C2Ts Lhf5S+CVkPCSbz+7cUEUnivpmxF2ajGtE+XaCwJCDZ2uV7V4vwdcB1AQz8RP56AkpS8EMIv6/08 nfCeXIhwYLkQWKuZPgudnta4qsK1PzwGcwIPgb4YZcvSy8RnDsPcguPOeLz3zojTjMQ== X-Received: by 2002:a17:902:201:: with SMTP id 1mr44722111plc.89.1556263909728; Fri, 26 Apr 2019 00:31:49 -0700 (PDT) X-Google-Smtp-Source: APXvYqzRhoCC13X4g10o5nZfIWed4q3JdFpqSGvxYAkkG9NVs1PjSuAd5rmnHzL6nFN8rCcLk0oc X-Received: by 2002:a17:902:201:: with SMTP id 1mr44722009plc.89.1556263908548; Fri, 26 Apr 2019 00:31:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263908; cv=none; d=google.com; s=arc-20160816; b=EbNHQmCYhkTfFH07IA4U5JIxWnaQPOhjNdxMGBcycWV3SSS1p6NPs+xNYYbn4P8KYB oq+wQBIJ1m25UVFeWBneOrm+jUHrueAgJj5x+vrA+gaN+dUF1+VbTeX1S7pD/hZIEs6m 73a6lx9iVCjzBDk7Iv00gHglWu3oHzStY19ymZzz4sgfEQS0Sj6P6t41We0eLVQBXS8T vbLm7FVbJ4loz9/QAI+03OCw+WuCK0Hc4yPiokYzme4W0pumALWaAW52EUFn9BNeR5XB tgc5Hwu2BJD/mToghoH3N+OCz1MANpio1CIXdI7CN25zlHEVxqkCUbimZohqu+/wdWyC 8dDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=03x8MyH4ONo6RL8TDAuJcYQYBUTk8Xfu5EMEl5vv8zs=; b=gHi7VE3rO9tcb9Ele1xEdjm6Tsv43Un5rQaUF7mYhoOBTSlbuJ+V+13noSTjQo31y+ KciDlrtpTGEHrc8x/fxqmBHTOVmclxvz9U1903VmD1+0nvEKcXi88oL3O4WeyqHgE1Qg pSj2gir9JG5RcQ8zhZy/nQcpmwNGoDwPQvPE/kYuPEioH001AcbmftAJxNwczrWKm0CO 9M5uQcQ2j8C1sA3Xb7OABXJG3zPUY9kyM62/c31PbGaytUsQhHMbhGE8wpAyGfItqJYG s4oKgMEpVoDXIt+j5am+zC6+0fvVI7rvib1qu2uvbikOwvGlcE0ABKf8JNPidQDEzqo6 gr7w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-002.vmware.com (ex13-edg-ou-002.vmware.com. [208.91.0.190]) by mx.google.com with ESMTPS id f9si22844507pgq.347.2019.04.26.00.31.48 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:48 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) client-ip=208.91.0.190; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-002.vmware.com (10.113.208.156) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:44 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id 0758741298; Fri, 26 Apr 2019 00:31:46 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe , Daniel Borkmann , Alexei Starovoitov Subject: [PATCH v5 18/23] bpf: Use vmalloc special flag Date: Thu, 25 Apr 2019 17:11:38 -0700 Message-ID: <20190426001143.4983-19-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-002.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Rick Edgecombe Use new flag VM_FLUSH_RESET_PERMS for handling freeing of special permissioned memory in vmalloc and remove places where memory was set RW before freeing which is no longer needed. Don't track if the memory is RO anymore because it is now tracked in vmalloc. Cc: Daniel Borkmann Cc: Alexei Starovoitov Signed-off-by: Rick Edgecombe --- include/linux/filter.h | 17 +++-------------- kernel/bpf/core.c | 1 - 2 files changed, 3 insertions(+), 15 deletions(-) diff --git a/include/linux/filter.h b/include/linux/filter.h index 14ec3bdad9a9..7d3abde3f183 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -20,6 +20,7 @@ #include #include #include +#include #include @@ -503,7 +504,6 @@ struct bpf_prog { u16 pages; /* Number of allocated pages */ u16 jited:1, /* Is our filter JIT'ed? */ jit_requested:1,/* archs need to JIT the prog */ - undo_set_mem:1, /* Passed set_memory_ro() checkpoint */ gpl_compatible:1, /* Is filter GPL compatible? */ cb_access:1, /* Is control block accessed? */ dst_needed:1, /* Do we need dst entry? */ @@ -733,27 +733,17 @@ bpf_ctx_narrow_access_ok(u32 off, u32 size, u32 size_default) static inline void bpf_prog_lock_ro(struct bpf_prog *fp) { - fp->undo_set_mem = 1; + set_vm_flush_reset_perms(fp); set_memory_ro((unsigned long)fp, fp->pages); } -static inline void bpf_prog_unlock_ro(struct bpf_prog *fp) -{ - if (fp->undo_set_mem) - set_memory_rw((unsigned long)fp, fp->pages); -} - static inline void bpf_jit_binary_lock_ro(struct bpf_binary_header *hdr) { + set_vm_flush_reset_perms(hdr); set_memory_ro((unsigned long)hdr, hdr->pages); set_memory_x((unsigned long)hdr, hdr->pages); } -static inline void bpf_jit_binary_unlock_ro(struct bpf_binary_header *hdr) -{ - set_memory_rw((unsigned long)hdr, hdr->pages); -} - static inline struct bpf_binary_header * bpf_jit_binary_hdr(const struct bpf_prog *fp) { @@ -789,7 +779,6 @@ void __bpf_prog_free(struct bpf_prog *fp); static inline void bpf_prog_unlock_free(struct bpf_prog *fp) { - bpf_prog_unlock_ro(fp); __bpf_prog_free(fp); } diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index ff09d32a8a1b..c605397c79f0 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -848,7 +848,6 @@ void __weak bpf_jit_free(struct bpf_prog *fp) if (fp->jited) { struct bpf_binary_header *hdr = bpf_jit_binary_hdr(fp); - bpf_jit_binary_unlock_ro(hdr); bpf_jit_binary_free(hdr); WARN_ON_ONCE(!bpf_prog_kallsyms_verify_off(fp)); From patchwork Fri Apr 26 00:11:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918563 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 706ED92A for ; Fri, 26 Apr 2019 07:32:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5E3D228DA2 for ; Fri, 26 Apr 2019 07:32:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 527E028DA4; Fri, 26 Apr 2019 07:32:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CFEFA28DA2 for ; Fri, 26 Apr 2019 07:32:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 21F636B0271; Fri, 26 Apr 2019 03:31:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DF7B16B026C; Fri, 26 Apr 2019 03:31:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA23E6B026D; Fri, 26 Apr 2019 03:31:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id E624E6B026B for ; Fri, 26 Apr 2019 03:31:48 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id s26so1646146pfm.18 for ; Fri, 26 Apr 2019 00:31:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=RIE66crRSFais2XMEreovT7cuc4aWzwk2c6XjFrsy1k=; b=sO3PEOEu0zMwHfobvo/GIOgY/Kfk2z+Gp/Lfh3htVxppUgAFLdUbN9d3homGNR55Yo m+qar2aZJMPtyMgRwS+h+3zQMC/r9PppdpIGW4+Km/eRoG7LecnfkyAD06/k/4lG9FRS g+1wmZYWGQgMW91VLC1h3XBbHkaZLi4YGoIvcbj+FkKPybB8Ofy+bZnpW0uWUe0Qhu53 p8Qi4amuQzmGP9CViUnPOGZBFNZC+lbBX2XqWnDd5zy7vjUo1D8R4L/qEi0O1mvLtlaE z1nZLIMIIlC1i+KjzLcZv8gbR+5UqUEOB/V8mbGjP+4ADVu2VESOd+G+nHfEEl1djtI3 mX9A== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAVEctG1BSO00Hrm4yq7QPVgqJGkYN1Z6SGmPgeHlK4Gv8jkGCbE lHR97GDbWKZbOMQy5l5DUHHV4D7XlMxPo38Bk17tNesIY5eP+Mz4VU7On+b5IrqycSXVoQK2Po+ HcBeEUnKl/LU147OAmhKCICb8mRHQX8pgZ0y/pp9L/OwP7zB53ZTmrva0rfHVEscDAg== X-Received: by 2002:a63:f448:: with SMTP id p8mr42207797pgk.50.1556263908581; Fri, 26 Apr 2019 00:31:48 -0700 (PDT) X-Google-Smtp-Source: APXvYqzu3cjqKi8CgTRqqdft5ubUUxrPPTBrYsfd49GmA87QBn6VwaEMmwqMG5P28PH8Be5k02dN X-Received: by 2002:a63:f448:: with SMTP id p8mr42207710pgk.50.1556263907273; Fri, 26 Apr 2019 00:31:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263907; cv=none; d=google.com; s=arc-20160816; b=b7MMLCn1I8omPuvRsqzkP/x6g9OXst7Z28rdl+ghF4+ONBhPVl6Kys9w5tXTjjAzDP 5SRkbSiwM2/GIPjAo5pYm5/Hj4nwE3x4XW7uwZyANgcOCv3s84sqesRgbveEDm0lRztr IqdzjbRl0BqeMlE5VUP83siR2wTjlJGHNHGJgCZJtgSu1+tFk/nIyeAkXpFKFp/aowrz TBH9UrD0Brfs7HWrrEMQy0KJCrP4nNIMHEcdoLLKUm7tzdpMdx28ibpJk2MPW6nXAwqm y8y9Sa3skXNw0K0gsFieGXyZ0sbI40YtgJgqBGSIFlzvla+nMdgpigLv+HAmK/JSIskO ZPtg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=RIE66crRSFais2XMEreovT7cuc4aWzwk2c6XjFrsy1k=; b=WMuWdgzpFtSZg+UDbkmfLIreLp1Yg+M+BZ/uGB2kbnN9rR+t/z+bbLSwXCIfYYFj5F msMBzYnAFm0TSDCATsueOToIamV4bFGUlcVNCEj2EdRr/mI7wxbCTtY1Tc3zLsge4VPx Sx+v0R0RBvzL2VBdpkk2af49Jf6dpBM/p51DipXXpOXuNI8YFjzrLBpyFyi6A90Ptznq ldjQbPYwO067cldw7B1gdmf/X5sJUzm1VUZZ/39uhjZ/l47DiOXhKzP29n1cZAyQGrkR ohOyjFi73DWcYK5CCDln3/uqUGlX0OulSW7cULn4ZdUfr3ssA2sCjjVP/UkTCYPwfKIR sb8Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-001.vmware.com (ex13-edg-ou-001.vmware.com. [208.91.0.189]) by mx.google.com with ESMTPS id v82si25417769pfa.42.2019.04.26.00.31.47 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:47 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) client-ip=208.91.0.189; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-001.vmware.com (10.113.208.155) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:40 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id 1086741299; Fri, 26 Apr 2019 00:31:46 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe Subject: [PATCH v5 19/23] x86/ftrace: Use vmalloc special flag Date: Thu, 25 Apr 2019 17:11:39 -0700 Message-ID: <20190426001143.4983-20-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-001.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Rick Edgecombe Use new flag VM_FLUSH_RESET_PERMS for handling freeing of special permissioned memory in vmalloc and remove places where memory was set NX and RW before freeing which is no longer needed. Acked-by: Steven Rostedt (VMware) Tested-by: Steven Rostedt (VMware) Signed-off-by: Rick Edgecombe --- arch/x86/kernel/ftrace.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index 53ba1aa3a01f..0caf8122d680 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -678,12 +678,8 @@ static inline void *alloc_tramp(unsigned long size) { return module_alloc(size); } -static inline void tramp_free(void *tramp, int size) +static inline void tramp_free(void *tramp) { - int npages = PAGE_ALIGN(size) >> PAGE_SHIFT; - - set_memory_nx((unsigned long)tramp, npages); - set_memory_rw((unsigned long)tramp, npages); module_memfree(tramp); } #else @@ -692,7 +688,7 @@ static inline void *alloc_tramp(unsigned long size) { return NULL; } -static inline void tramp_free(void *tramp, int size) { } +static inline void tramp_free(void *tramp) { } #endif /* Defined as markers to the end of the ftrace default trampolines */ @@ -808,6 +804,8 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) /* ALLOC_TRAMP flags lets us know we created it */ ops->flags |= FTRACE_OPS_FL_ALLOC_TRAMP; + set_vm_flush_reset_perms(trampoline); + /* * Module allocation needs to be completed by making the page * executable. The page is still writable, which is a security hazard, @@ -816,7 +814,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size) set_memory_x((unsigned long)trampoline, npages); return (unsigned long)trampoline; fail: - tramp_free(trampoline, *tramp_size); + tramp_free(trampoline); return 0; } @@ -947,7 +945,7 @@ void arch_ftrace_trampoline_free(struct ftrace_ops *ops) if (!ops || !(ops->flags & FTRACE_OPS_FL_ALLOC_TRAMP)) return; - tramp_free((void *)ops->trampoline, ops->trampoline_size); + tramp_free((void *)ops->trampoline); ops->trampoline = 0; } From patchwork Fri Apr 26 00:11:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918585 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A188B14C0 for ; Fri, 26 Apr 2019 07:32:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9084C28DA2 for ; Fri, 26 Apr 2019 07:32:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 83C2F28DA4; Fri, 26 Apr 2019 07:32:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3592928DA2 for ; Fri, 26 Apr 2019 07:32:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 46D346B0007; Fri, 26 Apr 2019 03:31:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id F09726B0273; Fri, 26 Apr 2019 03:31:50 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C248D6B026D; Fri, 26 Apr 2019 03:31:50 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 513B86B026D for ; Fri, 26 Apr 2019 03:31:50 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id v9so1509195pgg.8 for ; Fri, 26 Apr 2019 00:31:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=PMP2Cxb893S0prGS0vCi085zHCDPTtmjtSNHdboWACs=; b=KRFhzxyQbxI5f2oe6IxhsiGnJsbNlagc7Z364tzeGWIoTulKEt1Iu4DRSzMje9ytUh nwVEl1y9P11/FYwndQtwMmUXvyC5bcKy7hB8yPZ8ydA8sGdvJ+1O69IGbR76APNDrDFW cxCzUt2oDJ1CbcgzvZ7PEx+LmVXIp6Lx1vHK2pnKnrPfF29Rn1Ih594Ajg1LHaOwMIuX aRhsZFPBnf/wd8NAvWG7RytrfvWNXZNn3yASXdCp6SXbVeoLnjtR6D60vuayDHcC5bYv wFBN8EmpxahZehNpL6AXHlkovhUdDGqyI5ZG8Y94vPi2fwCsXF2cQhScLulRsGZMHKfc jUMQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAVVEfXUgSwDV4Xyrp/TrLeHuhSlF2Gvk02umkjunfcZtxnMt++X omssGOlJ+JcqXiFwfIX7SvX2S/0zFOKPvwtEBrVJglMQCf7//Nn5z+PW4NGapDzFBYbtsZtqdhY nEHrcTZTCpiurKx++zfwxmuD0Rw6jo/aOw/pFFch3RdAjwYRnDPHKZ+qbx2vtZK2aLA== X-Received: by 2002:a63:165f:: with SMTP id 31mr42513628pgw.321.1556263909992; Fri, 26 Apr 2019 00:31:49 -0700 (PDT) X-Google-Smtp-Source: APXvYqzLJwkwhUGtfYO8PZHsAPxLQq916eGTtamDVeP2MnVhICTfBPlba9rHEANFOrWIRdMR0Wq3 X-Received: by 2002:a63:165f:: with SMTP id 31mr42513560pgw.321.1556263908999; Fri, 26 Apr 2019 00:31:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263908; cv=none; d=google.com; s=arc-20160816; b=ggb0xoJh3MsZ5zH902CeQY5bLhBgUfwxaydev5M3he/sQA7lNIKYeRUqVj2gbjBN5s kb83V6g5jHhXnUfJE+ekOnPYGmDskLTW84heWdcXMSMBQDxqeZ56ZV0LAYz+RXblvdI1 znrb70EHA0jpAt8OyluTBlMJxdCsAKbS8ZO4oxXEyuLlwnhe/h54e/7P4lM9ErLYJT2y jPxY5EIxh2MJUrA8RJaYoqkNsjd7cTMue2+LClrt8cICEO/jtyN0GS/jvEMFhqAa/UV4 cUYp8ZaTxGgAAkelCDBYTSVpXd0nTJG3rXaYgmIFGwmZEbNqwklndM4SFWhSGVl6isVx pARw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=PMP2Cxb893S0prGS0vCi085zHCDPTtmjtSNHdboWACs=; b=kEdGFvkvGBs7uCjIQCDI7n51gju2ys9f7K+a9wpsF4ioa/zNRga0bygBKUme0U+I6a AvZ8SvZr3OFPcVZkPw+BA840QRGFhSqYzlXFp0wHaAqXQbyvmK5iQG7XFIPGu0FCGRVF qTrEI5hOly7/8cVVvULpR1g1pSSC4ob2QNR8h4RHGak2J1g4Dv5PDN+RkB//e7IE84zi FYnxcGcgUnEinUJ7RWNgN5XaDakd6U0KHWZs+3dBqYOxSrdydvkQAmCYa10/DjG2p6Tc DsEelB8BglPeVSZQw0Uv1ANh6jGfmX1rm+H0Igz5ZcUPTWrbcIljTSV9kKwBDZQxiKC8 e4Kw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-002.vmware.com (ex13-edg-ou-002.vmware.com. [208.91.0.190]) by mx.google.com with ESMTPS id f9si22844507pgq.347.2019.04.26.00.31.48 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:48 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) client-ip=208.91.0.190; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-002.vmware.com (10.113.208.156) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:45 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id 186974129C; Fri, 26 Apr 2019 00:31:46 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe , Masami Hiramatsu Subject: [PATCH v5 20/23] x86/kprobes: Use vmalloc special flag Date: Thu, 25 Apr 2019 17:11:40 -0700 Message-ID: <20190426001143.4983-21-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-002.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Rick Edgecombe Use new flag VM_FLUSH_RESET_PERMS for handling freeing of special permissioned memory in vmalloc and remove places where memory was set NX and RW before freeing which is no longer needed. Cc: Masami Hiramatsu Signed-off-by: Rick Edgecombe --- arch/x86/kernel/kprobes/core.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index 1591852d3ac4..136695e4434a 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -434,6 +434,7 @@ void *alloc_insn_page(void) if (!page) return NULL; + set_vm_flush_reset_perms(page); /* * First make the page read-only, and only then make it executable to * prevent it from being W+X in between. @@ -452,12 +453,6 @@ void *alloc_insn_page(void) /* Recover page to RW mode before releasing it */ void free_insn_page(void *page) { - /* - * First make the page non-executable, and only then make it writable to - * prevent it from being W+X in between. - */ - set_memory_nx((unsigned long)page, 1); - set_memory_rw((unsigned long)page, 1); module_memfree(page); } From patchwork Fri Apr 26 00:11:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918587 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D5F9514C0 for ; Fri, 26 Apr 2019 07:32:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C451828DA2 for ; Fri, 26 Apr 2019 07:32:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B82D228DA4; Fri, 26 Apr 2019 07:32:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6A46A28DA2 for ; Fri, 26 Apr 2019 07:32:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 733AB6B026D; Fri, 26 Apr 2019 03:31:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 222B56B0270; Fri, 26 Apr 2019 03:31:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E13E36B0272; Fri, 26 Apr 2019 03:31:50 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 8D5FD6B0007 for ; Fri, 26 Apr 2019 03:31:50 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id f7so1491715pgi.20 for ; Fri, 26 Apr 2019 00:31:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=WkjX1OCtVjmmi3iE32HdXzI4HYBsQAWEqrRzLHows7E=; b=aAVmHcaL4FnJpCIukB/pmajJfo9YuB9C2awtEcLJQiRx96l7TFmmr8+MV86wiEK7Uh hA0IcKsx8u6F0VRocAJjKEo7wNz8A6jWk2lbEo7M5kZ74cGoy8tS9QvD8bZ0DT/Mo95a odIAKTgBhnk1KGjrGozw7Pg9sKc2DYxd9ZTMl9VUBHsIdUrLW6WpkCXkZ0GkIlW74Fx6 3/HkrK9Vg0e1VPTihnRmjyKO9i/zivhN8Rwpm7DWc2sNRJqGxNKc1dpJVsLP681S7Pdl 0/XyXMLs1Ze79cJZjhqTND847m+t17dcQ0QDh6hP8xj7najtRx1oos7d+cfHKu/TB8MT AURw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAXXrOF4Cx11O5daqtXPBDXAy7CQlyVbUw/Ij8DJi9XLTx+U9oeU NuFYcDV8+CxrylU0ao7p+eieYHCfos3EFNJHRPB8blj4ZLpn14pnIhZQk+vC22EuQEdcJLqTP0K 7a0PgU7pAX47idNtpYfBNPDsEeyQqsErVKhb3YYFa+Tbw4F8oI8RhYEwiY89ObOh8mg== X-Received: by 2002:a17:902:e485:: with SMTP id cj5mr18289706plb.280.1556263910238; Fri, 26 Apr 2019 00:31:50 -0700 (PDT) X-Google-Smtp-Source: APXvYqxpVmEQl/UKcLGs+2eEo+5yHAua6cLZgh6mruoKJvgssQz5wIeP+GlQcuNN35pkBrE6I2EU X-Received: by 2002:a17:902:e485:: with SMTP id cj5mr18289636plb.280.1556263909272; Fri, 26 Apr 2019 00:31:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263909; cv=none; d=google.com; s=arc-20160816; b=ncakRD0gU7Wd6YFbSw810rwpJsAVmQTioFX6aKBv7cmIOxPcgss2mI5UbEREslsOMs J2NC874dVHEluouYKGEFSyNXzocl8LJDLqmPu4sVHj1NB35ViniRzJikjMQDuNms4i/2 v5Kh7BahGwaTyp5fOPZ+fHHkkuTpsLWsjbgPSww08j87qiUhEzTBMw9RHqJtShKG6VyA bCMMgbwXX3vAmT9GY9DR0JoI5YVAHJshLkFzMISXQ2o05b8MMMwVcDSXAcaDITaWq+oS 9fuDyH+xRO1K0SyohzyrowAm6OSJMRpFqLuIbx+AigAEIRgz+C5RWGN2QRr27tBEm0WV Wi6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=WkjX1OCtVjmmi3iE32HdXzI4HYBsQAWEqrRzLHows7E=; b=Lj1hj2UfMz5T45865ZwDRb2zNb71aLzmzhSBsySgEKeTtfDNPl88auy1/EvAkb3gaK SjVU6GyDpukoVXdSrxXztaO2fqnUEjc0VAKXs0EerimjJmzivUidwkbmqTnVScOXEEIv XdSZQrP09Nf1a+R++eyKK6E6o92fi3EEi8Mlc45YmtyyS1SKLaEXVr7MIh6dwrn/ef+C Ej/wSxL5OGl2s8fvizMS56Wre1Si+Iqb4m55HyPeyRiotn3KLVrRmLLfx7fHe56rCM0Z /pA2BTzEZrm7EDAgdjEb2ZLeONmZeoDpAwILZzXv850iyOSlv+/CX5niG3lEAaAX8dXS LYew== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-002.vmware.com (ex13-edg-ou-002.vmware.com. [208.91.0.190]) by mx.google.com with ESMTPS id f9si22844507pgq.347.2019.04.26.00.31.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:49 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) client-ip=208.91.0.190; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-002.vmware.com (10.113.208.156) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:45 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id 2130D41225; Fri, 26 Apr 2019 00:31:46 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe , Nadav Amit , Masami Hiramatsu Subject: [PATCH v5 21/23] x86/alternative: Comment about module removal races Date: Thu, 25 Apr 2019 17:11:41 -0700 Message-ID: <20190426001143.4983-22-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-002.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add a comment to clarify that users of text_poke() must ensure that no races with module removal take place. Cc: Masami Hiramatsu Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/kernel/alternative.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 18f959975ea0..7b9b49dfc05a 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -810,6 +810,11 @@ static void *__text_poke(void *addr, const void *opcode, size_t len) * It means the size must be writable atomically and the address must be aligned * in a way that permits an atomic write. It also makes sure we fit on a single * page. + * + * Note that the caller must ensure that if the modified code is part of a + * module, the module would not be removed during poking. This can be achieved + * by registering a module notifier, and ordering module removal and patching + * trough a mutex. */ void *text_poke(void *addr, const void *opcode, size_t len) { From patchwork Fri Apr 26 00:11:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918663 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B085C1575 for ; Fri, 26 Apr 2019 07:33:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A0AD928DA3 for ; Fri, 26 Apr 2019 07:33:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 94E4A28DA2; Fri, 26 Apr 2019 07:33:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 41E0F28DA3 for ; Fri, 26 Apr 2019 07:33:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D2B36B0289; Fri, 26 Apr 2019 03:33:24 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 804DB6B028B; Fri, 26 Apr 2019 03:33:24 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E47C6B028C; Fri, 26 Apr 2019 03:33:24 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id 101466B028B for ; Fri, 26 Apr 2019 03:33:24 -0400 (EDT) Received: by mail-pl1-f200.google.com with SMTP id gn10so1422313plb.23 for ; Fri, 26 Apr 2019 00:33:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=PQm9c3rdPLBXTcMKjbO2GS15ACOI1ILM5Qbwtu4lFBw=; b=Su0K+KneqvAgN0zSEc5aJQgpZhcjaHd6XKwH+acGKr5cbIxWb7gWdYSEnxt2QgifLS 9QLG+15v4SaBx7TLGl36lm9r6+xY1Zpa1/BEopp6pzXlVrkkRxfV86zyz08cgsZsV0AM CuLL/rrjNNBnUkM6bz01nvuzQELUTIZ+LCyB+u83bZixxVoK+OsPj7LS0WG3yuAszq8f NGHjpOQQ2OUwudXjw/p6K9Nzwgms1e1auyDowrEaSYjOkqFX35SRHis3u+tiyhWT/TcM 3lj9HCyQq8iji31OancL+aHZmrNqViwi2MEBjxkEJyF/oSH/cGICkbegrtftDoRaZRWc vyLQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAWHse0WMEHh22ZAsgn+nYh0F4T55dAJ/wxVDumnH5hBKJIrIwed Do0ucFFAn8lAN4x5OyJdP+f78oOA10E3TbMg0IfKWycL6UYaCzSRxPCsbacMx9XMyJqWZx5Kx9j aYIgUI6gUiN+Fq+P7l7xJ3OUtSyqFqM8p6UTts3WIuH/wIBFxMc34RkjbNSlPJv8Q4Q== X-Received: by 2002:a63:fa46:: with SMTP id g6mr42574663pgk.382.1556264003706; Fri, 26 Apr 2019 00:33:23 -0700 (PDT) X-Google-Smtp-Source: APXvYqyCB2ybfwR+m/jXvenxIHEBHDizQSKWN6uHx6/IBhJBeZTQQ9VuMecMMBDD//sO9rjPH5Db X-Received: by 2002:a63:fa46:: with SMTP id g6mr42567682pgk.382.1556263907475; Fri, 26 Apr 2019 00:31:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263907; cv=none; d=google.com; s=arc-20160816; b=PxwtttOIZDgbVfPxiwNCMfVH1MifF4TW/Pcfi0qy7HgorUCcp1rnUSteSiklJ9REqR i9UQcFeI4XiHvmAzh6I9v8tLcgPiMMhXxtEllXsdKdlTx4mQHbXX8Pm30rbnB/7Fu365 cibnlmh1U3I0vT/HvIDIZ+Y/lF3/WwJlAXBmyhtAbiJUgCr66uKbV0JESTNX7vlUUCDD ePIbdYNoH4BG9yxr6NFsWeltqbz7qmQJzyez60AlDrYwd1zTtT7/2jgdUeSDvEcIlL+c Rx1InGJ4PxvjRSpbgmr+z9S6NORHTGzrqbMEvi/CjCNY6U6AwT1XbERZHRQcNFk2gIvM lHmQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=PQm9c3rdPLBXTcMKjbO2GS15ACOI1ILM5Qbwtu4lFBw=; b=o7hVE8cZZofekeDKLJhpQbkXOAINWrEarl9JoN5wEqSRSbho3Z4cPoMmurq08ptH/i o5djp4FcdGsQFVXYBdz2mzm3o/P+hnH/TeYEd1gkXKa9maL0KESeTArpwAHUiOH5248G A/WFcZsPtfBuqeSb3y74zwnqPXkuZprXP0Ii/nCs9nkF+5yJzR2FAey9lkz7ht+rAyr6 BHb8aMZvRo+HlM2ds5RKP9GS8gh86YhM25/YICqoWqeagJHznof2qgzXyxfkQ3iq8PZU z0f21tVyrLred5+YuiEJQFeIiv80+ebArgG9TFL7UiW9tKWifUXNb/yt69WR6IDGHh8C x/UA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-001.vmware.com (ex13-edg-ou-001.vmware.com. [208.91.0.189]) by mx.google.com with ESMTPS id v82si25417769pfa.42.2019.04.26.00.31.47 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:47 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) client-ip=208.91.0.189; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.189 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-001.vmware.com (10.113.208.155) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:41 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id 296F2412A3; Fri, 26 Apr 2019 00:31:46 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe , Nadav Amit Subject: [PATCH v5 22/23] mm/tlb: Provide default nmi_uaccess_okay() Date: Thu, 25 Apr 2019 17:11:42 -0700 Message-ID: <20190426001143.4983-23-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-001.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP x86 has an nmi_uaccess_okay(), but other architectures do not. Arch-independent code might need to know whether access to user addresses is ok in an NMI context or in other code whose execution context is unknown. Specifically, this function is needed for bpf_probe_write_user(). Add a default implementation of nmi_uaccess_okay() for architectures that do not have such a function. Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/tlbflush.h | 2 ++ include/asm-generic/tlb.h | 9 +++++++++ 2 files changed, 11 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 90926e8dd1f8..dee375831962 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -274,6 +274,8 @@ static inline bool nmi_uaccess_okay(void) return true; } +#define nmi_uaccess_okay nmi_uaccess_okay + /* Initialize cr4 shadow for this CPU. */ static inline void cr4_init_shadow(void) { diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index b9edc7608d90..480e5b2a5748 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -21,6 +21,15 @@ #include #include +/* + * Blindly accessing user memory from NMI context can be dangerous + * if we're in the middle of switching the current user task or switching + * the loaded mm. + */ +#ifndef nmi_uaccess_okay +# define nmi_uaccess_okay() true +#endif + #ifdef CONFIG_MMU /* From patchwork Fri Apr 26 00:11:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 10918603 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EFAC815E9 for ; Fri, 26 Apr 2019 07:32:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DF22B28DA2 for ; Fri, 26 Apr 2019 07:32:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D336428DA7; Fri, 26 Apr 2019 07:32:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.4 required=2.0 tests=BAYES_00,DATE_IN_PAST_06_12, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=no version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7D65E28DA2 for ; Fri, 26 Apr 2019 07:32:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 956EF6B0270; Fri, 26 Apr 2019 03:31:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 637CB6B0274; Fri, 26 Apr 2019 03:31:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D6B06B026D; Fri, 26 Apr 2019 03:31:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id E43BF6B0007 for ; Fri, 26 Apr 2019 03:31:50 -0400 (EDT) Received: by mail-pl1-f200.google.com with SMTP id d10so1438209plo.12 for ; Fri, 26 Apr 2019 00:31:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version; bh=64HAp1aiSsnJ+6SwHGBfPAaWomfs1X7LEUQ04wa1/ZY=; b=qLMtvlIF8QiRmEHMUS97VypPqUP2v444LE+I5rwEkUVy1+vw1Hp2F2sLMBsxkO9F7v 0KnJ+2BpAZwTWYudT7m3XTx3Mgu//O7VvNFh+gUexi5bek/jU80qlkVoF/Odr6R8alam EDZ9hJZRY7AAYbKjK/u75RiCA0in7OZRI9njl9ZRtlP/pQvA6F9cJ/bKGxgW6k5yp6cp pvaQsv2RAlkwM771nm5MzNbgaEihP5mTMoz1xXQq1ut+wuLoyBC3vNZulSpY/ZOtWKNM NRjm9i8nHOFlaSzphTX9QpHjxgnIyT5jtMSb4WKAH3OCKE6lEN92S6GwmdZaRzJUhphZ EDFA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com X-Gm-Message-State: APjAAAWHlIzXZ+c3CTmne0010LsbLVIo61F2ZoE5C6HS40bCxHfuY+zD ZHSdmxPeBx5KEYH1HoI/9Ywgjc5zsiFDCUg/TR0s8eQ8EziybFDQGhBF0KhvI+eJ6VXfODTzfNp zr7hkaR6Q6oZyv1lWeI4drIvi1cA2sZUHvnORj8mceAMcIkfUFiF2AHdsWDaQMU4myw== X-Received: by 2002:a62:5795:: with SMTP id i21mr16806404pfj.194.1556263910592; Fri, 26 Apr 2019 00:31:50 -0700 (PDT) X-Google-Smtp-Source: APXvYqwWH0t2lD8SJ7EkhKEPHEGLzJJYN3l7Q7NVVVBogB6TDePfI3NsPkmf6t95aH5LQdx8Bmwz X-Received: by 2002:a62:5795:: with SMTP id i21mr16806318pfj.194.1556263909468; Fri, 26 Apr 2019 00:31:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556263909; cv=none; d=google.com; s=arc-20160816; b=ui3b8z+dFWiGS78CYJ2vM6gyUuQMi8XwRLYQTDqFALe2GwTWHKACF9IfbKMJQjSHM9 n0gn5CB/XerE4A3CRUWB15rsOYYMIxoUjTP3nB7arV48FVRzKnpN9SKv99AEwFqxzcmm 6wEMd9U/zcUB2/p3jDsV8D0CiLTPnMpJAPJJ7U22dIFsPzOEYEVKTxCBh570KRMK8uFI 002GUzPqZims0j9XUGKcWVwXvDn5F1FpTKljJJCA0Qq2hLSxCrSVTj0j16tlx7zlKNSi DPEUhPfDg2aqcrltZOIUy2EJj+nYov95e/L/H+isuFN4rHnkcu675a+i42cbeVnK+vHJ 4QZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=64HAp1aiSsnJ+6SwHGBfPAaWomfs1X7LEUQ04wa1/ZY=; b=Ab8BruFXUGNliAQZmJyqotoBe8ha4jOrYZIZxQqVODzN/qVSIOfABSwKtKGMHHrbjB oX60sAuCY6ba9zwTjx0va+oWG/D6GoFyIicq09oydri7rbD8zV0OoKl1fhHQWAjttTgI XfiD1Z4OejZQMGn0ggFEd5ogsi4aUZG+mIZ3Jdp9OPv+bb5UA18RMahF44baUhnfxpO0 dNawx5/ZFfNIK3+XnpDoURkl253iovBwBTZ97dWyR9f56Pkwi4LGEIZt1Hk+4rY9KPqz PaADorcJjQyeMnn0ohZuBRll6KLVIR1Q7gSt46eL/fOCFZiTgaWypdaQNd/R1HEsmKfU u5Og== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from EX13-EDG-OU-002.vmware.com (ex13-edg-ou-002.vmware.com. [208.91.0.190]) by mx.google.com with ESMTPS id f9si22844507pgq.347.2019.04.26.00.31.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Apr 2019 00:31:49 -0700 (PDT) Received-SPF: pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) client-ip=208.91.0.190; Authentication-Results: mx.google.com; spf=pass (google.com: domain of namit@vmware.com designates 208.91.0.190 as permitted sender) smtp.mailfrom=namit@vmware.com; dmarc=pass (p=QUARANTINE sp=NONE dis=NONE) header.from=vmware.com Received: from sc9-mailhost3.vmware.com (10.113.161.73) by EX13-EDG-OU-002.vmware.com (10.113.208.156) with Microsoft SMTP Server id 15.0.1156.6; Fri, 26 Apr 2019 00:31:45 -0700 Received: from sc2-haas01-esx0118.eng.vmware.com (sc2-haas01-esx0118.eng.vmware.com [10.172.44.118]) by sc9-mailhost3.vmware.com (Postfix) with ESMTP id 32C2B4129A; Fri, 26 Apr 2019 00:31:46 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Borislav Petkov , Andy Lutomirski , Ingo Molnar CC: , , , Thomas Gleixner , Nadav Amit , Dave Hansen , , , , , , , , , , , Rick Edgecombe , Nadav Amit , Daniel Borkmann , Alexei Starovoitov Subject: [PATCH v5 23/23] bpf: Fail bpf_probe_write_user() while mm is switched Date: Thu, 25 Apr 2019 17:11:43 -0700 Message-ID: <20190426001143.4983-24-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190426001143.4983-1-namit@vmware.com> References: <20190426001143.4983-1-namit@vmware.com> MIME-Version: 1.0 Received-SPF: None (EX13-EDG-OU-002.vmware.com: namit@vmware.com does not designate permitted sender hosts) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When using a temporary mm, bpf_probe_write_user() should not be able to write to user memory, since user memory addresses may be used to map kernel memory. Detect these cases and fail bpf_probe_write_user() in such cases. Cc: Daniel Borkmann Cc: Alexei Starovoitov Reported-by: Jann Horn Suggested-by: Jann Horn Signed-off-by: Nadav Amit Signed-off-by: Rick Edgecombe --- kernel/trace/bpf_trace.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index d64c00afceb5..94b0e37d90ef 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -14,6 +14,8 @@ #include #include +#include + #include "trace_probe.h" #include "trace.h" @@ -163,6 +165,10 @@ BPF_CALL_3(bpf_probe_write_user, void *, unsafe_ptr, const void *, src, * access_ok() should prevent writing to non-user memory, but in * some situations (nommu, temporary switch, etc) access_ok() does * not provide enough validation, hence the check on KERNEL_DS. + * + * nmi_uaccess_okay() ensures the probe is not run in an interim + * state, when the task or mm are switched. This is specifically + * required to prevent the use of temporary mm. */ if (unlikely(in_interrupt() || @@ -170,6 +176,8 @@ BPF_CALL_3(bpf_probe_write_user, void *, unsafe_ptr, const void *, src, return -EPERM; if (unlikely(uaccess_kernel())) return -EPERM; + if (unlikely(!nmi_uaccess_okay())) + return -EPERM; if (!access_ok(unsafe_ptr, size)) return -EPERM;