From patchwork Wed Dec 12 00:03:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10725327 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6ECBC159A for ; Wed, 12 Dec 2018 00:12:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5E0492B550 for ; Wed, 12 Dec 2018 00:12:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 521762B583; Wed, 12 Dec 2018 00:12:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A3F582B550 for ; Wed, 12 Dec 2018 00:12:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E15078E00C9; Tue, 11 Dec 2018 19:12:11 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BFF6A8E00E9; Tue, 11 Dec 2018 19:12:11 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A777A8E00E7; Tue, 11 Dec 2018 19:12:11 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f199.google.com (mail-pf1-f199.google.com [209.85.210.199]) by kanga.kvack.org (Postfix) with ESMTP id 5A1988E00E5 for ; Tue, 11 Dec 2018 19:12:11 -0500 (EST) Received: by mail-pf1-f199.google.com with SMTP id u20so14031288pfa.1 for ; Tue, 11 Dec 2018 16:12:11 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=dxV3upF7F0dcmLEbdfWFB9vg7rmE7IE+NndnAUM1Ecc=; b=XvPVDd7ZFUAnI9A3OotdVBw2HrfYukFhx9j/bRfDRay4vxw8BbhVfiUbKzVwjwaeLt C/v65kPmC1etGxfswxBpz5EgmMUz4McgxV3OjgUAVNvQEFy3RwzHGPODTESootTIqZWl MmrfVENGuvkWDy9PGuT9Xst0FWfAkEUa/156AqFNm96UkEuRWM/TA4ZH3WogDWe0LxKf 9mQhOHikycq6OmSObJPlEoz3/JhJl8IdtvpwpsM8tTAmBKnANPoex5sDV1cY9BrAmdCM Twu35qeR0akKSQoJljxqe98Gx7q/tfj5ENya3CATJU+GIa1thEau3+7U1VpSO3ddaJtr VxPA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AA+aEWae9f6EWMoRnV8PPhrEvH6Uy0gipn95KAFmOvJDW1AIjmME154c C4USDhoVC9v/VUJYfytM+BLCxa5KqO83ukthwDPharqiZLRFL3HhJFaYGlne55F5Be/2r4nrPVp B+fKlstRvR/levhbPlKrICL3914yvnhlaaqYIhtgQ5YAdm3dGsH6e2PpKlxns/Ql7hg== X-Received: by 2002:a63:a84a:: with SMTP id i10mr16702485pgp.263.1544573530998; Tue, 11 Dec 2018 16:12:10 -0800 (PST) X-Google-Smtp-Source: AFSGD/X3/bQEEXoPwDsjfvcrmwzfWeAiiXSAt+uJ6Xd/pKo62qIvh6N302v4vLOAuyvzhyHYWVDh X-Received: by 2002:a63:a84a:: with SMTP id i10mr16702435pgp.263.1544573529937; Tue, 11 Dec 2018 16:12:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544573529; cv=none; d=google.com; s=arc-20160816; b=Uft9MUjdJnkwv4RDb6/C7ca73IbJaLxBMpibw3fX6PyJ3X4zBEsJxpthWydVPlHGnk oY7SpbpYdeOTSu48ghJVI0UyJ4DJT19Bq/I1spg41RsyJRnGL9zF3hKcq2IMyXVXursp F79wzgQx+R5fwbk9rUVD9Me53KtUORmhyr3Cx98DAKxATqeqfHNZK/Dz5/lp6sXO8EUQ dCnRNqwyFy4uYLYJxtdnH7tWZqXuveVy5Onu865V0rV14XDelaVflUVrz0vaAPTrbMxP QknAb1Wbui4Mx+yqosy2EjZ0aeteTHi++bvNJRdj9RLMVsqm2y0ltSszmjn2lGftFJHH 9uwA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=dxV3upF7F0dcmLEbdfWFB9vg7rmE7IE+NndnAUM1Ecc=; b=OVkOnoIRS1xGYC9Wa9Va5HR0y2Nh2GBvAyady0Hb2sOUKVMBGgbswwrqGlXXqe7RnV YNvB69rjROgXedca2h4EdeIhY32ZGqOWgImyTHQuXBfwGVBIP5bID5VFK+YYWNdGrQ3w M3NaZiBTAUuGSBCOj9vGXaoPjd8D95wcsbXpga149qk/S1DfLaGH0Kx4cAJCeEj7Gbd8 onE/mmoGCr1X+jaCGMGl/q3GOeQDOmjGlNUx2XEfbfCP7tnWuA8FHhmnqzb21nTpIeRo 3PwH/5Jmk5bTsJJauCIUJ+DjWXFeH8Ebqhq/bs/eE0CLkbPGxMqS9RPXAif39zbx0bXQ cAKQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id u7si14323549pfu.270.2018.12.11.16.12.09 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Dec 2018 16:12:09 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Dec 2018 16:12:07 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,343,1539673200"; d="scan'208";a="282839400" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.141]) by orsmga005.jf.intel.com with ESMTP; 11 Dec 2018 16:12:07 -0800 From: Rick Edgecombe To: akpm@linux-foundation.org, luto@kernel.org, will.deacon@arm.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-hardening@lists.openwall.com, naveen.n.rao@linux.vnet.ibm.com, anil.s.keshavamurthy@intel.com, davem@davemloft.net, mhiramat@kernel.org, rostedt@goodmis.org, mingo@redhat.com, ast@kernel.org, daniel@iogearbox.net, jeyu@kernel.org, namit@vmware.com, netdev@vger.kernel.org, ard.biesheuvel@linaro.org, jannh@google.com Cc: kristen@linux.intel.com, dave.hansen@intel.com, deneen.t.dock@intel.com, Rick Edgecombe Subject: [PATCH v2 1/4] vmalloc: New flags for safe vfree on special perms Date: Tue, 11 Dec 2018 16:03:51 -0800 Message-Id: <20181212000354.31955-2-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181212000354.31955-1-rick.p.edgecombe@intel.com> References: <20181212000354.31955-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This adds two new flags VM_IMMEDIATE_UNMAP and VM_HAS_SPECIAL_PERMS, for enabling vfree operations to immediately clear executable TLB entries to freed pages, and handle freeing memory with special permissions. In order to support vfree being called on memory that might be RO, the vfree deferred list node is moved to a kmalloc allocated struct, from where it is today, reusing the allocation being freed. arch_vunmap is a new __weak function that implements the actual unmapping and resetting of the direct map permissions. It can be overridden by more efficient architecture specific implementations. For the default implementation, it uses architecture agnostic methods which are equivalent to what most usages do before calling vfree. So now it is just centralized here. This implementation derives from two sketches from Dave Hansen and Andy Lutomirski. Suggested-by: Dave Hansen Suggested-by: Andy Lutomirski Suggested-by: Will Deacon Signed-off-by: Rick Edgecombe --- include/linux/vmalloc.h | 2 ++ mm/vmalloc.c | 73 +++++++++++++++++++++++++++++++++++++---- 2 files changed, 69 insertions(+), 6 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 398e9c95cd61..872bcde17aca 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -21,6 +21,8 @@ struct notifier_block; /* in notifier.h */ #define VM_UNINITIALIZED 0x00000020 /* vm_struct is not fully initialized */ #define VM_NO_GUARD 0x00000040 /* don't add guard page */ #define VM_KASAN 0x00000080 /* has allocated kasan shadow memory */ +#define VM_IMMEDIATE_UNMAP 0x00000200 /* flush before releasing pages */ +#define VM_HAS_SPECIAL_PERMS 0x00000400 /* may be freed with special perms */ /* bits [20..32] reserved for arch specific ioremap internals */ /* diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 97d4b25d0373..02b284d2245a 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -38,6 +39,11 @@ #include "internal.h" +struct vfree_work { + struct llist_node node; + void *addr; +}; + struct vfree_deferred { struct llist_head list; struct work_struct wq; @@ -50,9 +56,13 @@ static void free_work(struct work_struct *w) { struct vfree_deferred *p = container_of(w, struct vfree_deferred, wq); struct llist_node *t, *llnode; + struct vfree_work *cur; - llist_for_each_safe(llnode, t, llist_del_all(&p->list)) - __vunmap((void *)llnode, 1); + llist_for_each_safe(llnode, t, llist_del_all(&p->list)) { + cur = container_of(llnode, struct vfree_work, node); + __vunmap(cur->addr, 1); + kfree(cur); + } } /*** Page table manipulation functions ***/ @@ -1494,6 +1504,48 @@ struct vm_struct *remove_vm_area(const void *addr) return NULL; } +/* + * This function handles unmapping and resetting the direct map as efficiently + * as it can with cross arch functions. The three categories of architectures + * are: + * 1. Architectures with no set_memory implementations and no direct map + * permissions. + * 2. Architectures with set_memory implementations but no direct map + * permissions + * 3. Architectures with set_memory implementations and direct map permissions + */ +void __weak arch_vunmap(struct vm_struct *area, int deallocate_pages) +{ + unsigned long addr = (unsigned long)area->addr; + int immediate = area->flags & VM_IMMEDIATE_UNMAP; + int special = area->flags & VM_HAS_SPECIAL_PERMS; + + /* + * In case of 2 and 3, use this general way of resetting the permissions + * on the directmap. Do NX before RW, in case of X, so there is no W^X + * violation window. + * + * For case 1 these will be noops. + */ + if (immediate) + set_memory_nx(addr, area->nr_pages); + if (deallocate_pages && special) + set_memory_rw(addr, area->nr_pages); + + /* Always actually remove the area */ + remove_vm_area(area->addr); + + /* + * Need to flush the TLB before freeing pages in the case of this flag. + * As long as that's happening, unmap aliases. + * + * For 2 and 3, this will not be needed because of the set_memory_nx + * above, because the stale TLBs will be NX. + */ + if (immediate && !IS_ENABLED(ARCH_HAS_SET_MEMORY)) + vm_unmap_aliases(); +} + static void __vunmap(const void *addr, int deallocate_pages) { struct vm_struct *area; @@ -1515,7 +1567,8 @@ static void __vunmap(const void *addr, int deallocate_pages) debug_check_no_locks_freed(area->addr, get_vm_area_size(area)); debug_check_no_obj_freed(area->addr, get_vm_area_size(area)); - remove_vm_area(addr); + arch_vunmap(area, deallocate_pages); + if (deallocate_pages) { int i; @@ -1542,8 +1595,15 @@ static inline void __vfree_deferred(const void *addr) * nother cpu's list. schedule_work() should be fine with this too. */ struct vfree_deferred *p = raw_cpu_ptr(&vfree_deferred); + struct vfree_work *w = kmalloc(sizeof(struct vfree_work), GFP_ATOMIC); + + /* If no memory for the deferred list node, give up */ + if (!w) + return; - if (llist_add((struct llist_node *)addr, &p->list)) + w->addr = (void *)addr; + + if (llist_add(&w->node, &p->list)) schedule_work(&p->wq); } @@ -1925,8 +1985,9 @@ EXPORT_SYMBOL(vzalloc_node); void *vmalloc_exec(unsigned long size) { - return __vmalloc_node(size, 1, GFP_KERNEL, PAGE_KERNEL_EXEC, - NUMA_NO_NODE, __builtin_return_address(0)); + return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, + GFP_KERNEL, PAGE_KERNEL_EXEC, VM_IMMEDIATE_UNMAP, + NUMA_NO_NODE, __builtin_return_address(0)); } #if defined(CONFIG_64BIT) && defined(CONFIG_ZONE_DMA32) From patchwork Wed Dec 12 00:03:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10725325 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 04FFA159A for ; Wed, 12 Dec 2018 00:12:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E77502B550 for ; Wed, 12 Dec 2018 00:12:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DA9042B583; Wed, 12 Dec 2018 00:12:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 583A52B550 for ; Wed, 12 Dec 2018 00:12:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B61268E00E8; Tue, 11 Dec 2018 19:12:11 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B12268E00E5; Tue, 11 Dec 2018 19:12:11 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 969FF8E00E8; Tue, 11 Dec 2018 19:12:11 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 37A598E00E4 for ; Tue, 11 Dec 2018 19:12:11 -0500 (EST) Received: by mail-pf1-f200.google.com with SMTP id p15so13958736pfk.7 for ; Tue, 11 Dec 2018 16:12:11 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=aFkerf4UfjublRdDyI4ovkPEU9VLW1HJz1GA1vU6UUQ=; b=tEE5klbysEnOR8khuPacnCSZqsvMiZNqk/cHDO5pL5BdLRXm9cZ6JATy58lfBGoJ/v DHqcj2xh+Kj4AujdXChLRqLBlGVkoOnrCGtPHRDQ9vDjAaI7LOgz5uLelyCapTqR4m8X jnDtzz5CWhovXLY52gdAzmDgKa9q3H2uGwwhrSLmMpL8Jn6GueXY+5O7ipX5i3QZXnZ3 VQIindEjShgcQ60S4Sowg5HypOiRfiLczezHt2VfPKXTdD2pfe2kQGUwY+J8KF2Cjny6 pFhbnCIQuYUSuH6u3jkef0Ywz4x0T8WNfLM1WB6J7SN6Z6V2ZHK+WQHKtnyqGv3UlrSb bpKQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AA+aEWYbr8PrXPCXID1ISlJ8DWscDmdjn2etWEZ6qLCelbM+q9S3n6o4 3zwN1D3exH5yxsgWiN3zWC4nLJCAWpf4TWXM4fNvE+Y3CO9lgoY6Nr8TgcgFYsHfYVDPP1LlKfj iomqOlkwc2sscPnXbbgNyxufefCIfDYkxejBrlItmLolc0mx3R/KLSIDOzjDVAM6Ylg== X-Received: by 2002:a65:6392:: with SMTP id h18mr16705334pgv.107.1544573530810; Tue, 11 Dec 2018 16:12:10 -0800 (PST) X-Google-Smtp-Source: AFSGD/WB5PUraIH5RgpTWBo6+9sgi3p2dEThlB3lMDRPYb2Bz7WXBBv0c1uJBDjrjqOuFtR2OV6P X-Received: by 2002:a65:6392:: with SMTP id h18mr16705303pgv.107.1544573529939; Tue, 11 Dec 2018 16:12:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544573529; cv=none; d=google.com; s=arc-20160816; b=PzQ+j2AkS7Y2WlFISsp9/Pz90NGDFaLZEb+M0TF/EK2gEtRWegJyCnwCHem8uHMYt3 gnWEhgzabh6+Kexz2ixv17M9kAzKnTl7sqFgfQWnaZ5okgOEcm4gR8D4ZW/VzCwynqHr WdBf5JZPfM6IpdcpKqsqUHCdxXf24CzmDcO/SMBoxFYqlJPzAlgors0hFLghdQOVOaq6 kPNTJiGUyivqt3NK1FbjNDEEpCD2PUC3rZU7j4+nniF0Skpx/zd8JRSb8Wlr7uBl1W5c 9VHp56XtxaIhSvd1g3sqhNvMLCRkVzdzPopsHk4tM9dSRYHyqR3wPRSBSBVvixaV6GGn HH3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=aFkerf4UfjublRdDyI4ovkPEU9VLW1HJz1GA1vU6UUQ=; b=KHeJch/2XQFmRy4h5D6cHIvkbjqvZFo1Vwj88NbRg90K31z8LUikTDrV8VX5SwpYat TUtG7ZktYixzmm3E/iU1ZJseYxtVV8vwhGSu/TUC0E55OyRiWD0HK23F99nuoxrKeYrx 8KZnICP6XKy7ljmZdzgu/xv979ReWtV4XvRjD1xfxOqaxh4RR811P41TBrWpPNbDZJ0s DyZTxhDmswK0AKxwIisCb0I6gkXhYgH4EOj3Y6lAhvLxy1mJPqbBsv2PP9z0Y8PUa0wh cP/gVhzW0+7AO3D+5RC2G7bp8aBDpkJVW8EpATTmVJ5HyGs5pufv+Gk7n3D3wFkix9cK u9wQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id f18si13139318pgl.457.2018.12.11.16.12.09 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Dec 2018 16:12:09 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Dec 2018 16:12:07 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,343,1539673200"; d="scan'208";a="282839403" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.141]) by orsmga005.jf.intel.com with ESMTP; 11 Dec 2018 16:12:07 -0800 From: Rick Edgecombe To: akpm@linux-foundation.org, luto@kernel.org, will.deacon@arm.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-hardening@lists.openwall.com, naveen.n.rao@linux.vnet.ibm.com, anil.s.keshavamurthy@intel.com, davem@davemloft.net, mhiramat@kernel.org, rostedt@goodmis.org, mingo@redhat.com, ast@kernel.org, daniel@iogearbox.net, jeyu@kernel.org, namit@vmware.com, netdev@vger.kernel.org, ard.biesheuvel@linaro.org, jannh@google.com Cc: kristen@linux.intel.com, dave.hansen@intel.com, deneen.t.dock@intel.com, Rick Edgecombe Subject: [PATCH v2 2/4] modules: Add new special vfree flags Date: Tue, 11 Dec 2018 16:03:52 -0800 Message-Id: <20181212000354.31955-3-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181212000354.31955-1-rick.p.edgecombe@intel.com> References: <20181212000354.31955-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add new flags for handling freeing of special permissioned memory in vmalloc, and remove places where the handling was done in module.c. This will enable this flag for all architectures. Signed-off-by: Rick Edgecombe --- kernel/module.c | 43 ++++++++++++------------------------------- 1 file changed, 12 insertions(+), 31 deletions(-) diff --git a/kernel/module.c b/kernel/module.c index 49a405891587..910f92b402f8 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -1941,11 +1941,23 @@ void module_disable_ro(const struct module *mod) frob_rodata(&mod->init_layout, set_memory_rw); } +static void module_set_vm_flags(const struct module_layout *layout) +{ + struct vm_struct *vm = find_vm_area(layout->base); + + if (vm) { + vm->flags |= VM_HAS_SPECIAL_PERMS; + vm->flags |= VM_IMMEDIATE_UNMAP; + } +} + void module_enable_ro(const struct module *mod, bool after_init) { if (!rodata_enabled) return; + module_set_vm_flags(&mod->core_layout); + module_set_vm_flags(&mod->init_layout); frob_text(&mod->core_layout, set_memory_ro); frob_rodata(&mod->core_layout, set_memory_ro); frob_text(&mod->init_layout, set_memory_ro); @@ -1964,15 +1976,6 @@ static void module_enable_nx(const struct module *mod) frob_writable_data(&mod->init_layout, set_memory_nx); } -static void module_disable_nx(const struct module *mod) -{ - frob_rodata(&mod->core_layout, set_memory_x); - frob_ro_after_init(&mod->core_layout, set_memory_x); - frob_writable_data(&mod->core_layout, set_memory_x); - frob_rodata(&mod->init_layout, set_memory_x); - frob_writable_data(&mod->init_layout, set_memory_x); -} - /* Iterate through all modules and set each module's text as RW */ void set_all_modules_text_rw(void) { @@ -2016,23 +2019,8 @@ void set_all_modules_text_ro(void) } mutex_unlock(&module_mutex); } - -static void disable_ro_nx(const struct module_layout *layout) -{ - if (rodata_enabled) { - frob_text(layout, set_memory_rw); - frob_rodata(layout, set_memory_rw); - frob_ro_after_init(layout, set_memory_rw); - } - frob_rodata(layout, set_memory_x); - frob_ro_after_init(layout, set_memory_x); - frob_writable_data(layout, set_memory_x); -} - #else -static void disable_ro_nx(const struct module_layout *layout) { } static void module_enable_nx(const struct module *mod) { } -static void module_disable_nx(const struct module *mod) { } #endif #ifdef CONFIG_LIVEPATCH @@ -2163,7 +2151,6 @@ static void free_module(struct module *mod) mutex_unlock(&module_mutex); /* This may be empty, but that's OK */ - disable_ro_nx(&mod->init_layout); module_arch_freeing_init(mod); module_memfree(mod->init_layout.base); kfree(mod->args); @@ -2173,7 +2160,6 @@ static void free_module(struct module *mod) lockdep_free_key_range(mod->core_layout.base, mod->core_layout.size); /* Finally, free the core (containing the module structure) */ - disable_ro_nx(&mod->core_layout); module_memfree(mod->core_layout.base); } @@ -3497,7 +3483,6 @@ static noinline int do_init_module(struct module *mod) #endif module_enable_ro(mod, true); mod_tree_remove_init(mod); - disable_ro_nx(&mod->init_layout); module_arch_freeing_init(mod); mod->init_layout.base = NULL; mod->init_layout.size = 0; @@ -3812,10 +3797,6 @@ static int load_module(struct load_info *info, const char __user *uargs, module_bug_cleanup(mod); mutex_unlock(&module_mutex); - /* we can't deallocate the module until we clear memory protection */ - module_disable_ro(mod); - module_disable_nx(mod); - ddebug_cleanup: ftrace_release_mod(mod); dynamic_debug_remove(mod, info->debug); From patchwork Wed Dec 12 00:03:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10725329 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B4F266C5 for ; Wed, 12 Dec 2018 00:12:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A452D2B550 for ; Wed, 12 Dec 2018 00:12:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 98A382B583; Wed, 12 Dec 2018 00:12:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C83162B550 for ; Wed, 12 Dec 2018 00:12:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1924C8E00E7; Tue, 11 Dec 2018 19:12:12 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 08A438E00E4; Tue, 11 Dec 2018 19:12:11 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC0CC8E00E7; Tue, 11 Dec 2018 19:12:11 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 6D1B38E00C9 for ; Tue, 11 Dec 2018 19:12:11 -0500 (EST) Received: by mail-pg1-f200.google.com with SMTP id v72so10975839pgb.10 for ; Tue, 11 Dec 2018 16:12:11 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Hp1GEthNazPZ/TRqQgCHn0d9BdgUz5hb+WmE6z4c0IY=; b=g1VhVKp7HVQR+HN8+7/XlS1cgqAu2DmkPgwqY/HGX018SSr2GyDg/FDlu6vsFm1C1i D256nfXsbKN9NO3/km1Ttwegc3hxUO78VslmHHiWKwiMA6gW9EhOkYaQAIhltrzTjpJn YQvIpqGCh0plBgohA9jjXiDgolmNZsIU4pftvlyF3vjOLgwj0jx7ghI31OlQgSro/zdR Qc9nTUROhR8wJ6MQdu0WeMp8RdFQc7b6SJ4ladYaatSCsZu3K+mUcslts557fBR2q3Ou KR/aer8/ENehOs4OueLHrLbruOLqvn4RPshxbqUnlv4gleQGjXkGKFCasGTR7SvnmKOJ P62g== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AA+aEWbzDcYBg68Qjd7oDjRJZY3/ftUQZv817DLrH6fMpLSx9kVND/+X kUXgHbD9umVYD0XNTPTyBfPX9Ai5hPdgFE+pOK95KZLwj46BQYweobwnYqLv8IcTFj0a82R23qQ 5yRYzY40uwImTigXHX1FvzYmxbZswsvO/2jsiN+z4ZEjlSAx91anJpqOhXSZmzvb04A== X-Received: by 2002:a63:5a57:: with SMTP id k23mr16361205pgm.5.1544573530946; Tue, 11 Dec 2018 16:12:10 -0800 (PST) X-Google-Smtp-Source: AFSGD/VqHCmy7SlZGZvQ3Q0tAV3t5rzst3t+fD9bxSglCwlqfjHhXtPiEsC7xV4XxoepUmsq7uoo X-Received: by 2002:a63:5a57:: with SMTP id k23mr16361175pgm.5.1544573530123; Tue, 11 Dec 2018 16:12:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544573530; cv=none; d=google.com; s=arc-20160816; b=Ot9X5Hir9W6dZuGqMNuxF/KLqNWF/okCj1jv0j8zf4ebsPRdLC4Wn/2Huf7le/MNuI 0v0yMPJOHGGWATOBbgQt1YePAEG4OAxpk3qWka07Bc7gXWYAbE6ojktWmC6/imUcVhAc L0WM+7Js9In+Cso7tzhNTc651pIV/Tm6vhKJwH07FFsVeA45Dy/cvxX/niNaWQoGakgp DKIWFzYjwy9xV3wWGDeuWVkiJ6Eu+R0qfCg7opgSKGXoNwE9xnVJ4ktxJ7YVSR3p5y0d Gpgw/8T7YbAjKbW7tVcpZFDRIITvgNonaNVDKyymRVDi84EvXmaYDBqtpGBK9O5KuSON Lv7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=Hp1GEthNazPZ/TRqQgCHn0d9BdgUz5hb+WmE6z4c0IY=; b=lN6J9+fGTqLOE3sSYyt0qPxJL5OBu+oyiH7lh40rDTSi5tWA3TwbFFp+zSPre4ZVcA TjRaTnQOzGdZMhwgE/N3J7qHDZnIK5WxWSMrqSKM8ybTnaCaxg2OR5o+6u/ApnwwAbaN d7yd5xyC0g0GkMIzJI9wFEMHJro+vXeqJXsren4ZvjdR43Qwp4B7/4BrBKj/NFYryIdc p9C0zPWccZwLTtvE3k0wWqbbdh6SE7HBu8D5qGYDHaBbSKcZj5UeXIVglte58Z/wFZju 5fR+XKXUPU5ZC7gIQ7JLngyqfSB5R7251pgrEeyc02W/HjNkJsisWgr4PPFbYiPemM/w SZBA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id f18si13139318pgl.457.2018.12.11.16.12.09 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Dec 2018 16:12:10 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Dec 2018 16:12:07 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,343,1539673200"; d="scan'208";a="282839406" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.141]) by orsmga005.jf.intel.com with ESMTP; 11 Dec 2018 16:12:07 -0800 From: Rick Edgecombe To: akpm@linux-foundation.org, luto@kernel.org, will.deacon@arm.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-hardening@lists.openwall.com, naveen.n.rao@linux.vnet.ibm.com, anil.s.keshavamurthy@intel.com, davem@davemloft.net, mhiramat@kernel.org, rostedt@goodmis.org, mingo@redhat.com, ast@kernel.org, daniel@iogearbox.net, jeyu@kernel.org, namit@vmware.com, netdev@vger.kernel.org, ard.biesheuvel@linaro.org, jannh@google.com Cc: kristen@linux.intel.com, dave.hansen@intel.com, deneen.t.dock@intel.com, Rick Edgecombe Subject: [PATCH v2 3/4] bpf: switch to new vmalloc vfree flags Date: Tue, 11 Dec 2018 16:03:53 -0800 Message-Id: <20181212000354.31955-4-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181212000354.31955-1-rick.p.edgecombe@intel.com> References: <20181212000354.31955-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This switches to use the new vmalloc flags to control freeing memory with special permissions. Signed-off-by: Rick Edgecombe --- include/linux/filter.h | 26 ++++++++++++-------------- kernel/bpf/core.c | 1 - 2 files changed, 12 insertions(+), 15 deletions(-) diff --git a/include/linux/filter.h b/include/linux/filter.h index 795ff0b869bb..2aeb93d3337d 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -20,6 +20,7 @@ #include #include #include +#include #include @@ -487,7 +488,6 @@ struct bpf_prog { u16 pages; /* Number of allocated pages */ u16 jited:1, /* Is our filter JIT'ed? */ jit_requested:1,/* archs need to JIT the prog */ - undo_set_mem:1, /* Passed set_memory_ro() checkpoint */ gpl_compatible:1, /* Is filter GPL compatible? */ cb_access:1, /* Is control block accessed? */ dst_needed:1, /* Do we need dst entry? */ @@ -699,24 +699,23 @@ bpf_ctx_narrow_access_ok(u32 off, u32 size, u32 size_default) static inline void bpf_prog_lock_ro(struct bpf_prog *fp) { - fp->undo_set_mem = 1; - set_memory_ro((unsigned long)fp, fp->pages); -} + struct vm_struct *vm = find_vm_area(fp); -static inline void bpf_prog_unlock_ro(struct bpf_prog *fp) -{ - if (fp->undo_set_mem) - set_memory_rw((unsigned long)fp, fp->pages); + if (vm) + vm->flags |= VM_HAS_SPECIAL_PERMS; + set_memory_ro((unsigned long)fp, fp->pages); } static inline void bpf_jit_binary_lock_ro(struct bpf_binary_header *hdr) { - set_memory_ro((unsigned long)hdr, hdr->pages); -} + struct vm_struct *vm = find_vm_area(hdr); -static inline void bpf_jit_binary_unlock_ro(struct bpf_binary_header *hdr) -{ - set_memory_rw((unsigned long)hdr, hdr->pages); + if (vm) { + vm->flags |= VM_HAS_SPECIAL_PERMS; + vm->flags |= VM_IMMEDIATE_UNMAP; + } + + set_memory_ro((unsigned long)hdr, hdr->pages); } static inline struct bpf_binary_header * @@ -746,7 +745,6 @@ void __bpf_prog_free(struct bpf_prog *fp); static inline void bpf_prog_unlock_free(struct bpf_prog *fp) { - bpf_prog_unlock_ro(fp); __bpf_prog_free(fp); } diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index b1a3545d0ec8..bd3efd7ce526 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -663,7 +663,6 @@ void __weak bpf_jit_free(struct bpf_prog *fp) if (fp->jited) { struct bpf_binary_header *hdr = bpf_jit_binary_hdr(fp); - bpf_jit_binary_unlock_ro(hdr); bpf_jit_binary_free(hdr); WARN_ON_ONCE(!bpf_prog_kallsyms_verify_off(fp)); From patchwork Wed Dec 12 00:03:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10725331 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9AB50159A for ; Wed, 12 Dec 2018 00:12:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 89B282B550 for ; Wed, 12 Dec 2018 00:12:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7D0FC2B583; Wed, 12 Dec 2018 00:12:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D39D22B550 for ; Wed, 12 Dec 2018 00:12:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4AC728E00E4; Tue, 11 Dec 2018 19:12:12 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2D8098E00E9; Tue, 11 Dec 2018 19:12:12 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 015A58E00E5; Tue, 11 Dec 2018 19:12:11 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by kanga.kvack.org (Postfix) with ESMTP id 9D4148E00E4 for ; Tue, 11 Dec 2018 19:12:11 -0500 (EST) Received: by mail-pl1-f197.google.com with SMTP id y2so11711993plr.8 for ; Tue, 11 Dec 2018 16:12:11 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=XUKGacGC7ZO0g/7SiSvDvW+P41nI5jnYqNQ7GSYGm1E=; b=NzovkFBWECAMRHPHMXbjZDL6uJIWmmHNc4UsERSRMMwtxpOokvncJML9PXl3R7wRaw ABjahJUyWS2XVn9sWtNJcDn/rcFGRa44jwyhNq3N2YdLp3279Rex94oZMmzkNAiwmqXp 4KnrAsaQ3chMPyY3T3m3nSKPU344s3yQ63AjqOR4QJFycxFPYu0+T7g7zscBN9LZj7gX +j4JcRc008gqooOfZeQI58ZOL1/LTm9oQDMHfAPGxd5FW73tx8ow6mHhjayObIUASwPr 5VexWk5aM4rtTed5gF5/ffxuGDCPelE80i/Lc2MxmGSQ8nIZRwOVQOvElPoTiQYVEjgk 2dig== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AA+aEWZ2kWGrVJu77A7O2GRqu4OH660+MVNLh0z9ezkBIu+AodT89QWw KYCWyz164w1mfCwUzOH0/kY4vF14Qap6QS3X7QF+x8/Xx+UOwOy0ZF/9Avrku10bmaVlEp9DxNM MQrpTSmeqORm1FdBZ8RmDHcGA8PU5PEusYU4OBq6ujXnNB0tif5qMoNypqf6/NCmO/A== X-Received: by 2002:a62:1447:: with SMTP id 68mr17977336pfu.257.1544573531147; Tue, 11 Dec 2018 16:12:11 -0800 (PST) X-Google-Smtp-Source: AFSGD/XdpJyJbYoBVWRY+x/kwthYuRKMm1KTQp7h9VzDEOEU6fBCDgLPr1leH+v4KMoHuUtXjTKQ X-Received: by 2002:a62:1447:: with SMTP id 68mr17977301pfu.257.1544573530181; Tue, 11 Dec 2018 16:12:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544573530; cv=none; d=google.com; s=arc-20160816; b=fvlWp3qkLkHN47Lx09Dqq1ajQ+UjK9H8+9/E8ypLRlO53q9iflZ6pdaIGvX7kmRFdR vq1TA+tsQivKk77zrwCXCjl6UeaZCFgD+czOA+4xsSetg75ExN2qcYXv4b0/Ei3wkxdO Z5KHnWBKmMZRGyF5WB+51AHiLIL/gIH9y9Rtktbq/FhNB2yGtpO+aU5liGu5utD6v/45 2NbfW5kRISX9LMt14Rx98WHwtit7y3gBy+9fQjCfiTEpajGWrwY9bvgiTNKWC2KV/6YT j6hoStmud8X7MbgTNs5uBnEwytdr2hB4vNG/IAjL7hcseSIHenZ1MFb3QebfZHY1OKaM Hdug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=XUKGacGC7ZO0g/7SiSvDvW+P41nI5jnYqNQ7GSYGm1E=; b=RmVqunfLB9eyNHZzSJmHr1VWWd3e28ojU/HknqYhbP8nk+PCvnpysQGWTnG6dlx5bf x7SMluKrgYGdLmNWvSYOKtA71x1zKL5RN2lkrnMWP0fSfp9eBbs0vNdmF6GiDOYYc7qY fBj3xj9zvYMqcFuSJbdhX9KCzI6Rt68toGriLexwUEw/jpvwDjVjkEgQz5HKc8dLFFCV fsuH0N/2Xv4AR9uCqd434pFvauwtyaDPtp+JuyVzQClOZLJ1ac9MnuoCVUwqalS8j+G4 iSX4s5RyceI+YezqnoQ3us19EQqIzroGbfAAe1V+Ot/gvHjCPfRfSJsyz0WYQySWuiFE u8Yw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id u7si14323549pfu.270.2018.12.11.16.12.10 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 11 Dec 2018 16:12:10 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Dec 2018 16:12:09 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,343,1539673200"; d="scan'208";a="282839408" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.141]) by orsmga005.jf.intel.com with ESMTP; 11 Dec 2018 16:12:07 -0800 From: Rick Edgecombe To: akpm@linux-foundation.org, luto@kernel.org, will.deacon@arm.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-hardening@lists.openwall.com, naveen.n.rao@linux.vnet.ibm.com, anil.s.keshavamurthy@intel.com, davem@davemloft.net, mhiramat@kernel.org, rostedt@goodmis.org, mingo@redhat.com, ast@kernel.org, daniel@iogearbox.net, jeyu@kernel.org, namit@vmware.com, netdev@vger.kernel.org, ard.biesheuvel@linaro.org, jannh@google.com Cc: kristen@linux.intel.com, dave.hansen@intel.com, deneen.t.dock@intel.com, Rick Edgecombe Subject: [PATCH v2 4/4] x86/vmalloc: Add TLB efficient x86 arch_vunmap Date: Tue, 11 Dec 2018 16:03:54 -0800 Message-Id: <20181212000354.31955-5-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181212000354.31955-1-rick.p.edgecombe@intel.com> References: <20181212000354.31955-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This adds a more efficient x86 architecture specific implementation of arch_vunmap, that can free any type of special permission memory with only 1 TLB flush. In order to enable this, _set_pages_p and _set_pages_np are made non-static and renamed set_pages_p_noflush and set_pages_np_noflush to better communicate their different (non-flushing) behavior from the rest of the set_pages_* functions. The method for doing this with only 1 TLB flush was suggested by Andy Lutomirski. Suggested-by: Andy Lutomirski Signed-off-by: Rick Edgecombe --- arch/x86/include/asm/set_memory.h | 2 + arch/x86/mm/Makefile | 3 +- arch/x86/mm/pageattr.c | 11 +++-- arch/x86/mm/vmalloc.c | 71 +++++++++++++++++++++++++++++++ 4 files changed, 80 insertions(+), 7 deletions(-) create mode 100644 arch/x86/mm/vmalloc.c diff --git a/arch/x86/include/asm/set_memory.h b/arch/x86/include/asm/set_memory.h index 07a25753e85c..70ee81e8914b 100644 --- a/arch/x86/include/asm/set_memory.h +++ b/arch/x86/include/asm/set_memory.h @@ -84,6 +84,8 @@ int set_pages_x(struct page *page, int numpages); int set_pages_nx(struct page *page, int numpages); int set_pages_ro(struct page *page, int numpages); int set_pages_rw(struct page *page, int numpages); +int set_pages_np_noflush(struct page *page, int numpages); +int set_pages_p_noflush(struct page *page, int numpages); extern int kernel_set_to_readonly; void set_kernel_text_rw(void); diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index 4b101dd6e52f..189681f863a6 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -13,7 +13,8 @@ CFLAGS_REMOVE_mem_encrypt_identity.o = -pg endif obj-y := init.o init_$(BITS).o fault.o ioremap.o extable.o pageattr.o mmap.o \ - pat.o pgtable.o physaddr.o setup_nx.o tlb.o cpu_entry_area.o + pat.o pgtable.o physaddr.o setup_nx.o tlb.o cpu_entry_area.o \ + vmalloc.o # Make sure __phys_addr has no stackprotector nostackp := $(call cc-option, -fno-stack-protector) diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c index db7a10082238..db0a4dfb5a7f 100644 --- a/arch/x86/mm/pageattr.c +++ b/arch/x86/mm/pageattr.c @@ -2248,9 +2248,7 @@ int set_pages_rw(struct page *page, int numpages) return set_memory_rw(addr, numpages); } -#ifdef CONFIG_DEBUG_PAGEALLOC - -static int __set_pages_p(struct page *page, int numpages) +int set_pages_p_noflush(struct page *page, int numpages) { unsigned long tempaddr = (unsigned long) page_address(page); struct cpa_data cpa = { .vaddr = &tempaddr, @@ -2269,7 +2267,7 @@ static int __set_pages_p(struct page *page, int numpages) return __change_page_attr_set_clr(&cpa, 0); } -static int __set_pages_np(struct page *page, int numpages) +int set_pages_np_noflush(struct page *page, int numpages) { unsigned long tempaddr = (unsigned long) page_address(page); struct cpa_data cpa = { .vaddr = &tempaddr, @@ -2288,6 +2286,7 @@ static int __set_pages_np(struct page *page, int numpages) return __change_page_attr_set_clr(&cpa, 0); } +#ifdef CONFIG_DEBUG_PAGEALLOC void __kernel_map_pages(struct page *page, int numpages, int enable) { if (PageHighMem(page)) @@ -2303,9 +2302,9 @@ void __kernel_map_pages(struct page *page, int numpages, int enable) * and hence no memory allocations during large page split. */ if (enable) - __set_pages_p(page, numpages); + set_pages_p_noflush(page, numpages); else - __set_pages_np(page, numpages); + set_pages_np_noflush(page, numpages); /* * We should perform an IPI and flush all tlbs, diff --git a/arch/x86/mm/vmalloc.c b/arch/x86/mm/vmalloc.c new file mode 100644 index 000000000000..be9ea42c3dfe --- /dev/null +++ b/arch/x86/mm/vmalloc.c @@ -0,0 +1,71 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * vmalloc.c: x86 arch version of vmalloc.c + * + * (C) Copyright 2018 Intel Corporation + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; version 2 + * of the License. + */ + +#include +#include +#include + +static void set_area_direct_np(struct vm_struct *area) +{ + int i; + + for (i = 0; i < area->nr_pages; i++) + set_pages_np_noflush(area->pages[i], 1); +} + +static void set_area_direct_prw(struct vm_struct *area) +{ + int i; + + for (i = 0; i < area->nr_pages; i++) + set_pages_p_noflush(area->pages[i], 1); +} + +void arch_vunmap(struct vm_struct *area, int deallocate_pages) +{ + int immediate = area->flags & VM_IMMEDIATE_UNMAP; + int special = area->flags & VM_HAS_SPECIAL_PERMS; + + /* Unmap from vmalloc area */ + remove_vm_area(area->addr); + + /* If no need to reset directmap perms, just check if need to flush */ + if (!(deallocate_pages || special)) { + if (immediate) + vm_unmap_aliases(); + return; + } + + /* From here we need to make sure to reset the direct map perms */ + + /* + * If the area being freed does not have any extra capabilities, we can + * just reset the directmap to RW before freeing. + */ + if (!immediate) { + set_area_direct_prw(area); + vm_unmap_aliases(); + return; + } + + /* + * If the vm being freed has security sensitive capabilities such as + * executable we need to make sure there is no W window on the directmap + * before removing the X in the TLB. So we set not present first so we + * can flush without any other CPU picking up the mapping. Then we reset + * RW+P without a flush, since NP prevented it from being cached by + * other cpus. + */ + set_area_direct_np(area); + vm_unmap_aliases(); + set_area_direct_prw(area); +}