From patchwork Mon Apr 22 18:57:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10911475 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8D2DB1575 for ; Mon, 22 Apr 2019 19:00:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 80EA1286B0 for ; Mon, 22 Apr 2019 19:00:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 74EA3287A6; Mon, 22 Apr 2019 19:00:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4780F286B0 for ; Mon, 22 Apr 2019 19:00:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 879136B0280; Mon, 22 Apr 2019 15:00:26 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7C5376B0282; Mon, 22 Apr 2019 15:00:26 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6662E6B0284; Mon, 22 Apr 2019 15:00:26 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 28F786B0280 for ; Mon, 22 Apr 2019 15:00:26 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id x9so8851688pln.0 for ; Mon, 22 Apr 2019 12:00:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=Ad8lsaTX3gMSj0SQ10uKFMNKwQGvntfT8IOy2gn/yIE=; b=rSk0W5dnrPmcAUEVnMIxIR2+FJIKgVKQr0UsXCuZT/0/qk9sO+Xpzx9hElAqOCXPkT EC+ez161yRBMjM0NM20eDorzrtT///KvSYraZnJ7apH4kzvvpp+3x2nm81GClM0SyemJ Hb2wqzs57GBja4DFAJ3PLT9T894PKN+FT+0zr57kqxoGrwqy60gEhfMV1+Gi3ipqh4d4 lB1KWkm8RXLvJ/x2TYasUWO4Oe5DU7LEleTJ5CFQELCLkU34TqLHNRA+tKQ+LeO8TDVp q+RDGOobl45rUpTcDwE6bSEhwxbJ8iNRm4QWz78SCF3BWGlRqWIdkr6T63qrnhN7zZP8 zGdQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APjAAAXG3Q8+Svz7r1NHIOc5P0ZxiCr8VZZ46OUBbqWvDaWNGaXhIp11 OKjtZRwtbvQvdSUQKYLqKz8prB7rLFEu2gQ3AucMPtgxiQqV1Ks+X3IwqvMgrnCiz2N4xB6d+oa RQsDgKaPUaig5rdotfWliI0f96u5ryfliuGXqMORk9WFDXsKucxpCiIDO0Emc2zyzGw== X-Received: by 2002:aa7:91d5:: with SMTP id z21mr22217009pfa.222.1555959625811; Mon, 22 Apr 2019 12:00:25 -0700 (PDT) X-Google-Smtp-Source: APXvYqyC5fdFY/sB3ZFIIWPVPAkiL4KYThr7otguJ1leQO7QKLMOjdnQG7JKlBO9uk1TtXUj0DW7 X-Received: by 2002:aa7:91d5:: with SMTP id z21mr22209255pfa.222.1555959524150; Mon, 22 Apr 2019 11:58:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555959524; cv=none; d=google.com; s=arc-20160816; b=aOdC1Wz2XXaMD8wXbu/w5rhcipEshCjk/ILUHYSBAx80Lgq5fNmo2sU2P+3D7cyF2k l3lqlyUsQUxyy5ZeyhZuSAD7a7ye0wv2LaFWhFHGC7ON90ygHSUByTOcyazuU2kYK6xE yY0bId+e5F/2TqnI+pkb3wTD19j+0Yy4/yxwuZM9eCJocdK+9U79X0OhzWDSE+3B/knY hBMujZQeLLOSRlk6luqHhOUu8U4SG1A5UqH3LxiQgLb8K9NCYS2/AKzTVWQAxzWZymby IOa4r1GkthbNgvxY/EgfjKMS5iBCGHx3fNwJTBvyI+HQAafe+A1utqruq4uxt4YGnLp7 k5sg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=Ad8lsaTX3gMSj0SQ10uKFMNKwQGvntfT8IOy2gn/yIE=; b=Te++ohD+Z7JdNi2ftZkOVL1aHio6hDvI3vnYigrpusPII2bXgkvdAzSAzP/iWkXQ7N sx5BrfqbbiGstjDP4AKZBQY6wfg5WZHwmhBF15x4q0LzeBnXFKdtA4nLXT+75a46ZSXZ 34f0+E8mtNTdGgxsj9ejKPOAU2LUeKoQcU0on1aPikD0PX90to9MijZI60f8UA9wFNDu rtpdddIdQFW/euooBeSP2lrtpBmPacgb7FgYIex0fN6l+kZYw/dtOMIKHBjZNaAGPyzP BY3mP0qx6brqOu8tCMhr7syumacL2jyIaHZxl1PCbGlWFW3fgOULZFdqSh2mnaaEsuGz xe2g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga02.intel.com (mga02.intel.com. [134.134.136.20]) by mx.google.com with ESMTPS id w15si615875pga.591.2019.04.22.11.58.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Apr 2019 11:58:44 -0700 (PDT) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) client-ip=134.134.136.20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Apr 2019 11:58:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,382,1549958400"; d="scan'208";a="136417168" Received: from linksys13920.jf.intel.com (HELO rpedgeco-DESK5.jf.intel.com) ([10.54.75.11]) by orsmga008.jf.intel.com with ESMTP; 22 Apr 2019 11:58:42 -0700 From: Rick Edgecombe To: Borislav Petkov , Andy Lutomirski , Ingo Molnar Cc: linux-kernel@vger.kernel.org, x86@kernel.org, hpa@zytor.com, Thomas Gleixner , Nadav Amit , Dave Hansen , Peter Zijlstra , linux_dti@icloud.com, linux-integrity@vger.kernel.org, linux-security-module@vger.kernel.org, akpm@linux-foundation.org, kernel-hardening@lists.openwall.com, linux-mm@kvack.org, will.deacon@arm.com, ard.biesheuvel@linaro.org, kristen@linux.intel.com, deneen.t.dock@intel.com, Rick Edgecombe , Jessica Yu , Steven Rostedt Subject: [PATCH v4 17/23] modules: Use vmalloc special flag Date: Mon, 22 Apr 2019 11:57:59 -0700 Message-Id: <20190422185805.1169-18-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190422185805.1169-1-rick.p.edgecombe@intel.com> References: <20190422185805.1169-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Use new flag for handling freeing of special permissioned memory in vmalloc and remove places where memory was set RW before freeing which is no longer needed. Since freeing of VM_FLUSH_RESET_PERMS memory is not supported in an interrupt by vmalloc, the freeing of init sections is moved to a work queue. Instead of call_rcu it now uses synchronize_rcu() in the work queue. Lastly, there is now a WARN_ON in module_memfree since it should not be called in an interrupt with special memory as is required for VM_FLUSH_RESET_PERMS. Cc: Jessica Yu Cc: Steven Rostedt Signed-off-by: Rick Edgecombe --- kernel/module.c | 77 +++++++++++++++++++++++++------------------------ 1 file changed, 39 insertions(+), 38 deletions(-) diff --git a/kernel/module.c b/kernel/module.c index 2b2845ae983e..a9020bdd4cf6 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -98,6 +98,10 @@ DEFINE_MUTEX(module_mutex); EXPORT_SYMBOL_GPL(module_mutex); static LIST_HEAD(modules); +/* Work queue for freeing init sections in success case */ +static struct work_struct init_free_wq; +static struct llist_head init_free_list; + #ifdef CONFIG_MODULES_TREE_LOOKUP /* @@ -1949,6 +1953,8 @@ void module_enable_ro(const struct module *mod, bool after_init) if (!rodata_enabled) return; + set_vm_flush_reset_perms(mod->core_layout.base); + set_vm_flush_reset_perms(mod->init_layout.base); frob_text(&mod->core_layout, set_memory_ro); frob_text(&mod->core_layout, set_memory_x); @@ -1972,15 +1978,6 @@ static void module_enable_nx(const struct module *mod) frob_writable_data(&mod->init_layout, set_memory_nx); } -static void module_disable_nx(const struct module *mod) -{ - frob_rodata(&mod->core_layout, set_memory_x); - frob_ro_after_init(&mod->core_layout, set_memory_x); - frob_writable_data(&mod->core_layout, set_memory_x); - frob_rodata(&mod->init_layout, set_memory_x); - frob_writable_data(&mod->init_layout, set_memory_x); -} - /* Iterate through all modules and set each module's text as RW */ void set_all_modules_text_rw(void) { @@ -2024,23 +2021,8 @@ void set_all_modules_text_ro(void) } mutex_unlock(&module_mutex); } - -static void disable_ro_nx(const struct module_layout *layout) -{ - if (rodata_enabled) { - frob_text(layout, set_memory_rw); - frob_rodata(layout, set_memory_rw); - frob_ro_after_init(layout, set_memory_rw); - } - frob_rodata(layout, set_memory_x); - frob_ro_after_init(layout, set_memory_x); - frob_writable_data(layout, set_memory_x); -} - #else -static void disable_ro_nx(const struct module_layout *layout) { } static void module_enable_nx(const struct module *mod) { } -static void module_disable_nx(const struct module *mod) { } #endif #ifdef CONFIG_LIVEPATCH @@ -2120,6 +2102,11 @@ static void free_module_elf(struct module *mod) void __weak module_memfree(void *module_region) { + /* + * This memory may be RO, and freeing RO memory in an interrupt is not + * supported by vmalloc. + */ + WARN_ON(in_interrupt()); vfree(module_region); } @@ -2171,7 +2158,6 @@ static void free_module(struct module *mod) mutex_unlock(&module_mutex); /* This may be empty, but that's OK */ - disable_ro_nx(&mod->init_layout); module_arch_freeing_init(mod); module_memfree(mod->init_layout.base); kfree(mod->args); @@ -2181,7 +2167,6 @@ static void free_module(struct module *mod) lockdep_free_key_range(mod->core_layout.base, mod->core_layout.size); /* Finally, free the core (containing the module structure) */ - disable_ro_nx(&mod->core_layout); module_memfree(mod->core_layout.base); } @@ -3420,17 +3405,34 @@ static void do_mod_ctors(struct module *mod) /* For freeing module_init on success, in case kallsyms traversing */ struct mod_initfree { - struct rcu_head rcu; + struct llist_node node; void *module_init; }; -static void do_free_init(struct rcu_head *head) +static void do_free_init(struct work_struct *w) { - struct mod_initfree *m = container_of(head, struct mod_initfree, rcu); - module_memfree(m->module_init); - kfree(m); + struct llist_node *pos, *n, *list; + struct mod_initfree *initfree; + + list = llist_del_all(&init_free_list); + + synchronize_rcu(); + + llist_for_each_safe(pos, n, list) { + initfree = container_of(pos, struct mod_initfree, node); + module_memfree(initfree->module_init); + kfree(initfree); + } } +static int __init modules_wq_init(void) +{ + INIT_WORK(&init_free_wq, do_free_init); + init_llist_head(&init_free_list); + return 0; +} +module_init(modules_wq_init); + /* * This is where the real work happens. * @@ -3507,7 +3509,6 @@ static noinline int do_init_module(struct module *mod) #endif module_enable_ro(mod, true); mod_tree_remove_init(mod); - disable_ro_nx(&mod->init_layout); module_arch_freeing_init(mod); mod->init_layout.base = NULL; mod->init_layout.size = 0; @@ -3518,14 +3519,18 @@ static noinline int do_init_module(struct module *mod) * We want to free module_init, but be aware that kallsyms may be * walking this with preempt disabled. In all the failure paths, we * call synchronize_rcu(), but we don't want to slow down the success - * path, so use actual RCU here. + * path. module_memfree() cannot be called in an interrupt, so do the + * work and call synchronize_rcu() in a work queue. + * * Note that module_alloc() on most architectures creates W+X page * mappings which won't be cleaned up until do_free_init() runs. Any * code such as mark_rodata_ro() which depends on those mappings to * be cleaned up needs to sync with the queued work - ie * rcu_barrier() */ - call_rcu(&freeinit->rcu, do_free_init); + if (llist_add(&freeinit->node, &init_free_list)) + schedule_work(&init_free_wq); + mutex_unlock(&module_mutex); wake_up_all(&module_wq); @@ -3822,10 +3827,6 @@ static int load_module(struct load_info *info, const char __user *uargs, module_bug_cleanup(mod); mutex_unlock(&module_mutex); - /* we can't deallocate the module until we clear memory protection */ - module_disable_ro(mod); - module_disable_nx(mod); - ddebug_cleanup: ftrace_release_mod(mod); dynamic_debug_remove(mod, info->debug);