From patchwork Wed Nov 28 00:07:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 10701657 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AD8D913BB for ; Wed, 28 Nov 2018 00:34:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9ACFB2C8D3 for ; Wed, 28 Nov 2018 00:34:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8ED422C8D9; Wed, 28 Nov 2018 00:34:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0A9752C8D3 for ; Wed, 28 Nov 2018 00:34:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5D2456B467E; Tue, 27 Nov 2018 19:34:09 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 519686B467D; Tue, 27 Nov 2018 19:34:09 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B9806B4683; Tue, 27 Nov 2018 19:34:09 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 057746B4682 for ; Tue, 27 Nov 2018 19:34:08 -0500 (EST) Received: by mail-pf1-f197.google.com with SMTP id u20so13239699pfa.1 for ; Tue, 27 Nov 2018 16:34:08 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=fsN4TEM+Jd846VfbpOFhinDwRxIYax74GBQHpii2IVI=; b=Y/rtykRFFodZpE7bFa4bq00cBib1Xnw47ebgLVlLyWEe3AoJu2i7EoIX/SmMrFJfYr ZGEfuc1rW3MNMZTTsb0a0q/sWsbLZ34BUJO4H0llYeB2QBMdDXiZCKePU5aDYJk5g/2R vd4W8POxCHLyeDYBcPz62ZbCj8ysyJswa4LwWarmpBoSVzHqNhDmtGXFyDpIuKqisw76 Qm46OFd+GB+QNBqoTo7fPZrOuWNbBL7wer86gyl5NjshmEt/KYr0rYJVg84hkVAKhmlN 9A/btux9IWBilwu2D8cjQu4DCLjgUgVUI6fFWAh3Djsxqf8KtMHdwP4RyrOzPK9oaZ5U t0pg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AA+aEWb6571HNXECF0AsuhQnY818pGCRU21CkPql32PONnv6U8xJ6YNG jeWeXf2m3L1uGlKlTiAjQoEK6zjlFN4NMnvnOeljwaw4hkGk92CCgLV0OTEfgHFXDA70nKcLAuZ y47J7b3kdRom3JOH7SPsEz53k027L2sgsCItJylZ7R8XtmtFfySOpjnikAqElRPi7Kg== X-Received: by 2002:a17:902:e002:: with SMTP id ca2mr35213697plb.103.1543365248578; Tue, 27 Nov 2018 16:34:08 -0800 (PST) X-Google-Smtp-Source: AFSGD/W53i5MoQTIzAMIg7XWhLan+qJzMAyMgflH/2ZKzNsUQWPvnUJkX4p46UGYOSF9mqYSyR52 X-Received: by 2002:a17:902:e002:: with SMTP id ca2mr35213654plb.103.1543365247576; Tue, 27 Nov 2018 16:34:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1543365247; cv=none; d=google.com; s=arc-20160816; b=PNbRo4I/dsaz/JffaPWCy7oeJaSvDjcU3MhUA0ZOiKL762RwtCFSiI4qka3kN2YNS7 Usg8N8buROyeDU8Qs2xLlVsNEUONV0ddzR+bOQWl1GjytQ/PREIu7ebIIkrQ1ABtwezj dm9yzGlgUPdgQ+K/lWP3crkEFvK40M8wmucQO5ZqGYdL45qMLqN1ivgnWnwiLJzEnTJ0 6RRbM2iTz5GpqG3bXWQKzPmKlQk7H9kxol9FC1TzfPMzBf/5iIGGwP9J8kTznvQUI3Yo 8oEllBJr1T6zS/Oof4OL+UQ6N1PKaP6yp9zHVcSA4NFORDLuFGMISl7Rz3T1RXG1/4Su DeVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=fsN4TEM+Jd846VfbpOFhinDwRxIYax74GBQHpii2IVI=; b=N70o2LB4B6s529q6kFn2eqJ7AH1JqkGjtyiq6Th3vACe/EbqhcxPWboOX+3Ti3lIEu SIFCqxORDKKhLYjg05zv4vl1LwzISxz01DcmrZLna7f3kG6Kv7zRtxlRXRXJWa2ijACb 80q0pIiJ4kT3zVBXfECcv8ig2u6B+ijAysTYR0oCAY5iGbQDmj4WprVYXEs6gW0rlJmC fSnuaKxZ3CRrvmVOrEUK0ZsoyF0UOJHo/Dw2X7DUxOgTTjKLvOndHS7LigCyj4g41GB1 w505ntXH7e6nvduqQHuT+lWEjD8dfqhkEk5eX8Bt4xWV0qm/97g7sgb/Pje0Dpa0Azsy EhBQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga17.intel.com (mga17.intel.com. [192.55.52.151]) by mx.google.com with ESMTPS id s5si5346864pfi.134.2018.11.27.16.34.07 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 27 Nov 2018 16:34:07 -0800 (PST) Received-SPF: pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.151 as permitted sender) client-ip=192.55.52.151; Authentication-Results: mx.google.com; spf=pass (google.com: domain of rick.p.edgecombe@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 27 Nov 2018 16:34:07 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,288,1539673200"; d="scan'208";a="91641478" Received: from rpedgeco-desk5.jf.intel.com ([10.54.75.128]) by fmsmga007.fm.intel.com with ESMTP; 27 Nov 2018 16:34:06 -0800 From: Rick Edgecombe To: akpm@linux-foundation.org, luto@kernel.org, will.deacon@arm.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-hardening@lists.openwall.com, naveen.n.rao@linux.vnet.ibm.com, anil.s.keshavamurthy@intel.com, davem@davemloft.net, mhiramat@kernel.org, rostedt@goodmis.org, mingo@redhat.com, ast@kernel.org, daniel@iogearbox.net, jeyu@kernel.org, netdev@vger.kernel.org, ard.biesheuvel@linaro.org, jannh@google.com Cc: kristen@linux.intel.com, dave.hansen@intel.com, deneen.t.dock@intel.com, Rick Edgecombe Subject: [PATCH 1/2] vmalloc: New flag for flush before releasing pages Date: Tue, 27 Nov 2018 16:07:53 -0800 Message-Id: <20181128000754.18056-2-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181128000754.18056-1-rick.p.edgecombe@intel.com> References: <20181128000754.18056-1-rick.p.edgecombe@intel.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Since vfree will lazily flush the TLB, but not lazily free the underlying pages, it often leaves stale TLB entries to freed pages that could get re-used. This is undesirable for cases where the memory being freed has special permissions such as executable. Having callers flush the TLB after calling vfree still leaves a window where the pages are freed, but the TLB entry remains. Also the entire operation can be deferred if the vfree is called from an interrupt and so a TLB flush after calling vfree would miss the entire operation. So in order to support this use case, a new flag VM_IMMEDIATE_UNMAP is added, that will cause the free operation to take place like this: 1. Unmap 2. Flush TLB/Unmap aliases 3. Free pages In the deferred case these steps are all done by the work queue. This implementation derives from two sketches from Dave Hansen and Andy Lutomirski. Suggested-by: Dave Hansen Suggested-by: Andy Lutomirski Suggested-by: Will Deacon Signed-off-by: Rick Edgecombe --- include/linux/vmalloc.h | 1 + mm/vmalloc.c | 13 +++++++++++-- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 398e9c95cd61..cca6b6b83cf0 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -21,6 +21,7 @@ struct notifier_block; /* in notifier.h */ #define VM_UNINITIALIZED 0x00000020 /* vm_struct is not fully initialized */ #define VM_NO_GUARD 0x00000040 /* don't add guard page */ #define VM_KASAN 0x00000080 /* has allocated kasan shadow memory */ +#define VM_IMMEDIATE_UNMAP 0x00000200 /* flush before releasing pages */ /* bits [20..32] reserved for arch specific ioremap internals */ /* diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 97d4b25d0373..68766651b5a7 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1516,6 +1516,14 @@ static void __vunmap(const void *addr, int deallocate_pages) debug_check_no_obj_freed(area->addr, get_vm_area_size(area)); remove_vm_area(addr); + + /* + * Need to flush the TLB before freeing pages in the case of this flag. + * As long as that's happening, unmap aliases. + */ + if (area->flags & VM_IMMEDIATE_UNMAP) + vm_unmap_aliases(); + if (deallocate_pages) { int i; @@ -1925,8 +1933,9 @@ EXPORT_SYMBOL(vzalloc_node); void *vmalloc_exec(unsigned long size) { - return __vmalloc_node(size, 1, GFP_KERNEL, PAGE_KERNEL_EXEC, - NUMA_NO_NODE, __builtin_return_address(0)); + return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, + GFP_KERNEL, PAGE_KERNEL_EXEC, VM_IMMEDIATE_UNMAP, + NUMA_NO_NODE, __builtin_return_address(0)); } #if defined(CONFIG_64BIT) && defined(CONFIG_ZONE_DMA32)