From patchwork Thu Feb 15 23:13:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Edgecombe, Rick P" X-Patchwork-Id: 13559260 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90F36C48BC4 for ; Thu, 15 Feb 2024 23:14:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 34ADA6B0088; Thu, 15 Feb 2024 18:14:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2D1996B008A; Thu, 15 Feb 2024 18:14:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0FFC66B0092; Thu, 15 Feb 2024 18:14:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C34EF6B008A for ; Thu, 15 Feb 2024 18:14:44 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8D1FE120842 for ; Thu, 15 Feb 2024 23:14:44 +0000 (UTC) X-FDA: 81795594888.17.7290AB7 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.21]) by imf09.hostedemail.com (Postfix) with ESMTP id 9D3EA140015 for ; Thu, 15 Feb 2024 23:14:42 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=NfpFrd8Q; spf=pass (imf09.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 198.175.65.21 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708038882; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=y0eyua5VLqarJkFbZIkOnrGFAFnMrr4VgPsmAogZaFc=; b=2RxfUBVltD3+Dxo+ieNdY5c443FXReGzUT1B45WSeBpNHr9WfvPyoa4mqESBKC58gyZ9ah foLkXTex0Z6r4kpQXMcIhMXAjbeIL/AZy0BXH5k71K7N7Mrj7bou56V4nxQt3GofjbK9G1 rejco0yX4PRjP60qsWzdbiwXIr4oVMc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708038882; a=rsa-sha256; cv=none; b=HPpdf9n0q5WMrvnQGgui3/tfqKWMqe782IV/xBg7Q7Z1TxGbn5McIW7ggRQ5lwhBXmCqsh 4dLrdgJdJCjsqoXUFraINywWeDPghay7T6aV7X+HPRQXVfz2urZ+WTQ9UQnGXzEOQSQY+w 2HLDdTTRbMMytMO2k4zozWNPmJmGJgA= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=NfpFrd8Q; spf=pass (imf09.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 198.175.65.21 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708038883; x=1739574883; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=XHhdhuxOKWz8EspMQDfPASDrP/vRFk3wVhxiIKVJyZE=; b=NfpFrd8QKeI9hUSsEoGQdlTQLTlezIbtpPp8Y4K6b2oPY9jS6+Wvohsm 2vvbo9RRbf9w7K80AvrYN+7sm+rvOYqOKCPuyhyzXKQBAynW5cstBsKNF OSIlrxc06ExNllCwHwuym+USb6RXSx8RBsjlgdvIqvYsA31ULgejFJ0SY XsL5ww3hdqlK+i27Qh1s9X4vkADpL81xt7EPzKOFTDYAI8+WHpKt0hh54 TxEWKe4CAu/IwDzTl3BFj1l8ufffzWSOd9k61b89Qd1YTW8d/3+R44VkM Zsb2vKgtuE4L+eeoNW1ckkVHMhYLvekTND0Fq9hE5rFaFNPY9W+ALI0TH w==; X-IronPort-AV: E=McAfee;i="6600,9927,10985"; a="2066347" X-IronPort-AV: E=Sophos;i="6.06,162,1705392000"; d="scan'208";a="2066347" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orvoesa113.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2024 15:14:41 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10985"; a="912250197" X-IronPort-AV: E=Sophos;i="6.06,162,1705392000"; d="scan'208";a="912250197" Received: from yshin-mobl1.amr.corp.intel.com (HELO rpedgeco-desk4.intel.com) ([10.209.95.133]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Feb 2024 15:14:39 -0800 From: Rick Edgecombe To: Liam.Howlett@oracle.com, akpm@linux-foundation.org, debug@rivosinc.com, broonie@kernel.org, kirill.shutemov@linux.intel.com, keescook@chromium.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, peterz@infradead.org, hpa@zytor.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: rick.p.edgecombe@intel.com Subject: [RFC PATCH 4/8] thp: Add thp_get_unmapped_area_vmflags() Date: Thu, 15 Feb 2024 15:13:28 -0800 Message-Id: <20240215231332.1556787-5-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240215231332.1556787-1-rick.p.edgecombe@intel.com> References: <20240215231332.1556787-1-rick.p.edgecombe@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9D3EA140015 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 3ew9zr6kw8cz7zo7a3535cxo8pqq743n X-HE-Tag: 1708038882-276667 X-HE-Meta: U2FsdGVkX1/j/RH3h/XHa2KhnOpSJLGwanXeUTGPFcKDGm50oU0bCTc572ij4+dUXcKYia0uVTTcsP/g099HXJciJHJFSExK7VI/G6m0y7vfGPRwmy5567frb/rXeGgHtwvVjseLrsVDJ+RRNvz0rieJUFSE2cHmW9c0mCODc3R0OVQta7cE6xZsZML7LHg7S4itkn6JUQIKSA3lpDI6ytqexroIGHPb7INf2BX6YjdfZvAOaMFWKqtjc1c2Oc/VupRreSqqSYL1yK4vNcAhZkM/ZovKIJIwsNC96BnioEQ6FT7tZ3/9edvRm6yPGhKk/PcZ9peqAOlv/3hkiYMGBG57Ja78p7ilLbHGBWuoqCJSHh2/0XxRwhF7nB+qRIaU5VGWOvhIltVIXFRjhRGASvuoB/5ZW++l9iyuYhBKkLAf2Or7M4/m+TYBhCCzM/ouavbewj8KIRvMi09wXwp+z5i9XjdJMvO3U3j3QleNLX24NiMJ0WUrB/YPrWQK2bmtzb+QPanZCB86e5lzICelQuiAVbBFyYMNcittqztJ36r6YgpQUE4MU9LkvdREDA7g13JZu8vdeZZvCDnZqnUW5Ec366hHFJv/Ee/8NsawHgeaYWJW3NegJiMc4vj85PLuckAg0CIpOHHy1Bp9gUCYKLRrjDlvVxgEarhIvg/f2UW1+cvL3ztdFpkN/YdhSXjfFQ5Q+4OFSoT6UpxNHwh9XJuPId+Jfr4sEVXy69Yi3K5S4sVfMyn8f/OIfgLL1hkU7gwqprRMyrCCbt2VHrs5VPjGWPZ1Tls/iaspw6IqVkvfkvec6s4gtzUBRcIQSw5psSwzXueukaTQ5XCKGfrVJFt1Gs9HOvj3UpyFj7nZlvQKBr2XSg0zuYdeKrtzr73Jh09pS35xrrE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things: 1. That the new mapping isn’t placed in an any existing mappings guard gaps. 2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps. The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA. Add a THP implementations of the vm_flags variant of get_unmapped_area(). Future changes will call this from mmap.c in the do_mmap() path to allow shadow stacks to be placed with consideration taken for the start guard gap. Shadow stack memory is always private and anonymous and so special guard gap logic is not needed in a lot of caseis, but it can be mapped by THP, so needs to be handled. Signed-off-by: Rick Edgecombe --- include/linux/huge_mm.h | 11 +++++++++++ mm/huge_memory.c | 23 ++++++++++++++++------- mm/mmap.c | 12 +++++++----- 3 files changed, 34 insertions(+), 12 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index fa0350b0812a..ef7251dfd9f9 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -139,6 +139,9 @@ bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags); +unsigned long thp_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, unsigned long flags, + vm_flags_t vm_flags); void folio_prep_large_rmappable(struct folio *folio); bool can_split_folio(struct folio *folio, int *pextra_pins); @@ -286,6 +289,14 @@ static inline void folio_prep_large_rmappable(struct folio *folio) {} #define thp_get_unmapped_area NULL +static inline unsigned long +thp_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, + unsigned long flags, vm_flags_t vm_flags) +{ + return 0; +} + static inline bool can_split_folio(struct folio *folio, int *pextra_pins) { diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e9ef43a719a5..f235f6d3ff62 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -628,7 +628,8 @@ static inline bool is_transparent_hugepage(struct folio *folio) static unsigned long __thp_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, - loff_t off, unsigned long flags, unsigned long size) + loff_t off, unsigned long flags, unsigned long size, + vm_flags_t vm_flags) { loff_t off_end = off + len; loff_t off_align = round_up(off, size); @@ -641,8 +642,8 @@ static unsigned long __thp_get_unmapped_area(struct file *filp, if (len_pad < len || (off + len_pad) < off) return 0; - ret = mm_get_unmapped_area(current->mm, filp, addr, len_pad, - off >> PAGE_SHIFT, flags); + ret = mm_get_unmapped_area_vmflags(current->mm, filp, addr, len_pad, + off >> PAGE_SHIFT, flags, vm_flags); /* * The failure might be due to length padding. The caller will retry @@ -662,17 +663,25 @@ static unsigned long __thp_get_unmapped_area(struct file *filp, return ret; } -unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, - unsigned long len, unsigned long pgoff, unsigned long flags) +unsigned long thp_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, unsigned long flags, + vm_flags_t vm_flags) { unsigned long ret; loff_t off = (loff_t)pgoff << PAGE_SHIFT; - ret = __thp_get_unmapped_area(filp, addr, len, off, flags, PMD_SIZE); + ret = __thp_get_unmapped_area(filp, addr, len, off, flags, PMD_SIZE, vm_flags); if (ret) return ret; - return mm_get_unmapped_area(current->mm, filp, addr, len, pgoff, flags); + return mm_get_unmapped_area_vmflags(current->mm, filp, addr, len, pgoff, flags, + vm_flags); +} + +unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, unsigned long flags) +{ + return thp_get_unmapped_area_vmflags(filp, addr, len, pgoff, flags, 0); } EXPORT_SYMBOL_GPL(thp_get_unmapped_area); diff --git a/mm/mmap.c b/mm/mmap.c index 2723f26f7c62..936d728ba1ca 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1857,16 +1857,18 @@ __get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, */ pgoff = 0; get_area = shmem_get_unmapped_area; - } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { - /* Ensures that larger anonymous mappings are THP aligned. */ - get_area = thp_get_unmapped_area; } - if (get_area) + if (get_area) { addr = get_area(file, addr, len, pgoff, flags); - else + } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + /* Ensures that larger anonymous mappings are THP aligned. */ + addr = thp_get_unmapped_area_vmflags(file, addr, len, + pgoff, flags, vm_flags); + } else { addr = mm_get_unmapped_area_vmflags(current->mm, file, addr, len, pgoff, flags, vm_flags); + } if (IS_ERR_VALUE(addr)) return addr;