From patchwork Tue Mar 12 22:28:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Rick Edgecombe X-Patchwork-Id: 13590669 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB04AC54E58 for ; Tue, 12 Mar 2024 22:29:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E9AC88E0021; Tue, 12 Mar 2024 18:29:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E222C8E0011; Tue, 12 Mar 2024 18:29:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CEA738E0021; Tue, 12 Mar 2024 18:29:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BBB768E0011 for ; Tue, 12 Mar 2024 18:29:08 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 83A2D40730 for ; Tue, 12 Mar 2024 22:29:08 +0000 (UTC) X-FDA: 81889828776.24.7714EF2 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.15]) by imf11.hostedemail.com (Postfix) with ESMTP id 83DF04000C for ; Tue, 12 Mar 2024 22:29:06 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=UlTlTOpD; spf=pass (imf11.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.198.163.15 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710282546; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=E4bFmXjAMRGmz8BILlOEqF3bzfCSQ9N6uf10SfMcabk=; b=3LtGWj61j5IpMoZPgAvuCSlYe14GNqiXCWWTNEZqmyFZhC86aw0IgM6u+wu19k0TJHpRGI 3Aup0nWe/aq1/4xKmWmIjmpSxcWo7T8ToHU8EuERRguP/jle1jZFKGN7NQcNQUnle0XRcv dHCXxOlQsM3c6qogIvJJLabMMvQI1Cw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710282546; a=rsa-sha256; cv=none; b=7hq+NQ8BfoK5Ckprt6RHidwxuauRff7ETqMm1QZVbP1a47WsikIVLt/KOLlP1TXZnWyCJZ uuHNmpLVDvO5pPMXAaC0S2IsPqD05gxJNIuNXkNp50kmUD71YUy3NU0YhhPI+iGe7tAiW7 ROrwxUseUfGpHywre5kJfNIP2HUt7fY= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=UlTlTOpD; spf=pass (imf11.hostedemail.com: domain of rick.p.edgecombe@intel.com designates 192.198.163.15 as permitted sender) smtp.mailfrom=rick.p.edgecombe@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1710282547; x=1741818547; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=/QgkcmxaszXiFcioo4194oXRAE7nWH9S6PPTH54Wib0=; b=UlTlTOpDaiSaolUkzychZ89iIx5Rqx5x/cxcjjwfxVY/0yZuApvXo7Po B1JSYKALI55yOd2CEIXOOSubg49O6bW5TuykJtNvFh8l9zsxPb60b3j7n PsQK+FyjilpQUTFti1DbpRNFBqXw/ifdaT0rUCkHvFU36Ya6IQqvBufFs Y8OjcAlSkQMoaQ3de4/h7wawX6yZ8jZiU7iqwOoWwhyqRVTYYsd+V9Nye XiyffL3zZZSCaU88oUayodXMKlQAgenQ0jff3Xn75Z8TsWV9dbI9YXWWY 5L3PbV9JvOf6Cntu4/JIfcbKbo/FVcilGNYV1kk8eipHtlvNVRLM4GwlM g==; X-IronPort-AV: E=McAfee;i="6600,9927,11011"; a="5191958" X-IronPort-AV: E=Sophos;i="6.07,119,1708416000"; d="scan'208";a="5191958" Received: from orviesa004.jf.intel.com ([10.64.159.144]) by fmvoesa109.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2024 15:29:03 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,119,1708416000"; d="scan'208";a="16356841" Received: from gargayus-mobl1.amr.corp.intel.com (HELO rpedgeco-desk4.intel.com) ([10.255.231.196]) by orviesa004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Mar 2024 15:29:02 -0700 From: Rick Edgecombe To: Liam.Howlett@oracle.com, akpm@linux-foundation.org, bp@alien8.de, broonie@kernel.org, dave.hansen@linux.intel.com, debug@rivosinc.com, hpa@zytor.com, keescook@chromium.org, kirill.shutemov@linux.intel.com, luto@kernel.org, mingo@redhat.com, peterz@infradead.org, tglx@linutronix.de, x86@kernel.org, christophe.leroy@csgroup.eu Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, rick.p.edgecombe@intel.com Subject: [PATCH v3 04/12] thp: Add thp_get_unmapped_area_vmflags() Date: Tue, 12 Mar 2024 15:28:35 -0700 Message-Id: <20240312222843.2505560-5-rick.p.edgecombe@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240312222843.2505560-1-rick.p.edgecombe@intel.com> References: <20240312222843.2505560-1-rick.p.edgecombe@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 83DF04000C X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: h1nm8stof81hrtmyebxz3xourjpn93fb X-HE-Tag: 1710282546-248550 X-HE-Meta: U2FsdGVkX19D7MxvyjuRiaszRe9RhRRyCWDNwLK/UkuAizskL/LRz8cc1+47LAPHIiHV4i+QNBjMUxr9YA3CSJgIjIpGRDBsfNOcEd4qk3HvElAGCpkt+C8VEgDECGXt7/i+SY/fo1y91Z4SAignvYdPi5zHR9vq8OKj682xDf5hWTwPvNjABZyvUcIQdzqzzD8gI6dQqcCFyDU50SRqdQMhtcm+Zk0eV+KIPAQYgclpI1DnGVUnMbWl8Bj1gVo6BN+tdcBTDrFRQ8M/lhDwPz4vXmavby9NJrS7+ejz6104XkABkMcratYBbnOAqWMA9jxuomYUm32b9ahizMXKJvLvRoWRLblQhN4+jQlF0AZtWcQjyzlCBMIlvU06e2LgcQid1CkQy/wFfDgJEsF5i3jflD7EafMLKk1Q9yzRjJoxe/n/XHg0VoFhkQoMHVp4V3dcar3lBX7vNGB7OcF81x2U+FjbiBGKGkx8iHzIeLvlH5BQPUO1Ve+2WmbOsl7v/zvE0Df19DnxZPhRHLvKkSTFLGO7cW8GSAdvvklc+No/aBPbE1URA0WAipLy2Cx0+hlaJoAsfsZzExfy9rCGng075x8eym9+RNWv7DFN1KTcXmf+rhknbEMcN8FJCOTMDdH7TRuYMNdhXIzYlx2gLnRV28GHmed3lz2aNJTZLgJS06Gy0IelNaYvotibZIHBiuQfll1+WhocJVngSFoU1Tum0LjQ6R+dDtaHP79F0HdGtvUFbu9kX/dQOeXw2l/8Ptj52jEf4eylsDu3GxvBOduPtQ7A1o+lcLtLEKQUlosaZjWKZhHkipFwsZn5qJZEnknv720oh0WChZNB0dSB+S7wcHmWD5lCMxge9bH3jRBZpZ1rKzFtxJuTGfk3tX5fDmK/zr0ISPU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When memory is being placed, mmap() will take care to respect the guard gaps of certain types of memory (VM_SHADOWSTACK, VM_GROWSUP and VM_GROWSDOWN). In order to ensure guard gaps between mappings, mmap() needs to consider two things: 1. That the new mapping isn’t placed in an any existing mappings guard gaps. 2. That the new mapping isn’t placed such that any existing mappings are not in *its* guard gaps. The long standing behavior of mmap() is to ensure 1, but not take any care around 2. So for example, if there is a PAGE_SIZE free area, and a mmap() with a PAGE_SIZE size, and a type that has a guard gap is being placed, mmap() may place the shadow stack in the PAGE_SIZE free area. Then the mapping that is supposed to have a guard gap will not have a gap to the adjacent VMA. Add a THP implementations of the vm_flags variant of get_unmapped_area(). Future changes will call this from mmap.c in the do_mmap() path to allow shadow stacks to be placed with consideration taken for the start guard gap. Shadow stack memory is always private and anonymous and so special guard gap logic is not needed in a lot of caseis, but it can be mapped by THP, so needs to be handled. Signed-off-by: Rick Edgecombe Reviewed-by: Christophe Leroy --- include/linux/huge_mm.h | 11 +++++++++++ mm/huge_memory.c | 23 ++++++++++++++++------- mm/mmap.c | 12 +++++++----- 3 files changed, 34 insertions(+), 12 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 5adb86af35fc..8744c808d380 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -262,6 +262,9 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags); +unsigned long thp_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, unsigned long flags, + vm_flags_t vm_flags); void folio_prep_large_rmappable(struct folio *folio); bool can_split_folio(struct folio *folio, int *pextra_pins); @@ -416,6 +419,14 @@ static inline void folio_prep_large_rmappable(struct folio *folio) {} #define thp_get_unmapped_area NULL +static inline unsigned long +thp_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, + unsigned long flags, vm_flags_t vm_flags) +{ + return 0; +} + static inline bool can_split_folio(struct folio *folio, int *pextra_pins) { diff --git a/mm/huge_memory.c b/mm/huge_memory.c index bc3bf441e768..349c93a1a7c3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -806,7 +806,8 @@ static inline bool is_transparent_hugepage(struct folio *folio) static unsigned long __thp_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, - loff_t off, unsigned long flags, unsigned long size) + loff_t off, unsigned long flags, unsigned long size, + vm_flags_t vm_flags) { loff_t off_end = off + len; loff_t off_align = round_up(off, size); @@ -822,8 +823,8 @@ static unsigned long __thp_get_unmapped_area(struct file *filp, if (len_pad < len || (off + len_pad) < off) return 0; - ret = mm_get_unmapped_area(current->mm, filp, addr, len_pad, - off >> PAGE_SHIFT, flags); + ret = mm_get_unmapped_area_vmflags(current->mm, filp, addr, len_pad, + off >> PAGE_SHIFT, flags, vm_flags); /* * The failure might be due to length padding. The caller will retry @@ -848,17 +849,25 @@ static unsigned long __thp_get_unmapped_area(struct file *filp, return ret; } -unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, - unsigned long len, unsigned long pgoff, unsigned long flags) +unsigned long thp_get_unmapped_area_vmflags(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, unsigned long flags, + vm_flags_t vm_flags) { unsigned long ret; loff_t off = (loff_t)pgoff << PAGE_SHIFT; - ret = __thp_get_unmapped_area(filp, addr, len, off, flags, PMD_SIZE); + ret = __thp_get_unmapped_area(filp, addr, len, off, flags, PMD_SIZE, vm_flags); if (ret) return ret; - return mm_get_unmapped_area(current->mm, filp, addr, len, pgoff, flags); + return mm_get_unmapped_area_vmflags(current->mm, filp, addr, len, pgoff, flags, + vm_flags); +} + +unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, unsigned long flags) +{ + return thp_get_unmapped_area_vmflags(filp, addr, len, pgoff, flags, 0); } EXPORT_SYMBOL_GPL(thp_get_unmapped_area); diff --git a/mm/mmap.c b/mm/mmap.c index a3128ed26676..68381b90f906 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1863,20 +1863,22 @@ __get_unmapped_area(struct file *file, unsigned long addr, unsigned long len, * so use shmem's get_unmapped_area in case it can be huge. */ get_area = shmem_get_unmapped_area; - } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { - /* Ensures that larger anonymous mappings are THP aligned. */ - get_area = thp_get_unmapped_area; } /* Always treat pgoff as zero for anonymous memory. */ if (!file) pgoff = 0; - if (get_area) + if (get_area) { addr = get_area(file, addr, len, pgoff, flags); - else + } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + /* Ensures that larger anonymous mappings are THP aligned. */ + addr = thp_get_unmapped_area_vmflags(file, addr, len, + pgoff, flags, vm_flags); + } else { addr = mm_get_unmapped_area_vmflags(current->mm, file, addr, len, pgoff, flags, vm_flags); + } if (IS_ERR_VALUE(addr)) return addr;