From patchwork Tue Mar 14 07:05:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhenhua Huang X-Patchwork-Id: 13173773 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 280FBC6FD1F for ; Tue, 14 Mar 2023 07:05:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 53E6D6B0072; Tue, 14 Mar 2023 03:05:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 51E8B6B0074; Tue, 14 Mar 2023 03:05:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3DCED6B0075; Tue, 14 Mar 2023 03:05:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2D2306B0072 for ; Tue, 14 Mar 2023 03:05:35 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id DA2ED80BDD for ; Tue, 14 Mar 2023 07:05:34 +0000 (UTC) X-FDA: 80566618188.10.784F346 Received: from mx0b-0031df01.pphosted.com (mx0b-0031df01.pphosted.com [205.220.180.131]) by imf11.hostedemail.com (Postfix) with ESMTP id D2CF340004 for ; Tue, 14 Mar 2023 07:05:31 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=quicinc.com header.s=qcppdkim1 header.b=GqSVXa1Z; spf=pass (imf11.hostedemail.com: domain of quic_zhenhuah@quicinc.com designates 205.220.180.131 as permitted sender) smtp.mailfrom=quic_zhenhuah@quicinc.com; dmarc=pass (policy=none) header.from=quicinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678777532; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=tiUmcexDo0RBs7bw61Wmb4mvOXf5Hp4R+5QOCWVmynI=; b=bggy8JnIIHo6bGlluIOA8O86WsFRd/3j5ZbQcynHQyswYAbe72XarZ1KSDXVj1wtdjulUI dbqXvoSwhALm4ltnDsm9WdfT1LuAceHkAWcdXloYF/5FsSFHFUzV0tYRwe6Tb44xa5XPn2 NIqQV9up7o9+C74orvfa6FHj4k1gya8= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=quicinc.com header.s=qcppdkim1 header.b=GqSVXa1Z; spf=pass (imf11.hostedemail.com: domain of quic_zhenhuah@quicinc.com designates 205.220.180.131 as permitted sender) smtp.mailfrom=quic_zhenhuah@quicinc.com; dmarc=pass (policy=none) header.from=quicinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678777532; a=rsa-sha256; cv=none; b=Oq95qNhy5oCNGDXr8qZizpRpHtkw9MMC6Q9CREYUUCDEidmKLdReEWH5gtsdrhnc7y4ua2 GKnaYlECaSrFHnVCwGzQFAN9VYgS6Q6RwtNJqyQgnXne9eERh8Qz9pItVPB/5lg4zHylxQ NpKFgesD6fsAwvCTTVnIRdaLJ90WcEM= Received: from pps.filterd (m0279871.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 32E3YQ8Q007143; Tue, 14 Mar 2023 07:05:16 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=from : to : cc : subject : date : message-id : mime-version : content-type; s=qcppdkim1; bh=tiUmcexDo0RBs7bw61Wmb4mvOXf5Hp4R+5QOCWVmynI=; b=GqSVXa1ZuSiT/LBAtJ/CgoKcvkRyXp8A6VrpaEJzGpl8KLhqm2BL0CZv3YiCJg0aAAfi mHm4YVw5KnPtcyNznPj137fo4udFJvtirZ+3WqBDTzoL4u/L/XTNDj5lPXemRCIs1WSQ nS9vxxaoKmBn5DVjmb/xVB2VrO3UJzpAbje2Dmb/XS0eWJMjCpVZwjopMnjj4brAM28N sgpsvUbx5AHcpWbL/snE4Ar+mez57xqb/C1sQBMZrUUhXav2+fqchHXFpB2CnghNqwRM oDQK9ihfRy8ImjsuOfOG0vagOC2fF4FLFT018/LDe8QpvJ0W53bfHbK0GEDrrLXz/fNN OA== Received: from nalasppmta05.qualcomm.com (Global_NAT1.qualcomm.com [129.46.96.20]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3pa3pttmwj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Mar 2023 07:05:16 +0000 Received: from nalasex01a.na.qualcomm.com (nalasex01a.na.qualcomm.com [10.47.209.196]) by NALASPPMTA05.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 32E75EJA018045 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 14 Mar 2023 07:05:14 GMT Received: from zhenhuah-gv.qualcomm.com (10.80.80.8) by nalasex01a.na.qualcomm.com (10.47.209.196) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 14 Mar 2023 00:05:10 -0700 From: Zhenhua Huang To: , , , , , , , , , , CC: Zhenhua Huang , , , , , , Subject: [PATCH v8] mm,kfence: decouple kfence from page granularity mapping judgement Date: Tue, 14 Mar 2023 15:05:02 +0800 Message-ID: <1678777502-6933-1-git-send-email-quic_zhenhuah@quicinc.com> X-Mailer: git-send-email 2.7.4 MIME-Version: 1.0 X-Originating-IP: [10.80.80.8] X-ClientProxiedBy: nasanex01b.na.qualcomm.com (10.46.141.250) To nalasex01a.na.qualcomm.com (10.47.209.196) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: BFosTgJeiPDhsjPCVbSXxpDRtTHdbmcX X-Proofpoint-ORIG-GUID: BFosTgJeiPDhsjPCVbSXxpDRtTHdbmcX X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-03-13_13,2023-03-14_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 spamscore=0 priorityscore=1501 malwarescore=0 mlxscore=0 clxscore=1015 phishscore=0 impostorscore=0 adultscore=0 mlxlogscore=999 bulkscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2303140059 X-Rspamd-Queue-Id: D2CF340004 X-Stat-Signature: bw153t9x7pce4b1rh6cusbmjycppe9jb X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1678777531-411629 X-HE-Meta: U2FsdGVkX1+cRE5a/5VkAcDQZEvCw8/8gbzVSu/Zqaw2aVie6ekTRiiN4Ti3cEmhDpqAY990H8dX+aD50DMCtFM5Xq05ecbWvDWDq68Jh+4Aser076Ok6P5DpZL7Qdcb37Zp2xDF4s/mEBuj3r5zUOcTUdr5eJXAB7oR8OFk1zgisQBFfP+0kzw+rcVYDdWlA/y7t95kmYzbJF78kLaj7Gc9Uw6x5LNXb5+6R09SPszu9LQfUh5ldLmI7lq+RSJb54O2TA7EhmJ1PQ2laJY9f3jg3Tj9ACZ94kc7VsVkWK7L3gDvVR/XnmARcZpkaIlD4yOP3eZ+3LtDEpWGYzapXp+b21UPA6kECdbMlHqhZs2lIF5Nxi0SdZYa0ioFeOwgYo3tySk3HJitaV2H4qYM1dAaEHaaI8aCuP4DSl06yC3+cTDQGgkzZkY26pcX+t14SzNSshSWBXiNEMEcWVcdCmqzGmY7EStopdzAHgx6XA4v6ZlwcZ67922xFcpfaqaRv62UoiqnIZsKTxFRMI/id3/Pe55DrrE81rY41SkVFJWns8vN3Q8YtQ9gXwwuYEQS0qEoqGbiAPhiOnyhri+ymFUehR2y5CJwWd/NcZ4jUhEhweZ1PMMF88+ZZ210KJs/UdXvD2woLnuch3TnXG/T5yF/pTwGuJ4SIJatCOionX56wRaEvHVgDupXKRns8aadDCHqtrklVEWFZfeyKwm6elFIIw2V1vNHGtpMzjUpHUdwH98uLybZZlpBCpUmpURygfFs4Qm4rNwjT0oPcYza24/vG/c+50FvOQOsjXPyXrN5tdzvItkpwbUjgsF1Pr6XRd+sAnqZ5/+NA8JLj//Lcko4swlr8vbt6+7AhI2RrM/GtSUbHzOHTBnayc552vTlhoXYVuoRoTh53w5fgEY1nk2VsGbp/pg5erAdj5Lb6fGnYDr+gNMntXFG4l7pa+cRKFJjAvejOL2GUtNf6ah DL/jia+G TUdFjM8jB0gvP3UZW/w7BIauZpi0Ndb48XOi9Wo8naNXO4OqSdSeK08XewqiW+2izSTSMShNmHpH8mmtM0txuW24PTcapAsMxOnKaQSMpZjBYRoidT3+E6PsLwQrm5Zrv83uaGtkUfujhFqVtCDbRXnbaxKCqxutKZrnqhI0AmbtcJKCMrTA6hVR5OrsLFBfEOaQBkKYHbv8yjDzo4epuI0Koyq2A6fVqvPkt6G2SjZhw4koHtlRVjCDPjXA14Q/rCeS1pdQ3E1QHN4Zhehs4TtU+bpY6DvPeWIUxDnkWGZOcDyPmG5jNI0h1FLiF9dBEkGIP3hKSNR8Qm0L5PLOHAZjtAOlrbOWmFepp X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Kfence only needs its pool to be mapped as page granularity, if it is inited early. Previous judgement was a bit over protected. From [1], Mark suggested to "just map the KFENCE region a page granularity". So I decouple it from judgement and do page granularity mapping for kfence pool only. Need to be noticed that late init of kfence pool still requires page granularity mapping. Page granularity mapping in theory cost more(2M per 1GB) memory on arm64 platform. Like what I've tested on QEMU(emulated 1GB RAM) with gki_defconfig, also turning off rodata protection: Before: [root@liebao ]# cat /proc/meminfo MemTotal: 999484 kB After: [root@liebao ]# cat /proc/meminfo MemTotal: 1001480 kB To implement this, also relocate the kfence pool allocation before the linear mapping setting up, arm64_kfence_alloc_pool is to allocate phys addr, __kfence_pool is to be set after linear mapping set up. LINK: [1] https://lore.kernel.org/linux-arm-kernel/Y+IsdrvDNILA59UN@FVFF77S0Q05N/ Suggested-by: Mark Rutland Signed-off-by: Zhenhua Huang --- arch/arm64/include/asm/kfence.h | 2 ++ arch/arm64/mm/mmu.c | 44 +++++++++++++++++++++++++++++++++++++++++ arch/arm64/mm/pageattr.c | 9 +++++++-- include/linux/kfence.h | 8 ++++++++ mm/kfence/core.c | 9 +++++++++ 5 files changed, 70 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kfence.h b/arch/arm64/include/asm/kfence.h index aa855c6..f1f9ca2d 100644 --- a/arch/arm64/include/asm/kfence.h +++ b/arch/arm64/include/asm/kfence.h @@ -10,6 +10,8 @@ #include +extern phys_addr_t early_kfence_pool; + static inline bool arch_kfence_init_pool(void) { return true; } static inline bool kfence_protect_page(unsigned long addr, bool protect) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 6f9d889..7fbf2ed 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include @@ -38,6 +39,7 @@ #include #include #include +#include #define NO_BLOCK_MAPPINGS BIT(0) #define NO_CONT_MAPPINGS BIT(1) @@ -525,6 +527,33 @@ static int __init enable_crash_mem_map(char *arg) } early_param("crashkernel", enable_crash_mem_map); +#ifdef CONFIG_KFENCE + +static phys_addr_t arm64_kfence_alloc_pool(void) +{ + phys_addr_t kfence_pool; + + if (!kfence_sample_interval) + return 0; + + kfence_pool = memblock_phys_alloc(KFENCE_POOL_SIZE, PAGE_SIZE); + if (!kfence_pool) + pr_err("failed to allocate kfence pool\n"); + + return kfence_pool; +} + +#else + +static phys_addr_t arm64_kfence_alloc_pool(void) +{ + return 0; +} + +#endif + +phys_addr_t early_kfence_pool; + static void __init map_mem(pgd_t *pgdp) { static const u64 direct_map_end = _PAGE_END(VA_BITS_MIN); @@ -543,6 +572,10 @@ static void __init map_mem(pgd_t *pgdp) */ BUILD_BUG_ON(pgd_index(direct_map_end - 1) == pgd_index(direct_map_end)); + early_kfence_pool = arm64_kfence_alloc_pool(); + if (early_kfence_pool) + memblock_mark_nomap(early_kfence_pool, KFENCE_POOL_SIZE); + if (can_set_direct_map()) flags |= NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; @@ -608,6 +641,17 @@ static void __init map_mem(pgd_t *pgdp) } } #endif + + /* Kfence pool needs page-level mapping */ + if (early_kfence_pool) { + __map_memblock(pgdp, early_kfence_pool, + early_kfence_pool + KFENCE_POOL_SIZE, + pgprot_tagged(PAGE_KERNEL), + NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS); + memblock_clear_nomap(early_kfence_pool, KFENCE_POOL_SIZE); + /* kfence_pool really mapped now */ + kfence_set_pool(early_kfence_pool); + } } void mark_rodata_ro(void) diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 79dd201..7ce5295 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -7,10 +7,12 @@ #include #include #include +#include #include #include #include +#include struct page_change_data { pgprot_t set_mask; @@ -22,12 +24,15 @@ bool rodata_full __ro_after_init = IS_ENABLED(CONFIG_RODATA_FULL_DEFAULT_ENABLED bool can_set_direct_map(void) { /* - * rodata_full, DEBUG_PAGEALLOC and KFENCE require linear map to be + * rodata_full and DEBUG_PAGEALLOC require linear map to be * mapped at page granularity, so that it is possible to * protect/unprotect single pages. + * + * Kfence pool requires page granularity mapping also if we init it + * late. */ return (rodata_enabled && rodata_full) || debug_pagealloc_enabled() || - IS_ENABLED(CONFIG_KFENCE); + (IS_ENABLED(CONFIG_KFENCE) && !early_kfence_pool); } static int change_page_range(pte_t *ptep, unsigned long addr, void *data) diff --git a/include/linux/kfence.h b/include/linux/kfence.h index 726857a..570d4e3 100644 --- a/include/linux/kfence.h +++ b/include/linux/kfence.h @@ -64,6 +64,12 @@ static __always_inline bool is_kfence_address(const void *addr) void __init kfence_alloc_pool(void); /** + * kfence_set_pool() - allows an arch to set the + * KFENCE pool during early init + */ +void __init kfence_set_pool(phys_addr_t addr); + +/** * kfence_init() - perform KFENCE initialization at boot time * * Requires that kfence_alloc_pool() was called before. This sets up the @@ -222,8 +228,10 @@ bool __kfence_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *sla #else /* CONFIG_KFENCE */ +#define KFENCE_POOL_SIZE 0 static inline bool is_kfence_address(const void *addr) { return false; } static inline void kfence_alloc_pool(void) { } +static inline void kfence_set_pool(phys_addr_t addr) { } static inline void kfence_init(void) { } static inline void kfence_shutdown_cache(struct kmem_cache *s) { } static inline void *kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags) { return NULL; } diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 5349c37..0765395 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -814,12 +814,21 @@ void __init kfence_alloc_pool(void) if (!kfence_sample_interval) return; + /* if the pool has already been initialized by arch, skip the below */ + if (__kfence_pool) + return; + __kfence_pool = memblock_alloc(KFENCE_POOL_SIZE, PAGE_SIZE); if (!__kfence_pool) pr_err("failed to allocate pool\n"); } +void __init kfence_set_pool(phys_addr_t addr) +{ + __kfence_pool = phys_to_virt(addr); +} + static void kfence_init_enable(void) { if (!IS_ENABLED(CONFIG_KFENCE_STATIC_KEYS))