From patchwork Wed Jun 19 19:33:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13704466 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 11978155759; Wed, 19 Jun 2024 19:33:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718825638; cv=none; b=cwK4McwYrzAiJwSq2ZD/XH21qmAZUGk59QGeLUmnTtr4/Yl8r0Jv0jXfW4/v4RGKfo4vHwzEHUNxl9+37p5/1RHmKM4JAiJ9zhArUU/VWrbehIUl5OyeYyO0+XxOuJPIWSgCBtAhZh91jdmC0ctPSW1mMbPTk4cL9eXexyE4/Gc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718825638; c=relaxed/simple; bh=jbXDHZFgMTEqVuIbGpjVHki9R28/opuEqbu8dA+PhS0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ocvNig5MeyxMT5aMn7Pob4T0T6Yjb8nFYWdmoWb3gPsAVqaPJnq6b0BBXxcvhKVhE+550Bfe+N4gg+EfHMDYxkD39jBjb4cyD6eg1Y3R9P8wKoduF1tWufmP5BOUMqwncYGqjhSIx7vvXTYWvAFhV8D0wfJqWFLp2Dn5rg/fRxc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=BCyQBt5W; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="BCyQBt5W" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A50BDC4AF08; Wed, 19 Jun 2024 19:33:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718825637; bh=jbXDHZFgMTEqVuIbGpjVHki9R28/opuEqbu8dA+PhS0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BCyQBt5W1LB+VIF0YRwWomc9oJnTKvhC4csl7t1FMqLzr79+LCzMZ4GfPx8Bjh64c K5A5x5XbgSx5bgRgdJgtfY+lYnkb6uee9IXIWSJq2FAtOvKwmMk2w5rYr6qbEb/nMh t+0gx1Sef/GMlGPyl/jpTXLHHwqH5h2rPL3mPe8HNZrXFJKXE0/mz5/9OnvGTG1QFj CCb8pT3aO3iucUMGBBD+k7BbUpejJRBIqxb7hTUa3H77t576ojcEVt9ITWRL832ZtH +kDguxYTHditwu84pRw6xbaV/NsenVniruEylET1/8p1QmUYSru3OQ9+5iIBWPKe6F GgEkkm9+qdgeA== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , "GONG, Ruiqi" , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , jvoisin , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v5 1/6] mm/slab: Introduce kmem_buckets typedef Date: Wed, 19 Jun 2024 12:33:49 -0700 Message-Id: <20240619193357.1333772-1-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240619192131.do.115-kees@kernel.org> References: <20240619192131.do.115-kees@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1406; i=kees@kernel.org; h=from:subject; bh=jbXDHZFgMTEqVuIbGpjVHki9R28/opuEqbu8dA+PhS0=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmczKh6T8pqxaDs77JF9cx/6sa+j5FaKjCYQYlt 8Av1OrjnquJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZnMyoQAKCRCJcvTf3G3A JnDmD/4lPT3IQnslSsmLm0/93PDrMo6dacPrg3fO6iYYge5SRvS8tMrXtla/rIN7FC2Vv9yTvN8 3aMsnyY8X5gSVBi6NnUdOanCs2w+MziAmdDjP4Gaevg1LQRWCzOfOak5shXpArYv+qoZdPw0Ttt LD1n9zWURQ830r1QQ/RiHGcwhlQNTeHINjMW18GIve5hEb2ZaV19FJPBMQP6GyIMeboDi5Zy0m/ RiXCicXZkU2K/T6W7jvQfVD85odLnhPoVFL98TUkeaPAn2aBj54T6GgYlPyc4B1N07E2F7TfLiy OfVAoefI78Tcp2cB03HV8piNNGzhTPM+siCg8NfUxReQDbmLjmWDYw0cPwbZBLrPP+xZ+1vvP0J wmve26MopNZXLnm6/NWvBZQs4gQxk/7OUINSKKWjdRYwClbZWqq0iqM+jyoeuaHxGeb5tpNYp9M FHMNBNVA95dzB3FexZlV/JwN2CkvN4ZnJd163nXWHOHdkxwPQSZcT7/uB1h7U1LJb1urQasc4Pj NUiE6wks+0i9AhkcnSPdNbCqXJMzNe7e9vTU2fFtj6CnSGGGha+HRGmubAwnN7u1F0tTw7tujn+ J+njOxDXJY1DPMy0lGGI+GwZY+8vnPIeWOLFnRXSn/4ph5gnyx7KOyiXxIkxiLJTt5dJbbJJHZF pcTcJxJ0nTDzcHQ== X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Encapsulate the concept of a single set of kmem_caches that are used for the kmalloc size buckets. Redefine kmalloc_caches as an array of these buckets (for the different global cache buckets). Signed-off-by: Kees Cook --- include/linux/slab.h | 5 +++-- mm/slab_common.c | 3 +-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index ed6bee5ec2b6..8a006fac57c6 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -426,8 +426,9 @@ enum kmalloc_cache_type { NR_KMALLOC_TYPES }; -extern struct kmem_cache * -kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1]; +typedef struct kmem_cache * kmem_buckets[KMALLOC_SHIFT_HIGH + 1]; + +extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES]; /* * Define gfp bits that should not be set for KMALLOC_NORMAL. diff --git a/mm/slab_common.c b/mm/slab_common.c index 1560a1546bb1..e0b1c109bed2 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -653,8 +653,7 @@ static struct kmem_cache *__init create_kmalloc_cache(const char *name, return s; } -struct kmem_cache * -kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1] __ro_after_init = +kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES] __ro_after_init = { /* initialization for https://llvm.org/pr42570 */ }; EXPORT_SYMBOL(kmalloc_caches); From patchwork Wed Jun 19 19:33:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13704467 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 119B5156C6A; Wed, 19 Jun 2024 19:33:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718825638; cv=none; b=ZRsytdJEMmQ5KFw4MzMVzJBQ6qAZAnlvkLlPi29GOAIcHTD/UwcHDjkwH5ShS4kQwHNJIkXhoWxZHVwOEHbFnj33nqtCo7qWTl3s5TOIiwyo19cA+5Tl2ZpXh9UaNMUhwsVEWq8AGtrp0rxLX7+v0x0LMIje9cSghEckQ+44K00= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718825638; c=relaxed/simple; bh=NjQz1ofxl1LDT7gykHG8dwBhCnofJ5FfmI7SBhvteU8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=KEjBQ1LHz+1wPO1M4+QOSk4D8s9KxCnUWrfeWsCl4mihXUlvUzGwKzSBiNiehd1c7G1X1MBojZnV7/CGDoRMMj95kvZBzgPXRNp+kfTReTHEMH59Ne8VPMWzcqISm0jtkxusTUcGRIJpqSxOTfuX1nxPlcTnsT/zgBvq+LMtDYU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=D7Yv9EWF; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="D7Yv9EWF" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9D6DEC2BBFC; Wed, 19 Jun 2024 19:33:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718825637; bh=NjQz1ofxl1LDT7gykHG8dwBhCnofJ5FfmI7SBhvteU8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=D7Yv9EWFE5EMKDGgobMPhyAUV9x3AHFCr/pIIZUu98B3OuPUZA8r3svdbB4AFpa5H MxcR2veMw4/vuH1WAWRajznqvscJehUz+cZzYAt+UoaS+ER1GKSl+zQlZz39R8H4nP Ur30YYJyb3nI0TPHIoqcbIf7ywAjK3Sg4s3Jt4Cm1eEtZYOBAN/xHMRq/IsqDcDir6 dLDTGDH14fHedHAd4qCXcU7IoKvalI09yiqYt4Co/g9Fw7bjyoixQn7tU5aaKXN1rr 8Iuvyq44D4JzWws0H/tWuIt8rAohqd6VZtGnSj8n3S9IfPbOP8J3DHjEfdxMkJO1Qz l/IGcFUt5/iZw== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , "GONG, Ruiqi" , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , jvoisin , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v5 2/6] mm/slab: Plumb kmem_buckets into __do_kmalloc_node() Date: Wed, 19 Jun 2024 12:33:50 -0700 Message-Id: <20240619193357.1333772-2-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240619192131.do.115-kees@kernel.org> References: <20240619192131.do.115-kees@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=9065; i=kees@kernel.org; h=from:subject; bh=NjQz1ofxl1LDT7gykHG8dwBhCnofJ5FfmI7SBhvteU8=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmczKhW14Hz9vvOfff9HWHD+CFC5tA89EoayyE9 eX0d7oLv5iJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZnMyoQAKCRCJcvTf3G3A JtTBEACiEpfE7E1Acag1iHwH7KdlseT3HcROajzW5EkeJHpi5+yOPr+lqzOJReVLbn3lC6lfE79 JP81uJK4J3AYRlSy13q1S9JyM0CdH3JzuJrMEZb1CE50OKe9Pn4rwEm23ccHUO4nRAPtihpYBTa Xo1arWYQIEOFn59OQ71Etz23tvsOD2QroL2Anq7VfcV8rOykPLRZYIXLVbDuOadjr72kH9PVkbP OLLkfTpboMqlc8PEXgG3C2zjIhFbYXWWwjXJHZHX/HXCrJFU/So9qi2HdIxIRL3N+BGXKrzW7RM eLR0+XcuHfoXrwsrVt6ofbAXFYtTQEg1a512936/8GAH/36lnwIYSmhnnPNuzH4UrYQ4eDt4dBh THVLE7v7FHg5Z/xoVBrzSbDMPU7pHvUXpkIZVbeGjwhZD0pyCe+wD858zkX6lkOHFmENqVA82Ob FyCZ2SLjzULRC77CTNjMaMixE10e81dBFUsjzVdVZ+TzrYsZpq/3XLor7SXdm7L1d6Zewp4RqvQ bf/aMUAhyxSq9OQQSKkSzHet2KUeldXaIv1Lojont8kdtStLjRJb73wd9/rAhHPsqE1glIj2wSP jlt5ATx6uemqSzJrDFkEQEgi6L6deG36lS1q0i2NaQA167U0HtgliVWczwNV65TXwUZkKMp51m0 bD5LifceuU+wxNQ== X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Introduce CONFIG_SLAB_BUCKETS which provides the infrastructure to support separated kmalloc buckets (in the following kmem_buckets_create() patches and future codetag-based separation). Since this will provide a mitigation for a very common case of exploits, enable it by default. To be able to choose which buckets to allocate from, make the buckets available to the internal kmalloc interfaces by adding them as the first argument, rather than depending on the buckets being chosen from the fixed set of global buckets. Where the bucket is not available, pass NULL, which means "use the default system kmalloc bucket set" (the prior existing behavior), as implemented in kmalloc_slab(). To avoid adding the extra argument when !CONFIG_SLAB_BUCKETS, only the top-level macros and static inlines use the buckets argument (where they are stripped out and compiled out respectively). The actual extern functions can then been built without the argument, and the internals fall back to the global kmalloc buckets unconditionally. Co-developed-by: Vlastimil Babka Signed-off-by: Vlastimil Babka Signed-off-by: Kees Cook --- include/linux/slab.h | 27 ++++++++++++++++++++++----- mm/Kconfig | 16 ++++++++++++++++ mm/slab.h | 6 ++++-- mm/slab_common.c | 2 +- mm/slub.c | 20 ++++++++++---------- 5 files changed, 53 insertions(+), 18 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 8a006fac57c6..708bde6039f0 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -570,6 +570,21 @@ void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t flags, int node) __assume_slab_alignment __malloc; #define kmem_cache_alloc_node(...) alloc_hooks(kmem_cache_alloc_node_noprof(__VA_ARGS__)) +/* + * These macros allow declaring a kmem_buckets * parameter alongside size, which + * can be compiled out with CONFIG_SLAB_BUCKETS=n so that a large number of call + * sites don't have to pass NULL. + */ +#ifdef CONFIG_SLAB_BUCKETS +#define DECL_BUCKET_PARAMS(_size, _b) size_t (_size), kmem_buckets *(_b) +#define PASS_BUCKET_PARAMS(_size, _b) (_size), (_b) +#define PASS_BUCKET_PARAM(_b) (_b) +#else +#define DECL_BUCKET_PARAMS(_size, _b) size_t (_size) +#define PASS_BUCKET_PARAMS(_size, _b) (_size) +#define PASS_BUCKET_PARAM(_b) NULL +#endif + /* * The following functions are not to be used directly and are intended only * for internal use from kmalloc() and kmalloc_node() @@ -579,7 +594,7 @@ void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t flags, void *__kmalloc_noprof(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_size(1); -void *__kmalloc_node_noprof(size_t size, gfp_t flags, int node) +void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) __assume_kmalloc_alignment __alloc_size(1); void *__kmalloc_cache_noprof(struct kmem_cache *s, gfp_t flags, size_t size) @@ -679,7 +694,7 @@ static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gf kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], flags, node, size); } - return __kmalloc_node_noprof(size, flags, node); + return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node); } #define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__)) @@ -730,8 +745,10 @@ static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(voi */ #define kcalloc(n, size, flags) kmalloc_array(n, size, (flags) | __GFP_ZERO) -void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, int node, - unsigned long caller) __alloc_size(1); +void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node, + unsigned long caller) __alloc_size(1); +#define kmalloc_node_track_caller_noprof(size, flags, node, caller) \ + __kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node, caller) #define kmalloc_node_track_caller(...) \ alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_)) @@ -757,7 +774,7 @@ static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_ return NULL; if (__builtin_constant_p(n) && __builtin_constant_p(size)) return kmalloc_node_noprof(bytes, flags, node); - return __kmalloc_node_noprof(bytes, flags, node); + return __kmalloc_node_noprof(PASS_BUCKET_PARAMS(bytes, NULL), flags, node); } #define kmalloc_array_node(...) alloc_hooks(kmalloc_array_node_noprof(__VA_ARGS__)) diff --git a/mm/Kconfig b/mm/Kconfig index b4cb45255a54..20bb71e241c3 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -273,6 +273,22 @@ config SLAB_FREELIST_HARDENED sacrifices to harden the kernel slab allocator against common freelist exploit methods. +config SLAB_BUCKETS + bool "Support allocation from separate kmalloc buckets" + depends on !SLUB_TINY + help + Kernel heap attacks frequently depend on being able to create + specifically-sized allocations with user-controlled contents + that will be allocated into the same kmalloc bucket as a + target object. To avoid sharing these allocation buckets, + provide an explicitly separated set of buckets to be used for + user-controlled allocations. This may very slightly increase + memory fragmentation, though in practice it's only a handful + of extra pages since the bulk of user-controlled allocations + are relatively long-lived. + + If unsure, say Y. + config SLUB_STATS default n bool "Enable performance statistics" diff --git a/mm/slab.h b/mm/slab.h index b16e63191578..d5e8034af9d5 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -403,16 +403,18 @@ static inline unsigned int size_index_elem(unsigned int bytes) * KMALLOC_MAX_CACHE_SIZE and the caller must check that. */ static inline struct kmem_cache * -kmalloc_slab(size_t size, gfp_t flags, unsigned long caller) +kmalloc_slab(size_t size, kmem_buckets *b, gfp_t flags, unsigned long caller) { unsigned int index; + if (!b) + b = &kmalloc_caches[kmalloc_type(flags, caller)]; if (size <= 192) index = kmalloc_size_index[size_index_elem(size)]; else index = fls(size - 1); - return kmalloc_caches[kmalloc_type(flags, caller)][index]; + return (*b)[index]; } gfp_t kmalloc_fix_flags(gfp_t flags); diff --git a/mm/slab_common.c b/mm/slab_common.c index e0b1c109bed2..9b0f2ef951f1 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -702,7 +702,7 @@ size_t kmalloc_size_roundup(size_t size) * The flags don't matter since size_index is common to all. * Neither does the caller for just getting ->object_size. */ - return kmalloc_slab(size, GFP_KERNEL, 0)->object_size; + return kmalloc_slab(size, NULL, GFP_KERNEL, 0)->object_size; } /* Above the smaller buckets, size is a multiple of page size. */ diff --git a/mm/slub.c b/mm/slub.c index 3d19a0ee411f..80f0a51242d1 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4117,7 +4117,7 @@ void *__kmalloc_large_node_noprof(size_t size, gfp_t flags, int node) EXPORT_SYMBOL(__kmalloc_large_node_noprof); static __always_inline -void *__do_kmalloc_node(size_t size, gfp_t flags, int node, +void *__do_kmalloc_node(size_t size, kmem_buckets *b, gfp_t flags, int node, unsigned long caller) { struct kmem_cache *s; @@ -4133,32 +4133,32 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, if (unlikely(!size)) return ZERO_SIZE_PTR; - s = kmalloc_slab(size, flags, caller); + s = kmalloc_slab(size, b, flags, caller); ret = slab_alloc_node(s, NULL, flags, node, caller, size); ret = kasan_kmalloc(s, ret, size, flags); trace_kmalloc(caller, ret, size, s->size, flags, node); return ret; } - -void *__kmalloc_node_noprof(size_t size, gfp_t flags, int node) +void *__kmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) { - return __do_kmalloc_node(size, flags, node, _RET_IP_); + return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, _RET_IP_); } EXPORT_SYMBOL(__kmalloc_node_noprof); void *__kmalloc_noprof(size_t size, gfp_t flags) { - return __do_kmalloc_node(size, flags, NUMA_NO_NODE, _RET_IP_); + return __do_kmalloc_node(size, NULL, flags, NUMA_NO_NODE, _RET_IP_); } EXPORT_SYMBOL(__kmalloc_noprof); -void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, - int node, unsigned long caller) +void *__kmalloc_node_track_caller_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, + int node, unsigned long caller) { - return __do_kmalloc_node(size, flags, node, caller); + return __do_kmalloc_node(size, PASS_BUCKET_PARAM(b), flags, node, caller); + } -EXPORT_SYMBOL(kmalloc_node_track_caller_noprof); +EXPORT_SYMBOL(__kmalloc_node_track_caller_noprof); void *__kmalloc_cache_noprof(struct kmem_cache *s, gfp_t gfpflags, size_t size) { From patchwork Wed Jun 19 19:33:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13704468 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 185F21586C9; Wed, 19 Jun 2024 19:33:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718825638; cv=none; b=trAmQiAcAuuodvDB40yuAy2f+bbrNwQ0leEXa3jgsGyD5qZWUYeH3GUiH4I1GZm8UkT08UifmVAc2JQTaN8z9w38jGNGrpOE+SqIrcBu3iPOYPFuUvcnCzi1+5L/jbaz/+UgbXeHHqxv6vIFQS3doxjkRjvWR3Dh/rTIm6/CLXk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718825638; c=relaxed/simple; bh=xxet30UEfeVq/dWdKRNFzCWZboWOxErsseLhti/TXwg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=LGMYIxg270vhxIPmDJPG/tLt7M9gnxS9y1SsNKXLsBah6HQ9covjQi+ye/S+d2rYa7zrAe/gPWkDJZJMKaHsMiSDSt1JgHlET8HNUN9xcSyZGlQ7NsdnwqppODIHzupPsw+bH7F4rTJfvF2ZgBKJJS4DZylS5HZmgmDcnXG4IeI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=sQhXPL1C; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="sQhXPL1C" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B91B1C4AF0B; Wed, 19 Jun 2024 19:33:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718825637; bh=xxet30UEfeVq/dWdKRNFzCWZboWOxErsseLhti/TXwg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sQhXPL1C3OHH54U1cVXlN7BmiNba91nmMpHH/d0Mozd9pLGODxYsU7/g/xaYKeK74 uso2CWUwa29eAbBPUx1NxMjoMXhj69h8q+Zp/a+uOzTPKuQhzXxbPmbRZy8MRamEGm Eq5OqHkAcxj6qcRj+HbzAxcJ3YK6x+2cdgEc8OtPdakCmbRx7cc2M5BRuSgQGJFXvI +erQKYRCHeYgMRlL070hUjRJ/cXNQmSS+bkGqwddmeGyhgNESjn27Oez2GGjmPrsfT v1HHi19jFtlWwsrVClErAHoUMqrOOjoxKqxj//KZ1AUrI+BUpNCZAbbiMwkmVGOULG Th+do/dczDw0g== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , "GONG, Ruiqi" , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , jvoisin , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v5 3/6] mm/slab: Introduce kvmalloc_buckets_node() that can take kmem_buckets argument Date: Wed, 19 Jun 2024 12:33:51 -0700 Message-Id: <20240619193357.1333772-3-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240619192131.do.115-kees@kernel.org> References: <20240619192131.do.115-kees@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2960; i=kees@kernel.org; h=from:subject; bh=xxet30UEfeVq/dWdKRNFzCWZboWOxErsseLhti/TXwg=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmczKhrAHw0U+CEnyXA/4XClLK6QAyqHm8wswfo OqdafG1Bj+JAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZnMyoQAKCRCJcvTf3G3A JpVcD/99ig6OdPl7DaH+0pAyKM3SG549CREJbtJi4lqSfP7oUzPpDpw8cC8JaJM3I8Gdu+iTiYd KOcNq3k5dY7YRMKVw2/U/+T4xUbQ5Z60fgOHRiBp0PHwY2C0TqjCVz4MfH23F9rTtFTteXh+jba gUikGWelLqlSjwS57roBtFMNzGUG3o+1/PXg6bar8KmwUY1yb/PQmLTy901tuQ8FD3qgBJ+5PWV PMi0V2v1xjAn5dLtUGtl6vX2+CB5sVP4LiSls41wOtBLOGDMzo/OV84jqpO5uZsAzaS7dcEX9NQ 1MU1fY3W2R8Ne9sPruomOQ7CHvTiRjRG3w1GQPMGqJb6iyHSwPe858NullL22+vxFriTW9+kDSN QmNul/Briq2wF6Xh/UYxhuMNLRbIcK6MVtJZebpKyfw7D5aXV2/In1sY/pUmT0+MKrb4CSBCgAm 4Qbky6Wd7FowZT8jYT2c1gI+SrjZ46v3qqjw0v9dmtfheYHutq5I9VYvtTBuJlA1cnlECnD9tTk l+W4f58PKW0aibacXzUL8jzPF9YBzqqJTb2urpfvAT57xT4bZ7K7ihli33K0/D01oCCYM/1bmyQ XbZKRQRIyRpMaOdFRLiZdAjDpphSYA9yPZgKBmyeXHZRsg/wFujguqKPECQKlqdl0uR0CG5N9Uh wXG82K7TQBZbhbQ== X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Plumb kmem_buckets arguments through kvmalloc_node_noprof() so it is possible to provide an API to perform kvmalloc-style allocations with a particular set of buckets. Introduce kvmalloc_buckets_node() that takes a kmem_buckets argument. Signed-off-by: Kees Cook --- include/linux/slab.h | 4 +++- mm/util.c | 9 +++++---- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 708bde6039f0..8d0800c7579a 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -798,7 +798,9 @@ static inline __alloc_size(1) void *kzalloc_noprof(size_t size, gfp_t flags) #define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__)) #define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node) -extern void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) __alloc_size(1); +void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) __alloc_size(1); +#define kvmalloc_node_noprof(size, flags, node) \ + __kvmalloc_node_noprof(PASS_BUCKET_PARAMS(size, NULL), flags, node) #define kvmalloc_node(...) alloc_hooks(kvmalloc_node_noprof(__VA_ARGS__)) #define kvmalloc(_size, _flags) kvmalloc_node(_size, _flags, NUMA_NO_NODE) diff --git a/mm/util.c b/mm/util.c index c9e519e6811f..28c5356b9f1c 100644 --- a/mm/util.c +++ b/mm/util.c @@ -594,9 +594,10 @@ unsigned long vm_mmap(struct file *file, unsigned long addr, EXPORT_SYMBOL(vm_mmap); /** - * kvmalloc_node - attempt to allocate physically contiguous memory, but upon + * __kvmalloc_node - attempt to allocate physically contiguous memory, but upon * failure, fall back to non-contiguous (vmalloc) allocation. * @size: size of the request. + * @b: which set of kmalloc buckets to allocate from. * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL. * @node: numa node to allocate from * @@ -609,7 +610,7 @@ EXPORT_SYMBOL(vm_mmap); * * Return: pointer to the allocated memory of %NULL in case of failure */ -void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) +void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) { gfp_t kmalloc_flags = flags; void *ret; @@ -631,7 +632,7 @@ void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) kmalloc_flags &= ~__GFP_NOFAIL; } - ret = kmalloc_node_noprof(size, kmalloc_flags, node); + ret = __kmalloc_node_noprof(PASS_BUCKET_PARAMS(size, b), kmalloc_flags, node); /* * It doesn't really make sense to fallback to vmalloc for sub page @@ -660,7 +661,7 @@ void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(kvmalloc_node_noprof); +EXPORT_SYMBOL(__kvmalloc_node_noprof); /** * kvfree() - Free memory. From patchwork Wed Jun 19 19:33:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13704469 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 18632158A2D; Wed, 19 Jun 2024 19:33:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718825638; cv=none; b=qPo4qJ8DOmFxr31/uen8adoTut1xLB6DABllB+lpZjkM9CLkc6gvudbVSI9w9rcAyL39sBNzSNowDKGYmdKHREeZTEHwOyk8RwntlDd4/JUDId8bNEeJfrAvyobLzo7dsx0Xa/L8KDBX+F22EkjgpIHqz1UHaIzsOuZL1fCRVac= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718825638; c=relaxed/simple; bh=/RmCJ4F/4ng/Khq7qR9N0HAZnD/jFStWnmxtCKBK+GE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=TGnCdRexNyCDHSp5a/yvMKd9PW5VxLecy4GDIO0B/5rxh3fBrhnedmStnwwYzQXl9NOxE8jGWAxMgT5GrIbwd7oMqs/WL7+KJk7h0mU2YERZtezJFwW3oI5uFNbcFwlAs5kfIhRP+YSXOVFhb2w6xeDSQHP7HzaRtukSdlCoChg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Yg3uS92q; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Yg3uS92q" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B4E7DC4AF09; Wed, 19 Jun 2024 19:33:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718825637; bh=/RmCJ4F/4ng/Khq7qR9N0HAZnD/jFStWnmxtCKBK+GE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Yg3uS92qwownEjYk+KrHqmmBxErM9MDG9GWHoFjllUlVSitrJzN/cCkCIxz9m8kG9 kKoYDXC55Gf2FwYsZgznS1yoHOWjYGoi+I0Z6o0Gm5TXB/JtXNVfbtLFKKjgll+gOR CVuFaIrDPLPbOhR1s/D38SWARmQueikUJkNoAqYKhvnRvsXp23H82MaQHLHr5bjLze 9djumIEXDuJJaKmToURyjFbfB9pEA6aj9VayWX/uc1mN3yBbyG5UQB7bA5tqX7afHH qvlaU7v3WU6YTKxQi8wrAE3TOP/f0TTQhbZb/ao4H72plbBlMPtDyeYKOxizgoQpN+ infour+aBOrTA== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , "GONG, Ruiqi" , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , jvoisin , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v5 4/6] mm/slab: Introduce kmem_buckets_create() and family Date: Wed, 19 Jun 2024 12:33:52 -0700 Message-Id: <20240619193357.1333772-4-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240619192131.do.115-kees@kernel.org> References: <20240619192131.do.115-kees@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6964; i=kees@kernel.org; h=from:subject; bh=/RmCJ4F/4ng/Khq7qR9N0HAZnD/jFStWnmxtCKBK+GE=; b=owEBbAKT/ZANAwAKAYly9N/cbcAmAcsmYgBmczKh21zJZgyIFpVjar+3GzcqV57oOAwiaPyFf 9lg6kUOpTKJAjIEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZnMyoQAKCRCJcvTf3G3A JnTnD/i2CjxP6n3FrmWS5Y2ZF/qTuu2v5aeKrAb1+69yKucXGvlFE1sS0j8g/4CcTgYD2Knvk0Z HJqxN0EMNzWhABAciSgOkzbDhOQyDLUvP+JcxuHiDxOp8K73MMFTWwXPiVjTcCUoqXiMGq3JNzx qesEPnvQM+Tg2kCxyZKlrcdSgRSpNAyOzt/YguC+V5/5tBzhRwfv/4wT8D50QkT/RgzPLjEM/cF LLXlmYAH4ftrzzTwHVPFydtRp3z/LXpUVh5wphBnpJ5tYkkBcEY8WNG5VJCoZbd2HIp58ex6Ypg mlp4bEfdmLCY0m8uqpdycuoTdXpRplIW30Y+VT8NG6JUz/Pmo3TPkDTlyIkFXRmgCwAY7yJ9lcO t5PXQjfhG3j4+JzW0kkDAi8PCZSuB5K+mMSpHGg/KUxut0vaVf/FUqZLaKFbtDtDt3j5Hi2Zxsc McIfFUN7qSQw1oeXD+WFrbTDLVTt7Kvs/PYdqdnswzMcTHKQVyl7k2mwUtibR2uTSs0F1GW3zgb 5kFMEd+FNidxfUztUA449ujJ8F1IbS2inC6b/xfGpEt0PqtKBDgpX2GcDhveoJO/0N4g5S7s5jF 08LkkH5XKfseBz0goioeJ+0E2Lercac875NHpziSFZtCVDzjqgDLDH5gQKsxmNXSXcJ5xqI33VM LRyiGlHiaNTFf X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Dedicated caches are available for fixed size allocations via kmem_cache_alloc(), but for dynamically sized allocations there is only the global kmalloc API's set of buckets available. This means it isn't possible to separate specific sets of dynamically sized allocations into a separate collection of caches. This leads to a use-after-free exploitation weakness in the Linux kernel since many heap memory spraying/grooming attacks depend on using userspace-controllable dynamically sized allocations to collide with fixed size allocations that end up in same cache. While CONFIG_RANDOM_KMALLOC_CACHES provides a probabilistic defense against these kinds of "type confusion" attacks, including for fixed same-size heap objects, we can create a complementary deterministic defense for dynamically sized allocations that are directly user controlled. Addressing these cases is limited in scope, so isolating these kinds of interfaces will not become an unbounded game of whack-a-mole. For example, many pass through memdup_user(), making isolation there very effective. In order to isolate user-controllable dynamically-sized allocations from the common system kmalloc allocations, introduce kmem_buckets_create(), which behaves like kmem_cache_create(). Introduce kmem_buckets_alloc(), which behaves like kmem_cache_alloc(). Introduce kmem_buckets_alloc_track_caller() for where caller tracking is needed. Introduce kmem_buckets_valloc() for cases where vmalloc fallback is needed. This can also be used in the future to extend allocation profiling's use of code tagging to implement per-caller allocation cache isolation[1] even for dynamic allocations. Memory allocation pinning[2] is still needed to plug the Use-After-Free cross-allocator weakness, but that is an existing and separate issue which is complementary to this improvement. Development continues for that feature via the SLAB_VIRTUAL[3] series (which could also provide guard pages -- another complementary improvement). Link: https://lore.kernel.org/lkml/202402211449.401382D2AF@keescook [1] Link: https://googleprojectzero.blogspot.com/2021/10/how-simple-linux-kernel-memory.html [2] Link: https://lore.kernel.org/lkml/20230915105933.495735-1-matteorizzo@google.com/ [3] Signed-off-by: Kees Cook --- include/linux/slab.h | 13 ++++++++ mm/slab_common.c | 78 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index 8d0800c7579a..3698b15b6138 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -549,6 +549,11 @@ void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru, void kmem_cache_free(struct kmem_cache *s, void *objp); +kmem_buckets *kmem_buckets_create(const char *name, unsigned int align, + slab_flags_t flags, + unsigned int useroffset, unsigned int usersize, + void (*ctor)(void *)); + /* * Bulk allocation and freeing operations. These are accelerated in an * allocator specific way to avoid taking locks repeatedly or building @@ -681,6 +686,12 @@ static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t f } #define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__)) +#define kmem_buckets_alloc(_b, _size, _flags) \ + alloc_hooks(__kmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE)) + +#define kmem_buckets_alloc_track_caller(_b, _size, _flags) \ + alloc_hooks(__kmalloc_node_track_caller_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE, _RET_IP_)) + static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gfp_t flags, int node) { if (__builtin_constant_p(size) && size) { @@ -808,6 +819,8 @@ void *__kvmalloc_node_noprof(DECL_BUCKET_PARAMS(size, b), gfp_t flags, int node) #define kvzalloc(_size, _flags) kvmalloc(_size, (_flags)|__GFP_ZERO) #define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, (_flags)|__GFP_ZERO, _node) +#define kmem_buckets_valloc(_b, _size, _flags) \ + alloc_hooks(__kvmalloc_node_noprof(PASS_BUCKET_PARAMS(_size, _b), _flags, NUMA_NO_NODE)) static inline __alloc_size(1, 2) void * kvmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node) diff --git a/mm/slab_common.c b/mm/slab_common.c index 9b0f2ef951f1..453bc4ec8b57 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -392,6 +392,80 @@ kmem_cache_create(const char *name, unsigned int size, unsigned int align, } EXPORT_SYMBOL(kmem_cache_create); +static struct kmem_cache *kmem_buckets_cache __ro_after_init; + +kmem_buckets *kmem_buckets_create(const char *name, unsigned int align, + slab_flags_t flags, + unsigned int useroffset, + unsigned int usersize, + void (*ctor)(void *)) +{ + kmem_buckets *b; + int idx; + + /* + * When the separate buckets API is not built in, just return + * a non-NULL value for the kmem_buckets pointer, which will be + * unused when performing allocations. + */ + if (!IS_ENABLED(CONFIG_SLAB_BUCKETS)) + return ZERO_SIZE_PTR; + + if (WARN_ON(!kmem_buckets_cache)) + return NULL; + + b = kmem_cache_alloc(kmem_buckets_cache, GFP_KERNEL|__GFP_ZERO); + if (WARN_ON(!b)) + return NULL; + + flags |= SLAB_NO_MERGE; + + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { + char *short_size, *cache_name; + unsigned int cache_useroffset, cache_usersize; + unsigned int size; + + if (!kmalloc_caches[KMALLOC_NORMAL][idx]) + continue; + + size = kmalloc_caches[KMALLOC_NORMAL][idx]->object_size; + if (!size) + continue; + + short_size = strchr(kmalloc_caches[KMALLOC_NORMAL][idx]->name, '-'); + if (WARN_ON(!short_size)) + goto fail; + + cache_name = kasprintf(GFP_KERNEL, "%s-%s", name, short_size + 1); + if (WARN_ON(!cache_name)) + goto fail; + + if (useroffset >= size) { + cache_useroffset = 0; + cache_usersize = 0; + } else { + cache_useroffset = useroffset; + cache_usersize = min(size - cache_useroffset, usersize); + } + (*b)[idx] = kmem_cache_create_usercopy(cache_name, size, + align, flags, cache_useroffset, + cache_usersize, ctor); + kfree(cache_name); + if (WARN_ON(!(*b)[idx])) + goto fail; + } + + return b; + +fail: + for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) + kmem_cache_destroy((*b)[idx]); + kfree(b); + + return NULL; +} +EXPORT_SYMBOL(kmem_buckets_create); + #ifdef SLAB_SUPPORTS_SYSFS /* * For a given kmem_cache, kmem_cache_destroy() should only be called @@ -931,6 +1005,10 @@ void __init create_kmalloc_caches(void) /* Kmalloc array is now usable */ slab_state = UP; + + kmem_buckets_cache = kmem_cache_create("kmalloc_buckets", + sizeof(kmem_buckets), + 0, SLAB_NO_MERGE, NULL); } /** From patchwork Wed Jun 19 19:33:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13704471 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE0F115990E; Wed, 19 Jun 2024 19:33:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718825638; cv=none; b=kQ4zEqOWPfQTVPBp7empGKinmi9L3CZC0+npQyRV86hk+LwZC5UwjAKHEuw81GKy4IjV+kUdwiP0u2nUpquXZ1trts/4XwvK94hqqvMagpbGrTZf0AlvYJ9ZWrujugQYphf+uMMnVqNSDVBFyQLoki2TJlfE0TkedZzsZZFC+/Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718825638; c=relaxed/simple; bh=l6yqTeNyOBrP23PYN3nun8WZu+CjjUQ/unbct0M2mXI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=kQkLGMQGKiTLcQQTD6117CjYyghrcyYMRmbb4xGXEyy3GRXeW+TQypzpjP/c9oPzZs3RsP3wIQrSlzTMofzN2GwMl5N6jd3BywnMcE8czgu0hOAAF8jLsWE2Iq2FLDLR3FeXYnYMKvSEEzv0b2GO8iPSRFmfiDJ1BL+J8Qos7t4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ab6SRa4A; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ab6SRa4A" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 40ADBC2BBFC; Wed, 19 Jun 2024 19:33:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718825638; bh=l6yqTeNyOBrP23PYN3nun8WZu+CjjUQ/unbct0M2mXI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ab6SRa4A4O3te1DOOmVV9LKCuiJpDJdfI2FJQBq7Kv17C7DHW9zsNHeb7bueSUyci Lw8BamiL2ZhJaGhfx8G68H5MVVl5rP0PoFMr3MXdBanf8G0+uamQxXGnta/B3S9dDw hxeMRHH2RbBfswtUdPsiANqaMFIchVgdwbI9OYciFSLtTNAUo6Y550TibRKn1EDO+N jFMP5K79mMw1Gt13ZqhVo7pLGtcBu3PyeZYSN0KNkStMGjUKsvFOBnXvq0ajXDw3Km hKwtApCwZSJRbY1AcU0nqYvTcz9qlLvkzi8luHa6uZNhZmCKnFcJPfiqqXBd6ZjO8p iynQrnp0MQn5A== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , "GONG, Ruiqi" , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , jvoisin , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v5 5/6] ipc, msg: Use dedicated slab buckets for alloc_msg() Date: Wed, 19 Jun 2024 12:33:53 -0700 Message-Id: <20240619193357.1333772-5-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240619192131.do.115-kees@kernel.org> References: <20240619192131.do.115-kees@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1957; i=kees@kernel.org; h=from:subject; bh=l6yqTeNyOBrP23PYN3nun8WZu+CjjUQ/unbct0M2mXI=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmczKhrdqjrPyL3pEgr89Wc1pCFROhedjcMhzfD 1ktUaK7VaKJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZnMyoQAKCRCJcvTf3G3A JiQ3D/9sneRcmARSURll7MrwE/KHnzRzFcQfEmQHROWakjylRhL3Z0qgUV2xaUp+i+YCu8W3lfd vjdODHDdoXjT/ExemxxivY16a/qE6rBCZGnPKOVxK3ieSuA9ZzetcC5zX+W+++FilPd31YyrXlJ mesSod6rWwnUA5JSzWJM3SRCSq9v826dVuBa/5LuO3DEBHKEJXdmuKjh0omg1zf2f5vPlkq6/dg QfbRJ89z7CY8mHavgoYcCnepTFBhqheIookn2IiB/Uml9NSas9I2zkAKvhknI/QSKeR1uJ+JKe+ Xvyod/C1OCjXxY2Mh9vpESjonkiOcjBwbvmUVemIO6/HtIRKr639BOBbvBgpjz3z70L4U4M7idu Y23rYQDQxyjeSsys0C0w3USaByxJj+rz7bwMoX+DwBhJdCchT/xvk4TlXJ9/j7Cd7FRjKYVvgrq swSCdTWUMlIcP+hMvlV7t0lUti3RUgOtnuLdtg08vETEz0mVTjgKy/fhn8z+N3MUPBwC2WEPqSq fE7nx3DOP8Eeo1yl4iIITBECv/k7Cppw1SaBEDd7IInEJ4sM+V5kIZU1+1LCQQLrffItFFCReOT MF6cMh4pmxuG4jeXxARn7mBikJcVkRfWhv2hbIQE/LCdwAEJRwNg5tKV6icRA6I5m7Oy2Gnm6Vp OMux8ZOHO1evhkg== X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 The msg subsystem is a common target for exploiting[1][2][3][4][5][6][7] use-after-free type confusion flaws in the kernel for both read and write primitives. Avoid having a user-controlled dynamically-size allocation share the global kmalloc cache by using a separate set of kmalloc buckets via the kmem_buckets API. Link: https://blog.hacktivesecurity.com/index.php/2022/06/13/linux-kernel-exploit-development-1day-case-study/ [1] Link: https://hardenedvault.net/blog/2022-11-13-msg_msg-recon-mitigation-ved/ [2] Link: https://www.willsroot.io/2021/08/corctf-2021-fire-of-salvation-writeup.html [3] Link: https://a13xp0p0v.github.io/2021/02/09/CVE-2021-26708.html [4] Link: https://google.github.io/security-research/pocs/linux/cve-2021-22555/writeup.html [5] Link: https://zplin.me/papers/ELOISE.pdf [6] Link: https://syst3mfailure.io/wall-of-perdition/ [7] Signed-off-by: Kees Cook --- ipc/msgutil.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/ipc/msgutil.c b/ipc/msgutil.c index d0a0e877cadd..f392f30a057a 100644 --- a/ipc/msgutil.c +++ b/ipc/msgutil.c @@ -42,6 +42,17 @@ struct msg_msgseg { #define DATALEN_MSG ((size_t)PAGE_SIZE-sizeof(struct msg_msg)) #define DATALEN_SEG ((size_t)PAGE_SIZE-sizeof(struct msg_msgseg)) +static kmem_buckets *msg_buckets __ro_after_init; + +static int __init init_msg_buckets(void) +{ + msg_buckets = kmem_buckets_create("msg_msg", 0, SLAB_ACCOUNT, + sizeof(struct msg_msg), + DATALEN_MSG, NULL); + + return 0; +} +subsys_initcall(init_msg_buckets); static struct msg_msg *alloc_msg(size_t len) { @@ -50,7 +61,7 @@ static struct msg_msg *alloc_msg(size_t len) size_t alen; alen = min(len, DATALEN_MSG); - msg = kmalloc(sizeof(*msg) + alen, GFP_KERNEL_ACCOUNT); + msg = kmem_buckets_alloc(msg_buckets, sizeof(*msg) + alen, GFP_KERNEL); if (msg == NULL) return NULL; From patchwork Wed Jun 19 19:33:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13704470 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 858C015921D; Wed, 19 Jun 2024 19:33:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718825638; cv=none; b=k57D+O9B8kun7RBMxFIBOWM00CsT837xceeZ3ALtWULwCeXF55OQWFYZ0lbIFadcHzGNuPTj2h9lOBYy6Hmux4WYPBe2ooKrm9wdLZn1Br4Ai85X27QTROuEnN78DCs+rfFnwjmYzTim9jsnMQUqy1rl1OPshs015c1t75e6M/o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718825638; c=relaxed/simple; bh=LrnolFNHWclGZ9p3hY/QGRuJsr6+6G+8GF/8xJCY8DU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=sYG+zqaNvNblG7g8GGyaUT99qmzktI1gyGulXz17l8oSZ8oe3QW6i37E0F6zUz9XFwgfhwfkqdhBWhxVmzN8JVFP4Fx+r/scDYvHlA9kH+lyNtDg0nM8NsySbZm/GTEAec5x5ECqKmOOJpF9nmOxxxQ/d8YX1qjKKiwe/JR/D+Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Pc6m2Jyd; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Pc6m2Jyd" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2C6FAC4AF0A; Wed, 19 Jun 2024 19:33:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718825638; bh=LrnolFNHWclGZ9p3hY/QGRuJsr6+6G+8GF/8xJCY8DU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Pc6m2JydDigHX1wPdvV7BSO6pjR4OiuefHdkKa1fdpttENlQLKGuUfbGFFSu2l4Lm lQD5iAks3gvBoAIfaXZXsbn9df6WJYs3vBKiEbv3bJ4PaaLahoIa4QSNZ1SYfuLbBH EC/g9SUx1BXID0KIUyOFIJTu1RoIyozImXz+TkjeVy0SusKmCLRASggim94z6rYRCM tO4C6jMpK8ZWUlJKguJ9sfCfZZJc1MVWVRiBXSV+GO5TXMdXWOzrdcFw/b8i8cgg7U lBK0Q0+bcSuO5G7slNZJMYU9DBvzN/fxnRHAqOvcn3r1IAoqVZP9OzadrVNBk/NTEU Fodm/pnVMavLA== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , "GONG, Ruiqi" , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , jvoisin , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v5 6/6] mm/util: Use dedicated slab buckets for memdup_user() Date: Wed, 19 Jun 2024 12:33:54 -0700 Message-Id: <20240619193357.1333772-6-kees@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240619192131.do.115-kees@kernel.org> References: <20240619192131.do.115-kees@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2799; i=kees@kernel.org; h=from:subject; bh=LrnolFNHWclGZ9p3hY/QGRuJsr6+6G+8GF/8xJCY8DU=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmczKhEAWYa/Ji7QXOW1GAYZ9rnB+j1DB6dbkqF TcWKMWXK8GJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZnMyoQAKCRCJcvTf3G3A Jna9D/9jn8FgmhtJGIsF1k6FXqxnOb5H/Edjil/UGnza0ThNC5KYFYysA7NmGEt75u5Ul8yTo/m mPSLD5BsKR+jO9ra+Jb1otzerSK7RPkiM6Imdxr4fUVqJRDIyZfow5kmiqU9kP3XohZkW6L9tCf Gg3Gf3IyVqjtN6r7kMQfjfEwb0asp7l19n+RBG+Y+rCjuwW9f0zVuora95LsN+puXykCTSHPd8l triyydYzJHXgpSTHex+MpUFovzc22Spf+qmwDTaHqtewlJLtF0Yv2hU8zovuwGlpfr8ByOElpfK Vk+S54XYZnxOLQ67p3popKVW0MElvIx5m8o5iUKZDmwhhLvPBOzUH1bT6we+7qxY02Rjs8l7tcq 2WQdGuzFjXnYo2mAqRN7l7S7Cq1nPwRpwtPoUinjqbvdFIV/NRq9GABynRSpDqxH5qrbas9689/ gvJkHTXbL+PSnY6Qs32jdRqdMuCqHzXkPa02gx1uuVpAPC8z9C0JsTzVOz30ZPQ9b0RmeBt9pE6 lT0nq+2zyjP5Wa+1oEckM/FhVlBq0s4VsGGABVI6DWi7oWRyz/ZZfn5gV0N4qWgfnHJalyM3cFC 3O6fBUVeH3zAlDMwpz+DEypq18nKLYt6AtEq5WMurqTQR+ES3H6DKAtjc5xSYw6hrbgAMWMSOLR smdpkS6OJkKmNzw== X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Both memdup_user() and vmemdup_user() handle allocations that are regularly used for exploiting use-after-free type confusion flaws in the kernel (e.g. prctl() PR_SET_VMA_ANON_NAME[1] and setxattr[2][3][4] respectively). Since both are designed for contents coming from userspace, it allows for userspace-controlled allocation sizes. Use a dedicated set of kmalloc buckets so these allocations do not share caches with the global kmalloc buckets. After a fresh boot under Ubuntu 23.10, we can see the caches are already in active use: # grep ^memdup /proc/slabinfo memdup_user-8k 4 4 8192 4 8 : ... memdup_user-4k 8 8 4096 8 8 : ... memdup_user-2k 16 16 2048 16 8 : ... memdup_user-1k 0 0 1024 16 4 : ... memdup_user-512 0 0 512 16 2 : ... memdup_user-256 0 0 256 16 1 : ... memdup_user-128 0 0 128 32 1 : ... memdup_user-64 256 256 64 64 1 : ... memdup_user-32 512 512 32 128 1 : ... memdup_user-16 1024 1024 16 256 1 : ... memdup_user-8 2048 2048 8 512 1 : ... memdup_user-192 0 0 192 21 1 : ... memdup_user-96 168 168 96 42 1 : ... Link: https://starlabs.sg/blog/2023/07-prctl-anon_vma_name-an-amusing-heap-spray/ [1] Link: https://duasynt.com/blog/linux-kernel-heap-spray [2] Link: https://etenal.me/archives/1336 [3] Link: https://github.com/a13xp0p0v/kernel-hack-drill/blob/master/drill_exploit_uaf.c [4] Signed-off-by: Kees Cook --- mm/util.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/mm/util.c b/mm/util.c index 28c5356b9f1c..6f0fcc5f4243 100644 --- a/mm/util.c +++ b/mm/util.c @@ -198,6 +198,16 @@ char *kmemdup_nul(const char *s, size_t len, gfp_t gfp) } EXPORT_SYMBOL(kmemdup_nul); +static kmem_buckets *user_buckets __ro_after_init; + +static int __init init_user_buckets(void) +{ + user_buckets = kmem_buckets_create("memdup_user", 0, 0, 0, INT_MAX, NULL); + + return 0; +} +subsys_initcall(init_user_buckets); + /** * memdup_user - duplicate memory region from user space * @@ -211,7 +221,7 @@ void *memdup_user(const void __user *src, size_t len) { void *p; - p = kmalloc_track_caller(len, GFP_USER | __GFP_NOWARN); + p = kmem_buckets_alloc_track_caller(user_buckets, len, GFP_USER | __GFP_NOWARN); if (!p) return ERR_PTR(-ENOMEM); @@ -237,7 +247,7 @@ void *vmemdup_user(const void __user *src, size_t len) { void *p; - p = kvmalloc(len, GFP_USER); + p = kmem_buckets_valloc(user_buckets, len, GFP_USER); if (!p) return ERR_PTR(-ENOMEM);