From patchwork Wed Jun 19 19:33:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13704457 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED009C27C53 for ; Wed, 19 Jun 2024 19:34:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B5216B044F; Wed, 19 Jun 2024 15:34:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4500D6B044E; Wed, 19 Jun 2024 15:34:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2B1AB6B0450; Wed, 19 Jun 2024 15:34:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 01A1B6B019D for ; Wed, 19 Jun 2024 15:34:00 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B1A5DA2F0A for ; Wed, 19 Jun 2024 19:34:00 +0000 (UTC) X-FDA: 82248638640.20.C1BB178 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf05.hostedemail.com (Postfix) with ESMTP id 08569100010 for ; Wed, 19 Jun 2024 19:33:58 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=ILO1WzP4; spf=pass (imf05.hostedemail.com: domain of kees@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=kees@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718825634; a=rsa-sha256; cv=none; b=JywN8akB+jqph6qcUtaU1YTPRex5JmFL4wOlL39Kk6q7EBjpjkpQZMBC+qY+f4YiI5Ij+1 MWV2cx/yndxYBtKlOt2jGNfNmqQAvWyilB9wYmWRXEZDTMDQN15m6VSo+zdJ2xv1cOdckc nypzv6gU3yOTrfRHbssV36C4k+UgF2k= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=ILO1WzP4; spf=pass (imf05.hostedemail.com: domain of kees@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=kees@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718825634; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=1t1MK1g6QcDdPIHRlr37myW5AOLaY9efMFJ0W5ITSnU=; b=Ont83+jfzzc3+xOu7si3wwXOF209jsxXWVrZBdZ1Ve0HPXoAaV76AX5sOuDD6/aQ6Mso3A G9d9ifY6o/CDVxojzb3JG7LsHOHPC0TD+qco2X4ZFfeacR71ddcBzO02kp/Ehyi4d8hKPp loztpKvYy3HqMwelEWK29lGDiBGAkR8= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 05A8461F5B; Wed, 19 Jun 2024 19:33:58 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A06EDC4AF07; Wed, 19 Jun 2024 19:33:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718825637; bh=rCuC147fYmDtvDKRsay/TPMA580R1cV0CqcWtYXITNQ=; h=From:To:Cc:Subject:Date:From; b=ILO1WzP4rJTLoTLKi4PkpSpRq9c5N/PhEDFECsHrTkFl1QACtTgY+rJW9KMm4BWfx /E4qKMwezvvz+vg0/xBy8DkaLMirjNY6DuvEAfzF2+sBMQLTLA5cMSZ/W/CfJJEAbS WuiHIBRkicBOUlJ3X1Lvb3A9rWr5/XErJC7b+sD4RTGU/0uI+YAWz6sH7lE4zuAR+n WGk6zFz30g/B+bCG1RkJTnzwa7GFQVzMGWyDq8WkdWb0BjIUgCz/WqATwqEber5h5O w7ddILFyb2XPdJZeq+xblfZYvcVGdxlwsFY0CyaVcF1UEcfwLVrsaZ3o5egseJAO7k sfzRLblD7QKRA== From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , "GONG, Ruiqi" , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , jvoisin , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Thomas Graf , Herbert Xu , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org, netdev@vger.kernel.org Subject: [PATCH v5 0/6] slab: Introduce dedicated bucket allocator Date: Wed, 19 Jun 2024 12:33:48 -0700 Message-Id: <20240619192131.do.115-kees@kernel.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6065; i=kees@kernel.org; h=from:subject:message-id; bh=rCuC147fYmDtvDKRsay/TPMA580R1cV0CqcWtYXITNQ=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBmczKhRgUdrjMlqp+DXoOV1DQo8UHBJkd/m+b/9 E8XxeZuB7SJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZnMyoQAKCRCJcvTf3G3A JoY1D/4ylvVu7defFNoTvkPBIvUnSotH37+jE2Ii4yHqdGyXcw6IxVKniJ6wiQJ5Hcj5qavdAEe lewv2kVY5lA69EKPpEsuZ53pH6M0ywnmh0Qb6ojBTpj/aPKB3WyY2fYhOKicNcxWw/k1tpvhj/V aII8Z5yaqizEuEkjjghQn5yERDk9UZEUGWwcNbentM4KmRkuB9e5qJTlSb5/JJP2UrlG4VwOyzX vr+L/sMNfqC9C/PbJwAuhu17HpcPzH4lt8BrhYYYSU+rM35ObCB+GLbsvVhy//bYGMjyZPqNtuh aeQ7o6d53Bjs/mB3T2AgGaDY1Ht3zj+5w+2ABdEELpsHIL8bTt3Vqak9zs/Lcr+iMfhG+5XSZMb 4ZXDYrTuRuidL3wNp88Rg2XGfSs/9tIfsYhPpbJ8nHVHxVelXIkTTYr0vE1aQFkSMLzn9IyDIiL NSFpEnzewhfDGAytMEN8A7LvexYaHs6pvr1uk++G4d9BVL+pl2ThEKmajJHacX6X7xS7iVqhkrR regafqOdddTgSthM4B4oYCBH00MJAsZzPxBzYdIb4IROOPQmGLNhfxIYqEfSLu+ZDhDJ9okiMbZ yxa3uQCfjfYg2IL9lA71NQwl8SuVKd+f4s6s4ww+hEBCDZDngM5cDR7AtXEdyYz4ELy4tcdeDUb +mYf3 3wGGRTVv Fg== X-Developer-Key: i=kees@kernel.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Stat-Signature: gtidx8d41j4qsdur1ztrt7nf7xk9hrkq X-Rspamd-Queue-Id: 08569100010 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1718825638-716065 X-HE-Meta: U2FsdGVkX1/YSXUGJBLYdzeiLPylFdj5vtUJ38mO/aedKTryGlJhHU0n/UWox8xSCWD8NwkOllJ9sCos5+o4iaLArmRAUu8kG/1jqUoG+E5z0VTI0odT1II633rX1r6dnpW1pdlv5MJI0ZjElH+6xzXTPGU1P+aMEADt81alB76tYu8nVgMhKOFxOvGGv6bIOJmkVYb/TyjdxY2lsx/B0sJGvMUMYZ0FoebqtJvZ6XjO2NX6eQKYOkDFoDkLAmAe/guTaqWpxGJ4BSacwWQXAKsTl8StzyCBOTTfjQo7MLuSMSeIsHrut5CF9wZVT9fqXWhOBkSn0PBBcMd0DpSCWKvylIg2jav48eUMRLpe9lQeiZLf9E5NNlZRQVSFx5FePgJlpUHrKmvGzNFOPSmaFM/g9RB4XUTf625Nnw3gOfLn4Wgs8H/LbSsH+72mdPlmopuWX0jDVRt2Bs4mTKQwGe8JIl778uD9bQpMj/M07ALli8HWTCm0oD/a3/Sg5EOfd9hHqO7gtFAbvMTbsSTEXa6mqR67KeKnaw+w/sNCvyYTPY3/FoIx0eZgqmXhEjmjgcTo6V/4awMSAbvAorV/4G75Pc22apBerrXFl4HlIddsjWnHWxHeYTUNJv5zMRC7O/hXGlfJRqpoakGq6X/+uNaEVsoiD8VC03sXO9uTeoaPmgtOoIDMUdDQidEojDL9ZvJVZXBMoFFfH2orbUli3zzcZgX2ZogfBDGDOR2B2rGF6sXhz8DxAELCgDKPURy/cnxPYOCp9znD7JFdFnyqkH29+jO9ta+PND+meeaTxwg+2ZoJslDN4HAt/DAuY558TXIV/xELAZuQ3T0PVxVxV7ZM0tp5+qan/1TFt56J2fXYru6rcduQXCsWKbTOfMo9hiZiFt/Cu7JHoQf8Lnqpmg1AtjXOh5TqW+jXSCMTTGppK9C63eeN41WsUcKIaNujoVleJJbSkM9dhMQK1n4 bQXciVac mRsCnsDuS3ElcSSyZGqDsB7aTEuGxmEmPHNTlax+FW9LiGDCxaGI4w+Wd6MiM0AGqEK27Ev1IyKxKbU9W8TprS9N5J1Uj5MBRk9nYJu/azKbuEkKvgcgx1RogKub/TPf2487NsYMYrQzqF0pZ3zdAV8hzsZ5z3vjX3ovP1HsOq2t8LS/r8Bz0nhYJDcHgmmY0Y+nP1jbmP6EE23c0zAZl5/EIhIH3PRLu1P1yL+QGtmqzoDhHr75yn7tmxtfXaWt5VzF3aSeYMdQe9ajTTxC0t7VlypZmQODa58h1KgIRMAlO2ntvVvIP/p2q5DoLX5/fp3XPjLh+xeaNAE+1gNtuwNxyiRJFilfLxT3Kkjh69X9JRjHnAzE4TN/KDTL43tqQGO/+4ITpfA6bHt0QNBXgSTaMJExE5ZnHHWfM5iM6m73iXZveWA23tFbq+vZ8SXSiD5EOvKLGgBy/9DoUibOo22R9xd5cLSpf2lfFcZ2W99lAxT3Q+zvqDiEhmEA0EhX/dE7RBJa9cX44vIwz88FHR06tPhDbGooPezRd+nwzeFIpeiZXqEqpS0BQ2DAwuNoC9wyzIRdBgZpDOSJ537dXRcqA1RPBxOLjcXV7cPdsPN9VRTSE4DQDedoSoymzu+VLLLGr4w8X+dTzGbHjXap9J9sUKeVidjE2CwUEOsC9baKNZCEj2d+3FcBdC2oqwfGUbvnwBUbqQRScIMeCTfDqd0sQDzjRZCYz130PmJh101aEJHZ8n5gv6xcME7hD9t93mTrE0QomvNRQYGLTDjmCuWAhCWEt1KJc6lQWboLpp1bE7OfH2LKs0+6xooTpsUfNzyir55o8qC6d4JgSYwDiXobFAVnYrM8J/bniQJ3u7nW5Vkg1BzeyVsHyCc3SuNIVGZX9M78UKyJHqVIXZjyf5oaTOuhXcVriyBzbFADBJGGSgBi2Sl2vdYZDnNo7mwQ0xloCynE4Xm/wUawzD67kQ4WWT9B3 NwqiUWOL GdmIOe/GVT5B410tVv+ZJwlMSKC/3orSgxRac6aLqFHHdItXjWKV4MPLL2iayB7iYTckpslFSQO9+Mid6WZQBluq3ea/qmLEUgdbd2z6sWfsslsIeO/QDUStbckHnLq0T7C/cGCqvsVnXUqkt6giQmhrkEGBJLqjCPlb5tIp5gK7wOlZyZu7K+l2Tc84eBm99pPbKFaAWFUaKpsmV8naknyRanEEUhnanDI4QDMX2QpDxp/XFjyQ+6C6PuI5WZVb X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi, v5: - Use vbabka's macros for optional arguments (thank you! I added a Co-developed-by and S-o-b) - Do not make Kconfig "default y", but recommend that it be enabled (vbabka) - Do not check for NULL before kmem_cache_destroy() on error path (horms) - Adjust size/bucket argument ordering on slab_alloc() - Make sure kmem_buckets cache itself is SLAB_NO_MERGE - Do not include "_noprof" in kern-doc (it is redundant) - Fix kern-doc argument ordering v4: https://lore.kernel.org/lkml/20240531191304.it.853-kees@kernel.org/ v3: https://lore.kernel.org/lkml/20240424213019.make.366-kees@kernel.org/ v2: https://lore.kernel.org/lkml/20240305100933.it.923-kees@kernel.org/ v1: https://lore.kernel.org/lkml/20240304184252.work.496-kees@kernel.org/ For the cover letter, I'm repeating the commit log for patch 4 here, which has additional clarifications and rationale since v2: Dedicated caches are available for fixed size allocations via kmem_cache_alloc(), but for dynamically sized allocations there is only the global kmalloc API's set of buckets available. This means it isn't possible to separate specific sets of dynamically sized allocations into a separate collection of caches. This leads to a use-after-free exploitation weakness in the Linux kernel since many heap memory spraying/grooming attacks depend on using userspace-controllable dynamically sized allocations to collide with fixed size allocations that end up in same cache. While CONFIG_RANDOM_KMALLOC_CACHES provides a probabilistic defense against these kinds of "type confusion" attacks, including for fixed same-size heap objects, we can create a complementary deterministic defense for dynamically sized allocations that are directly user controlled. Addressing these cases is limited in scope, so isolation these kinds of interfaces will not become an unbounded game of whack-a-mole. For example, pass through memdup_user(), making isolation there very effective. In order to isolate user-controllable sized allocations from system allocations, introduce kmem_buckets_create(), which behaves like kmem_cache_create(). Introduce kmem_buckets_alloc(), which behaves like kmem_cache_alloc(). Introduce kmem_buckets_alloc_track_caller() for where caller tracking is needed. Introduce kmem_buckets_valloc() for cases where vmalloc callback is needed. Allows for confining allocations to a dedicated set of sized caches (which have the same layout as the kmalloc caches). This can also be used in the future to extend codetag allocation annotations to implement per-caller allocation cache isolation[1] even for dynamic allocations. Memory allocation pinning[2] is still needed to plug the Use-After-Free cross-allocator weakness, but that is an existing and separate issue which is complementary to this improvement. Development continues for that feature via the SLAB_VIRTUAL[3] series (which could also provide guard pages -- another complementary improvement). Link: https://lore.kernel.org/lkml/202402211449.401382D2AF@keescook [1] Link: https://googleprojectzero.blogspot.com/2021/10/how-simple-linux-kernel-memory.html [2] Link: https://lore.kernel.org/lkml/20230915105933.495735-1-matteorizzo@google.com/ [3] After the core implementation are 2 patches that cover the most heavily abused "repeat offenders" used in exploits. Repeating those details here: The msg subsystem is a common target for exploiting[1][2][3][4][5][6] use-after-free type confusion flaws in the kernel for both read and write primitives. Avoid having a user-controlled size cache share the global kmalloc allocator by using a separate set of kmalloc buckets. Link: https://blog.hacktivesecurity.com/index.php/2022/06/13/linux-kernel-exploit-development-1day-case-study/ [1] Link: https://hardenedvault.net/blog/2022-11-13-msg_msg-recon-mitigation-ved/ [2] Link: https://www.willsroot.io/2021/08/corctf-2021-fire-of-salvation-writeup.html [3] Link: https://a13xp0p0v.github.io/2021/02/09/CVE-2021-26708.html [4] Link: https://google.github.io/security-research/pocs/linux/cve-2021-22555/writeup.html [5] Link: https://zplin.me/papers/ELOISE.pdf [6] Link: https://syst3mfailure.io/wall-of-perdition/ [7] Both memdup_user() and vmemdup_user() handle allocations that are regularly used for exploiting use-after-free type confusion flaws in the kernel (e.g. prctl() PR_SET_VMA_ANON_NAME[1] and setxattr[2][3][4] respectively). Since both are designed for contents coming from userspace, it allows for userspace-controlled allocation sizes. Use a dedicated set of kmalloc buckets so these allocations do not share caches with the global kmalloc buckets. Link: https://starlabs.sg/blog/2023/07-prctl-anon_vma_name-an-amusing-heap-spray/ [1] Link: https://duasynt.com/blog/linux-kernel-heap-spray [2] Link: https://etenal.me/archives/1336 [3] Link: https://github.com/a13xp0p0v/kernel-hack-drill/blob/master/drill_exploit_uaf.c [4] Thanks! -Kees Kees Cook (6): mm/slab: Introduce kmem_buckets typedef mm/slab: Plumb kmem_buckets into __do_kmalloc_node() mm/slab: Introduce kvmalloc_buckets_node() that can take kmem_buckets argument mm/slab: Introduce kmem_buckets_create() and family ipc, msg: Use dedicated slab buckets for alloc_msg() mm/util: Use dedicated slab buckets for memdup_user() include/linux/slab.h | 49 +++++++++++++++++++++----- ipc/msgutil.c | 13 ++++++- mm/Kconfig | 16 +++++++++ mm/slab.h | 6 ++-- mm/slab_common.c | 83 ++++++++++++++++++++++++++++++++++++++++++-- mm/slub.c | 20 +++++------ mm/util.c | 23 ++++++++---- 7 files changed, 180 insertions(+), 30 deletions(-)