From patchwork Tue Mar 5 10:10:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kees Cook X-Patchwork-Id: 13581910 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3226DC54E41 for ; Tue, 5 Mar 2024 10:10:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 771B36B0098; Tue, 5 Mar 2024 05:10:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6FB9D6B0095; Tue, 5 Mar 2024 05:10:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B1A46B0098; Tue, 5 Mar 2024 05:10:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 365CB6B0095 for ; Tue, 5 Mar 2024 05:10:34 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D76F61C0221 for ; Tue, 5 Mar 2024 10:10:33 +0000 (UTC) X-FDA: 81862565946.21.69C3BD6 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf30.hostedemail.com (Postfix) with ESMTP id D01C58001D for ; Tue, 5 Mar 2024 10:10:31 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=BNoCoHdN; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf30.hostedemail.com: domain of keescook@chromium.org designates 209.85.214.174 as permitted sender) smtp.mailfrom=keescook@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709633431; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DCac96MRWhgh1/2EEmwcpolePHb0x1wbAjOzVa8BY+0=; b=ohyEEp9A4QTDSH3794fCxMwWHBvniQAgP6B1+nuoS0VLuyWdl+UHoNcaeiAJiB1IZHTgEa sCUBxuaGdWuSXFq/JdUV75OFFStgbsOW5csMyGvIaYzJ4GZv0jjgWd0kUhHt6OUTVlnaUt YepO5ZtFjixzbGeRrzJhz37piRXsXQo= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=BNoCoHdN; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf30.hostedemail.com: domain of keescook@chromium.org designates 209.85.214.174 as permitted sender) smtp.mailfrom=keescook@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709633431; a=rsa-sha256; cv=none; b=0yA35rZ6NT9o98fd+PsCQ/HZKcAkQ/9EM01oju2rWLZbAMBFRdgvOYga8yAtAKO7OiLr64 vltY+ir6ihjzonmEaZRt6QQccMgiN5wC/yuP85T/V7hffGUIHxfYsK+qmaz+W6X4lTyIkL b3IVgu4ugSIOAWiRpcOPG58gqfOuXCE= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-1dc1ff58fe4so47588845ad.1 for ; Tue, 05 Mar 2024 02:10:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709633431; x=1710238231; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DCac96MRWhgh1/2EEmwcpolePHb0x1wbAjOzVa8BY+0=; b=BNoCoHdN7890973QOtoQErf4zHirwl10BqwfIMAybJrRuaSw2yUJUDxAnCyVPW2yrC RLAqwiJPKvO63S+MZATqlMzp1pGXO2JgJETZAmB7quU9FUE7+FcFlM3SOptJ+pYIy+hm 45q4FC+FlGZt2HDobVBLjUUA/RuE3viVenuYQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709633431; x=1710238231; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DCac96MRWhgh1/2EEmwcpolePHb0x1wbAjOzVa8BY+0=; b=UybFbl7jXAOXNOVqHJpYHQqGyJybLdvq8XZsn7whAhVKKcLoD/onr9CbthjGqB1O7b Vyof+ugl/YiB56vbdtT51VCpqdwb15uuFCRgZrh1D5i4XJJwgav6bSdwDuilT9royBD2 2C9sWCuSETB4irGr9+sbjDNRoZdkCfATCrB8IyuD2teRXIMbpJHelUdqri1BCqv47PHl rUugSfcVW76eFBeoW49OaHvNYo2FQF7RIuSY0LNx1dtnWlx3izW8KUrqOHcQS403IOLo iPh7eLHhQL3snlXQbAKFeMi2kZqQuXC8E/oE0SL4h6ZLLRxFyDtkSoWG4BLlsJlscn1w EpxA== X-Forwarded-Encrypted: i=1; AJvYcCVbX3dRMY9MdUqU5jIgG2uxt81tssBfO3sac9bgg/s3CFyuoA0iGfG+RVSLgsnUEqE6QiIQqH/QBy2AW23pAScOUgY= X-Gm-Message-State: AOJu0Yxx60KfBkS1sQPqPByagV7n/5Nw77PvUkL1r893k/0Vc1RqMGK6 IHNTUNa66MyVhj98uFC9VYKmlZnEX2puhu/e/oDPZBlwc2V9IhjvOY9H7w2rqcD8FpPNpgLlvdw = X-Google-Smtp-Source: AGHT+IGQD7egJY3N2jy+qTOLk98RlzGpfSgXtQ8qNR83/EYIZpLvDqeKxbLyOSOxTxFrRo9HzKuI6Q== X-Received: by 2002:a17:903:230a:b0:1dc:fb9d:402d with SMTP id d10-20020a170903230a00b001dcfb9d402dmr1584663plh.58.1709633430745; Tue, 05 Mar 2024 02:10:30 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id n7-20020a170903110700b001dc9893b03bsm10063682plh.272.2024.03.05.02.10.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 02:10:27 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-hardening@vger.kernel.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , linux-kernel@vger.kernel.org Subject: [PATCH v2 2/9] slub: Plumb kmem_buckets into __do_kmalloc_node() Date: Tue, 5 Mar 2024 02:10:18 -0800 Message-Id: <20240305101026.694758-2-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240305100933.it.923-kees@kernel.org> References: <20240305100933.it.923-kees@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5763; i=keescook@chromium.org; h=from:subject; bh=dsYwePKLsCLMzShH9MEQL3PgSxerM2N+02XNBrgQB3U=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBl5u+Qh5OvUkS2WLPejBFffVYr3X7AdnuJ+g+ss wawW9pNc9iJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZebvkAAKCRCJcvTf3G3A Jpz5D/90L2rsJJvl6+P+wDTk00KFpS0yvEOEnTFGCVFHdm1gLjjRgBEZOCCPWclrHgGTkc8hBia nOccJ3xAylGH42H9q91nYChjlw/IZUry4o+uK9hKbxXcdC57QMGm38XenTvoOd4Eeqry07ryFmS Vtt0qurq/+5ki/6NecKkeipCrrNoA5RCxIDY0d95UiJV9EO4VZiWoeJ6j+YW6sCHuYqojJUmth8 nASULi9PDpCRxUlhbDJGHsgd7vhSSVKhr+iAVeILIgKu0ABrT+JLWABfwImg0azZt5Ls1q6akOw gnS+llQjZPPJ3vLk3Yqy3vdLff1gpUvad4G0TTJ69rOlneNSxmYTtrZZcxO/7jqHYl3Y3+pZWcI Ag/9AGf2AlCALy0WdUlBXXNfJE4lwywJnBfsr7GkeTtnd8JhTwB2w92l+xuHv5JUmzB5cSk6EOK dLaYXgNTlKnMTIq8eiOLrYhtEZGOpGgC15pVSSGiUBGviTyv+Rdi59+L+NVkvcHyG9z+2qVDc9/ Lvm8+g1VDnXM8KSNOfUFbX48M+wlbPFG1fhjcwxttYIsH2AqUtWu8K48KlXf5ugIsLGlhRQmjEX 8bFw/IhfXee/9Bt9phiU1TThOqIbodEUWlAb14Y/A/0YMUIOJbkeUWTR5mjkPZDaIFUh0MEGkE/ lg0QYFHUdAVLRJg== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 X-Rspamd-Queue-Id: D01C58001D X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: a5gyrprjg4i856u9hxfbdweatk1ao9qh X-HE-Tag: 1709633431-564793 X-HE-Meta: U2FsdGVkX1/qRF4pRmQUJCfPzE84onkNeniSuW8qfEVFk75sgNGeb53hI3UAZvJMBdnlaiGygH7pHtxFpE7wdMHPsQPPH1Fnx34VEpvpLLCyJ7XMa+6BV5+0zVcOsvzGqyPx3fIJrhSgr9TbnJSyAsYD8WDwfWzKxqXMrmek9EsY0AjWXT6BvcPmoK5aFkZeROOSwg5dSEd1yVK3KDzNJ/0qPynA7zUDuR6gDXJtJ31XKW448/lAeuLc4+xt+A2mIwY/hL5PI6tdbciU9VfUkkwQwXWxTLUu+jULM0fSsxoESs1Uj4W9pz4mr7UacBsWQlTTsLKJcC3vc1AcOKyoiUacZ9dHRBjpKLfEWwkqdQRP4McBQl1nRsZwlm502T++vsm393idmBZabp8T9GGFCLW7li+8l+VZcY06afDPtv8yCJql2Ae6LT9g7MEfPyhsLueAFvZl3MSv8lzSJIMCuZQmYXuVOdP5r4tEqJbXY0U2L4F9QsJcIEFcW1vKCzIo9W6hM62PS49j8tMiPeB3m4FZQsDWsEL70/8HGL/s9Av+Mx9StEytB/TtmlgGSed8POz9zwlxNu/Ec5wW8ncj6tib2c/vPHEXq3GQR0iEcsybOfJdvPRCfLpCl68VTqGOY3jGL8MXZH7zVNB3206k4EKamtvkjxOx3YZ6IJ/5SeSYzSt7A/MqspZfS+jW/bR7ouIi1MO+EvLudUaeNP1K/LexqwNwyx7mRj6WXgtD0eJi7lVk6wqdzu1SPlQOBcnJZKUKktm93Qr5jv8Vsris5goT3Y2KqDJz/c2RNzwD0EH1gRkrW5+59gbZpChEBNrVAXn2ihE5NNSp9hFsL6RJaOeR47g0IXV4j6aRlUokfAwNm0XUscPuE0l4NRzkqrthkfn1WLwhYyQuERi7K0/E2w4MuCnpXpDzZ0Uryq0iQCZrJxrA6h/uuLl8NsWs+oSwLmYH8kZTS8elHDGkjhQ saXbYYMB /SdSQON/RnslS9RkUvf6GI6vhg7DAhtUV/KTzGPpfl/4hO2EB8Ngljvh4BQwG+oscF7EY4Q4wVAaEg8KWajlf++dKPadbJHqzvYtYY+dM/I05U9bwyrVdGQRN3Rg09v5YqHCriXImeI+I3HLVsMBC5Q33GY0lwTKgi1f2uPeEaVf8Ck+4XSoEiI1Zc8qEtKVcu2EZas27lSekHTGKmy3PDVbBfQ4Tz0QKcDnNJWQrVlXF8TYC+/Z/nl30Xb6NKZT/iKxSeHDM93pD9saQCkQr7L9iU79kn0JrI0K0BvHYHpgWbt6PuLQKnQBAdk6yYZ0NDnmpWAeQLi6bHEuhpu4l/1pPRVaCb2xaSzeDPuX147jXbQyaQa/3Ln1OH0N0TzQgEbnnGAlyDsRTWl7jUk6IFx86dey+OBowdFLUWkYl2+4JzIB0t8ke6qq9A14xzv9Fo588OUQpfGxCKhf94A6lNlpRxD2IVLLHAvUkoUkf16N3YgcbQQPic+8h1Vk9WeCnZFBZzUOdS2yB/xJg7LBFD5fvGPcE/YIZD52bg46EZ1pCdVi8bbEOGud4M+mRwaPIo+NVQvu+bcqqpgbV5b2vVOUhuShqPFh7VNP5idvqojaQfrgSySCNGUI9np7OyxvN7ZBjjIsA2RSsx/bAIlQ5kcY4e0/nKkyNaux84lhe71mDimv2c06VF8uqQL2KIydcW4IeFG0dWRtZ7+EHSqZKmah4R5O3pRarqhWFSwtmx1EVgtNcdellQ3PeRRx6N5+8FnY201TDlHrJo5mVtkUCJBVopS5K16RAHSQV/GkIm116uqcw3JL5E6WfbSWEQWkWk+aNQFA26YtSkc11V10ZCwkwZ7eod4UUCTsfJWo9SkUnUIs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To be able to choose which buckets to allocate from, make the buckets available to the lower level kmalloc interfaces. Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org Cc: linux-hardening@vger.kernel.org --- include/linux/slab.h | 8 ++++---- lib/fortify_kunit.c | 2 +- mm/slab.h | 6 ++++-- mm/slab_common.c | 2 +- mm/slub.c | 12 ++++++------ 5 files changed, 16 insertions(+), 14 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 55059faf166c..1cc1a7637b56 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -508,8 +508,8 @@ static __always_inline void kfree_bulk(size_t size, void **p) kmem_cache_free_bulk(NULL, size, p); } -void *__kmalloc_node(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment - __alloc_size(1); +void *__kmalloc_node(kmem_buckets *b, size_t size, gfp_t flags, int node) + __assume_kmalloc_alignment __alloc_size(2); void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t flags, int node) __assume_slab_alignment __malloc; @@ -608,7 +608,7 @@ static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t fla kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], flags, node, size); } - return __kmalloc_node(size, flags, node); + return __kmalloc_node(NULL, size, flags, node); } /** @@ -686,7 +686,7 @@ static inline __alloc_size(1, 2) void *kmalloc_array_node(size_t n, size_t size, return NULL; if (__builtin_constant_p(n) && __builtin_constant_p(size)) return kmalloc_node(bytes, flags, node); - return __kmalloc_node(bytes, flags, node); + return __kmalloc_node(NULL, bytes, flags, node); } static inline __alloc_size(1, 2) void *kcalloc_node(size_t n, size_t size, gfp_t flags, int node) diff --git a/lib/fortify_kunit.c b/lib/fortify_kunit.c index 2e4fedc81621..c44400b577f3 100644 --- a/lib/fortify_kunit.c +++ b/lib/fortify_kunit.c @@ -182,7 +182,7 @@ static void alloc_size_##allocator##_dynamic_test(struct kunit *test) \ checker(expected_size, __kmalloc(alloc_size, gfp), \ kfree(p)); \ checker(expected_size, \ - __kmalloc_node(alloc_size, gfp, NUMA_NO_NODE), \ + __kmalloc_node(NULL, alloc_size, gfp, NUMA_NO_NODE), \ kfree(p)); \ \ orig = kmalloc(alloc_size, gfp); \ diff --git a/mm/slab.h b/mm/slab.h index 54deeb0428c6..931f261bde48 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -404,16 +404,18 @@ static inline unsigned int size_index_elem(unsigned int bytes) * KMALLOC_MAX_CACHE_SIZE and the caller must check that. */ static inline struct kmem_cache * -kmalloc_slab(size_t size, gfp_t flags, unsigned long caller) +kmalloc_slab(kmem_buckets *b, size_t size, gfp_t flags, unsigned long caller) { unsigned int index; + if (!b) + b = &kmalloc_caches[kmalloc_type(flags, caller)]; if (size <= 192) index = kmalloc_size_index[size_index_elem(size)]; else index = fls(size - 1); - return kmalloc_caches[kmalloc_type(flags, caller)][index]; + return (*b)[index]; } gfp_t kmalloc_fix_flags(gfp_t flags); diff --git a/mm/slab_common.c b/mm/slab_common.c index 8787cf17d6e4..1d0f25b6ae91 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -698,7 +698,7 @@ size_t kmalloc_size_roundup(size_t size) * The flags don't matter since size_index is common to all. * Neither does the caller for just getting ->object_size. */ - return kmalloc_slab(size, GFP_KERNEL, 0)->object_size; + return kmalloc_slab(NULL, size, GFP_KERNEL, 0)->object_size; } /* Above the smaller buckets, size is a multiple of page size. */ diff --git a/mm/slub.c b/mm/slub.c index 2ef88bbf56a3..71220b4b1f79 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3959,7 +3959,7 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node) EXPORT_SYMBOL(kmalloc_large_node); static __always_inline -void *__do_kmalloc_node(size_t size, gfp_t flags, int node, +void *__do_kmalloc_node(kmem_buckets *b, size_t size, gfp_t flags, int node, unsigned long caller) { struct kmem_cache *s; @@ -3975,7 +3975,7 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, if (unlikely(!size)) return ZERO_SIZE_PTR; - s = kmalloc_slab(size, flags, caller); + s = kmalloc_slab(b, size, flags, caller); ret = slab_alloc_node(s, NULL, flags, node, caller, size); ret = kasan_kmalloc(s, ret, size, flags); @@ -3983,22 +3983,22 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, return ret; } -void *__kmalloc_node(size_t size, gfp_t flags, int node) +void *__kmalloc_node(kmem_buckets *b, size_t size, gfp_t flags, int node) { - return __do_kmalloc_node(size, flags, node, _RET_IP_); + return __do_kmalloc_node(b, size, flags, node, _RET_IP_); } EXPORT_SYMBOL(__kmalloc_node); void *__kmalloc(size_t size, gfp_t flags) { - return __do_kmalloc_node(size, flags, NUMA_NO_NODE, _RET_IP_); + return __do_kmalloc_node(NULL, size, flags, NUMA_NO_NODE, _RET_IP_); } EXPORT_SYMBOL(__kmalloc); void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, unsigned long caller) { - return __do_kmalloc_node(size, flags, node, caller); + return __do_kmalloc_node(NULL, size, flags, node, caller); } EXPORT_SYMBOL(__kmalloc_node_track_caller);