From patchwork Mon Feb 21 10:53:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12753433 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39D93C433EF for ; Mon, 21 Feb 2022 10:54:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AECAA8D0003; Mon, 21 Feb 2022 05:54:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A76108D0001; Mon, 21 Feb 2022 05:54:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C9AF8D0003; Mon, 21 Feb 2022 05:54:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.a.hostedemail.com [64.99.140.24]) by kanga.kvack.org (Postfix) with ESMTP id 7871C8D0001 for ; Mon, 21 Feb 2022 05:54:00 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 45206608DC for ; Mon, 21 Feb 2022 10:54:00 +0000 (UTC) X-FDA: 79166477040.03.E51C3FC Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf20.hostedemail.com (Postfix) with ESMTP id BBAB61C0004 for ; Mon, 21 Feb 2022 10:53:59 +0000 (UTC) Received: by mail-pj1-f42.google.com with SMTP id m1-20020a17090a668100b001bc023c6f34so4675406pjj.3 for ; Mon, 21 Feb 2022 02:53:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=aA5sKZP46Br2IXGSlOJ6tqSDR6UxBlE/5/p8sTxFW2A=; b=TWXXfAQAoJokZM5wJBrVvEyYs+WaUxvPz/zifG36IeQ28CHce0CzCreiT2Lh78cklx tSBjiq3VySZhLEMOmAiUqrEKlLPMUhTQX9D01txRno+rzAJ0/PBum81r1/qFAR9BcYhf c7Y0wlGCn6ZCvmuSZbEre5JidSb/Lfd+0fr+oGqCQyahOo57+LHcEQbLb8SwYvsRUK+k mHcYuRl/BAu6IeHexEL4G32rL5+GNPosCx3ZHoyit0PBwyt50XLsZ8idVJt9bFbB2UTx M3RS4IIG1m3c3yIwATCQV4lDOaRg+enMG+GvFJRnfe9k6PjfS0dslpHqmztoZhSv4IRs 0m6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aA5sKZP46Br2IXGSlOJ6tqSDR6UxBlE/5/p8sTxFW2A=; b=RLu6aYp4R77VmoTef8bhf5ytrBTqcacAkHh69+Uj9wOc6Gm0K0hCC3gmL6vnHcHeMq s4Sxa5x0XBdRBO+11E19muPo/vOGXqQLP1rrwMW5ixhQP9rUKLZz1cnjK0feJyzyZljP Ja/zHdmO+rMhvwhAsXSvKO//1L7e63VCugS6mHHos04Fp20yhV9IjuoaE/otYEvPwign 9if6B+uK9ywWn4LrX9QSqWswrhuC6bOVfMjjv4hHvA/GuO3Kx/Mw1V9vSVylJJqB9LhW zNtdzfthGvl1q9QK0cRisgty05l48+wDwYzsizt65qGru+Hl0Dm69rIn7dtJ77Ov/koL iSnA== X-Gm-Message-State: AOAM531al5S9OOf5cvO6iT1lqkjtc5MioCY4fnKQLXMXuDiTtGJZ0c9i 5DhKb0KwG1/3cZ2c+GIwzKKeQNucFxMhPw== X-Google-Smtp-Source: ABdhPJzkBcA6oUH/qAaVd478izfxy2f+MQeMcLSeJRQBVr6AW6QXp3xyg6hHzj0PclNb0Z1q3auSqQ== X-Received: by 2002:a17:90b:240f:b0:1b8:e0b1:9667 with SMTP id nr15-20020a17090b240f00b001b8e0b19667mr25349719pjb.244.1645440838606; Mon, 21 Feb 2022 02:53:58 -0800 (PST) Received: from ip-172-31-19-208.ap-northeast-1.compute.internal (ec2-18-181-137-102.ap-northeast-1.compute.amazonaws.com. [18.181.137.102]) by smtp.gmail.com with ESMTPSA id u6sm2214725pfk.203.2022.02.21.02.53.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Feb 2022 02:53:58 -0800 (PST) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: linux-mm@kvack.org Cc: Roman Gushchin , Andrew Morton , Vlastimil Babka , linux-kernel@vger.kernel.org, Joonsoo Kim , David Rientjes , Christoph Lameter , Pekka Enberg , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH 1/5] mm/sl[au]b: Unify __ksize() Date: Mon, 21 Feb 2022 10:53:32 +0000 Message-Id: <20220221105336.522086-2-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20220221105336.522086-1-42.hyeyoo@gmail.com> References: <20220221105336.522086-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BBAB61C0004 X-Stat-Signature: qmtfrwk6u94n576hnefqwzxk5mjhf1ch Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=TWXXfAQA; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1645440839-538110 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Only SLOB need to implement __ksize() separately because SLOB records size in object header for kmalloc objects. Unify SLAB/SLUB's __ksize(). Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slab.c | 23 ----------------------- mm/slab_common.c | 29 +++++++++++++++++++++++++++++ mm/slub.c | 16 ---------------- 3 files changed, 29 insertions(+), 39 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index ddf5737c63d9..eb73d2499480 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -4199,27 +4199,4 @@ void __check_heap_object(const void *ptr, unsigned long n, } #endif /* CONFIG_HARDENED_USERCOPY */ -/** - * __ksize -- Uninstrumented ksize. - * @objp: pointer to the object - * - * Unlike ksize(), __ksize() is uninstrumented, and does not provide the same - * safety checks as ksize() with KASAN instrumentation enabled. - * - * Return: size of the actual memory used by @objp in bytes - */ -size_t __ksize(const void *objp) -{ - struct kmem_cache *c; - size_t size; - BUG_ON(!objp); - if (unlikely(objp == ZERO_SIZE_PTR)) - return 0; - - c = virt_to_cache(objp); - size = c ? c->object_size : 0; - - return size; -} -EXPORT_SYMBOL(__ksize); diff --git a/mm/slab_common.c b/mm/slab_common.c index 23f2ab0713b7..488997db0d97 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1245,6 +1245,35 @@ void kfree_sensitive(const void *p) } EXPORT_SYMBOL(kfree_sensitive); +#ifndef CONFIG_SLOB +/** + * __ksize -- Uninstrumented ksize. + * @objp: pointer to the object + * + * Unlike ksize(), __ksize() is uninstrumented, and does not provide the same + * safety checks as ksize() with KASAN instrumentation enabled. + * + * Return: size of the actual memory used by @objp in bytes + */ +size_t __ksize(const void *object) +{ + struct folio *folio; + + if (unlikely(object == ZERO_SIZE_PTR)) + return 0; + + folio = virt_to_folio(object); + +#ifdef CONFIG_SLUB + if (unlikely(!folio_test_slab(folio))) + return folio_size(folio); +#endif + + return slab_ksize(folio_slab(folio)->slab_cache); +} +EXPORT_SYMBOL(__ksize); +#endif + /** * ksize - get the actual amount of memory allocated for a given object * @objp: Pointer to the object diff --git a/mm/slub.c b/mm/slub.c index 261474092e43..3a4458976ab7 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4526,22 +4526,6 @@ void __check_heap_object(const void *ptr, unsigned long n, } #endif /* CONFIG_HARDENED_USERCOPY */ -size_t __ksize(const void *object) -{ - struct folio *folio; - - if (unlikely(object == ZERO_SIZE_PTR)) - return 0; - - folio = virt_to_folio(object); - - if (unlikely(!folio_test_slab(folio))) - return folio_size(folio); - - return slab_ksize(folio_slab(folio)->slab_cache); -} -EXPORT_SYMBOL(__ksize); - void kfree(const void *x) { struct folio *folio; From patchwork Mon Feb 21 10:53:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12753434 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D848C433EF for ; Mon, 21 Feb 2022 10:54:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E963A8D0005; Mon, 21 Feb 2022 05:54:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E1F308D0001; Mon, 21 Feb 2022 05:54:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C98DB8D0005; Mon, 21 Feb 2022 05:54:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0007.hostedemail.com [216.40.44.7]) by kanga.kvack.org (Postfix) with ESMTP id B4F798D0001 for ; Mon, 21 Feb 2022 05:54:02 -0500 (EST) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 74B58A15D7 for ; Mon, 21 Feb 2022 10:54:02 +0000 (UTC) X-FDA: 79166477124.14.8E82C53 Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by imf20.hostedemail.com (Postfix) with ESMTP id 0B3041C0008 for ; Mon, 21 Feb 2022 10:54:01 +0000 (UTC) Received: by mail-pj1-f46.google.com with SMTP id qe15so14800853pjb.3 for ; Mon, 21 Feb 2022 02:54:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=a0dsHEkkjYMDgDk/stCFwxaKiFUsmcZuEuqNP0XmmQ4=; b=KQWzIh6iHJeXbHTKWVjQ9D41ZTsqUYpa1zh7KmZnFyY8LTHgMsfbPldAhLtHRazhbF iyJ2tHidCoinbQysLDxGZ/K9/DSJmfxe+VR28huzDlGIgyZNW+M5fJNFDxTodOZuIoA7 o7F6WpsjI5aJlu4Ixla5LTpCWx4oKE+SVEbaMjEJ5oCbRqmSAMywCddBpedDxZMcmznC SVjT2Eo3CdxQkHjEG5HCPm/iUKUF3BBORX0eRVMh+8DzOq+ZkzsconTuqcuubqDFfL+/ jZ0iM1Zi/NHOm+Fds1x1RfGt0SNXPgriZU6gQGSCT6GfnJbKY/HC8BJLP00wkdC/Rl0g uHOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=a0dsHEkkjYMDgDk/stCFwxaKiFUsmcZuEuqNP0XmmQ4=; b=Z8y52jpJd58OAUGOCC87nPVbrfC9+oWoHyNpJPUm27TPsfa+nYWj/cCwac/CKMLneu 3i2fc6jEqz4oWDypQ4hIW1rzocfr4awhxNwn/oQCWe2FrdJBD7lIEFc2lfe8/bT8IOkR HD26iu/yhzq3FI5kQPq6NKxJ5mYPoZMs2bRoML/flbm0088IeFFPKFL52kRQJ1yP3gIJ a2dyDA3aY5ZAS6tkmg2zfsjKrrPc9RLIT3bJI7wA9sRDMGZl2/Cyymdv1FoYE02w3yE3 NZP47lmcYAohnTUmrIsfxT1wAPVphIFRAqt8jqQGxb6DLYEf3ziuZpM3E8wMlxtTnJNH 4L/A== X-Gm-Message-State: AOAM530bch1YP4UJNYfObowH3tUj6UKYxXZW+1OIuFw6MMJ/Af0fpBxv NgiQPqS4gHj4jpJ8EdSQfc5/S4Wyon7/gA== X-Google-Smtp-Source: ABdhPJykGrMP/3jkaSti52P2kS6FzFtdySBLx2jd5Kbdy9BLiESNB++I0iuWS6uFkrxzJj+Drnww5g== X-Received: by 2002:a17:90b:1c83:b0:1b9:caa:8230 with SMTP id oo3-20020a17090b1c8300b001b90caa8230mr25247369pjb.26.1645440841051; Mon, 21 Feb 2022 02:54:01 -0800 (PST) Received: from ip-172-31-19-208.ap-northeast-1.compute.internal (ec2-18-181-137-102.ap-northeast-1.compute.amazonaws.com. [18.181.137.102]) by smtp.gmail.com with ESMTPSA id u6sm2214725pfk.203.2022.02.21.02.53.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Feb 2022 02:54:00 -0800 (PST) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: linux-mm@kvack.org Cc: Roman Gushchin , Andrew Morton , Vlastimil Babka , linux-kernel@vger.kernel.org, Joonsoo Kim , David Rientjes , Christoph Lameter , Pekka Enberg , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH 2/5] mm/sl[auo]b: Do not export __ksize() Date: Mon, 21 Feb 2022 10:53:33 +0000 Message-Id: <20220221105336.522086-3-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20220221105336.522086-1-42.hyeyoo@gmail.com> References: <20220221105336.522086-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 X-Rspam-User: Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=KQWzIh6i; spf=pass (imf20.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 0B3041C0008 X-Stat-Signature: akske5amce81ozw4utxb1itknpk76ub4 X-HE-Tag: 1645440841-462234 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Do not export __ksize(). Only kasan calls __ksize() directly. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slab_common.c | 1 - mm/slob.c | 1 - 2 files changed, 2 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 488997db0d97..f2c021b57579 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1271,7 +1271,6 @@ size_t __ksize(const void *object) return slab_ksize(folio_slab(folio)->slab_cache); } -EXPORT_SYMBOL(__ksize); #endif /** diff --git a/mm/slob.c b/mm/slob.c index 60c5842215f1..d8af6c54f133 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -588,7 +588,6 @@ size_t __ksize(const void *block) m = (unsigned int *)(block - align); return SLOB_UNITS(*m) * SLOB_UNIT; } -EXPORT_SYMBOL(__ksize); int __kmem_cache_create(struct kmem_cache *c, slab_flags_t flags) { From patchwork Mon Feb 21 10:53:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12753435 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03609C433EF for ; Mon, 21 Feb 2022 10:54:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 76AEA8D0006; Mon, 21 Feb 2022 05:54:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6F3C68D0001; Mon, 21 Feb 2022 05:54:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 593BB8D0006; Mon, 21 Feb 2022 05:54:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0183.hostedemail.com [216.40.44.183]) by kanga.kvack.org (Postfix) with ESMTP id 47F9F8D0001 for ; Mon, 21 Feb 2022 05:54:05 -0500 (EST) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id E351DA15CB for ; Mon, 21 Feb 2022 10:54:04 +0000 (UTC) X-FDA: 79166477208.08.59A4A93 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by imf16.hostedemail.com (Postfix) with ESMTP id 6A207180004 for ; Mon, 21 Feb 2022 10:54:04 +0000 (UTC) Received: by mail-pj1-f52.google.com with SMTP id cp23-20020a17090afb9700b001bbfe0fbe94so5298294pjb.3 for ; Mon, 21 Feb 2022 02:54:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ehNQdlI9mPXdldG87wHGJGXW6GsgTB2HkJJp4vDiMvs=; b=qlaVWsO4wSTU3J0DH2Hpj3dMsi97mWHbdhECMnS3joUeasrk2uCdT2Jsmn+qIr9fLe rViNtkASpFYzpkolR6zh0SSWu6KdwdyStVhbZtvJPmsetjJYkEodcgwRZToKocOqYzXp 2aONPUD1l/wjsooqbu2AT9/xm2Es0hJdg6t1ldwpTs1qoGcNK3P0NOimVfqpqulx8+jY A9E6Y7dM5wGAqY3f7dDQHAVGyZZ2KSNUulpKOWtTqp5ppCG94DDmaysD/n5TwvITPAj3 wRKyP8MX8+1pwmDbCM0zemfWRrbDL5678sUQC/T8/T70yB9PPlieHMoRTd3hFflEyvqy ROkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ehNQdlI9mPXdldG87wHGJGXW6GsgTB2HkJJp4vDiMvs=; b=H0bnIe5NgZ8cDzddOKRunfsAtOJWSDPd8eGop3NN9eDbVEa/035znSqkGyjzxaWWkM 8FpjLQr4N5SClvouPBsxDzxrbSYJNPy/yNamwP8sAhRiysueOpCnKBl8CZ72V2MZinFg /ikqDPRXbCfXe/LAPy1ufVmlmZxeoxJpuMQ1w104hCHqH9O6fuivrSYQrFiTfI+loky6 WMXna2DlVL3mM1k7t0cHjshpRetbfsvKboDxhTmL7QnlNvrQlFt+Qol7+oEr2uQQSUyN lXoJuNIMl60BvOZ71Uq3YL2EihBD7tFwDVumGnGqkw+6uFFVBHogsH7vXx2cpGSn06cY 1yVQ== X-Gm-Message-State: AOAM533tlfR0din3IDv7mFnfLoIO0xT/UgztduCzf5ttejNuw6bYGTHa DqN+tRNq6GM+0shoL9ZitZrdfpcMeq8Cpg== X-Google-Smtp-Source: ABdhPJxy1hS2KmSmwp4qNm1k1FKMOHVd+tQ3HzzZmVsC3p1rRBWwpEZvPh4IIAH8r4LS58T0mexUGQ== X-Received: by 2002:a17:902:b091:b0:14b:4b27:48d1 with SMTP id p17-20020a170902b09100b0014b4b2748d1mr18959339plr.52.1645440843472; Mon, 21 Feb 2022 02:54:03 -0800 (PST) Received: from ip-172-31-19-208.ap-northeast-1.compute.internal (ec2-18-181-137-102.ap-northeast-1.compute.amazonaws.com. [18.181.137.102]) by smtp.gmail.com with ESMTPSA id u6sm2214725pfk.203.2022.02.21.02.54.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Feb 2022 02:54:03 -0800 (PST) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: linux-mm@kvack.org Cc: Roman Gushchin , Andrew Morton , Vlastimil Babka , linux-kernel@vger.kernel.org, Joonsoo Kim , David Rientjes , Christoph Lameter , Pekka Enberg , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH 3/5] mm/slab: Do not call kmalloc_large() for unsupported size Date: Mon, 21 Feb 2022 10:53:34 +0000 Message-Id: <20220221105336.522086-4-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20220221105336.522086-1-42.hyeyoo@gmail.com> References: <20220221105336.522086-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 6A207180004 X-Rspam-User: Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=qlaVWsO4; spf=pass (imf16.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.52 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: k41jyghrdjrmrx4cbtcxxo71nt7oj6z6 X-HE-Tag: 1645440844-523998 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: SLAB's kfree() does not support freeing an object that is allocated from kmalloc_large(). Fix this as SLAB do not pass requests larger than KMALLOC_MAX_CACHE_SIZE directly to page allocator. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- include/linux/slab.h | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 37bde99b74af..aeda3e863f2b 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -564,15 +564,19 @@ static __always_inline __alloc_size(1) void *kmalloc_large(size_t size, gfp_t fl * Try really hard to succeed the allocation but fail * eventually. */ +#ifndef CONFIG_SLOB static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) { if (__builtin_constant_p(size)) { -#ifndef CONFIG_SLOB unsigned int index; -#endif - if (size > KMALLOC_MAX_CACHE_SIZE) - return kmalloc_large(size, flags); -#ifndef CONFIG_SLOB + + if (size > KMALLOC_MAX_CACHE_SIZE) { + if (IS_ENABLED(CONFIG_SLUB)) + return kmalloc_large(size, flags); + else + return NULL; + } + index = kmalloc_index(size); if (!index) @@ -581,10 +585,17 @@ static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) return kmem_cache_alloc_trace( kmalloc_caches[kmalloc_type(flags)][index], flags, size); -#endif } return __kmalloc(size, flags); } +#else +static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) +{ + if (__builtin_constant_p(size) && size > KMALLOC_MAX_CACHE_SIZE) + return kmalloc_large(size, flags); + return __kmalloc(size, flags); +} +#endif static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node) { From patchwork Mon Feb 21 10:53:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12753436 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47955C433EF for ; Mon, 21 Feb 2022 10:54:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D389A8D0007; Mon, 21 Feb 2022 05:54:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CBF1E8D0001; Mon, 21 Feb 2022 05:54:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B60448D0007; Mon, 21 Feb 2022 05:54:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.28]) by kanga.kvack.org (Postfix) with ESMTP id A1C498D0001 for ; Mon, 21 Feb 2022 05:54:07 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7599D608DC for ; Mon, 21 Feb 2022 10:54:07 +0000 (UTC) X-FDA: 79166477334.04.B80052E Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by imf10.hostedemail.com (Postfix) with ESMTP id EEFE2C0004 for ; Mon, 21 Feb 2022 10:54:06 +0000 (UTC) Received: by mail-pj1-f52.google.com with SMTP id j10-20020a17090a94ca00b001bc2a9596f6so3489040pjw.5 for ; Mon, 21 Feb 2022 02:54:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2UEo/Stc+iYNXbwbi51AqwpgyfC/DeYqJj3Sep1UETg=; b=TQt9smYJlqZvzDOc9o4bZM6zroYbhcyVxV0DCbsexdb+fnHDrG3ZNC1Q19EAk0Ao63 62jNfhPKSpSYPZTo+tuDPbrYVI4z300rrE2lkOz0yuGIFjjjg2R8aSc/LGuUyT6n3/wa tEWUlZkuCKsHGUD7VwVlHgOJDLPU97mLW2j0c0+kdZeCY5TJr7TPVUi0+R1hqlrfuTmd h/X2ZsALJKksPfhJw9pnNn3+Er5/IYKz2nhA93DkEGGq3O1L+pd5bROH6NkpKr2NIVif ch3qRTEBpyHSOR5lP5Kk1eLqF5vseOBeoEP2WSgyUfOIDrJDCoGFqzsdpLf2IkArULuB cEIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2UEo/Stc+iYNXbwbi51AqwpgyfC/DeYqJj3Sep1UETg=; b=A6AaXpx29q7Txpktsj7AUlhndrL4uxzfDRTV3A9pkIPHZC5RqYUtSkDZaZBnS7nytz gipdetD7WTS1FZ/5BF4we67LZFBXPXxJHtHTb6jFVGBrlMg2u+5Pd7byjTTAMe4OSSHY 7934SEpI+uYRMpGShp9PXq08zXEpaU2IKZfLQMYgHJ8AQdtC1Kw6dW/qNpFm9Dcjshjp gUlwypIARiqKZGnjtkwRQg4T9rhUUrGruHzxIfQrRsvm1Zwbaa57F8ZZCLZiN9lhPak/ DzzdUEaHb9NpxKg2oQoWBpi1uzAF/4aePk6/azvxQ+goYacAAP5s2T3w5el+KStCJNrn lt4w== X-Gm-Message-State: AOAM531Xon9jSOVR3N5EHk8oITI3SoCLltc6HbhjX1rUse5hM0qWGiMV d0wDINK34RJ/nD5u+0Mvv2quUQhcu5xJZg== X-Google-Smtp-Source: ABdhPJyoPW7XVnrEz+Uimph+0Op7tRbd6nES2sCgwyi1ygR/jFYNy8MlQDyqUcT1S8O4S6ye7lb0dQ== X-Received: by 2002:a17:90b:2390:b0:1b9:c392:ab8d with SMTP id mr16-20020a17090b239000b001b9c392ab8dmr20876794pjb.30.1645440845915; Mon, 21 Feb 2022 02:54:05 -0800 (PST) Received: from ip-172-31-19-208.ap-northeast-1.compute.internal (ec2-18-181-137-102.ap-northeast-1.compute.amazonaws.com. [18.181.137.102]) by smtp.gmail.com with ESMTPSA id u6sm2214725pfk.203.2022.02.21.02.54.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Feb 2022 02:54:05 -0800 (PST) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: linux-mm@kvack.org Cc: Roman Gushchin , Andrew Morton , Vlastimil Babka , linux-kernel@vger.kernel.org, Joonsoo Kim , David Rientjes , Christoph Lameter , Pekka Enberg , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH 4/5] mm/slub: Limit min_partial only in cache creation Date: Mon, 21 Feb 2022 10:53:35 +0000 Message-Id: <20220221105336.522086-5-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20220221105336.522086-1-42.hyeyoo@gmail.com> References: <20220221105336.522086-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 X-Rspam-User: Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=TQt9smYJ; spf=pass (imf10.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.52 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: EEFE2C0004 X-Stat-Signature: 9r75h6mi5o3jsa66rrqrg6audekhqdfh X-HE-Tag: 1645440846-94189 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: SLUB sets number of minimum partial slabs for node (min_partial) using set_min_partial(). SLUB holds at least min_partial slabs even if they're empty to avoid excessive use of page allocator. set_min_partial() limits value of min_partial between MIN_PARTIAL and MAX_PARTIAL. As set_min_partial() can be called by min_partial_store() too, Only limit value of min_partial in kmem_cache_open() so that it can be changed to value that a user wants. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slub.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 3a4458976ab7..a4964deccb61 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4002,10 +4002,6 @@ static int init_kmem_cache_nodes(struct kmem_cache *s) static void set_min_partial(struct kmem_cache *s, unsigned long min) { - if (min < MIN_PARTIAL) - min = MIN_PARTIAL; - else if (min > MAX_PARTIAL) - min = MAX_PARTIAL; s->min_partial = min; } @@ -4184,6 +4180,8 @@ static int calculate_sizes(struct kmem_cache *s, int forced_order) static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) { + int min_partial; + s->flags = kmem_cache_flags(s->size, flags, s->name); #ifdef CONFIG_SLAB_FREELIST_HARDENED s->random = get_random_long(); @@ -4215,7 +4213,10 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) * The larger the object size is, the more slabs we want on the partial * list to avoid pounding the page allocator excessively. */ - set_min_partial(s, ilog2(s->size) / 2); + min_partial = min(MAX_PARTIAL, ilog2(s->size) / 2); + min_partial = max(MIN_PARTIAL, min_partial); + + set_min_partial(s, min_partial); set_cpu_partial(s); From patchwork Mon Feb 21 10:53:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12753437 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1202AC433FE for ; Mon, 21 Feb 2022 10:54:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C95E8D0008; Mon, 21 Feb 2022 05:54:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 854308D0001; Mon, 21 Feb 2022 05:54:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 67F0A8D0008; Mon, 21 Feb 2022 05:54:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0168.hostedemail.com [216.40.44.168]) by kanga.kvack.org (Postfix) with ESMTP id 5249F8D0001 for ; Mon, 21 Feb 2022 05:54:10 -0500 (EST) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id EEBC8180E8950 for ; Mon, 21 Feb 2022 10:54:09 +0000 (UTC) X-FDA: 79166477418.30.3E9CA43 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) by imf17.hostedemail.com (Postfix) with ESMTP id 67D1E40009 for ; Mon, 21 Feb 2022 10:54:09 +0000 (UTC) Received: by mail-pj1-f45.google.com with SMTP id gi6so5232237pjb.1 for ; Mon, 21 Feb 2022 02:54:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=mNqSGZPctSWXqZ6gYN0FXGhiJj1MEcwtFP+jmJt2lAM=; b=RFTPXk5DrwGPw8J5yfqw2l+chmPeFSAdBQcKMCI2hS+QHbqTp47fbGdwOgCxkQjL/5 s4KOqNYdUwQnAXGGSon/S4gwtsNgaUQm+hPClmpa6dsyOiUsk4bIbjjXDMN/Mr2LxJbF 8iglITyjVsZYvBOGkg8UpLRM/fXaOMnQJleiLza6uMPjkMb/zZEeyzuW3RjXegMQEwQW 5MfkgPkZqcSZ8OXKsXauuKkYUDFECPq7K/51WtVTzGXzDDDE5FHKMFo7H6Q04KmMXAxr TmgkL1ED1/zGJod/kfDAgFV7FFH/oMNp1EaundBNuAFgPhyWacQ4ZfNwxfK6yHdTy9nY vRxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mNqSGZPctSWXqZ6gYN0FXGhiJj1MEcwtFP+jmJt2lAM=; b=GoeLz9u8CxuW8usOu8ctCPcWdtEVsL9NUNnmia0AHH1tWSeor/qkhFL0d57jhBi537 jJ3jW1d98TtSlUoBK2WiUmvWZO2vde8zWpZR68EC5cGi2Ug4XHZAEHL5AtWe04J+TjKD VKTfMijAiGzza5pO0+ZbhM+s3d4KUKSLHw+AT+dDjeX9SU9U7FG/2PSf1dQMWfp22+KI JyjAdvB8IvZ+aTr2SKxXF2i20Pk1xWOWSYP+EfXENzlWPLfkWbmFxkwt98ElvBPZY20G dUxzNK6jZXk2KqhH7xx8ngB+/OMtBG1ypT8ro+Ny92n4IeiPHugPfgX9TPXjBNo6NWpO nkkg== X-Gm-Message-State: AOAM531zLQ2PWT6VDuI3TkY0ZPBm6TUD2QDrpwFEtm/fMtjMd1pI052u O/tXAJZmqOBiJWiJdBkGwd1dTyf5k0z9dA== X-Google-Smtp-Source: ABdhPJywNTHU63XkzHt51xLMmCuOt0KCbfUmRMm0QdINnoAQ9l+TE+h6FliiSaxwldW7bQVaCkRlDQ== X-Received: by 2002:a17:90a:8048:b0:1b9:55dd:b72d with SMTP id e8-20020a17090a804800b001b955ddb72dmr20733044pjw.90.1645440848421; Mon, 21 Feb 2022 02:54:08 -0800 (PST) Received: from ip-172-31-19-208.ap-northeast-1.compute.internal (ec2-18-181-137-102.ap-northeast-1.compute.amazonaws.com. [18.181.137.102]) by smtp.gmail.com with ESMTPSA id u6sm2214725pfk.203.2022.02.21.02.54.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Feb 2022 02:54:08 -0800 (PST) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: linux-mm@kvack.org Cc: Roman Gushchin , Andrew Morton , Vlastimil Babka , linux-kernel@vger.kernel.org, Joonsoo Kim , David Rientjes , Christoph Lameter , Pekka Enberg , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH 5/5] mm/slub: Refactor deactivate_slab() Date: Mon, 21 Feb 2022 10:53:36 +0000 Message-Id: <20220221105336.522086-6-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20220221105336.522086-1-42.hyeyoo@gmail.com> References: <20220221105336.522086-1-42.hyeyoo@gmail.com> MIME-Version: 1.0 X-Stat-Signature: 9qx3sodqrcb8qobyupj1oux4m6g1erm4 X-Rspam-User: Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=RFTPXk5D; spf=pass (imf17.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.45 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 67D1E40009 X-HE-Tag: 1645440849-969872 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Simply deactivate_slab() by removing variable 'lock' and replacing 'l' and 'm' with 'mode'. Instead, remove slab from list and unlock n->list_lock when cmpxchg_double() fails, and then retry. One slight functional change is releasing and taking n->list_lock again when cmpxchg_double() fails. This is not harmful because SLUB avoids deactivating slabs as much as possible. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slub.c | 74 +++++++++++++++++++++++++------------------------------ 1 file changed, 33 insertions(+), 41 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index a4964deccb61..2d0663befb9e 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2350,8 +2350,8 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, { enum slab_modes { M_NONE, M_PARTIAL, M_FULL, M_FREE }; struct kmem_cache_node *n = get_node(s, slab_nid(slab)); - int lock = 0, free_delta = 0; - enum slab_modes l = M_NONE, m = M_NONE; + int free_delta = 0; + enum slab_modes mode = M_NONE; void *nextfree, *freelist_iter, *freelist_tail; int tail = DEACTIVATE_TO_HEAD; unsigned long flags = 0; @@ -2420,57 +2420,49 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, new.frozen = 0; if (!new.inuse && n->nr_partial >= s->min_partial) - m = M_FREE; + mode = M_FREE; else if (new.freelist) { - m = M_PARTIAL; - if (!lock) { - lock = 1; - /* - * Taking the spinlock removes the possibility that - * acquire_slab() will see a slab that is frozen - */ - spin_lock_irqsave(&n->list_lock, flags); - } - } else { - m = M_FULL; - if (kmem_cache_debug_flags(s, SLAB_STORE_USER) && !lock) { - lock = 1; - /* - * This also ensures that the scanning of full - * slabs from diagnostic functions will not see - * any frozen slabs. - */ - spin_lock_irqsave(&n->list_lock, flags); - } + mode = M_PARTIAL; + /* + * Taking the spinlock removes the possibility that + * acquire_slab() will see a slab that is frozen + */ + spin_lock_irqsave(&n->list_lock, flags); + add_partial(n, slab, tail); + } else if (kmem_cache_debug_flags(s, SLAB_STORE_USER)) { + mode = M_FULL; + /* + * This also ensures that the scanning of full + * slabs from diagnostic functions will not see + * any frozen slabs. + */ + spin_lock_irqsave(&n->list_lock, flags); + add_full(s, n, slab); } - if (l != m) { - if (l == M_PARTIAL) - remove_partial(n, slab); - else if (l == M_FULL) - remove_full(s, n, slab); - if (m == M_PARTIAL) - add_partial(n, slab, tail); - else if (m == M_FULL) - add_full(s, n, slab); - } - - l = m; if (!cmpxchg_double_slab(s, slab, old.freelist, old.counters, new.freelist, new.counters, - "unfreezing slab")) + "unfreezing slab")) { + if (mode == M_PARTIAL) { + remove_partial(n, slab); + spin_unlock_irqrestore(&n->list_lock, flags); + } else if (mode == M_FULL) { + remove_full(s, n, slab); + spin_unlock_irqrestore(&n->list_lock, flags); + } goto redo; + } - if (lock) - spin_unlock_irqrestore(&n->list_lock, flags); - if (m == M_PARTIAL) + if (mode == M_PARTIAL) { + spin_unlock_irqrestore(&n->list_lock, flags); stat(s, tail); - else if (m == M_FULL) + } else if (mode == M_FULL) { + spin_unlock_irqrestore(&n->list_lock, flags); stat(s, DEACTIVATE_FULL); - else if (m == M_FREE) { + } else if (mode == M_FREE) { stat(s, DEACTIVATE_EMPTY); discard_slab(s, slab); stat(s, FREE_SLAB);