From patchwork Mon Dec 12 00:37:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13070697 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15845C4167B for ; Mon, 12 Dec 2022 00:38:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A78C38E0008; Sun, 11 Dec 2022 19:37:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A1FD38E0002; Sun, 11 Dec 2022 19:37:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C1E18E0008; Sun, 11 Dec 2022 19:37:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7C8F88E0002 for ; Sun, 11 Dec 2022 19:37:59 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 4ADE7A0330 for ; Mon, 12 Dec 2022 00:37:59 +0000 (UTC) X-FDA: 80231791878.10.B9DDC58 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf24.hostedemail.com (Postfix) with ESMTP id ACF1E180012 for ; Mon, 12 Dec 2022 00:37:57 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=R1Dq44NI; spf=pass (imf24.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670805477; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QHvieJOPfKU228A/FcGpl6NQ2QtETLo/uwnKW3/KnGM=; b=Q3Ie6EMb/PEZzvuY2alPHVlp/auEdrZXrE/so5U2wAlmUvF4YrtyNjcBNig2z3FgvFGDYp DnTfL9249eGwdWYOlDIgK02ohWm8s63uOJ+4etKAfM9zlDLmce9wbzZr4mX/S8dr8t0FDp 5FGMvyKsdJc9h4fbc+YM+IOvHVCCOh0= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=R1Dq44NI; spf=pass (imf24.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670805477; a=rsa-sha256; cv=none; b=RCdEkAwber/C9abOeeEFDf8tVzrK5PydKTmIoCtKL+4yGwaOVkDrCLTp13aw+6+nLSLzbG D+8G4LS0l4bLDozyS/Cn/anS8OcPkIMmuSqJxOw5wArclH+cxnn914mR3JHfk+++WafMms /YzCUVP+NyzLYaPv/LFyaMBRAoHaHuc= Received: by mail-pj1-f43.google.com with SMTP id t17so10285326pjo.3 for ; Sun, 11 Dec 2022 16:37:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QHvieJOPfKU228A/FcGpl6NQ2QtETLo/uwnKW3/KnGM=; b=R1Dq44NIQGVO0+qq21tEHH6CncuZ3dTgE2cbw8HLO8saI8/5z0r/X0CECMdjYOgRv5 SMjSPHWZ8K2mAGRes/5tIu7FBuiyDmZfqOd6zZtvSmpU2LipHlsLhWkPY4mEV5JoKNRi BpaWYokf6VLHMRv8IXvOqTUQ1BqQZlb6F1ZsovRUuujPt0efJ9O2cbw3ryYciICij6gR W2NclCv2pqtr7MdFxnuVO1kO83DIlfndrqXivZo6F/9XzBbyauW/h1+LcLsNpSphAhP/ WgRcDWX/f/2LzKBVFNKMEqblIw2fLaqSAgkzDYeF1+tSuC9j2136W3PmA2kqcTyA5rAL 3iEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QHvieJOPfKU228A/FcGpl6NQ2QtETLo/uwnKW3/KnGM=; b=Ng9+gIBfuw7kPYm9bqC50ImpqDtZbl8EzbbDqofelvJRMcfCnPNc7dwuj1tIn7ucyL kzqn5viX+CXXdwddqgZfrHZKCPrX+/cJMLwg+t3Le7RjGYpnqdLbQlJGDsa9HLGQo/40 BxkpbmkDWtg5/7jxn3kWASxL0x9N4k5d6fFEEBjTH43oHFNryicTzpmqe+egl/B1q9Rz 6Y6RLVFDNCh3qoiOQ2pXcjKkOLYAP2dO0tSikmVolidWgYbhQFY50GbmjL0AtktrOTT+ yRfYO1UfSYdaLl3A46Dx4A1CkcszXcchm+0nET/F8lVt1HPHC82anNyn4zuY7yhKhscU /X0w== X-Gm-Message-State: ANoB5pkNgG2ZUNbcW+mUFf4+mRGlHsxkKGvTaj3Ji15jqrc777QdtKmj 4YkPQz0iBKOTvQqRPrGlVLw= X-Google-Smtp-Source: AA0mqf6GDUpbXr/OdqXwQUyMcN2xVFgzoCfs01LkdidF7uxzxZaISdRKNaWHUuHKNdtKC3oYE3Ycsg== X-Received: by 2002:a17:902:c14c:b0:189:9733:59d3 with SMTP id 12-20020a170902c14c00b00189973359d3mr14030488plj.29.1670805476623; Sun, 11 Dec 2022 16:37:56 -0800 (PST) Received: from vultr.guest ([2001:19f0:7002:8c7:5400:4ff:fe3d:656a]) by smtp.gmail.com with ESMTPSA id w9-20020a170902e88900b00177fb862a87sm4895960plg.20.2022.12.11.16.37.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 11 Dec 2022 16:37:55 -0800 (PST) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 4/9] mm: slab: Account active vm for slab Date: Mon, 12 Dec 2022 00:37:06 +0000 Message-Id: <20221212003711.24977-5-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20221212003711.24977-1-laoar.shao@gmail.com> References: <20221212003711.24977-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: ACF1E180012 X-Stat-Signature: shy88rantrry7ezbd4eh76rwgjipsys7 X-HE-Tag: 1670805477-296656 X-HE-Meta: U2FsdGVkX1/xOkQdsN9oGJ2CEoYNuJvWeNJGZ4LmdwVRlawi+97zMlB7Mn6yHQG/jGsEjnm9d3OQY6uyapL4hNlhRGg0MY6tE99wGXlDtcIft4LZhdAq8dV7CJhVSoGbkJ3lPvJ0h9UznnkFqBvPKhItAbFxFlPhCw6NT0RHuTdHE71I8y4YvG58zxn3YMm7YoHo8SbjkeDEG9gSaRgDS9Lojs8H+SlPdOUdim2RW6gvL2zdDsWf8aOwpKCswVsBdYJJQGtMoQgBa8bIqg11NBSeNFklaBlKjDt6C8FvF61w/S2AToiomA27oeFbQcc9WgbgMuzFnNqPAW6Tkn2YVOwbA5i1rk3XAyVK37peV7EBMxOMKE4uRABaTig3Gymm5OR55KkwVxjgsV1ThJIJpnEISkEint7RrMCqfSXXY2f1bv9kvUY+hmIOIvKjZOgs9mURzBns2lnUU60/M4RydLxS/J7N8eyxgY1r2tZEcYBmTVLGBI1qT/udrLo+BSKH75zLOn/uaH4xGQK42WkzQb8xcDbeYjEl9yj63lMOaFaHH6U3X/HoWvZyRBqrUYtK1jDHVJ+eVc5Yf5ACQGC6wl2KhliFl2YMjnWr4lm2zK3Y4cvUI54oFmS+dhbZ1wwBqnt7ymlD5JVITYwIvMbFy+ygCQWidX7NSs8I4tiTE2VsjHV7+bh1WZrjYi8a8HTncDnXWkFTPAnaKLdIv7ZKVgNYa1IqXONjLdYfMUyJg+mK0h2QSJNPh9U+ljmfbkrrpzeXc0oYhG+wD9xGAH79zomm9i6Ad6Jc0YamT6QsgVga6xfL7PoA9Wc7M59DSpLWSjmBAhJFB1DnpgdHeOpHkrl3vmpkqYe2tZuLyA1bvf/p4blF+uTlPtaH5Sg2emRNOWon4bt7d7Q31ODdn2VsbY8XcEc5cC0vJCpx+doJ74u4otRz2bZ/erbaZNzd5BlXSE/nJZhWZGTsrfyBpX4 4oxYYtKg BMnkH8wiu1N7uEaKhMz61n+2DCstD3foHKDMS9HwUMu2qmkhfYge0P5K/o1Z7jc8w4U/wvnSbln3BU5X2aqNmfWZrE5B6Qy2jxDeBW3R9Jo7TgKKfe4lk21engPsj/HquXzaXAYqT8TqGvT2wdjGucPBEPE7mPyJdY3dZTNpk6GYN+7lHRFgEObsh3DLqodjNiVbKidSko7ryrTG3BDTU0973sBcEHp9DeFI2PvOjLwmVFcs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a slab object is allocated, we will mark this object in this slab and check it at slab object freeing. That said we need extra memory to store the information of each object in a slab. The information of each object in a slab can be stored in the new introduced page extension active_vm, so a new member is added into struct active_vm. Signed-off-by: Yafang Shao --- mm/active_vm.c | 111 +++++++++++++++++++++++++++++++++++++++++++++++++ mm/active_vm.h | 16 +++++++ mm/slab.h | 7 ++++ mm/slub.c | 2 + 4 files changed, 136 insertions(+) diff --git a/mm/active_vm.c b/mm/active_vm.c index 541b2ba22da9..ee38047a4adc 100644 --- a/mm/active_vm.c +++ b/mm/active_vm.c @@ -1,6 +1,11 @@ // SPDX-License-Identifier: GPL-2.0 +#include #include #include +#include + +#include "active_vm.h" +#include "slab.h" static bool __active_vm_enabled __initdata = IS_ENABLED(CONFIG_ACTIVE_VM); @@ -28,7 +33,12 @@ static void __init init_active_vm(void) static_branch_disable(&active_vm_disabled); } +struct active_vm { + int *slab_data; /* for slab */ +}; + struct page_ext_operations active_vm_ops = { + .size = sizeof(struct active_vm), .need = need_active_vm, .init = init_active_vm, }; @@ -54,3 +64,104 @@ long active_vm_item_sum(int item) return sum; } + +static int *active_vm_from_slab(struct page_ext *page_ext) +{ + struct active_vm *av; + + if (static_branch_likely(&active_vm_disabled)) + return NULL; + + av = (void *)(page_ext) + active_vm_ops.offset; + return READ_ONCE(av->slab_data); +} + +void active_vm_slab_free(struct slab *slab) +{ + struct page_ext *page_ext; + struct active_vm *av; + struct page *page; + + page = slab_page(slab); + page_ext = page_ext_get(page); + if (!page_ext) + return; + + av = (void *)(page_ext) + active_vm_ops.offset; + kfree(av->slab_data); + av->slab_data = NULL; + page_ext_put(page_ext); +} + +static bool active_vm_slab_cmpxchg(struct page_ext *page_ext, int *new) +{ + struct active_vm *av; + + av = (void *)(page_ext) + active_vm_ops.offset; + return cmpxchg(&av->slab_data, NULL, new) == NULL; +} + +void active_vm_slab_add(struct kmem_cache *s, gfp_t flags, size_t size, void **p) +{ + struct page_ext *page_ext; + struct slab *slab; + struct page *page; + int *vec; + int item; + int off; + int i; + + item = active_vm_item(); + for (i = 0; i < size; i++) { + slab = virt_to_slab(p[i]); + page = slab_page(slab); + page_ext = page_ext_get(page); + + if (!page_ext) + continue; + + off = obj_to_index(s, slab, p[i]); + vec = active_vm_from_slab(page_ext); + if (!vec) { + vec = kcalloc_node(objs_per_slab(s, slab), sizeof(int), + flags & ~__GFP_ACCOUNT, slab_nid(slab)); + if (!vec) { + page_ext_put(page_ext); + continue; + } + + if (!active_vm_slab_cmpxchg(page_ext, vec)) { + kfree(vec); + vec = active_vm_from_slab(page_ext); + } + } + + vec[off] = item; + active_vm_item_add(item, obj_full_size(s)); + page_ext_put(page_ext); + } +} + +void active_vm_slab_sub(struct kmem_cache *s, struct slab *slab, void **p, int cnt) +{ + struct page *page = slab_page(slab); + struct page_ext *page_ext = page_ext_get(page); + int *vec; + int off; + int i; + + if (!page_ext) + return; + + for (i = 0; i < cnt; i++) { + vec = active_vm_from_slab(page_ext); + if (vec) { + off = obj_to_index(s, slab, p[i]); + if (vec[off] > 0) { + active_vm_item_sub(vec[off], obj_full_size(s)); + vec[off] = 0; + } + } + } + page_ext_put(page_ext); +} diff --git a/mm/active_vm.h b/mm/active_vm.h index 1df088d768ef..cf80b35412c5 100644 --- a/mm/active_vm.h +++ b/mm/active_vm.h @@ -4,8 +4,12 @@ #ifdef CONFIG_ACTIVE_VM #include +#include extern struct page_ext_operations active_vm_ops; +void active_vm_slab_add(struct kmem_cache *s, gfp_t flags, size_t size, void **p); +void active_vm_slab_sub(struct kmem_cache *s, struct slab *slab, void **p, int cnt); +void active_vm_slab_free(struct slab *slab); static inline int active_vm_item(void) { @@ -42,5 +46,17 @@ static inline void active_vm_item_add(int item, long delta) static inline void active_vm_item_sub(int item, long delta) { } + +static inline void active_vm_slab_add(struct kmem_cache *s, gfp_t flags, size_t size, void **p) +{ +} + +static inline void active_vm_slab_sub(struct kmem_cache *s, struct slab *slab, void **p, int cnt) +{ +} + +static inline void active_vm_slab_free(struct slab *slab) +{ +} #endif /* CONFIG_ACTIVE_VM */ #endif /* __MM_ACTIVE_VM_H */ diff --git a/mm/slab.h b/mm/slab.h index 0202a8c2f0d2..e8a4c16c29cb 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -232,6 +232,8 @@ struct kmem_cache { #include #include #include +#include +#include "active_vm.h" /* * State of the slab allocator. @@ -644,6 +646,9 @@ static __always_inline void unaccount_slab(struct slab *slab, int order, if (memcg_kmem_enabled()) memcg_free_slab_cgroups(slab); + if (active_vm_enabled()) + active_vm_slab_free(slab); + mod_node_page_state(slab_pgdat(slab), cache_vmstat_idx(s), -(PAGE_SIZE << order)); } @@ -742,6 +747,8 @@ static inline void slab_post_alloc_hook(struct kmem_cache *s, kmsan_slab_alloc(s, p[i], flags); } + if (active_vm_enabled() && (flags & __GFP_ACCOUNT) && active_vm_item() > 0) + active_vm_slab_add(s, flags, size, p); memcg_slab_post_alloc_hook(s, objcg, flags, size, p); } diff --git a/mm/slub.c b/mm/slub.c index 157527d7101b..21bd714c1182 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -45,6 +45,7 @@ #include #include "internal.h" +#include "active_vm.h" /* * Lock order: @@ -3654,6 +3655,7 @@ static __always_inline void slab_free(struct kmem_cache *s, struct slab *slab, unsigned long addr) { memcg_slab_free_hook(s, slab, p, cnt); + active_vm_slab_sub(s, slab, p, cnt); /* * With KASAN enabled slab_free_freelist_hook modifies the freelist * to remove objects, whose reuse must be delayed.