From patchwork Wed Mar 6 18:23:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584411 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED304C54E41 for ; Wed, 6 Mar 2024 18:24:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5A6B86B0081; Wed, 6 Mar 2024 13:24:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 555296B0082; Wed, 6 Mar 2024 13:24:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3827E6B0083; Wed, 6 Mar 2024 13:24:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1F23A6B0081 for ; Wed, 6 Mar 2024 13:24:51 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E52E11C0154 for ; Wed, 6 Mar 2024 18:24:50 +0000 (UTC) X-FDA: 81867440340.09.8AC1508 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf27.hostedemail.com (Postfix) with ESMTP id 327EA4001A for ; Wed, 6 Mar 2024 18:24:47 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Ux5qt5vH; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of 37rToZQYKCDEfheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=37rToZQYKCDEfheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749488; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8RGvR6a/4QfDJFETwF9ERXeFC4vNBRLgWjcd9YtwopU=; b=xW4YN49yBTikGnNMrFh22jSNYOdCo+LVYr5tFiO4Ihs6M2k1fQyJjlZ8BVf0JM/WBnx51g mRwQIyhUMQun7z8aDXXIMdRU7F7p6kfKf0LflWj2h+3EmbkQPbLMKngbNNYHAxxtR0jOiC HIEgp1w/OqiucCxRqzqo/tNQgBBYoLg= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Ux5qt5vH; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of 37rToZQYKCDEfheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=37rToZQYKCDEfheRaOTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749488; a=rsa-sha256; cv=none; b=hQEDx3B4lPvxyGaP3KupV5Q3u/Y+QzxM+Z/fAHA/NqG6WdOHqZymN6nbe08ubIdIss4caY rCHQR+FXyEtx3E+dq6qvq5IbO7oYAFOH1yf5O4UEqtVLCBOKcg3YrIPy1JLWjMM9YCSIch RGYD1gxE7BgCXqVC/1nfVR7jiztZPXY= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc693399655so13298390276.1 for ; Wed, 06 Mar 2024 10:24:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749487; x=1710354287; darn=kvack.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=8RGvR6a/4QfDJFETwF9ERXeFC4vNBRLgWjcd9YtwopU=; b=Ux5qt5vHXNXuepYB9Ke40TTf8EsJElw+Yo3ORRDk5D+IT2rPd5+usynoGYhhIEViSS o6bzOPy2cEelOT+/p1ew57INFpEnAGhDgBIgM4jEpr2j+RYV+cxMRlKz3ua7ofvJhHjE zEJx7gEGqrdjogwaVAPNcX5IMx0usbfjEJ/Z6Khp8y0uUXmL4XPIqFuIp9qsd1Huo7S+ BaS/mtNRqLuGTg/j0tYh3Ht5Qq92XvXTzIP/W+n7l6nwHenri5RbrQjnvwCe8kZaw2hS EtYRpr8Tb0tdwpfPzVGFb2qsjbotUAN6olAo3DRDKa//BBwuG4fVKWb0o5n0HC0uciAc NH7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749487; x=1710354287; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=8RGvR6a/4QfDJFETwF9ERXeFC4vNBRLgWjcd9YtwopU=; b=JqFEFiMMS9cMBLp4XuVC3c+iFG3ruVI6/xQVUxuOUTJzJFjw3ed9XXMowookidQmJV KK1MMutDAUoVB7x3C2S6YxSfxJKQeOqY6rSsoSMbnVEvO4QtVK+bjGOTyys/bPTZFt0r 7Bt6zNvSVNXupsqD7WvFROUKRa6Y+MDi9fy9Pj8CfaKXgm8r7FtOTpWFQN5Gn/E5h7yj mshlI80AIAYRpSxdJnHWLCgpkH1TAsmM2owMztrYETogQ+F6/lk40BuNPZDWKaD9qkfx m4b2MNpf8acw96vePhLBBtMFCpQpozUOlHNm0mU5EYg1gaRYKBJuwYWSQOEneeLXZ6+X YUpQ== X-Forwarded-Encrypted: i=1; AJvYcCXNn7sC9UJxh1JNxQwL+PLwMZdf3G9/35PuPnm8cX+Xz8TExbW4CcGZcgW8G/UbvgTvzD19Ebp6Pc2At8xwFXz+AfI= X-Gm-Message-State: AOJu0YzBQd0zzEWOpKndhkmxAOUtQMzsxRnmBy4oq2Gr87RlLBKRs1Ii bXGVJAti9bNsQ2JCekYE776/M9Pz+bBndZA1h9sh5NGc5ipNwZLCrGPDsvd9kn6758ZyPhKqE2A 0bg== X-Google-Smtp-Source: AGHT+IExM7WM2bpsJNZENlKCPTFTDsrMgfG+eMZ3sGJR2MljRFuXizgR+iOh98I5L1VZeGWDVIiMjT0x7X8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:1101:b0:dc6:fec4:1c26 with SMTP id o1-20020a056902110100b00dc6fec41c26mr4010569ybu.1.1709749486956; Wed, 06 Mar 2024 10:24:46 -0800 (PST) Date: Wed, 6 Mar 2024 10:23:59 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-2-surenb@google.com> Subject: [PATCH v5 01/37] fix missing vmalloc.h includes From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Stat-Signature: 3c8fhbhny63isfib9nm61dpifd7yxkuq X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 327EA4001A X-HE-Tag: 1709749487-701680 X-HE-Meta: U2FsdGVkX19rGO0VQDLyHesj/eKOpWNcM29NK9nb85FBFoTiQ5xELn/EbQzzpKRPYrcxZyLLDJoPDGukAgPoXObYAXJaHTaA0ZcwNnSXyC89talzjnsIabDOwwGm9HMzdojgYgZbh8LwKE4AIn7LOICYmuUQ3NXCfeEPC8RUYMtVrgEczC4sRgeslXon7JPxyVrsDg9y0wyR/EdvZGvXoO7cuMZkQfFzi7Wqe55e5MtCCbAdjvDPYOeqh2X6pdv6aoDvB7wiLDacmaEsuqG1DxmUhDLDYEsDGFdRQ+4IEoRaRDKpG0cHJ1BoOAInzZk6NWsMqOWA0M7gGinFsjrJuFAAW2FrlVh0lAPkG1GY9ZjJ32lYeWciNJrIWNN/1Mnqzb6VSs8AzIA/jiyHotqnzwEPTEHIos7AoL3zC7gx3O2rNRekTv0vUKMQ0gCl2ySYVWFHNeza+Q46RWEdyRARGBYYMR3lRR2tNAu4Ps6YNy4S9YZB3WEkYNjFGKR13bgnsEH0TUYk6A8D4EqtsGAsslKihZW+ma0IlFx1dKgMFNJ7XekAH2xDf6jDTmg2ftVTVuvYWuvBzegh5FLAl2fea6YIhhjsaKNfrb8xGRKhMZRS3Ni2m/yGsodx7A6nOKqUQSz8jh+uikUF37P0z0gDvKUY2km2ysn143EAdaCayM04Fbsa9DHr19X6gXG8+fjp0qTpwAZrEjYEKAmrxlO7Le8pWip+bp458Ydb1a3cY0IQe+Fda/8QyIiVRIWlIC0mAk3SFr3YV9GwSE8U0/n4oJkjVINeTJkzHEwm93JwNvSyZIPH3WxiyQKyYeb995nH1uUiAFyiEUEyId0fwFqBkK1eH8IVgcGrHoHMyQ1/NNAbIYrKg4LpdmyqfX3CIzws219A7XcH0FMRCGwlg1rFJFcQQ/ELVbgzHpjFQyuqVRxuKHaJ8NspfgNOloB3JTJf6p67TUNMSxRMQZbovty RdL7qXdf ealIa7qKbhg3xj1xDenF6IkAZ6M6depL4NsJInK15C2fq3ssoNgX1NxIK9jPuxstJbmWr6exjTgYqNY5c903XZrEyuTCUKAdgFoW4fOO92hEyPngkxySIeAj8F6QBUp4RcwEJpeu9Vv/W/v0pg5wWzpKdzbPAeL9dBjHLFkApR2J+S8gM8W5Fn1FOtCKH/hi0don5ie/xrliDsmkguM9jOh9sTDzId1AoMbIm/ac/bcyTNq9/P6KCFi9kOjmlQMrBfaCV4RhQWrq8JpgtZbMDsEEI/UUN2qC/C5v58085ZDGTHzvPlNK1k6vsL6005BplxTxST9ln396Kpq3oH3Psp2Vh2g0Mz9GfqpDpRDFBjPBofAD48kkebUqszOoPKBkAD0YTuwX0Zj8+bWEnku2OmDR6Q55NUmCSKe3P X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet The next patch drops vmalloc.h from a system header in order to fix a circular dependency; this adds it to all the files that were pulling it in implicitly. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Reviewed-by: Pasha Tatashin --- arch/alpha/lib/checksum.c | 1 + arch/alpha/lib/fpreg.c | 1 + arch/alpha/lib/memcpy.c | 1 + arch/arm/kernel/irq.c | 1 + arch/arm/kernel/traps.c | 1 + arch/arm64/kernel/efi.c | 1 + arch/loongarch/include/asm/kfence.h | 1 + arch/powerpc/kernel/iommu.c | 1 + arch/powerpc/mm/mem.c | 1 + arch/riscv/kernel/elf_kexec.c | 1 + arch/riscv/kernel/probes/kprobes.c | 1 + arch/s390/kernel/cert_store.c | 1 + arch/s390/kernel/ipl.c | 1 + arch/x86/include/asm/io.h | 1 + arch/x86/kernel/cpu/sgx/main.c | 1 + arch/x86/kernel/irq_64.c | 1 + arch/x86/mm/fault.c | 1 + drivers/accel/ivpu/ivpu_mmu_context.c | 1 + drivers/gpu/drm/gma500/mmu.c | 1 + drivers/gpu/drm/i915/gem/i915_gem_pages.c | 1 + drivers/gpu/drm/i915/gem/selftests/mock_dmabuf.c | 1 + drivers/gpu/drm/i915/gt/shmem_utils.c | 1 + drivers/gpu/drm/i915/gvt/firmware.c | 1 + drivers/gpu/drm/i915/gvt/gtt.c | 1 + drivers/gpu/drm/i915/gvt/handlers.c | 1 + drivers/gpu/drm/i915/gvt/mmio.c | 1 + drivers/gpu/drm/i915/gvt/vgpu.c | 1 + drivers/gpu/drm/i915/intel_gvt.c | 1 + drivers/gpu/drm/imagination/pvr_vm_mips.c | 1 + drivers/gpu/drm/mediatek/mtk_drm_gem.c | 1 + drivers/gpu/drm/omapdrm/omap_gem.c | 1 + drivers/gpu/drm/v3d/v3d_bo.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_binding.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_cmd.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_devcaps.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_ioctl.c | 1 + drivers/gpu/drm/xen/xen_drm_front_gem.c | 1 + drivers/hwtracing/coresight/coresight-trbe.c | 1 + drivers/net/ethernet/marvell/octeon_ep/octep_pfvf_mbox.c | 1 + drivers/net/ethernet/microsoft/mana/hw_channel.c | 1 + drivers/platform/x86/uv_sysfs.c | 1 + drivers/scsi/mpi3mr/mpi3mr_transport.c | 2 ++ drivers/vfio/pci/pds/dirty.c | 1 + drivers/virt/acrn/mm.c | 1 + drivers/virtio/virtio_mem.c | 1 + include/linux/pds/pds_common.h | 2 ++ include/rdma/rdmavt_qp.h | 1 + mm/debug_vm_pgtable.c | 1 + sound/pci/hda/cs35l41_hda.c | 1 + 51 files changed, 53 insertions(+) diff --git a/arch/alpha/lib/checksum.c b/arch/alpha/lib/checksum.c index 3f35c3ed6948..c29b98ef9c82 100644 --- a/arch/alpha/lib/checksum.c +++ b/arch/alpha/lib/checksum.c @@ -14,6 +14,7 @@ #include #include +#include static inline unsigned short from64to16(unsigned long x) { diff --git a/arch/alpha/lib/fpreg.c b/arch/alpha/lib/fpreg.c index 7c08b225261c..3d32165043f8 100644 --- a/arch/alpha/lib/fpreg.c +++ b/arch/alpha/lib/fpreg.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #if defined(CONFIG_ALPHA_EV6) || defined(CONFIG_ALPHA_EV67) diff --git a/arch/alpha/lib/memcpy.c b/arch/alpha/lib/memcpy.c index cbac3dc6d963..0e536a1a39ff 100644 --- a/arch/alpha/lib/memcpy.c +++ b/arch/alpha/lib/memcpy.c @@ -18,6 +18,7 @@ #include #include +#include /* * This should be done in one go with ldq_u*2/mask/stq_u. Do it diff --git a/arch/arm/kernel/irq.c b/arch/arm/kernel/irq.c index fe28fc1f759d..dab42d066d06 100644 --- a/arch/arm/kernel/irq.c +++ b/arch/arm/kernel/irq.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c index 3bad79db5d6e..27addbf0f98c 100644 --- a/arch/arm/kernel/traps.c +++ b/arch/arm/kernel/traps.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include diff --git a/arch/arm64/kernel/efi.c b/arch/arm64/kernel/efi.c index 9afcc690fe73..4a92096db34e 100644 --- a/arch/arm64/kernel/efi.c +++ b/arch/arm64/kernel/efi.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include diff --git a/arch/loongarch/include/asm/kfence.h b/arch/loongarch/include/asm/kfence.h index 6c82aea1c993..54062656dc7b 100644 --- a/arch/loongarch/include/asm/kfence.h +++ b/arch/loongarch/include/asm/kfence.h @@ -10,6 +10,7 @@ #define _ASM_LOONGARCH_KFENCE_H #include +#include #include #include diff --git a/arch/powerpc/kernel/iommu.c b/arch/powerpc/kernel/iommu.c index a9bebfd56b3b..25782d361884 100644 --- a/arch/powerpc/kernel/iommu.c +++ b/arch/powerpc/kernel/iommu.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include #include diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c index 3a440004b97d..a197d4c2244b 100644 --- a/arch/powerpc/mm/mem.c +++ b/arch/powerpc/mm/mem.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include diff --git a/arch/riscv/kernel/elf_kexec.c b/arch/riscv/kernel/elf_kexec.c index 54260c16f991..11c0d2e0becf 100644 --- a/arch/riscv/kernel/elf_kexec.c +++ b/arch/riscv/kernel/elf_kexec.c @@ -19,6 +19,7 @@ #include #include #include +#include #include int arch_kimage_file_post_load_cleanup(struct kimage *image) diff --git a/arch/riscv/kernel/probes/kprobes.c b/arch/riscv/kernel/probes/kprobes.c index 2f08c14a933d..71a8b8945b26 100644 --- a/arch/riscv/kernel/probes/kprobes.c +++ b/arch/riscv/kernel/probes/kprobes.c @@ -6,6 +6,7 @@ #include #include #include +#include #include #include #include diff --git a/arch/s390/kernel/cert_store.c b/arch/s390/kernel/cert_store.c index 554447768bdd..bf983513dd33 100644 --- a/arch/s390/kernel/cert_store.c +++ b/arch/s390/kernel/cert_store.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include diff --git a/arch/s390/kernel/ipl.c b/arch/s390/kernel/ipl.c index ba75f6bee774..0854a8450a6e 100644 --- a/arch/s390/kernel/ipl.c +++ b/arch/s390/kernel/ipl.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include diff --git a/arch/x86/include/asm/io.h b/arch/x86/include/asm/io.h index 3814a9263d64..c6b799d28126 100644 --- a/arch/x86/include/asm/io.h +++ b/arch/x86/include/asm/io.h @@ -42,6 +42,7 @@ #include #include #include +#include #define build_mmio_read(name, size, type, reg, barrier) \ static inline type name(const volatile void __iomem *addr) \ diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c index 166692f2d501..27892e57c4ef 100644 --- a/arch/x86/kernel/cpu/sgx/main.c +++ b/arch/x86/kernel/cpu/sgx/main.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include "driver.h" #include "encl.h" diff --git a/arch/x86/kernel/irq_64.c b/arch/x86/kernel/irq_64.c index fe0c859873d1..ade0043ce56e 100644 --- a/arch/x86/kernel/irq_64.c +++ b/arch/x86/kernel/irq_64.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 679b09cfe241..af223e57aa63 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -20,6 +20,7 @@ #include /* efi_crash_gracefully_on_page_fault()*/ #include #include /* find_and_lock_vma() */ +#include #include /* boot_cpu_has, ... */ #include /* dotraplinkage, ... */ diff --git a/drivers/accel/ivpu/ivpu_mmu_context.c b/drivers/accel/ivpu/ivpu_mmu_context.c index fe6161299236..128aef8e5a19 100644 --- a/drivers/accel/ivpu/ivpu_mmu_context.c +++ b/drivers/accel/ivpu/ivpu_mmu_context.c @@ -6,6 +6,7 @@ #include #include #include +#include #include diff --git a/drivers/gpu/drm/gma500/mmu.c b/drivers/gpu/drm/gma500/mmu.c index a70b01ccdf70..4d78b33eaa82 100644 --- a/drivers/gpu/drm/gma500/mmu.c +++ b/drivers/gpu/drm/gma500/mmu.c @@ -5,6 +5,7 @@ **************************************************************************/ #include +#include #include "mmu.h" #include "psb_drv.h" diff --git a/drivers/gpu/drm/i915/gem/i915_gem_pages.c b/drivers/gpu/drm/i915/gem/i915_gem_pages.c index 0ba955611dfb..8780aa243105 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_pages.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_pages.c @@ -5,6 +5,7 @@ */ #include +#include #include "gt/intel_gt.h" #include "gt/intel_tlb.h" diff --git a/drivers/gpu/drm/i915/gem/selftests/mock_dmabuf.c b/drivers/gpu/drm/i915/gem/selftests/mock_dmabuf.c index b2a5882b8f81..075657018739 100644 --- a/drivers/gpu/drm/i915/gem/selftests/mock_dmabuf.c +++ b/drivers/gpu/drm/i915/gem/selftests/mock_dmabuf.c @@ -4,6 +4,7 @@ * Copyright © 2016 Intel Corporation */ +#include #include "mock_dmabuf.h" static struct sg_table *mock_map_dma_buf(struct dma_buf_attachment *attachment, diff --git a/drivers/gpu/drm/i915/gt/shmem_utils.c b/drivers/gpu/drm/i915/gt/shmem_utils.c index bccc3a1200bc..1fb6ff77fd89 100644 --- a/drivers/gpu/drm/i915/gt/shmem_utils.c +++ b/drivers/gpu/drm/i915/gt/shmem_utils.c @@ -7,6 +7,7 @@ #include #include #include +#include #include "i915_drv.h" #include "gem/i915_gem_object.h" diff --git a/drivers/gpu/drm/i915/gvt/firmware.c b/drivers/gpu/drm/i915/gvt/firmware.c index 4dd52ac2043e..d800d267f0e9 100644 --- a/drivers/gpu/drm/i915/gvt/firmware.c +++ b/drivers/gpu/drm/i915/gvt/firmware.c @@ -30,6 +30,7 @@ #include #include +#include #include "i915_drv.h" #include "gvt.h" diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c index 094fca9b0e73..58cca4906f41 100644 --- a/drivers/gpu/drm/i915/gvt/gtt.c +++ b/drivers/gpu/drm/i915/gvt/gtt.c @@ -39,6 +39,7 @@ #include "trace.h" #include "gt/intel_gt_regs.h" +#include #if defined(VERBOSE_DEBUG) #define gvt_vdbg_mm(fmt, args...) gvt_dbg_mm(fmt, ##args) diff --git a/drivers/gpu/drm/i915/gvt/handlers.c b/drivers/gpu/drm/i915/gvt/handlers.c index efcb00472be2..ea9c30092767 100644 --- a/drivers/gpu/drm/i915/gvt/handlers.c +++ b/drivers/gpu/drm/i915/gvt/handlers.c @@ -52,6 +52,7 @@ #include "display/skl_watermark_regs.h" #include "display/vlv_dsi_pll_regs.h" #include "gt/intel_gt_regs.h" +#include /* XXX FIXME i915 has changed PP_XXX definition */ #define PCH_PP_STATUS _MMIO(0xc7200) diff --git a/drivers/gpu/drm/i915/gvt/mmio.c b/drivers/gpu/drm/i915/gvt/mmio.c index 5b5def6ddef7..780762f28aa4 100644 --- a/drivers/gpu/drm/i915/gvt/mmio.c +++ b/drivers/gpu/drm/i915/gvt/mmio.c @@ -33,6 +33,7 @@ * */ +#include #include "i915_drv.h" #include "i915_reg.h" #include "gvt.h" diff --git a/drivers/gpu/drm/i915/gvt/vgpu.c b/drivers/gpu/drm/i915/gvt/vgpu.c index 08ad1bd651f1..63c751ca4119 100644 --- a/drivers/gpu/drm/i915/gvt/vgpu.c +++ b/drivers/gpu/drm/i915/gvt/vgpu.c @@ -34,6 +34,7 @@ #include "i915_drv.h" #include "gvt.h" #include "i915_pvinfo.h" +#include void populate_pvinfo_page(struct intel_vgpu *vgpu) { diff --git a/drivers/gpu/drm/i915/intel_gvt.c b/drivers/gpu/drm/i915/intel_gvt.c index 9b6d87c8b583..5a01d60e5186 100644 --- a/drivers/gpu/drm/i915/intel_gvt.c +++ b/drivers/gpu/drm/i915/intel_gvt.c @@ -28,6 +28,7 @@ #include "gt/intel_context.h" #include "gt/intel_ring.h" #include "gt/shmem_utils.h" +#include /** * DOC: Intel GVT-g host support diff --git a/drivers/gpu/drm/imagination/pvr_vm_mips.c b/drivers/gpu/drm/imagination/pvr_vm_mips.c index b7fef3c797e6..6563dcde109c 100644 --- a/drivers/gpu/drm/imagination/pvr_vm_mips.c +++ b/drivers/gpu/drm/imagination/pvr_vm_mips.c @@ -14,6 +14,7 @@ #include #include #include +#include /** * pvr_vm_mips_init() - Initialise MIPS FW pagetable diff --git a/drivers/gpu/drm/mediatek/mtk_drm_gem.c b/drivers/gpu/drm/mediatek/mtk_drm_gem.c index 4f2e3feabc0f..3e519869b632 100644 --- a/drivers/gpu/drm/mediatek/mtk_drm_gem.c +++ b/drivers/gpu/drm/mediatek/mtk_drm_gem.c @@ -4,6 +4,7 @@ */ #include +#include #include #include diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c index 3421e8389222..9ea0c64c26b5 100644 --- a/drivers/gpu/drm/omapdrm/omap_gem.c +++ b/drivers/gpu/drm/omapdrm/omap_gem.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include diff --git a/drivers/gpu/drm/v3d/v3d_bo.c b/drivers/gpu/drm/v3d/v3d_bo.c index 1bdfac8beafd..bd078852cd60 100644 --- a/drivers/gpu/drm/v3d/v3d_bo.c +++ b/drivers/gpu/drm/v3d/v3d_bo.c @@ -21,6 +21,7 @@ #include #include +#include #include "v3d_drv.h" #include "uapi/drm/v3d_drm.h" diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_binding.c b/drivers/gpu/drm/vmwgfx/vmwgfx_binding.c index ae2de914eb89..2731f6ded1c2 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_binding.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_binding.c @@ -54,6 +54,7 @@ #include "vmwgfx_drv.h" #include "vmwgfx_binding.h" #include "device_include/svga3d_reg.h" +#include #define VMW_BINDING_RT_BIT 0 #define VMW_BINDING_PS_BIT 1 diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_cmd.c b/drivers/gpu/drm/vmwgfx/vmwgfx_cmd.c index 195ff8792e5a..dd4ca6a9c690 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_cmd.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_cmd.c @@ -31,6 +31,7 @@ #include #include +#include bool vmw_supports_3d(struct vmw_private *dev_priv) { diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_devcaps.c b/drivers/gpu/drm/vmwgfx/vmwgfx_devcaps.c index 829df395c2ed..6e6beff9e262 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_devcaps.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_devcaps.c @@ -25,6 +25,7 @@ * **************************************************************************/ +#include #include "vmwgfx_devcaps.h" #include "vmwgfx_drv.h" diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c index d3e308fdfd5b..7a451410ad77 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c @@ -53,6 +53,7 @@ #include #include #include +#include #define VMWGFX_DRIVER_DESC "Linux drm driver for VMware graphics devices" diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c b/drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c index 36987ef3fc30..4ce22843015e 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c @@ -35,6 +35,7 @@ #include #include +#include /* * Helper macro to get dx_ctx_node if available otherwise print an error diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_ioctl.c b/drivers/gpu/drm/vmwgfx/vmwgfx_ioctl.c index a1da5678c731..835d1eed8dd9 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_ioctl.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_ioctl.c @@ -31,6 +31,7 @@ #include #include +#include int vmw_getparam_ioctl(struct drm_device *dev, void *data, struct drm_file *file_priv) diff --git a/drivers/gpu/drm/xen/xen_drm_front_gem.c b/drivers/gpu/drm/xen/xen_drm_front_gem.c index 3ad2b4cfd1f0..63112ed975c4 100644 --- a/drivers/gpu/drm/xen/xen_drm_front_gem.c +++ b/drivers/gpu/drm/xen/xen_drm_front_gem.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c index 6136776482e6..96a32b213669 100644 --- a/drivers/hwtracing/coresight/coresight-trbe.c +++ b/drivers/hwtracing/coresight/coresight-trbe.c @@ -17,6 +17,7 @@ #include #include +#include #include "coresight-self-hosted-trace.h" #include "coresight-trbe.h" diff --git a/drivers/net/ethernet/marvell/octeon_ep/octep_pfvf_mbox.c b/drivers/net/ethernet/marvell/octeon_ep/octep_pfvf_mbox.c index 2e2c3be8a0b4..e6eb98d70f3c 100644 --- a/drivers/net/ethernet/marvell/octeon_ep/octep_pfvf_mbox.c +++ b/drivers/net/ethernet/marvell/octeon_ep/octep_pfvf_mbox.c @@ -15,6 +15,7 @@ #include #include #include +#include #include "octep_config.h" #include "octep_main.h" diff --git a/drivers/net/ethernet/microsoft/mana/hw_channel.c b/drivers/net/ethernet/microsoft/mana/hw_channel.c index 2729a2c5acf9..11021c34e47e 100644 --- a/drivers/net/ethernet/microsoft/mana/hw_channel.c +++ b/drivers/net/ethernet/microsoft/mana/hw_channel.c @@ -3,6 +3,7 @@ #include #include +#include static int mana_hwc_get_msg_index(struct hw_channel_context *hwc, u16 *msg_id) { diff --git a/drivers/platform/x86/uv_sysfs.c b/drivers/platform/x86/uv_sysfs.c index 38d1b692d3c0..40e010877189 100644 --- a/drivers/platform/x86/uv_sysfs.c +++ b/drivers/platform/x86/uv_sysfs.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include diff --git a/drivers/scsi/mpi3mr/mpi3mr_transport.c b/drivers/scsi/mpi3mr/mpi3mr_transport.c index c0c8ab586957..408a4023406b 100644 --- a/drivers/scsi/mpi3mr/mpi3mr_transport.c +++ b/drivers/scsi/mpi3mr/mpi3mr_transport.c @@ -7,6 +7,8 @@ * */ +#include + #include "mpi3mr.h" /** diff --git a/drivers/vfio/pci/pds/dirty.c b/drivers/vfio/pci/pds/dirty.c index 8ddf4346fcd5..0a161becd646 100644 --- a/drivers/vfio/pci/pds/dirty.c +++ b/drivers/vfio/pci/pds/dirty.c @@ -3,6 +3,7 @@ #include #include +#include #include #include diff --git a/drivers/virt/acrn/mm.c b/drivers/virt/acrn/mm.c index fa5d9ca6be57..c088ee1f1180 100644 --- a/drivers/virt/acrn/mm.c +++ b/drivers/virt/acrn/mm.c @@ -12,6 +12,7 @@ #include #include #include +#include #include "acrn_drv.h" diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c index 8e3223294442..e8355f55a8f7 100644 --- a/drivers/virtio/virtio_mem.c +++ b/drivers/virtio/virtio_mem.c @@ -21,6 +21,7 @@ #include #include #include +#include #include diff --git a/include/linux/pds/pds_common.h b/include/linux/pds/pds_common.h index 30581e2e04cc..5802e1deef24 100644 --- a/include/linux/pds/pds_common.h +++ b/include/linux/pds/pds_common.h @@ -4,6 +4,8 @@ #ifndef _PDS_COMMON_H_ #define _PDS_COMMON_H_ +#include + #define PDS_CORE_DRV_NAME "pds_core" /* the device's internal addressing uses up to 52 bits */ diff --git a/include/rdma/rdmavt_qp.h b/include/rdma/rdmavt_qp.h index 2e58d5e6ac0e..d67892944193 100644 --- a/include/rdma/rdmavt_qp.h +++ b/include/rdma/rdmavt_qp.h @@ -11,6 +11,7 @@ #include #include #include +#include /* * Atomic bit definitions for r_aflags. */ diff --git a/mm/debug_vm_pgtable.c b/mm/debug_vm_pgtable.c index 65c19025da3d..f1c9a2c5abc0 100644 --- a/mm/debug_vm_pgtable.c +++ b/mm/debug_vm_pgtable.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include diff --git a/sound/pci/hda/cs35l41_hda.c b/sound/pci/hda/cs35l41_hda.c index d3fa6e136744..990b5bd717a1 100644 --- a/sound/pci/hda/cs35l41_hda.c +++ b/sound/pci/hda/cs35l41_hda.c @@ -13,6 +13,7 @@ #include #include #include +#include #include "hda_local.h" #include "hda_auto_parser.h" #include "hda_jack.h" From patchwork Wed Mar 6 18:24:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584412 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67BC8C5475B for ; Wed, 6 Mar 2024 18:24:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A3B2D6B0082; Wed, 6 Mar 2024 13:24:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9741F6B0083; Wed, 6 Mar 2024 13:24:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 755E16B0085; Wed, 6 Mar 2024 13:24:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 605A86B0082 for ; Wed, 6 Mar 2024 13:24:52 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 428E51C0C6A for ; Wed, 6 Mar 2024 18:24:52 +0000 (UTC) X-FDA: 81867440424.01.9DCB4C6 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf09.hostedemail.com (Postfix) with ESMTP id 7E726140014 for ; Wed, 6 Mar 2024 18:24:50 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=hAh7cmmF; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of 38bToZQYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=38bToZQYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749490; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IB/ApQiOXGBR0WmAItdSsQq8CBMweolDZikHX0/GsD8=; b=6vq4BHSoYb7vk+PFwSAUIUFFFtiun4kcADULSuInNnhpANe5hsk7MXTCEg4nQXjkNpWVVI QO7w57KIGf8VFUCBauygVpqiGkzy8SIYjwHxFccrMjYYofNc09O55bOrE55jO+Sbh78qb0 3xSK1imDIBJo8kSOIGOdKRJBkdp9qM4= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=hAh7cmmF; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of 38bToZQYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=38bToZQYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749490; a=rsa-sha256; cv=none; b=nDzoIgTKxkNwSneW4ysSciuGWyp0DAuger0W/flXEoRUSW33ck4Glvju58BOB4N3HIJHN/ YcKqcotteF3JHp6G8xi6iwFtW5MJPwOKBa+TLaklmZxhtio6QYOieFcS3PL9VM9c77bs2C GmUI7mZhZM2p5Zn/ze0236eMbiuzpWk= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcdc3db67f0so1395616276.1 for ; Wed, 06 Mar 2024 10:24:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749489; x=1710354289; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IB/ApQiOXGBR0WmAItdSsQq8CBMweolDZikHX0/GsD8=; b=hAh7cmmFXbKOkY0ABVbBWP6qStnswMd6pVqbabx7F6JGA8zcun3rCUewGIDaeMnS8o t+MZbcvzR+ei/TrBOXCgr9Tkhb5/cRL7Y/07JKAtgQ5EjvRdA1KEwx5DuGHCCxDv7mxw S3gh9EHUg7if4Ss4SiaoLF7o7LW7khqVDTXQWmHADEgNGZ/estS57FaP9lx1lqdoMH4l J83XAGksk3olB6f7wq2oXiSFHlvvzFLEoNgBRGweVqcM0zPE8tA9QJmSzhLc0pKEJ354 0Z8tXA1hqQqVHY5+qhaNjUPJvb2/Slw75PeFfBlybu6wJez/JpHYlwFXV+FrFADEFFhu tW0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749489; x=1710354289; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IB/ApQiOXGBR0WmAItdSsQq8CBMweolDZikHX0/GsD8=; b=iaOP/w6t/P0J++fpNtRJ2dLGQ3iE1cUDT5BUVcqir1X88OUC6yipw60E6YzFj0O+Rk 7/fUh+y+y0pWS1noOpDodZCrHOjJDAC6hwPlEw1RfjAkodWmxg0BKIaCDMfZWw9vpyYy SFYKDCryqrnlQJ9lA19Xy72t6Ht+LmmZ4scQlee2SgqjqSjNc71HJ6QQ9qajnOmF/7fD /Rwp73zy1GqEWFqCMVpwpIwVrpm0agOhqBemhG0K7VwQB6Iulh6Uo75qGEB84xS4ATIv PC3oXUP+hh+SMqGfqvXHhI5/sryIAwA1jRsWzUbMejxNAUOIQ5JwB9MzjoMe9Q9mnCJp i0eQ== X-Forwarded-Encrypted: i=1; AJvYcCWFAoRsJP6jACHRhPoUwycCWOGOOugmOUs9U5uvj53e1XHN3epg3yDVBrC8TWWmSnl5RUQng4uLCLVXZyEn6LNjEmQ= X-Gm-Message-State: AOJu0YyMZ3Qj2Lc9HCr1fOgfiyJTak8hfl0BuPiJxHboK8ch7MRZ/WB2 nyHMVG1NHjvWGyUXb38JfnZUZU4xlL01e6ioN2c+65aSl7gO+iIVPArQDWvso93Q1wUSOTkx6vY DnA== X-Google-Smtp-Source: AGHT+IFVqihldBNv4wgyZ0iAVVxZ+bY4OMcJ5N94UurEgGn8+DdCU01VG5gLaZEk+WmQHH2kGW3d+2j2Wcc= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:110c:b0:dc6:e884:2342 with SMTP id o12-20020a056902110c00b00dc6e8842342mr1897962ybu.5.1709749489247; Wed, 06 Mar 2024 10:24:49 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:00 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-3-surenb@google.com> Subject: [PATCH v5 02/37] asm-generic/io.h: Kill vmalloc.h dependency From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Stat-Signature: e3h364jgqmb6ow1xdsq19xarskq5qqnh X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 7E726140014 X-HE-Tag: 1709749490-209155 X-HE-Meta: U2FsdGVkX1+kVUNTXvRUho67hYGkmGBcmuyu9M3yMUUfJ8n8VOayh7/hJNrVDP/cnCQ5Gv/9oja7JE8c4LU5Zzf2Z4yt14qVgsTB4cEWq8LY10wUraGpa/9KnvYfG/Av2IGNNnBtSiMj1w8Ik75QL2meKKDclkSV/szP1J/W4drCIkiLe4PmsegGRh1iiZyUHnHW2J5swqmRzojrl6UMcUzaiOhCA9rD+Pz7XlPHQISFGCDVgosP6P92btA9r/YOBzkMpH7gsYws+urzQMA6p3aA6perd/v+9rZJyrfI7nuIwPPsy4/1bN5x5OdhNBF9wV5zRgxYIfSEK9d+CEKgwZmfzd+9EdymexFqrgRTBPMEs2lVbJAZSChoSR62XiVJTme9xE5JAkQfZLi9BAhMY0aXaL5SFHp/skLTEcQM8xnEt78I+XdDWo1excD/uivBCY7pG4WoQ0BXs+gXa2BHNdmAMg13RsnWcxEVVNKDhCb2e+L9vvvUayYiIzVBTw0oEsSV86i4fnIDueE1YzAJl8WmP/H6VuBQAfLYwHzk/ZZcySr39du4m1CejShGZ0/04DibJVW9nBIreuW2ZvOkmAw6sokUzIzT7MMafRKBo27yOQjeCaFY26chCeOgcR3uC+FPvCQlmH+wazGtKA2XXFoMdk95fcE4acNym4ZtLyC77oGY/9SjTzZxAoAUar187y+iTqfTAGQJoTtE36HUWFgNu4PoDyX/HfVWm/2TUoFZgIfOx1u2KrKHt5XO/CB31C6u+gVTzcxg7bBlp/LAG/usJYNBdrFlUXwpJUJfvZdYFjPHuWSoEm88gOpfIMmvHf0ppdFOVbSgRULc+M7S1dL4mZAlB+crQatYkkECmGVkrZi0WXbFGhegjWUmMfZWbNEfMn52MG3KHF6TMkRaU6hC2itUaYf4nafneYbjhMcGiZjUYzXdiPi1EhXO+VpEN5wCKKRhrb4TclKsQRp FjlfsNzI xgz/hsyZsKlHkAScmIbp2zmWlCyPtQ4a9k74vS3RyRwSr8ijEPTAv3OhnV2I+HytgRK/J7F+Irj00FamGLoKsWZVx54+HuYBX1DCzBRPX11KU1UQd0U2ECtlbsAea+OQCDU+2cFf3sFeVx02mSj3MupGTR8H0qNxVSE/5wIk8A432OBKoCcYaoQ+APZ4UZAKVgVkQlE/u0LwVOrSWKHJtWv9TUCZd6ulxA494LtsLGE5LnTRycZYWqEMCKmP/TKv+fahCaa2MAsHg1tP3qVM2+CXQdC/UFYXF7hNu0MvEAEiQkxe62nwWyHXDL41jTCKg2Ay6bB0kMhy2Ywqyao4TulBt6hjRQVY/E3053jHzQkqJ9HDj18GAEquYZHet4YxI0jAoSqCNmrbbhwocr0QsZq7u3gZNFfr3+UxHw697iletePGXGO9Pm4xrV+FXT2Gq9gaXpDWpq2KFQ0L70Uu6h3CP4z3QFACd9TYQdTXnCFckhtcFwg3Xd4aad/nLWq+0XlM/fXTmjrFsvlhZQR3dNmAesSWHYiArNv7jVw0ZgZ4J7vTVRugvgKv9RyDeBbw13b4u2ZQy68oi0hSI84R+E0W77hYUfawJpDFej6CSGDzloYQKgnmM+e2ICrfws5ZiLqippPntE7plCoqi4KykN7+D/dlnq50YL/xSVzL1MPs7OD3GnT33FftB+wlVSgApE13nM1JXCV+BoBcr4pGLOSJcDGnp9YIgozD9M3yTL87UXSn7BJ3AO3R2z5fJZrE0x/iI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet Needed to avoid a new circular dependency with the memory allocation profiling series. Naturally, a whole bunch of files needed to include vmalloc.h that were previously getting it implicitly. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Reviewed-by: Pasha Tatashin --- include/asm-generic/io.h | 1 - 1 file changed, 1 deletion(-) diff --git a/include/asm-generic/io.h b/include/asm-generic/io.h index bac63e874c7b..c27313414a82 100644 --- a/include/asm-generic/io.h +++ b/include/asm-generic/io.h @@ -991,7 +991,6 @@ static inline void iowrite64_rep(volatile void __iomem *addr, #ifdef __KERNEL__ -#include #define __io_virt(x) ((void __force *)(x)) /* From patchwork Wed Mar 6 18:24:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584413 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03651C54E41 for ; Wed, 6 Mar 2024 18:24:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D96A96B0083; Wed, 6 Mar 2024 13:24:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D20F16B0085; Wed, 6 Mar 2024 13:24:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B4C486B0088; Wed, 6 Mar 2024 13:24:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 989996B0083 for ; Wed, 6 Mar 2024 13:24:54 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 2F91116074C for ; Wed, 6 Mar 2024 18:24:54 +0000 (UTC) X-FDA: 81867440508.15.DDACD19 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf18.hostedemail.com (Postfix) with ESMTP id 47C581C0007 for ; Wed, 6 Mar 2024 18:24:52 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=KB3j2cgW; spf=pass (imf18.hostedemail.com: domain of 387ToZQYKCDYkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=387ToZQYKCDYkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749492; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NeN+jv49ztz1DkBAYeWMzQyB4IxgpHbFtvet8rPjepw=; b=JKrjygA/URbnzpy5xn654UNpbHOdPal4Ag3WK0AYBwq1NTmMynXm9j3g7zSl1cAswRi7nH +QjnhP/46UhlXe2Tpa5nnZpDbnSd5KcVG2ITocXpLQbjqAumM5FAy9QVX9Ty2STiQL1EN2 5/6vJitaRQbJ5LkTfd6J7EMrWFAq9UI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749492; a=rsa-sha256; cv=none; b=7TvhgJTpcuU6zXVv3XSO18we/PVpgo/YuY6wkMFfmCy5Ap+VkBGrqlQd6KwbJu4/icpRD5 p5y1g77cjSamBAS8gR/Zu/AIppLlyI4tLEATg8rU5u1yw+olC+8+Bgi4DGhmJJiMQhRFz1 9kutZBGoUgZNNf0gZxPaJ4qjQdHjvMw= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=KB3j2cgW; spf=pass (imf18.hostedemail.com: domain of 387ToZQYKCDYkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=387ToZQYKCDYkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dccc49ef73eso12358149276.2 for ; Wed, 06 Mar 2024 10:24:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749491; x=1710354291; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NeN+jv49ztz1DkBAYeWMzQyB4IxgpHbFtvet8rPjepw=; b=KB3j2cgWHeaHxAoXgscKQgNE4MySRD441SU/HFnBvhpriW+uCkzortazp0xZgCBGBS AZZOsbxbujVfsdUZ3WWgM7szyAv4RU4H6DzLoO4GIYloImAthp2mpirmV2COKAFZA4Fb SpUlbaNJ218FKPKp6Gk9GjVJrLa7AX6t1RotH88YQW+qrS1XjpaKgGlllGQhNtFFUPuU XynGWePwWKo/YeQQp/ewdId1qUEMW6QlDIEratVZDljPr7WLZ3mYX3PpJDXoXkTK6Vbh 99Cq6jFcFDc5xPt79ZCVySncxYpeKeOHOOT0oxC+AjpD6JMF6gClHDey6LZh3nL/RNga BUFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749491; x=1710354291; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NeN+jv49ztz1DkBAYeWMzQyB4IxgpHbFtvet8rPjepw=; b=NCi8X12YHvmDXGV1LydarWB31ONCl9wDTrkK/L2kN2CoY/Iy5xutd3ZfaI15H0YVDN SsOurOl6ZwvJBjbEqwMYezuVnRWuroEsHYFndMsRoNw5MOO67n3Bv0S5/Usnwf1tkYAK gey0/4lejy77HVduVxTohyG08HimNLrmNomrluRTTqVkx4FhAgCbNMJcbSI3SncnciEg 0A+CFxI+7D7PA1ZGs75YqM0gxc88dW4G7kUSKNNvCHKFyCnSKLJ7M/fvHwBFMQlFcbuv PjAbRde4m2FrbheVVMK9bVjxycggBxGJc5zQ+PLehyUJfxHZefX94qhk1gOz/19OAibe 0X2g== X-Forwarded-Encrypted: i=1; AJvYcCWAy+4nmCfVqdZH1Xlfasz//LkM5fh/YfN0JW9xAgJja5pwVa9Kl6EpLLVix00w4aXrWak6+iboRusMoHdEG9JOhE8= X-Gm-Message-State: AOJu0Yz+oe/Z0gPDAB99SCURRvKL/D2NcsnNEQVOYtVbf2CGJbTqFobj Hl9bm0d5a1j17FwBALbGbOGF33BBFUlmOJaawFu5eqmv7HFx0wZbq1qACA8TPlWQaB9ElmrEPno HDA== X-Google-Smtp-Source: AGHT+IEhO5aEi5nmrgTX3yTDU5CUyz1hSmpU3f2fCr4To3c82DT/ejxK9TArxLseWogn2XsiLsKy5eHr87g= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:f0d:b0:dcd:4286:4498 with SMTP id et13-20020a0569020f0d00b00dcd42864498mr559510ybb.6.1709749491385; Wed, 06 Mar 2024 10:24:51 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:01 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-4-surenb@google.com> Subject: [PATCH v5 03/37] mm/slub: Mark slab_free_freelist_hook() __always_inline From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: buub513t7ypag96f6rg8w4148wttxuzd X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 47C581C0007 X-Rspam-User: X-HE-Tag: 1709749492-516283 X-HE-Meta: U2FsdGVkX1+2+F0CqrbSzZpZjCB0b2APnTxA6kxe/9pDABgLV1azdAWbZ/4jwvrE+1Ms0warn4HYImncR5+8kpHe5Y3qMA2WcE+WL5LP+/JkTYu82JUS7ODHtCxUyeOfMshCpWtosRbdEg5NzeSzByrXCc7/8CJrqdUZSTPKTb26YQkOpy29uo55hEtW9CLGf9VBV/ebLYn/KQlY+lCaPoTHLSDrbk6gJAMZzUO5AvqoDO7kAPHwY6UwoFIQo8EW2Ofa316fmvl2aAdv7wQJyNkXpevjew+t1FJ8eF1Whtao1HDBEbAjOkTVfRjepJ0wa2nJZ0t/hgj1Mx3uJSnbIpBBpl40PyDr5adKkQC2wNlXcfVEGIzKZ2R7iyWVSnlkSJSBdUnEH+iBadaiyg3XiCzhl7m/G0jXCVgfxTiLfioobdCofGd0MLQNQjtjPcCrwfvhJTk3KoXyBM/xLlc/5F1gNx73Ke/bjf6bbY6BVDCiysaS8vUPiPKLqjrAsYp1bcQJW2X0/wRFthNcgrmsivKaHV0xB7ycREGUfTyafv1+L2yt8PeJFGKTs6SLai3KqDbgdzm83UlcY6WXEgWrdhYT1uMpPmQy3X6vnkm5k8zy8ELpSO3hTwcVAAVTSHS0PgiOWHOqlbDWVjhPxFgHg9b2fSai0SS2UtzHSnv8xRefl4EZlZUbT0coEiFryitG0kImtuDiGYQIA8w/AxMLjOuGvlpTvLoRK9TgzptyF1lrbUXZkTV5v8X5y3dC4FRBrK3sy7kTEyudv+j6SbK/+s4eBTo+A2mWh/Od/JKPNlk5YADqjBP73FqRzRFelcWXc/x3+AS4KpzFl2aT02PzOAx7qlxLqB4h+6F/3ZFD9dL/EiMQT2yk3P4w108XCsdBH6UXIKxM8rlRH6ppG44dqw+umlkEGnC0vlwztRCKco1Q9oK9sXY9wm1DDsy04+o7cHGa+rnJsIslIdY5Azf 6tldx8Ve H7EYxVJugBQVod4dbEfwoau04q0rzyXZxrG4YYj8SSbFXUdJlJ7kj0w4zV1Uy+7KEAP5WLvSgq4pozder7DzQad/3cRTbQ5oeJZBqztPN5HGgHli0MRGqrxytLZXVVLIbMWuXanROewQfs7UyqGeFCC1Ll9L9+5+WAO3amweYHKmiKkAHAYYgppIBWU7hF2EC/uvV91QO2nhy3Dp+fTpEYnXgiShv3iCCrZJocj9lrJhw2uxNr3MDRNmRGn4h7akHyAy6IUkYVwV0ik+2ChCMNLZOMXSq89gnlZmujWqS/mr6PG/ibI8qzIaixZdyBISuCpM75rw2KXx9oTzwaLCHi4YgFY2A5fAXP6mo2latQbr7kO5WGbbTMo7/Ch5KaOF/m+5GAX9Zv0/3sZvZjCpkVDIjJh9kO5KVVNjEDyYwkyQ9d8f1i5fl4acNW0NlViJQyxsAswCHgjH7UPauf1zVOpBEVNyF+CeV/hVWTAPU1lz3G4k6F9BOouZKqE1/ilA+08skK29auMcgK5calxvPVie4OnjwKZe7th3Pto8YJhNEYyHf5xEslkX2I92YlJeaMhZxl1efFYVL+Z1WjwwtX30JAFXFyANy8KLPOAOqQq1DrQxkNNB14OCZC/9ByD9vE4vZl8eVaMBLwz+sjtpCIAr91JQuDAFlw6/8/BAKuh6zm9Jt5vAVr/L150e5VaLOxbruHCddwVkHzERAiKwzgL6H7RbAaMi4M4ZefIuODTEg0VOIM56SyvZySAwr+R1hdzOJR6387vOA9HCqX67bWP7MYssnO5bLoJ7zNn3Dov8S2FwP3Q81Q8YaZhmm3cYdfYk075Q4/FJmxwNO6pY9M2vcRw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet It seems we need to be more forceful with the compiler on this one. This is done for performance reasons only. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook Reviewed-by: Pasha Tatashin Reviewed-by: Vlastimil Babka --- mm/slub.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 2ef88bbf56a3..0f3369f6188b 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2121,9 +2121,9 @@ bool slab_free_hook(struct kmem_cache *s, void *x, bool init) return !kasan_slab_free(s, x, init); } -static inline bool slab_free_freelist_hook(struct kmem_cache *s, - void **head, void **tail, - int *cnt) +static __fastpath_inline +bool slab_free_freelist_hook(struct kmem_cache *s, void **head, void **tail, + int *cnt) { void *object; From patchwork Wed Mar 6 18:24:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584414 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0FE8C54E41 for ; Wed, 6 Mar 2024 18:24:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0EDDD6B0089; Wed, 6 Mar 2024 13:24:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 09C356B008C; Wed, 6 Mar 2024 13:24:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E7E8A6B008A; Wed, 6 Mar 2024 13:24:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B98BC6B0088 for ; Wed, 6 Mar 2024 13:24:56 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 783A8A0F35 for ; Wed, 6 Mar 2024 18:24:56 +0000 (UTC) X-FDA: 81867440592.26.110E182 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf24.hostedemail.com (Postfix) with ESMTP id 9E74A180028 for ; Wed, 6 Mar 2024 18:24:54 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=EIQIpxhQ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 39bToZQYKCDgmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=39bToZQYKCDgmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749494; a=rsa-sha256; cv=none; b=emccuJk29vyAeEtkMB2p0G0Shj+UWD4QYd0NxGwH7oKJWSAsjrT3COCSjVgq7It6J4WuPm 5/yxPWYStdEL3e/N2FN4do6NlRyMu5M6Zk2LkjJYpFqCi2Pz4u0Li1T4tsNNg0GI5DcLZQ GclEy1c7/kjKo2pkYcxBD5ljkYnrAxM= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=EIQIpxhQ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 39bToZQYKCDgmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=39bToZQYKCDgmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749494; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5nTBv3DNXNZIxIc1ZBoC1bDpLfB6IB4LOR7eT5PZErs=; b=vd03I5ZlOsetMFYm5vrhANkFg4cU9eyQCoEobgLP2bgJyA3zbNP2r9MVfRFt+iaw8rjmIq R54xEygsErdMzx1uu4IvdmK1+C9DFU4QXO+BnY8Gp35D6tQLGwtYXP3E9M+UL5pe3wb4ZR 6O7MsKdohF3ivTtpzsDcR/TPJC56Y+E= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60987370f06so415947b3.1 for ; Wed, 06 Mar 2024 10:24:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749494; x=1710354294; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5nTBv3DNXNZIxIc1ZBoC1bDpLfB6IB4LOR7eT5PZErs=; b=EIQIpxhQhdxUT8Gzf7BDTg/AeJlMrpL+Sd89mzrsd6UcTquwP0K8yvTzNxjRY90OhF 9MKWVBWb9Iay+tVtDP3YCobPPRVMSMZcMO6LvLu0q/HkymnoVRNhxfCaTR35gIbX3KpZ O/IYS/oOhIaueIRWe9KnYSO98Ix153dBZ7upUgkHjwSr+prcF3R0ZIgm0WYnKcMterWJ S09ptPOZspBppILiqthktwq1Ml/9JF21S8zGIT1tzF2qm/SDhGZqKZzQZtnQahdLA/EE 99jTMGcY3aQewg38BdeH7MWAyGr2g2lRlKl9vjug49lQK+cNLG939xRyAbYA2hlRWwmZ V10w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749494; x=1710354294; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5nTBv3DNXNZIxIc1ZBoC1bDpLfB6IB4LOR7eT5PZErs=; b=iOEayNBia2tH8ZEk6x0RP0i42gYJSmntVmTB2paeo60znOND9XBOUyJF42GSWnsjeK BepZ4XklSeXODSX1+jFaoaqZcLiFxerC/KWXguduRq4UMoAKwQSGaECqq30M4Ft/OH6k VhtXWzH5+1LQYU821faoNGbMLG0V/HziRCjkwuXK/mhBqb2NVbcPskSuC2nXQmD/dk80 QcOrmUA1IZ5MuRkxMEzZ+H/0smAQCQe1Sr6ewxOvF7jVJSu/VrEJjiYGiISgvjSezFNF FOaJ1cyd/l0zTsvcZp1nOAOR9G0LA+U6Doiyi0q2Cm+Amu/OiI2OjMFpdQOmLzwrX0gj 6uBw== X-Forwarded-Encrypted: i=1; AJvYcCXHW6cIHo0x3UHQItcEohz3d2vqBwDO8xa4xnTmHi78lUK9fC0ghnzg0Ud3Esv1/is0ss3dVMO83fL8ExVSOLmYVQE= X-Gm-Message-State: AOJu0YwQ5jOj+luj0Xr4/vgIeIoXWEP9zenXrQ6ZdWdoCfGbAi0hXQCk HnCcBRGJjhvrRiLOif82NJLb3eoPG/2mMoNv7HJc4PadcDXlMXFM6rW6S/KkudEhWqdhxR/fK1s H0A== X-Google-Smtp-Source: AGHT+IG5gxx5etlJS8XEklM9jb6RXbjr2MBzNYyK3K6zvzjzbtTe9XCBybhsxXG5rBRlQj+HJ+M1RmTOlz4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:1009:b0:dbe:387d:a8ef with SMTP id w9-20020a056902100900b00dbe387da8efmr534074ybt.1.1709749493516; Wed, 06 Mar 2024 10:24:53 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:02 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-5-surenb@google.com> Subject: [PATCH v5 04/37] scripts/kallysms: Always include __start and __stop symbols From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 9E74A180028 X-Stat-Signature: wym78od3mm5k1z5don18y7ht68dtwhm4 X-HE-Tag: 1709749494-482506 X-HE-Meta: U2FsdGVkX19xiflTWr0RJi9E6bCwx2jYCTSJ2mhsBrjmlNZzA7hYbTnnzkfBe/XoPmwFKKyrd9NHFmbwhz80pOhoDUjM4ycYHJJ19mORAcq8lIawzk7IQmYM3Vex0t59Pa/WrpF+/CqlFiYS3xMqemiAlwIYvnOOmbPlbc2kmbD2X8cHc5Q1oqMDmfNKxWGtJxYJwDuxFTifxV+fwUwMuDm8OTmAnBAxWhLHDdbVS6OMb4QxOKJJgeXvpvfq7DekQIbziVl4gfJ3kvr6BTlR2qtIaeiOAzSmAaSXTzT3XrKYRSEwriG8UoGIjIr3DbV6TJvo+6cALvTSAloAukM8aNKFLzQC9aPrl2/1ABmlN5zNp2Zh4K/gt6HJ7txnNyXVsmUL5epiuev6lowqNgE/5pUZjAI7IoCXKonSHdCAry+nefRvxMZE+EeFB29pWRmg+9GyH6a9q2GqJdylGOJX+Jw+OG9AH6kk+NWQetSVIloOpVSEtQxE/TtxGR8+aG3Yuh6r31Xw+hoJxXLUSU/WgibDARKJft/Rtcwk4cPfn8mOjNfPrhcfHzgodso1+NCKmwg8mVTDgRxciQBJSIuBb/mTzBiCz6UjzA46jKLtJdMrZGrQZQfN26w5n5QrqzGddWeTyjH411ADWSR2SzZA1t1viTf2qfQGUc+NnKFGWqSUfRV+8Um00ejr7jSDQOgKupdFLTx/MyEi96a7fgQumqYU2D9zVWWTCTmH/j5kydr+FH3NlECgFPVJcmOlxHU5T4nEmQIdrye2jkcX0LaysLrfdK/GeT8Uo9yoU3pIU7nLGXuFUJKEyDTJb4AfOjIJVdqS+nP+2I9h8q46k6QmVCP1slRyZxNJX6Bh5ioTHlAEacdNTxYzKvL282K+AvCAEoICkknsTBZ9ceL2w+XZOb+QmrPzlh2WXm3GjyRKbieIM/Bbnk0lUk8hfidCv5ldpdYuxpBMb20VcNrk4Di 6artbGlP bb+BRBC14ApOBudYN50xsvzrZnmfJGRWNWntcQOIPp/2KBWZfV3WgEtZXgR2+y4aHMuWepo7ZnzuOD7CWTQkxMQjfRA81S2SzcTV2bURUxHrCGrEHwCSDymiU+Mb5QRdC3gMB1gfnl4cvlA06bGDMsgoCkz4e0NdvTV8zXeF5TXdG4cnbS+aeV9AnIBaFoLK5tqsbLNFkc1AboRVAAJuXHRJeSEg8i/GilM4ULSzBcCfhRI1aWxH23woeQ/R+9iYfJFyo/TruKtesLTcWj07G3kx2hUs3xVHPKDhUnyJRvAuuGssL81t81d6H2aEfhCsvebwyUC/t/UtEo/LmK5CLAQxJDYGsUv5GPFLYIa834DV6sT0DR2qQ18+GVBqNfQeyotlvlOq683f82iTHpgx3S2kNzEzbP/LwM6083b1fcER3N062/OmkyyfL3Cgnq/IFP2N9RLh5mDzBXRaMKV+c8LeXcZrY7ZZWC9VjFUU39HU3Fk+P4AgYzmO7hq8JdxFsiFnFb+BWA9tpuHZARbi4BeUWj/6wxXsOYFBefUF84kPO5UHQBlB9HywTJKrvEvJTmkxFGGmKW1mWZoNXmXuLIpyrrVAPwsfhRAIzz1Fr4qH0upUj5zEi3Ah+HtaKByvohGGjU6taviIDfIh5vkxlRZZATQkvp9PL4MV4O1M5a1388NJXq+Z2ngMJaqnND7XRgsqOXSzNuXkp3h7zXxZO+PDvjNoovBOnmmEJ1Xj7PL5NebKIpirHPXSAwfI5SED3M7x7hyy/c/Yiga8TXzRCjDqxuel8VCgq0DrJbQ/Fr7PEQn9Q3/X8WJtEBDO6giCy5SLs7vnzAagMmPs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000033, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet These symbols are used to denote section boundaries: by always including them we can unify loading sections from modules with loading built-in sections, which leads to some significant cleanup. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook Reviewed-by: Pasha Tatashin --- scripts/kallsyms.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/scripts/kallsyms.c b/scripts/kallsyms.c index 653b92f6d4c8..47978efe4797 100644 --- a/scripts/kallsyms.c +++ b/scripts/kallsyms.c @@ -204,6 +204,11 @@ static int symbol_in_range(const struct sym_entry *s, return 0; } +static bool string_starts_with(const char *s, const char *prefix) +{ + return strncmp(s, prefix, strlen(prefix)) == 0; +} + static int symbol_valid(const struct sym_entry *s) { const char *name = sym_name(s); @@ -211,6 +216,14 @@ static int symbol_valid(const struct sym_entry *s) /* if --all-symbols is not specified, then symbols outside the text * and inittext sections are discarded */ if (!all_symbols) { + /* + * Symbols starting with __start and __stop are used to denote + * section boundaries, and should always be included: + */ + if (string_starts_with(name, "__start_") || + string_starts_with(name, "__stop_")) + return 1; + if (symbol_in_range(s, text_ranges, ARRAY_SIZE(text_ranges)) == 0) return 0; From patchwork Wed Mar 6 18:24:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584415 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03A02C54E41 for ; Wed, 6 Mar 2024 18:25:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 905A96B008C; Wed, 6 Mar 2024 13:25:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 88B116B0092; Wed, 6 Mar 2024 13:25:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6B7656B0093; Wed, 6 Mar 2024 13:25:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 556256B008C for ; Wed, 6 Mar 2024 13:25:00 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 68EFB1C0E52 for ; Wed, 6 Mar 2024 18:24:59 +0000 (UTC) X-FDA: 81867440718.24.B748197 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf26.hostedemail.com (Postfix) with ESMTP id 95A03140007 for ; Wed, 6 Mar 2024 18:24:56 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=bR8dI68U; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 397ToZQYKCDooqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=397ToZQYKCDooqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749496; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/+1MAxDJuAJQyho1HAnRuoUycvJ4xPriHfVlSiFl1RY=; b=tS918kFR8A9YCszCGUsOnEm18vqMbW7U/q7VUIN3Lp8JywIl8VXzTm6YhHz1HQpxRiNtTq 6gOZiS4dnA1CtKb9i0OYhdJe6SjHQzyUqp8omdQIkZTCDfmOklgcc2tgKfAIP1jIkDxG72 Yu0itcedr8NQ/3KPZCwTVC6wnILZiAI= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=bR8dI68U; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 397ToZQYKCDooqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=397ToZQYKCDooqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749496; a=rsa-sha256; cv=none; b=LiUwIeg6Xj9bffS0AxYKZkgdEdjcVKYbd8+40HwMfBQaRy4W+6UsGj8F1aB+OtiTPgDpQU ApBm6qITpbQSQ3OXGR3qdpWuGSyN8OxwThzlsNVemixx8qh6v515MYV/mWeAMwd65Bnbhy FTjlrfSsu92vbBor7IgkvsJFteBBa3M= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcc4563611cso11102887276.3 for ; Wed, 06 Mar 2024 10:24:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749495; x=1710354295; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/+1MAxDJuAJQyho1HAnRuoUycvJ4xPriHfVlSiFl1RY=; b=bR8dI68UIIwnBXys2pVaHkBEZcDL6ISWq977Oj4N1FSofETt/IjT41yEN3DbtELRx7 PtYyzA/oVpSjqcwAzkYP/hKcAwPATVQrAKS3lj0mVlRYKsN2nJQl6ZYy6ghGatLSnyBx iZ3w7tljOF4V1z56cvl+QtMGVTjI/tracr7kXnRPwOs5ta0IqLooqEirSc/oTq2Iq5MM GQoysOVlb2Zl9dJZKFtwGaOEOyfh1UxBWD5YsjxkKbi39Ty0RyMeODsDfsi01onQ3ujJ 1sNicJtvguJwerVXBo3ZTOMrFg3DJkIkcOt8f/1X5KmyF29SZ/BblUp/xQML/GLSX8sa U9TQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749495; x=1710354295; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/+1MAxDJuAJQyho1HAnRuoUycvJ4xPriHfVlSiFl1RY=; b=Nm4pWNnXBwXsn/PHvSeR/kdBBcoPm73ADsm/9lVgO1KRKX/HA+nq6+dXJ36OE4mRWS AKau83BBSOj0d2iq5rkJZjQL0/f5IyWbK1M/TDG74bcnP96PRs+k4HIZndZ5levu1bmU SJDxBdvLPi9mOT2dFaYISksYqNIjJVfV1q4OL9IZGG8DB40vQZBaKs8yVqPfQNOh9xVB F5+00J7wq5f6+9OyT+nKocV1fxSw63Ra/CVIWLO0+EYY38dvjHzNDcRZl5znNk/xjjrg ssr5vqH2whatBmM2sFbio3TdJiFIClL8yBXNyF2foxMA/+iO2+6Cjh3cYz7ggX/1ArHd IUAg== X-Forwarded-Encrypted: i=1; AJvYcCWzXiUASmc7iDldsh5B/cJPA9d75oVPaybdQgFkQj8+qN+v4wabGWRjB9NyaK/RVlO12KuoxXRWdgI8GdSxs9xD9O0= X-Gm-Message-State: AOJu0YxARQYkPUPv121MhRL/0VvT41d/i9yw+h+zKZz48HpynLGSyO+H +pumYoz8Ue+4YPuLcpxcuaFHEYWFhVjKXS1eRqGoL+4JhTSqK9Yz6gobKYt/C/i19SlN2Oo7pvX oHg== X-Google-Smtp-Source: AGHT+IEkw8OywS5NVn7/NXWRII+MpOBoDjeHrVnBsvvZCv8Bi13QwsyESjkt49rN57nFyogg4BGvGtr41mQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:1744:b0:dcf:6b50:9bd7 with SMTP id bz4-20020a056902174400b00dcf6b509bd7mr3993490ybb.7.1709749495639; Wed, 06 Mar 2024 10:24:55 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:03 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-6-surenb@google.com> Subject: [PATCH v5 05/37] fs: Convert alloc_inode_sb() to a macro From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org, Alexander Viro X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 95A03140007 X-Stat-Signature: gdp6ifwkqf1bid6u43gaq15bt41g7ieb X-Rspam-User: X-HE-Tag: 1709749496-624737 X-HE-Meta: U2FsdGVkX18Uc/Vt/h0A4YSKd5uPGvzr0PUbEJc5p3I50X2rLzKcAl6uI9SglrjB+VzwLZX0jOMbrHDI6hD2xGBGyKRMxj7IQ0FWrkFKNQdiBn5AEwk4pKhEwid0MqIYeAYehiFBZ4ffx1mBWdQMZtI4LEd4MSSSC12RMGApXW5V43b91B5LxLmTUKVEAxF/785+wgfxx8vx2zSFu3Zpv4PAi+P02FNvI2qLiylxEi+JttlfYW0x4CR45sFFoNRq+ps4RkDHAia3IIbwv6ulPOjZe9DmOJEaMavkkHeT0wQ5u9XQRnCefnt+4399XN6FbN4nHQv9cQ0MGrQCg68mH4G/RPGj+9I3tCNe98mubQnVRCnCwVPrQadeHWc7WVTap+MZXQDNq0b65TBEMTSOYnSdmb/z/THK/ffbwxoI4sAWUsE8lOpt+qALsaAmu628erqsvlpfz705fZ4qDVQ/qGIudhPNOu/seZZy9eAV9s0f+vS1GNJEHV8NsCdixhao4lUIfIimufaxLpsQ8Y37ntVbWkgJOYKB7YT1DJyaEmkOBQwPJjgyBT24fyY93epLuxTAGko1iAAP4TWb7x/YPAxugGESG11RbQBqX6KqKrOEJPBpTS4eSFdD3Re6+DZyKgFoNxG+/WupyCP6/uYpjIGbrODOshPTvcA+zwW0PR1rFiyJe19l9hLpKNjNcX7BATg/Kq8i5vkxDDa0Y6gJCMimc7vGXSKk9u1XY6vWjSVZ0H94ANopYi5ZEw4nBlF3ltvHQtIPxqtCTXraCsXpK2ss72fD6FE7yRfrgMY7LpR81pF1TGoSES3dQcXzMFQiL7hYPGN49HPOO+auAYjO33D+Bl81sf32kQQOHb8q3M22JAP1oWDnVs2Gh/qauP/niEiZ2FscK7qY5xD3qijLmXPYz8xnpm06Ck9AN3DXq+ldFpB2LXyGHIKNI5owrvHdy6A+QvptVqAtxE8D5Bn YUy+dinQ Is7wM1hQq/0ZKvCAmxukqIxQNzdem4rWbLwplJ9BIqiAJAWYJKQsShy5DMm3XBr1l/ZDppHWXlbT9/SzuNeijZqTU1SlkmUav+gla3MYEvkwFxegoVLcKM5VbvT6VBT6JDn87zpBGDZfMx7/+KwToxsung2VxT2P348LcQUmTlIAYfQbdAdOknIzfhNwnn3sdMwNzsfenJaipV7RxyAMyAgU47PHwuOPSnp2F+MVDiZPp8h380GHtZrSECRj0tYFr0F7pti+xb2jY91Clm6kt10dDg8rB/OpRYSEYwTLS9VcC5MwQxU+QkzANGhe+2ehGnMsfvrjUQMhHKTtd358wBAyr4CUysCopmVUETiGbIX/cXfgwyRurcuBJfSvX9Z0T58Bcvg2676MV3YjS6v97/6PpTuJMUrnSIQvhkb5sVLzsF2+wbvO7W/UBUuvR2sxgoFFilnV86OfFq6GdKgY1TOIVYg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet We're introducing alloc tagging, which tracks memory allocations by callsite. Converting alloc_inode_sb() to a macro means allocations will be tracked by its caller, which is a bit more useful. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Cc: Alexander Viro Reviewed-by: Kees Cook Reviewed-by: Pasha Tatashin --- include/linux/fs.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index 023f37c60709..08d8246399c3 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -3010,11 +3010,7 @@ int setattr_should_drop_sgid(struct mnt_idmap *idmap, * This must be used for allocating filesystems specific inodes to set * up the inode reclaim context correctly. */ -static inline void * -alloc_inode_sb(struct super_block *sb, struct kmem_cache *cache, gfp_t gfp) -{ - return kmem_cache_alloc_lru(cache, &sb->s_inode_lru, gfp); -} +#define alloc_inode_sb(_sb, _cache, _gfp) kmem_cache_alloc_lru(_cache, &_sb->s_inode_lru, _gfp) extern void __insert_inode_hash(struct inode *, unsigned long hashval); static inline void insert_inode_hash(struct inode *inode) From patchwork Wed Mar 6 18:24:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584416 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A662C5475B for ; Wed, 6 Mar 2024 18:25:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DBE2B6B0092; Wed, 6 Mar 2024 13:25:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D46B56B0093; Wed, 6 Mar 2024 13:25:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B98D96B0095; Wed, 6 Mar 2024 13:25:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 937026B0092 for ; Wed, 6 Mar 2024 13:25:01 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 2848841017 for ; Wed, 6 Mar 2024 18:25:01 +0000 (UTC) X-FDA: 81867440802.10.D40B8AD Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf20.hostedemail.com (Postfix) with ESMTP id EB0A91C000D for ; Wed, 6 Mar 2024 18:24:58 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=xN1082Id; spf=pass (imf20.hostedemail.com: domain of 3-bToZQYKCDwqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3-bToZQYKCDwqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749499; a=rsa-sha256; cv=none; b=DfD2fhif8MKYVo/YBWYhz3uXxxN/0cR8Zd+hxG+DKmkQ1aZocVb2EWb0vo1PPV/aN8EQfK 3FSkrMdO18VELGbHV/zL6DlJE2noCHnvSArPlMLG0iqpDx/AE6nR48gUd+Zd+5VNANqrEJ M+koR7q03h5z8aC48hhUJWxQ/yWAJf0= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=xN1082Id; spf=pass (imf20.hostedemail.com: domain of 3-bToZQYKCDwqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3-bToZQYKCDwqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749499; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6JMcGt6EYvTroht8h7kHXMFTQxZQYrHbNcyZWU/g6ow=; b=Cqj2eCzcXNCo+Pf5HWh0brUF0CHgE362jVPO1BxtTkDKPekpSK7uHvaizsYekArl5Nh75w m7968MhpX4p3OAkVgEaN9aZk5n9LyIz4DiSF2oLWl7hbKtzdELsemVITf6d4vExDa4X89K vk6Z9cD+ypukT5rJ4iF8n1cMtK7zzZs= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-609e2b87a6bso118657b3.3 for ; Wed, 06 Mar 2024 10:24:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749498; x=1710354298; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6JMcGt6EYvTroht8h7kHXMFTQxZQYrHbNcyZWU/g6ow=; b=xN1082Id1rIqV3lf9gBPFUTyL/O/+9KlZORiDquNfhwSNl2bx8M1HgwKg+/2OAkN13 6nb31xWUprqpTCmpD1RI2OjlmjoBA75SAxuFaRSJCqyfeyet8RwTb4C8imEAhPLnoJ9P VGpTd60odh0R44lY3M3PGT+tn0wtOAlFHsRbJ64F8xvSMvFhAlMbMSOSa2gFwUcqZrK7 t+VP11PGlXZBJTTii5k+FbSrM3XbS9xWhGZyE5L8yLcAIuPTjHYgSUc10rMhATgdoY5/ hg8Omxmef7t3aMdWoS/d1NAbBYXtWvxe4V1QUfy4A7FMi+bhGi74ZW8HR+CxGJZREtgp AMLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749498; x=1710354298; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6JMcGt6EYvTroht8h7kHXMFTQxZQYrHbNcyZWU/g6ow=; b=PiPlrVKMQA/FCNcKSvATtGUzH3IyO+q6GkkNRdX36vNkEp/XIjyrkzChdxOeTi1XNs T1V583QnAv2+4ERSRhSG76loQzVMkAIkuU2Dj6zC4rlu9vh+HstktJYhKQRTk4Ici7am wZSyOHw1VsUFVhqQZf7JJtmYoeyg02ASAySBsqtuHcIeqpTxiipDDaRzJLEAtE34T8t6 EHSsGoCu62CYH58D8IdBhs86BfzmfedHwuI5pmGZBMCa2cED8hP7NiJ46D7O3LLMqr/l /KAjdz24Ng3k7X8plqN4geRSr+5elZKYFVl9tFPuGNoRBCs4CAE0UeTX43+zJnb3/3hG hGUA== X-Forwarded-Encrypted: i=1; AJvYcCXasuDyjDMXgJJ6rpSyCyJLLJbi6bbejxlaH2z8+W/xyWHI6lOKK5+f1flRLUZz9XSpaIBSMgKbUnmgsqFkaUWzi/A= X-Gm-Message-State: AOJu0Yy1U/ai37yDGBygw2GMFIS1onCsMhjNIhIj1fEzpY0KvJv4s/kS 3u6jFhaICnBZ32Ve17/JyYDW+nZTvin1HNidxNV6VzxVSKC4JsTXu6eG/gmPAYXXp9UUNHzJJc7 R1g== X-Google-Smtp-Source: AGHT+IFmVfhA7vPBjAOd1wle9uwLugrDjdSWKUX3JS43rdSMKEljpI20cubFGlFPWCXFgrjwSeAgsiZP1KU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:722:b0:dcd:875:4c40 with SMTP id l2-20020a056902072200b00dcd08754c40mr4233991ybt.10.1709749497766; Wed, 06 Mar 2024 10:24:57 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:04 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-7-surenb@google.com> Subject: [PATCH v5 06/37] mm: introduce slabobj_ext to support slab object extensions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: EB0A91C000D X-Stat-Signature: ich7qzzsqs6brpgc7i7ifu8g3zxzkgrj X-Rspam-User: X-HE-Tag: 1709749498-346649 X-HE-Meta: U2FsdGVkX186b2mEXusjSy2TXRbGmlIPgjSA6nsTznTARyC8EkJSiQYIKBXWHJlfvTr8l45gPwLQEPRUBWI1Y1RQuOfiYKYcO8aFqSy2X9P073mbm3A9PbVvrf3vN8ihN9eD9tAp0wSGWHNPntWZ87iXJjqZJZk7pvWH114yMBc6ya+DG642dvhyvV5ACLLy5DXuhHluyAnCDJ4++wp+BNUJ2ZVz46KcmUy2qFVElvq4A9Hbi6t03zQitvaQPyNzfUu29YGT3ltu44SZ5o3jM2DMuOOAuz39iUi+gc/7T3zUibycrK4umwpwbNTwHPKkroeOL0L3W0g5y6aP5HtYKfsVZ9XdgIRBCQaIRl7H+Whrsz6IlJ+GtIO5SeZh5lieSOVEB2eJgrpNgtIr18XrFW1l27h9IsPeyN+0GCx/qYXD0PM5wK4g6yiSJTEVKpnrW8qR2fJvyzLuhhZ4SjIpoBhTuwxjuThKFDXWTXmIMRf+q4MCddidumtDRoZS2T+S+L4NeljS8qzY45zSaJgRBuRzTwnAzVxz873NPwokxDwE0XGzBMWZw4nTHB8TEpcY4YnjRxsoFpdC9sjwysBDDly60rpKGE3VnHvOjvNrPz83WkrScKowfIUG3qVuYEs55nb/+bEMhl38H0njpQbq88RP6segAdzLRSztCVIm/TTovxWIacHlXl8ZVe0e2M8Lo5PyKFQT5MNYFCqZMrGGTqcZ6/1/xqe43Z+0z1uLo2sOHKx7EbjNRWN4LHRlzhYj6y6IuG3WbYcnNLGp4/skKHI2NZFxFJZ+n3DAfR7XoYHhYRr0KcQIOt0lnFS51fEz4x++UCY+5H2QlqPCrCHpuC+g46ngtDT3XHA+gEMIuSaeGUlGdBSdWGsW2GW2Ixx92B9OaF2ntBtnnYMR483axXo5bLpA1MLQszeyxDYe2SK/HfQSBFQi1DMMT5PKhHahVUd29GKsxWTW4ibauXS J71E2foj ZB7VcbUmkXiDAP3aDYPAbQQAPHtZVuKP5doD3PxBCJ06/kGyQrT2H1cwtFIG+ojO3ep7RtHN8yjlDJFy7wWJWZaEtuUTdjRcnkFgNMhmBsayHCdVBVbvZkK9/qmgUvk5WVjZZqhyJFAccAktImClMFbXivldGIiyery+ojwlPtB5u8RrxUwX+xyGZeeVwzrLScjAdiv1kInmgu6BXkE2Ykqc8pmx1Mdlpn/nFIagele+jZE2WSkKG3PpZnR0fgu6IMijlgWKnrHRZS/i9gYhBmRQatE1xAuLAUhwNUJlPXaMJnLQlno5xLJHHZqhdzNNCzSsDjUsQDhBzd4Xkrcf2VSj+PotyTFuDoS836IY0m2lwwivhFs67v19LYGa8C9w1gjQaLouopcGrfHQ4+Qb8zWwPp025bPDAf7RhSiKQzEfg1LjHqVwLeA7YXB50vpAOA8GMeF2xdWNk9/VH9i6Uhf1d0k4C+4NEFvYLXMl4U3ZaH6ycQSW/rHP/xx5TuN16eM48ChiPGEdUv1Zxj9/xdWz+KFq1/qz3zOmpIp0ZYcynD9GhO/YjKoV+k0ii1hIVkQNJq+tMjhCzFn46m0pIv101n/gke/f6pbEgWya3RyNhmrtKYyEaqPNelp4dadFWbPBalzUS9jhqC5d9k2hRehN52AdkeVWZjiJr2tw+q83pZkiNxO3VdxLiVyfwFcPi7JX5Tr7OHfmzyTFjMzQdrlJRHeucGNE8TQns6xCsBeJUa+lzvPsQmkuGeuYh5VhqBeXQ0YncNr0hslaZzwAkxhnnqA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently slab pages can store only vectors of obj_cgroup pointers in page->memcg_data. Introduce slabobj_ext structure to allow more data to be stored for each slab object. Wrap obj_cgroup into slabobj_ext to support current functionality while allowing to extend slabobj_ext in the future. Signed-off-by: Suren Baghdasaryan Reviewed-by: Pasha Tatashin Reviewed-by: Vlastimil Babka --- include/linux/memcontrol.h | 20 ++++--- include/linux/mm_types.h | 4 +- init/Kconfig | 4 ++ mm/kfence/core.c | 14 ++--- mm/kfence/kfence.h | 4 +- mm/memcontrol.c | 56 +++----------------- mm/page_owner.c | 2 +- mm/slab.h | 52 +++++++++--------- mm/slub.c | 106 ++++++++++++++++++++++++++++--------- 9 files changed, 145 insertions(+), 117 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 394fd0a887ae..9a731523000d 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -349,8 +349,8 @@ struct mem_cgroup { extern struct mem_cgroup *root_mem_cgroup; enum page_memcg_data_flags { - /* page->memcg_data is a pointer to an objcgs vector */ - MEMCG_DATA_OBJCGS = (1UL << 0), + /* page->memcg_data is a pointer to an slabobj_ext vector */ + MEMCG_DATA_OBJEXTS = (1UL << 0), /* page has been accounted as a non-slab kernel page */ MEMCG_DATA_KMEM = (1UL << 1), /* the next bit after the last actual flag */ @@ -388,7 +388,7 @@ static inline struct mem_cgroup *__folio_memcg(struct folio *folio) unsigned long memcg_data = folio->memcg_data; VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); - VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJCGS, folio); + VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_KMEM, folio); return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); @@ -409,7 +409,7 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio) unsigned long memcg_data = folio->memcg_data; VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); - VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJCGS, folio); + VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(!(memcg_data & MEMCG_DATA_KMEM), folio); return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); @@ -506,7 +506,7 @@ static inline struct mem_cgroup *folio_memcg_check(struct folio *folio) */ unsigned long memcg_data = READ_ONCE(folio->memcg_data); - if (memcg_data & MEMCG_DATA_OBJCGS) + if (memcg_data & MEMCG_DATA_OBJEXTS) return NULL; if (memcg_data & MEMCG_DATA_KMEM) { @@ -552,7 +552,7 @@ static inline struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgroup *ob static inline bool folio_memcg_kmem(struct folio *folio) { VM_BUG_ON_PGFLAGS(PageTail(&folio->page), &folio->page); - VM_BUG_ON_FOLIO(folio->memcg_data & MEMCG_DATA_OBJCGS, folio); + VM_BUG_ON_FOLIO(folio->memcg_data & MEMCG_DATA_OBJEXTS, folio); return folio->memcg_data & MEMCG_DATA_KMEM; } @@ -1632,6 +1632,14 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, } #endif /* CONFIG_MEMCG */ +/* + * Extended information for slab objects stored as an array in page->memcg_data + * if MEMCG_DATA_OBJEXTS is set. + */ +struct slabobj_ext { + struct obj_cgroup *objcg; +} __aligned(8); + static inline void __inc_lruvec_kmem_state(void *p, enum node_stat_item idx) { __mod_lruvec_kmem_state(p, idx, 1); diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 5240bd7bca33..4ae4684d1add 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -169,7 +169,7 @@ struct page { /* Usage count. *DO NOT USE DIRECTLY*. See page_ref.h */ atomic_t _refcount; -#ifdef CONFIG_MEMCG +#ifdef CONFIG_SLAB_OBJ_EXT unsigned long memcg_data; #endif @@ -331,7 +331,7 @@ struct folio { }; atomic_t _mapcount; atomic_t _refcount; -#ifdef CONFIG_MEMCG +#ifdef CONFIG_SLAB_OBJ_EXT unsigned long memcg_data; #endif #if defined(WANT_PAGE_VIRTUAL) diff --git a/init/Kconfig b/init/Kconfig index bee58f7468c3..d160ab591308 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -929,6 +929,9 @@ config NUMA_BALANCING_DEFAULT_ENABLED If set, automatic NUMA balancing will be enabled if running on a NUMA machine. +config SLAB_OBJ_EXT + bool + menuconfig CGROUPS bool "Control Group support" select KERNFS @@ -962,6 +965,7 @@ config MEMCG bool "Memory controller" select PAGE_COUNTER select EVENTFD + select SLAB_OBJ_EXT help Provides control over the memory footprint of tasks in a cgroup. diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 8350f5c06f2e..964b8482275b 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -595,9 +595,9 @@ static unsigned long kfence_init_pool(void) continue; __folio_set_slab(slab_folio(slab)); -#ifdef CONFIG_MEMCG - slab->memcg_data = (unsigned long)&kfence_metadata_init[i / 2 - 1].objcg | - MEMCG_DATA_OBJCGS; +#ifdef CONFIG_MEMCG_KMEM + slab->obj_exts = (unsigned long)&kfence_metadata_init[i / 2 - 1].obj_exts | + MEMCG_DATA_OBJEXTS; #endif } @@ -645,8 +645,8 @@ static unsigned long kfence_init_pool(void) if (!i || (i % 2)) continue; -#ifdef CONFIG_MEMCG - slab->memcg_data = 0; +#ifdef CONFIG_MEMCG_KMEM + slab->obj_exts = 0; #endif __folio_clear_slab(slab_folio(slab)); } @@ -1139,8 +1139,8 @@ void __kfence_free(void *addr) { struct kfence_metadata *meta = addr_to_metadata((unsigned long)addr); -#ifdef CONFIG_MEMCG - KFENCE_WARN_ON(meta->objcg); +#ifdef CONFIG_MEMCG_KMEM + KFENCE_WARN_ON(meta->obj_exts.objcg); #endif /* * If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing diff --git a/mm/kfence/kfence.h b/mm/kfence/kfence.h index f46fbb03062b..084f5f36e8e7 100644 --- a/mm/kfence/kfence.h +++ b/mm/kfence/kfence.h @@ -97,8 +97,8 @@ struct kfence_metadata { struct kfence_track free_track; /* For updating alloc_covered on frees. */ u32 alloc_stack_hash; -#ifdef CONFIG_MEMCG - struct obj_cgroup *objcg; +#ifdef CONFIG_MEMCG_KMEM + struct slabobj_ext obj_exts; #endif }; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 138bcfa18234..f99c63d5e9f2 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2980,13 +2980,6 @@ void mem_cgroup_commit_charge(struct folio *folio, struct mem_cgroup *memcg) } #ifdef CONFIG_MEMCG_KMEM -/* - * The allocated objcg pointers array is not accounted directly. - * Moreover, it should not come from DMA buffer and is not readily - * reclaimable. So those GFP bits should be masked off. - */ -#define OBJCGS_CLEAR_MASK (__GFP_DMA | __GFP_RECLAIMABLE | \ - __GFP_ACCOUNT | __GFP_NOFAIL) /* * mod_objcg_mlstate() may be called with irq enabled, so @@ -3006,62 +2999,27 @@ static inline void mod_objcg_mlstate(struct obj_cgroup *objcg, rcu_read_unlock(); } -int memcg_alloc_slab_cgroups(struct slab *slab, struct kmem_cache *s, - gfp_t gfp, bool new_slab) -{ - unsigned int objects = objs_per_slab(s, slab); - unsigned long memcg_data; - void *vec; - - gfp &= ~OBJCGS_CLEAR_MASK; - vec = kcalloc_node(objects, sizeof(struct obj_cgroup *), gfp, - slab_nid(slab)); - if (!vec) - return -ENOMEM; - - memcg_data = (unsigned long) vec | MEMCG_DATA_OBJCGS; - if (new_slab) { - /* - * If the slab is brand new and nobody can yet access its - * memcg_data, no synchronization is required and memcg_data can - * be simply assigned. - */ - slab->memcg_data = memcg_data; - } else if (cmpxchg(&slab->memcg_data, 0, memcg_data)) { - /* - * If the slab is already in use, somebody can allocate and - * assign obj_cgroups in parallel. In this case the existing - * objcg vector should be reused. - */ - kfree(vec); - return 0; - } - - kmemleak_not_leak(vec); - return 0; -} - static __always_inline struct mem_cgroup *mem_cgroup_from_obj_folio(struct folio *folio, void *p) { /* * Slab objects are accounted individually, not per-page. * Memcg membership data for each individual object is saved in - * slab->memcg_data. + * slab->obj_exts. */ if (folio_test_slab(folio)) { - struct obj_cgroup **objcgs; + struct slabobj_ext *obj_exts; struct slab *slab; unsigned int off; slab = folio_slab(folio); - objcgs = slab_objcgs(slab); - if (!objcgs) + obj_exts = slab_obj_exts(slab); + if (!obj_exts) return NULL; off = obj_to_index(slab->slab_cache, slab, p); - if (objcgs[off]) - return obj_cgroup_memcg(objcgs[off]); + if (obj_exts[off].objcg) + return obj_cgroup_memcg(obj_exts[off].objcg); return NULL; } @@ -3069,7 +3027,7 @@ struct mem_cgroup *mem_cgroup_from_obj_folio(struct folio *folio, void *p) /* * folio_memcg_check() is used here, because in theory we can encounter * a folio where the slab flag has been cleared already, but - * slab->memcg_data has not been freed yet + * slab->obj_exts has not been freed yet * folio_memcg_check() will guarantee that a proper memory * cgroup pointer or NULL will be returned. */ diff --git a/mm/page_owner.c b/mm/page_owner.c index 033e349f6479..d21d00a04e98 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -469,7 +469,7 @@ static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret, if (!memcg_data) goto out_unlock; - if (memcg_data & MEMCG_DATA_OBJCGS) + if (memcg_data & MEMCG_DATA_OBJEXTS) ret += scnprintf(kbuf + ret, count - ret, "Slab cache page\n"); diff --git a/mm/slab.h b/mm/slab.h index 54deeb0428c6..0e61a5834c5f 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -87,8 +87,8 @@ struct slab { unsigned int __unused; atomic_t __page_refcount; -#ifdef CONFIG_MEMCG - unsigned long memcg_data; +#ifdef CONFIG_SLAB_OBJ_EXT + unsigned long obj_exts; #endif }; @@ -97,8 +97,8 @@ struct slab { SLAB_MATCH(flags, __page_flags); SLAB_MATCH(compound_head, slab_cache); /* Ensure bit 0 is clear */ SLAB_MATCH(_refcount, __page_refcount); -#ifdef CONFIG_MEMCG -SLAB_MATCH(memcg_data, memcg_data); +#ifdef CONFIG_SLAB_OBJ_EXT +SLAB_MATCH(memcg_data, obj_exts); #endif #undef SLAB_MATCH static_assert(sizeof(struct slab) <= sizeof(struct page)); @@ -541,42 +541,44 @@ static inline bool kmem_cache_debug_flags(struct kmem_cache *s, slab_flags_t fla return false; } -#ifdef CONFIG_MEMCG_KMEM +#ifdef CONFIG_SLAB_OBJ_EXT + /* - * slab_objcgs - get the object cgroups vector associated with a slab + * slab_obj_exts - get the pointer to the slab object extension vector + * associated with a slab. * @slab: a pointer to the slab struct * - * Returns a pointer to the object cgroups vector associated with the slab, + * Returns a pointer to the object extension vector associated with the slab, * or NULL if no such vector has been associated yet. */ -static inline struct obj_cgroup **slab_objcgs(struct slab *slab) +static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) { - unsigned long memcg_data = READ_ONCE(slab->memcg_data); + unsigned long obj_exts = READ_ONCE(slab->obj_exts); - VM_BUG_ON_PAGE(memcg_data && !(memcg_data & MEMCG_DATA_OBJCGS), +#ifdef CONFIG_MEMCG + VM_BUG_ON_PAGE(obj_exts && !(obj_exts & MEMCG_DATA_OBJEXTS), slab_page(slab)); - VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_KMEM, slab_page(slab)); + VM_BUG_ON_PAGE(obj_exts & MEMCG_DATA_KMEM, slab_page(slab)); - return (struct obj_cgroup **)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct slabobj_ext *)(obj_exts & ~MEMCG_DATA_FLAGS_MASK); +#else + return (struct slabobj_ext *)obj_exts; +#endif } -int memcg_alloc_slab_cgroups(struct slab *slab, struct kmem_cache *s, - gfp_t gfp, bool new_slab); -void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, - enum node_stat_item idx, int nr); -#else /* CONFIG_MEMCG_KMEM */ -static inline struct obj_cgroup **slab_objcgs(struct slab *slab) +#else /* CONFIG_SLAB_OBJ_EXT */ + +static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) { return NULL; } -static inline int memcg_alloc_slab_cgroups(struct slab *slab, - struct kmem_cache *s, gfp_t gfp, - bool new_slab) -{ - return 0; -} -#endif /* CONFIG_MEMCG_KMEM */ +#endif /* CONFIG_SLAB_OBJ_EXT */ + +#ifdef CONFIG_MEMCG_KMEM +void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, + enum node_stat_item idx, int nr); +#endif size_t __ksize(const void *objp); diff --git a/mm/slub.c b/mm/slub.c index 0f3369f6188b..6ab9f8f38ac5 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1881,13 +1881,78 @@ static inline enum node_stat_item cache_vmstat_idx(struct kmem_cache *s) NR_SLAB_RECLAIMABLE_B : NR_SLAB_UNRECLAIMABLE_B; } -#ifdef CONFIG_MEMCG_KMEM -static inline void memcg_free_slab_cgroups(struct slab *slab) +#ifdef CONFIG_SLAB_OBJ_EXT + +/* + * The allocated objcg pointers array is not accounted directly. + * Moreover, it should not come from DMA buffer and is not readily + * reclaimable. So those GFP bits should be masked off. + */ +#define OBJCGS_CLEAR_MASK (__GFP_DMA | __GFP_RECLAIMABLE | \ + __GFP_ACCOUNT | __GFP_NOFAIL) + +static int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, + gfp_t gfp, bool new_slab) { - kfree(slab_objcgs(slab)); - slab->memcg_data = 0; + unsigned int objects = objs_per_slab(s, slab); + unsigned long obj_exts; + void *vec; + + gfp &= ~OBJCGS_CLEAR_MASK; + vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, + slab_nid(slab)); + if (!vec) + return -ENOMEM; + + obj_exts = (unsigned long)vec; +#ifdef CONFIG_MEMCG + obj_exts |= MEMCG_DATA_OBJEXTS; +#endif + if (new_slab) { + /* + * If the slab is brand new and nobody can yet access its + * obj_exts, no synchronization is required and obj_exts can + * be simply assigned. + */ + slab->obj_exts = obj_exts; + } else if (cmpxchg(&slab->obj_exts, 0, obj_exts)) { + /* + * If the slab is already in use, somebody can allocate and + * assign slabobj_exts in parallel. In this case the existing + * objcg vector should be reused. + */ + kfree(vec); + return 0; + } + + kmemleak_not_leak(vec); + return 0; } +static inline void free_slab_obj_exts(struct slab *slab) +{ + struct slabobj_ext *obj_exts; + + obj_exts = slab_obj_exts(slab); + if (!obj_exts) + return; + + kfree(obj_exts); + slab->obj_exts = 0; +} +#else /* CONFIG_SLAB_OBJ_EXT */ +static int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, + gfp_t gfp, bool new_slab) +{ + return 0; +} + +static inline void free_slab_obj_exts(struct slab *slab) +{ +} +#endif /* CONFIG_SLAB_OBJ_EXT */ + +#ifdef CONFIG_MEMCG_KMEM static inline size_t obj_full_size(struct kmem_cache *s) { /* @@ -1966,15 +2031,15 @@ static void __memcg_slab_post_alloc_hook(struct kmem_cache *s, if (likely(p[i])) { slab = virt_to_slab(p[i]); - if (!slab_objcgs(slab) && - memcg_alloc_slab_cgroups(slab, s, flags, false)) { + if (!slab_obj_exts(slab) && + alloc_slab_obj_exts(slab, s, flags, false)) { obj_cgroup_uncharge(objcg, obj_full_size(s)); continue; } off = obj_to_index(s, slab, p[i]); obj_cgroup_get(objcg); - slab_objcgs(slab)[off] = objcg; + slab_obj_exts(slab)[off].objcg = objcg; mod_objcg_state(objcg, slab_pgdat(slab), cache_vmstat_idx(s), obj_full_size(s)); } else { @@ -1995,18 +2060,18 @@ void memcg_slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, static void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects, - struct obj_cgroup **objcgs) + struct slabobj_ext *obj_exts) { for (int i = 0; i < objects; i++) { struct obj_cgroup *objcg; unsigned int off; off = obj_to_index(s, slab, p[i]); - objcg = objcgs[off]; + objcg = obj_exts[off].objcg; if (!objcg) continue; - objcgs[off] = NULL; + obj_exts[off].objcg = NULL; obj_cgroup_uncharge(objcg, obj_full_size(s)); mod_objcg_state(objcg, slab_pgdat(slab), cache_vmstat_idx(s), -obj_full_size(s)); @@ -2018,16 +2083,16 @@ static __fastpath_inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects) { - struct obj_cgroup **objcgs; + struct slabobj_ext *obj_exts; if (!memcg_kmem_online()) return; - objcgs = slab_objcgs(slab); - if (likely(!objcgs)) + obj_exts = slab_obj_exts(slab); + if (likely(!obj_exts)) return; - __memcg_slab_free_hook(s, slab, p, objects, objcgs); + __memcg_slab_free_hook(s, slab, p, objects, obj_exts); } static inline @@ -2038,15 +2103,6 @@ void memcg_slab_alloc_error_hook(struct kmem_cache *s, int objects, obj_cgroup_uncharge(objcg, objects * obj_full_size(s)); } #else /* CONFIG_MEMCG_KMEM */ -static inline struct mem_cgroup *memcg_from_slab_obj(void *ptr) -{ - return NULL; -} - -static inline void memcg_free_slab_cgroups(struct slab *slab) -{ -} - static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, struct list_lru *lru, struct obj_cgroup **objcgp, @@ -2314,7 +2370,7 @@ static __always_inline void account_slab(struct slab *slab, int order, struct kmem_cache *s, gfp_t gfp) { if (memcg_kmem_online() && (s->flags & SLAB_ACCOUNT)) - memcg_alloc_slab_cgroups(slab, s, gfp, true); + alloc_slab_obj_exts(slab, s, gfp, true); mod_node_page_state(slab_pgdat(slab), cache_vmstat_idx(s), PAGE_SIZE << order); @@ -2324,7 +2380,7 @@ static __always_inline void unaccount_slab(struct slab *slab, int order, struct kmem_cache *s) { if (memcg_kmem_online()) - memcg_free_slab_cgroups(slab); + free_slab_obj_exts(slab); mod_node_page_state(slab_pgdat(slab), cache_vmstat_idx(s), -(PAGE_SIZE << order)); From patchwork Wed Mar 6 18:24:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584417 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24565C5475B for ; Wed, 6 Mar 2024 18:25:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4C01C6B0093; Wed, 6 Mar 2024 13:25:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 41E8A6B0095; Wed, 6 Mar 2024 13:25:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1FAA06B0096; Wed, 6 Mar 2024 13:25:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 09E4A6B0093 for ; Wed, 6 Mar 2024 13:25:03 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id DA9F21C11B5 for ; Wed, 6 Mar 2024 18:25:02 +0000 (UTC) X-FDA: 81867440844.21.198A15E Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf01.hostedemail.com (Postfix) with ESMTP id 10D8940005 for ; Wed, 6 Mar 2024 18:25:00 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="UYFf/gz8"; spf=pass (imf01.hostedemail.com: domain of 3-7ToZQYKCD4xzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3-7ToZQYKCD4xzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749501; a=rsa-sha256; cv=none; b=7nscL/nOcN8tn6Qo1VJpLjhbk/mW88hcRR8/lT8vZPgCScBh0yenrZYk0ytRN3HwdfM61q oVET1QAjeZ0UKeYhkfQ5WYGXc2GCMPDP8o/Nzer8WgbEYR8r7lkr4iWOPg34bgMBWeb+Is gx20r10EdEQEGk1nZa1QD8e+iVrJZN8= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="UYFf/gz8"; spf=pass (imf01.hostedemail.com: domain of 3-7ToZQYKCD4xzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3-7ToZQYKCD4xzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749501; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6rH9mI3mTJcaBoi8NvQC9FpkTZd7d5upxL04jrJdHuE=; b=xa93/cCsE02vfKSyD9YmIY3ZSIXAKNTEqiTWDNJ78nuB03yjLIOYWHBz8csvNpA3omRBpD t7XaC/flZrb6hl3pYKbfyuzlhePM5E0RnFil+q/c73tGbD+5bikuYmtBqS6EeP41Wi8yIO dzn4Wq9tC2LIZmMNMzxSaq1dr6urai0= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc64b659a9cso2344800276.3 for ; Wed, 06 Mar 2024 10:25:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749500; x=1710354300; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6rH9mI3mTJcaBoi8NvQC9FpkTZd7d5upxL04jrJdHuE=; b=UYFf/gz8dWKsmLbdZzs4JLdws5uAdWZHdrP/gU/NOnE9xtQhjm/xtXHibFX80O5LoH QJIORDu+GouyfiBtEEn8bVld9/P+5WiGfSXReA3ti+4Dungb4j84+lF0QcgR3rkN2muK dDdIE2Itbi5sOcMxl+wfFBLklbX6eXCI+mjKb+DS3RTbfsoU2vzHYA5nqRtDnO/vBIFj uZ/GaPWlAgPmZw+iuATXebFEyG6Z+J3fQaRLyI3RMwTcdAonIYrQyHkZm2IO/ZzeKbit CVPqo/UJkl0ea+luX48Ag62MkxrRup11BqXO3gWu1O+496WLXwFczZoKa8o8Z92zj5Oq XFPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749500; x=1710354300; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6rH9mI3mTJcaBoi8NvQC9FpkTZd7d5upxL04jrJdHuE=; b=FB4EzoShdrY8ZYDoU2lTJRFoiYGu9g5BAVTpysdSGqSWwD3MRMJ7P9kDwxemwZ2KCR XSn2RpseW94IDYlFtUlKp+hFbGRoQbsONELg8JFbCkzbfhP+XkicL3msI+NfQJbeC23R OuP8wwQ2ZMFZbsT35butIssOzJtVVHXvEKI7TZbgx4Zm/EXfHIbkBINoIENIEd3HLLmB Di1668n1dZ1lSyJhVTOZ2NfyHPWcO3rSkszDDoc/aR1IHpRjfDOPB0KEa7tVIzyQ/+fh c9nJ2iNuagXva4SM7Xyo42tYemzm8HdexzBW6z1UdTodbBHWG89oYB9Tsne+e8/Fx6rD EZBQ== X-Forwarded-Encrypted: i=1; AJvYcCXWbr75Gz7LYq4vWNJ8QeNME0U+nVh4BLR03Ig68Pu275eaf7grpWbL+x2aWx0Y0RxwYfcTHOKXpUGHM8IL07p4kgs= X-Gm-Message-State: AOJu0YwJw642HemTYQo3yxcTPO6D+yrXt7E3F0rb9uGBP+ESTM6G/25M ypODmEeVrsc6VAqdol4PFkFSAxpQZrCq4Ahat6vS9W5+kTzRVcdAaVWdiKpVE8t4rHU+3dHYTFC HUw== X-Google-Smtp-Source: AGHT+IHxR0olBWteUpSeEtMuXDZWp8I1meqV0Icf5ps5DXZSQPpSTLoMw3+arJQFndTmnzLC9JieP9mTFwI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:1885:b0:dc6:207e:e8b1 with SMTP id cj5-20020a056902188500b00dc6207ee8b1mr3919663ybb.2.1709749499865; Wed, 06 Mar 2024 10:24:59 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:05 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-8-surenb@google.com> Subject: [PATCH v5 07/37] mm: introduce __GFP_NO_OBJ_EXT flag to selectively prevent slabobj_ext creation From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 10D8940005 X-Stat-Signature: dcskr4f5dgspc1wkfzmd9eu614wydzx8 X-Rspam-User: X-HE-Tag: 1709749500-638338 X-HE-Meta: U2FsdGVkX1+glY4SVOLmZ76BXlExz3nXqZfUxSnugpyQhI0r1wZmFFmmtMs5FDt+mnm/myVDWW/axLqSWSDg7UjVPVwBTbdcdaEhOHt9TPEbM88A/VgNjS28X9VO7yhiyL1VmDJ9DPkZ6GY508NqqrciFK+oE2jet/TlwhhbLIm3UW4n7I+xO4tPpQFGPIwxt5bnS0Voayj8HcaZMAT5oGeVyTJlE/oLyWFN/4axqX9T9iwkQefeAuU/JECu/6jag498+gnZmUGcrDr/R1OqGRmG6SbGHnMAwHLxkHANECw2lHSGK0YlJ9pQRptT6i8qtKW2sExdOE/he5zJlmqTka0LpKNLMW0rw7dKsMuu6J2D1wDl43kbpyFnNnWLSMPytWdeoo1bbXO0KW1ysXP2XNQBQUI0J4JdD5umj/4EP9xrBEFWfDWxfxVYfG7h8zJYX3pys9zK15s3sYN8VtHeLcxKUezJzZkCduJ+xoErC8nbPtEetJwKVDDqTBiO6I/Zqjgo+YuUa5/uISw1vaYREyX/5EBdM41RGnhyG8yyWus4lN39dzdoM7G3gMPSup7wB1ra4sMh7EFIu69BoFxKZm99rNXAkG+wEkmuoh9xcpGjiWHRCLNH093YD+DH7rh63iTY7Z3EbNPKSnT5/jPZF7wIv99GeyEZQmby3XB6AWKSarZFx8+6Wpsjkk6hrl9CR8wPDdSw7Z51iSUhBgtDZfAeuAkSqscr1kv9DAdzgcYInjRak/AZLmGpWRJTTSFcfBDR5oa9RDzN1dCLraCF9d30WZG4vbojRP43587UbyOmORhdQYvup4uRmCuWp4SMWhKa0Y9K/TFUnlZSnl/jet0SdT0U7bHYM8nKn55B403hqhgGbdRT4zbEwHsKHmInqEUepsgRqscfhy8O0XZ3iEjC7l9Ak9Gs1yZhnFNvg6K9pZChMAnuR+qj6Gol4vgk7I2RlNaflIrmx0k7c3n dhuZTy0U nRRe963oNPMx7RJs84wLrim2gyMzxYMJgtLanmqWnqyB8YkEw5Hywwqz6AC/LOJfC0oRoTBhTs9zbZqCmEzJFVZ4Ium9HF74a0L/EosbADo77z8LR3U8Xbde67GuFpzs5zIcQr6ToMrVSRGorczNL6qVi9ct+d+hZEFl3JBDVP5TsE0EkGFM0xIgXGTBZIzVJn/5nLtkBz1pg12qFf7PmXlamvqH8qEU14KAmjRXuP/kRkVokbF6Xayv9obX0KyD62ikF9DJ+f+ltKQ1QFoAxAE39hxQNke/My4igOf+hzQYvCiGIUYwAWqKC7eesLZyfKv7dGQLi522uY0dieDea/gkwwGMUoc01LdpSy48RqGSoIE4pIq9Un3bxADdam4/FTTIwQaISq5N3hOyGoGPQSJDewHgRehzsNShNNL4j8shoD7MFgcrNfH41M0ebAsLOZk0TlwuK+bnpUb7Q36XFtGqPGA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce __GFP_NO_OBJ_EXT flag in order to prevent recursive allocations when allocating slabobj_ext on a slab. Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook Reviewed-by: Pasha Tatashin Reviewed-by: Vlastimil Babka --- include/linux/gfp_types.h | 11 +++++++++++ mm/slub.c | 2 ++ 2 files changed, 13 insertions(+) diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 868c8fb1bbc1..e36e168d8cfd 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -52,6 +52,9 @@ enum { #endif #ifdef CONFIG_LOCKDEP ___GFP_NOLOCKDEP_BIT, +#endif +#ifdef CONFIG_SLAB_OBJ_EXT + ___GFP_NO_OBJ_EXT_BIT, #endif ___GFP_LAST_BIT }; @@ -93,6 +96,11 @@ enum { #else #define ___GFP_NOLOCKDEP 0 #endif +#ifdef CONFIG_SLAB_OBJ_EXT +#define ___GFP_NO_OBJ_EXT BIT(___GFP_NO_OBJ_EXT_BIT) +#else +#define ___GFP_NO_OBJ_EXT 0 +#endif /* * Physical address zone modifiers (see linux/mmzone.h - low four bits) @@ -133,12 +141,15 @@ enum { * node with no fallbacks or placement policy enforcements. * * %__GFP_ACCOUNT causes the allocation to be accounted to kmemcg. + * + * %__GFP_NO_OBJ_EXT causes slab allocation to have no object extension. */ #define __GFP_RECLAIMABLE ((__force gfp_t)___GFP_RECLAIMABLE) #define __GFP_WRITE ((__force gfp_t)___GFP_WRITE) #define __GFP_HARDWALL ((__force gfp_t)___GFP_HARDWALL) #define __GFP_THISNODE ((__force gfp_t)___GFP_THISNODE) #define __GFP_ACCOUNT ((__force gfp_t)___GFP_ACCOUNT) +#define __GFP_NO_OBJ_EXT ((__force gfp_t)___GFP_NO_OBJ_EXT) /** * DOC: Watermark modifiers diff --git a/mm/slub.c b/mm/slub.c index 6ab9f8f38ac5..2ba5d7b2711d 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1899,6 +1899,8 @@ static int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, void *vec; gfp &= ~OBJCGS_CLEAR_MASK; + /* Prevent recursive extension vector allocation */ + gfp |= __GFP_NO_OBJ_EXT; vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, slab_nid(slab)); if (!vec) From patchwork Wed Mar 6 18:24:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584418 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92A4DC54E41 for ; Wed, 6 Mar 2024 18:25:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 422BD6B0096; Wed, 6 Mar 2024 13:25:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 383066B0098; Wed, 6 Mar 2024 13:25:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1616F6B0099; Wed, 6 Mar 2024 13:25:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0238F6B0096 for ; Wed, 6 Mar 2024 13:25:04 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id CB87BA0191 for ; Wed, 6 Mar 2024 18:25:04 +0000 (UTC) X-FDA: 81867440928.01.80C94D6 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf29.hostedemail.com (Postfix) with ESMTP id 2EC00120010 for ; Wed, 6 Mar 2024 18:25:03 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=HXzRUPyn; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3_rToZQYKCEEvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3_rToZQYKCEEvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749503; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MWCYv1v0m9ThdSU1Yjo1XvUgYihcegNMceRH273060M=; b=2EdyV/lzjKYtgXxzm48C/swRMa/YDHPJvqXzMHXupfNX2s5LyQ5ySuFGjgVFT/9VI7fD1G YUbItVoLzqMXXdmRTfXFICsGY71FX6me9RNqGfpYQx1ZjsHkGA7PFUfKu2gX6ns1tGxh8l elZndeW5iDCOip9Jb7FyK7igEMFR+04= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=HXzRUPyn; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3_rToZQYKCEEvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3_rToZQYKCEEvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749503; a=rsa-sha256; cv=none; b=mwg3qa5jLLFzA1l7HuEmJTbdujuOblMev1+DNPJRPsrCL0WL5ap7DBjMw4pyME5m4YsR6c eUt+ewTMqkITtSMmiFJzDUoS2O9c5mp5Oc82xdi0qyDIhufQgR+fNsTJDVz7p7CMqpkoC0 W7sxN0fUkTZKNAfW0OkMKhQxUXgIsQw= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcdc3db67f0so1395906276.1 for ; Wed, 06 Mar 2024 10:25:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749502; x=1710354302; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MWCYv1v0m9ThdSU1Yjo1XvUgYihcegNMceRH273060M=; b=HXzRUPynex3vNGhtMdLNSvSTBf0v1qNEq+91+qJBCxyig4/rd9E56cvmBqPs5TDb7n uYZdm15gKljGJwuvCfIYNTkMKvFm8syzc2re/setRQKEud+k+4ddg5KbH6kL4SUA4l2C Hh4Czu5gvLrbNG6KgLSGLiFiyh+A1GQ3m4x15fIL2TjpLl0abuiCeg3cYgcLpfIRy1vH 3+3CK4rd3Wu5R/FD1Wkke/+ogLXIa2iaAEcKMl3zYn31fOiTkp1lnjBCjvUdg25v+pVw bwjY/mVr8Yt+Q1oARnkE4VU0Ipm4c81jBDAOPqW4BpyecLSgSwJQS5I4Pz62p9r/yz8T rOhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749502; x=1710354302; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MWCYv1v0m9ThdSU1Yjo1XvUgYihcegNMceRH273060M=; b=Pkp++fX6FBJ4PSqa4wovbeBJ1IvaqRa5FvpHt3cmY9XHUnPuH9si1pa/hs+FpAQTLj FZZtGIqFgXxatCvCGYNks14cSjD+dbdrJzChyIRsFXXqjWCFVoPhuqWEZPjf9Ud7OHLg 9Ea1gEPL2Dc3reXpRGxdQ6nWcDgXygK5Uc56w/1rFwcpWqp2oZAilEP0/HccIQJbF/cU ckvHeRgxo0usFwlu2rzUh9MluBd3nK1ETnqmiait/EdVvi0hMtOYr5BAqpzyjtaLaLmz HeM8g9rmHvNZxjm38jasvr9ex83jL57Xjb18EWsuhT0feB6Bx6C64jWKU4ZVOoLKtujm b2tw== X-Forwarded-Encrypted: i=1; AJvYcCVyHOaQUTq3fiYq1SdGCv5pOYatEzxgUkdq6IgYaoYGB+GL8kj9Gd+XSiNr0Dj45188Qe9f3rAOVEDsYQPT4tBDi4Q= X-Gm-Message-State: AOJu0YyVzMjS6+PVHqqvn1abQ/6EiRyxSvRftDyVrWNWG6XzL3Kt16VP 6i4Hrp4KAUtRfWfh+hEPI1zVX9FaFpWxw/E7S/IPu8KAttJvRPmS3WOUMw8CA8H1+HG+1hdDmnk Jjg== X-Google-Smtp-Source: AGHT+IGnljfjTXizPPHvZAgx20zMOkYizKuctvx7GCLoCP3KSTZj+nHPn74pStdoBYj7G+M/rBJH3HI3tiw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a5b:54c:0:b0:dc7:5925:92d2 with SMTP id r12-20020a5b054c000000b00dc7592592d2mr1248065ybp.1.1709749502066; Wed, 06 Mar 2024 10:25:02 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:06 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-9-surenb@google.com> Subject: [PATCH v5 08/37] mm/slab: introduce SLAB_NO_OBJ_EXT to avoid obj_ext creation From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 2EC00120010 X-Stat-Signature: hpnkcn5x9h6gsr3c8nasbxsc9hk8suju X-HE-Tag: 1709749503-414281 X-HE-Meta: U2FsdGVkX18ter1Agriar63nOnbARfNQFFw6vgnyoEzcUYzOuBUc94y8jIKpDrwbO3/zqokobf4ePXqYOcUVBIZCf3731gLJB6QVi+vdBYkK1zkEiOwGjxiR/Iw181JH/E8KkOBhe9GJmYofc4XR9J8XOFcqKAQrSjr2HI8fS8eYVeNEO/zcCh7+ZJVHGJQAx28HXSfLc1gIvtetdeEccrz2+qN8WKwRJDaoiNCGAyGgqgGYyDv+67ih2ZDEGG8uXSrKeiPG8i9Z4lKvSMVQAlexWG8V3mMa5EBuoUdR1x7fIf1dG+S/hhoD0ikCtACSCcRQ/zbhqqvWbqiswoJP/J0kZq828B5DiGgRNv9dfHHTf3M5f5xDMYyiDIukbF8xFNAksoSOWcqF1KXM/UMjqRPFk5ipRTZgvg/oLIH5/sysU7OmFe0mPCzZcvbdPVBkCD+1j74ZfgHmokBgndRfo6ykkvVIsAsZ42PgdXJ4heWQiIID3ikPKwrnPHzhzjPeSWFo/HVhQeY3UJ944sSTLSy89bKJUPZ7R+aoc1eP6EI7yYSA/FLfVyWwQO08WfCQQOqgPrjREc51CCeAXCfkq8nyIuosBlfLFl2yf42Mdz+Uov6iRGoQZugZCk88c+bLvD4jNve221eNCtc1xa50tVe4wKca9fWvwnaE4O2d5vCQEnWVOK4xk6Psvxp4lq7tsO7/etEX/YN3s0YSdOMTmypFKGcrOdrKbvSsoC93ZNA6POP2/6jAk6GwjvM8iGqJULtnARTzoQ3RYz6WU6lgqLBH1XnyQPmiqMmTwWw7V3k4y5Rn3pXfys6yky1gVFSc6o+kvpLdUFXuF68EASYaN/ZzeXEGNeW/KJ7lu2F32G3K+m1OAyWDWozO9esuuZ/jouiRlUkRuh635ig2Y70++Y8hS3Z9hQndBSgc6ELPo3z0ksTmmkHmEGW/+RNdjYZjww0glG7Pb2tHDFvIo6g eyhg6/s4 FJDtCpbzxWhCxYZ30k3usF0FdY5WVcqKoIgZgBSlq03HoQT8HY60mQ7hFEz1hYHbEixCAfeE79mIFS3Qi1j6f79fMZ4gT07v/MOolJo0Y3fkjhn4YSOqn58tN4uC3qr6uuuRZ3qlXiN0xTsMt3G9OtnbTp8Qsn5IE+ZxC5VPXocsNB1NpjVTd6ZuNHbf3NCZvGkr6zPfxOC8Recl8QrGwttdmtti/zpblpgukGEul56kasIsqwbYCM6APKoi6vJsxB5Su0TyQCD7EZIpgrYauYl4HxcsK97RPYR1Px8nf1KjAkiTWyOFcDvoWOOL5uqXXUq2dCIf8KW6h8FjTnZ2vEK3Dq0utPTqNrhqb8XohWameXbBlQ0aCY5mEgSlI/dh3fChWznorqzMltEmlZORV555pQ4I/2ebldMX8xE4knfxu1D5ZLvmEZT1N3gHG1o37iMZ8PFon88GAwRM06h3uKtTqzQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Slab extension objects can't be allocated before slab infrastructure is initialized. Some caches, like kmem_cache and kmem_cache_node, are created before slab infrastructure is initialized. Objects from these caches can't have extension objects. Introduce SLAB_NO_OBJ_EXT slab flag to mark these caches and avoid creating extensions for objects allocated from these slabs. Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook Reviewed-by: Pasha Tatashin Reviewed-by: Vlastimil Babka --- include/linux/slab.h | 6 ++++++ mm/slub.c | 5 +++-- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index b5f5ee8308d0..58794043ab5b 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -28,6 +28,12 @@ */ /* DEBUG: Perform (expensive) checks on alloc/free */ #define SLAB_CONSISTENCY_CHECKS ((slab_flags_t __force)0x00000100U) +/* Slab created using create_boot_cache */ +#ifdef CONFIG_SLAB_OBJ_EXT +#define SLAB_NO_OBJ_EXT ((slab_flags_t __force)0x00000200U) +#else +#define SLAB_NO_OBJ_EXT 0 +#endif /* DEBUG: Red zone objs in a cache */ #define SLAB_RED_ZONE ((slab_flags_t __force)0x00000400U) /* DEBUG: Poison objects */ diff --git a/mm/slub.c b/mm/slub.c index 2ba5d7b2711d..e94d3cc1b270 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -5704,7 +5704,8 @@ void __init kmem_cache_init(void) node_set(node, slab_nodes); create_boot_cache(kmem_cache_node, "kmem_cache_node", - sizeof(struct kmem_cache_node), SLAB_HWCACHE_ALIGN, 0, 0); + sizeof(struct kmem_cache_node), + SLAB_HWCACHE_ALIGN | SLAB_NO_OBJ_EXT, 0, 0); hotplug_memory_notifier(slab_memory_callback, SLAB_CALLBACK_PRI); @@ -5714,7 +5715,7 @@ void __init kmem_cache_init(void) create_boot_cache(kmem_cache, "kmem_cache", offsetof(struct kmem_cache, node) + nr_node_ids * sizeof(struct kmem_cache_node *), - SLAB_HWCACHE_ALIGN, 0, 0); + SLAB_HWCACHE_ALIGN | SLAB_NO_OBJ_EXT, 0, 0); kmem_cache = bootstrap(&boot_kmem_cache); kmem_cache_node = bootstrap(&boot_kmem_cache_node); From patchwork Wed Mar 6 18:24:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584419 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6ADFDC54E41 for ; Wed, 6 Mar 2024 18:25:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 902FC6B0099; Wed, 6 Mar 2024 13:25:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7EFCA6B009A; Wed, 6 Mar 2024 13:25:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 68F9B6B009B; Wed, 6 Mar 2024 13:25:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 541386B0099 for ; Wed, 6 Mar 2024 13:25:07 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 1EDCF80304 for ; Wed, 6 Mar 2024 18:25:07 +0000 (UTC) X-FDA: 81867441054.01.6BC4E4B Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf02.hostedemail.com (Postfix) with ESMTP id 4ACDD80017 for ; Wed, 6 Mar 2024 18:25:05 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1M55IW1o; spf=pass (imf02.hostedemail.com: domain of 3ALXoZQYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3ALXoZQYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749505; a=rsa-sha256; cv=none; b=5cxbI9APrcY+C/deeys33MFoE0gRMLclzCbjpYhPA6ZqihCOVsIAO5tco2CrLCWhRx6R1y 6c3LPf0g2tINxQu0HsE2R5r+mSvGAi8Js2qaYeZkzuaq2P1GfutymzYlSQALbNGZYQXBdO db2V/HmIYqiNrUZD/80dZWnhVW7m7c8= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1M55IW1o; spf=pass (imf02.hostedemail.com: domain of 3ALXoZQYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3ALXoZQYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749505; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WRHB/CGZyDTcWTfW8LE9qYpRlSe4g56CTtmfKB0L28A=; b=MGrla6XsjweLZEN+g0BmZk6ryKSpizFa44eOOz9ldxJ3iWJ2IfQgigtmXauai4eiObOryC Z36exWdjMTB+w/OaVaYSinIINMfvcViCuO0F8u3PZaO+owGYGK+l/Y1VbQHTEqakU9Ifay N/5mjYHn0fvGyPJgNg9yU505SJKoEWE= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60948e99cd4so379087b3.1 for ; Wed, 06 Mar 2024 10:25:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749504; x=1710354304; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=WRHB/CGZyDTcWTfW8LE9qYpRlSe4g56CTtmfKB0L28A=; b=1M55IW1ozSfUndAxgbTjIqGjIODEZtkIwLMPvblshUEEeQo7odsDiENNLi+2MhMEVu qSQnWDZoUOslIQEc0cuw/jqm5sIygHhOI08QWhmDUJTKKTzMnF1rFAYqcKI2SfjVBAta VTe/EdLGISUDvD5Xt0u6XHqfM0MjKaLAqZSGw8Zi4fXE27DVR9Fo5fg3TDAff1MAGBQb KFD3SWs4ncBn71F8skTgC5OMFDU9J0kAFnJLPlszgXe350GRp10l2zDZQBHMdlVrHtdF WnFmUrfco+cxp86udg6F9eCI7D7S9AQEqVm1T313xbkLxoKlSDhG6DtPgYfGetL0vUR1 UiXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749504; x=1710354304; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WRHB/CGZyDTcWTfW8LE9qYpRlSe4g56CTtmfKB0L28A=; b=ZFO7IilXK5YeeEMHmdfAM+BxmJCy2MW8IrwFJcpmq0Hi2HrmcjAgp6iKrAgVcKspwx 6+/9gjos7b7sDFa8q1gYVx3T/rJWJPjLR/mbP+4QpzyPJbx10ntXKv1e3KRfPrRhLZN7 jvhxctaiFW66i11yPMZuZkdGnsxu7FB/aGMIq/0LmvtzKRrL8cwrN+UW2Kx1gjAmW1B2 Wh0iA7WX/oFZXb6nAzM+E/iaVDFlFTQVTR8ydVW6RfnG28MGCi5CZWtc9qMjLGPx11pZ T+jlf2NwxTwwQs5z4eFxzSbVIMCrvND8D3t9K5YLRaotz5mcS9w23my0CT2QWhAzFVBF P/lA== X-Forwarded-Encrypted: i=1; AJvYcCWFfFTsac1BNtUB73vOQW03XfV/yoU3qZ2djjW82tONNR+zNmsyreXwpbznlArZQs+y3Mavd2GU8yMGICeiW/gKca8= X-Gm-Message-State: AOJu0Yw4Eeu+bK66V9mrVmaj5p73rQToypoYAxIy9b39bVrBI0kAuIIW DF/+BJuyuUh9DORVfiPsaBAjXuWUMEiQQHcnvfgpzurLsA7wkNqInjcM9Hta9NoHlbfcLlcX8mL /nQ== X-Google-Smtp-Source: AGHT+IFUSmekR0M1FVWzKb1jjXcJpGUE8Fd5c5LGEWBTD5HNNgZzfSlizaoDyWgDdYfuybzaLrnil2CkymA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:690c:ed3:b0:609:3c46:1885 with SMTP id cs19-20020a05690c0ed300b006093c461885mr3528012ywb.10.1709749504202; Wed, 06 Mar 2024 10:25:04 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:07 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-10-surenb@google.com> Subject: [PATCH v5 09/37] slab: objext: introduce objext_flags as extension to page_memcg_data_flags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 4ACDD80017 X-Stat-Signature: mxwx19jbn4pseu4jun7b363q4qokbu4e X-Rspam-User: X-HE-Tag: 1709749505-707062 X-HE-Meta: U2FsdGVkX1+lFguJMHoBi8R0L4HCWMR0eM9L7TJBV5VSObhHx30qMOkK1c03g6kfkbPMGgVubpRi1ZWPKvE8dHgHq4DyWv11+r3e9pK8+1x2s5IMHwsAUlOuM6mcvQSEGhxtWXMQNz6eJd16TN+gf2a3oN7dntt+1fg8+Sxahl5hhMHQiK3rGq8bjjlr+BymsL0eTV+1epNHXtb57VyfMln7B2TYWSA+Ru8mzJO6BS9Lgczq/nPLt7xJjAJVHyynLJ9JmyulxBNgCaObMNOJX9wai0LnZmSr+oNWtvBFs2dKA0EEBjPeStd2prJ41ut4aklLUYj1ZGjIQo/rtgc+g01E5g9DArjpNVBefUmnCjHQDgISMNq320j7uf7GrxoyYCmTmBX4QkCTGKW8FZgqhA4mTYaKECzxTQo+ZPejcgAF2xxlgNZa5pKy7mrcj00HRMmNPZAR37gkVo4Ox0v77EfxU0lKO2yCkMEv48hHhNdpFuPR7GRIzrmUEoN5SgbcFp7Ml91gF5Wt7QkNEPi8zwyjNNsfOw3xoQpl7/3QLhuE2AYKvIBJtzLd14715ROCSr4APOWz99CSI61sjg7RuqKy6x74f7rPrnaHOMOaiKSNTC9Fi+aTwa6wH0YWqHa+YKQiQwCVCB7Fn5TbSOLT34BUD0VYJhdyRBviDnhZjCF4wNPRxRUvfRxQ9BD64ukZzgqLg7PCNqwyNtLaxJPpVGN8Wq2B+ctObtupOa1F8n9y4BmdmfkF2qrwf7K9ofmcvaSETSjhI5ZQiuF7dO4crh1TDtNrctcKxbUKkhW//2sTQ7lj62vSyF6vJNMmFcj8LluFt/i1gWE1XlZleId9iQ/Qfo0GAfSX1dB4vZZRulJ5o6/H58L4OlL7IXiC2E8qtbzDMUf0X0Qk2TOl3Zt7AiArCQBzlnwW/nBg2dLpns7yG/mbTX0sGjS2v/Lstr0/9XyJNZNItBNBGIj8Kti fajRcWph MzsZn1Ob9R2Q9r93Y8TK5mPV0rccFX3GGephtCEgMm4BrMV4b9W4QXKodUNjhuGhmVOgwGN5Wuh0dpTuLWe4+L1KxSiKIX1Pex/WSXeW2XOrj/aR1YCv6srEpg6kNB1OHXVF2FOoe/iQT/NDvfNeyqYa5jtKacmKuX6f93ueAxQTZEKpS+NU46CtercDKi2rnu76Ook4ADniEXCk0OpHY6bfKR9I6R0SHcRvs/PWBXmzcrJm5XrO3lf2uDsZop2yU/juXOIHYANhupSIVP/ZUTNvm9SC0Q3eeX8uCZrlZ3My+uygHA/pfCsroD4u8jjnnE/Zt0hK1pv7/YQlFe9yYmRekQeGjuQYFVULvbovPq86DwcxlVhUwmtbFX6JhTIpBam5A2WKrkWlMo9xaC7859jLim18DykZRcC4caHxI6YSm6JvLZ6HjQ1Yv05NynrdnT31GN6PFn1bgna5SZdP84gg36fnClgXk1DwMFG1T74EvlwNSXnPdX7ukn9L0l2b2aaBmOA2JHo+wpAO4jhSPUQDc+5adfQiLN31ZTgCcvs9W0fZyOxm332/7zppTqETPS/chv8ZH29virG6RSHCFFRcRXbH/zNOGP65IbnbF8ttf5Ke5Po4tWu6eFE19lMhlnvYw+UCQ6y8fLoL3wK8DDIY5IjZ8DDATJzMJAGz5X50/Qjg2KjzZXz3qhUN7FPeUStdy42L5YK+ba1uMSHhoUpEolqOhwoctGtXckjwwindAX8XCDn/ETYxbIBqbLb9YenSm0Cg1on/KM8CoqC5ZHEU4HFjKzqdPkzg38vRg7IkDkrPJf6LTO44ExnPsij6ckqJVJ5AjOx9UozXNZ3sQHbsCkg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce objext_flags to store additional objext flags unrelated to memcg. Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook Reviewed-by: Pasha Tatashin Reviewed-by: Vlastimil Babka --- include/linux/memcontrol.h | 29 ++++++++++++++++++++++------- mm/slab.h | 5 +---- 2 files changed, 23 insertions(+), 11 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 9a731523000d..7709fc3f8f5f 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -357,7 +357,22 @@ enum page_memcg_data_flags { __NR_MEMCG_DATA_FLAGS = (1UL << 2), }; -#define MEMCG_DATA_FLAGS_MASK (__NR_MEMCG_DATA_FLAGS - 1) +#define __FIRST_OBJEXT_FLAG __NR_MEMCG_DATA_FLAGS + +#else /* CONFIG_MEMCG */ + +#define __FIRST_OBJEXT_FLAG (1UL << 0) + +#endif /* CONFIG_MEMCG */ + +enum objext_flags { + /* the next bit after the last actual flag */ + __NR_OBJEXTS_FLAGS = __FIRST_OBJEXT_FLAG, +}; + +#define OBJEXTS_FLAGS_MASK (__NR_OBJEXTS_FLAGS - 1) + +#ifdef CONFIG_MEMCG static inline bool folio_memcg_kmem(struct folio *folio); @@ -391,7 +406,7 @@ static inline struct mem_cgroup *__folio_memcg(struct folio *folio) VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_KMEM, folio); - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } /* @@ -412,7 +427,7 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio) VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(!(memcg_data & MEMCG_DATA_KMEM), folio); - return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct obj_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } /* @@ -469,11 +484,11 @@ static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio) if (memcg_data & MEMCG_DATA_KMEM) { struct obj_cgroup *objcg; - objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK); return obj_cgroup_memcg(objcg); } - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } /* @@ -512,11 +527,11 @@ static inline struct mem_cgroup *folio_memcg_check(struct folio *folio) if (memcg_data & MEMCG_DATA_KMEM) { struct obj_cgroup *objcg; - objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK); return obj_cgroup_memcg(objcg); } - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } static inline struct mem_cgroup *page_memcg_check(struct page *page) diff --git a/mm/slab.h b/mm/slab.h index 0e61a5834c5f..c8504574f8fb 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -559,11 +559,8 @@ static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) VM_BUG_ON_PAGE(obj_exts && !(obj_exts & MEMCG_DATA_OBJEXTS), slab_page(slab)); VM_BUG_ON_PAGE(obj_exts & MEMCG_DATA_KMEM, slab_page(slab)); - - return (struct slabobj_ext *)(obj_exts & ~MEMCG_DATA_FLAGS_MASK); -#else - return (struct slabobj_ext *)obj_exts; #endif + return (struct slabobj_ext *)(obj_exts & ~OBJEXTS_FLAGS_MASK); } #else /* CONFIG_SLAB_OBJ_EXT */ From patchwork Wed Mar 6 18:24:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584420 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9466EC54E41 for ; Wed, 6 Mar 2024 18:25:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8B46D6B009A; Wed, 6 Mar 2024 13:25:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 83F106B009B; Wed, 6 Mar 2024 13:25:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 61D9E6B009C; Wed, 6 Mar 2024 13:25:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4BA006B009A for ; Wed, 6 Mar 2024 13:25:09 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0F2DB1C11BC for ; Wed, 6 Mar 2024 18:25:09 +0000 (UTC) X-FDA: 81867441138.07.7BB7297 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf22.hostedemail.com (Postfix) with ESMTP id 64CD1C0022 for ; Wed, 6 Mar 2024 18:25:07 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=EPLLtBxT; spf=pass (imf22.hostedemail.com: domain of 3ArXoZQYKCEUz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3ArXoZQYKCEUz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749507; a=rsa-sha256; cv=none; b=UtXThJg8mIMNzXXMb2pjMS072ySb9xdLWqhi9Rfwrs5vG6aIe2W2n5E1HbbWH66c3gZq2q fzoLj9jEZKMgn52tF+uKsSvozqyhgX+8iFJ6UZChNjTJz0Sid3sZtookSHtm402rQqTJPM GO7dV2ITSAi+rr1nlKb4neFkwJSmINc= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=EPLLtBxT; spf=pass (imf22.hostedemail.com: domain of 3ArXoZQYKCEUz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3ArXoZQYKCEUz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749507; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BNgaGcRDgPcaIXeO4Fkb/yIfx8W6judwG+Ary21N5qY=; b=cbTPFp7tnwFNPYUZ8/KYPNSRYMjt+SJ7pJagQ5OuBYChEWtEHBMj4AWUjAtEDKA7fvDg1r Tl9UCRhYnF/HP4HRW02XubmIXE4m09x3CYP/0JnXpu5Hx6zbx2OYLK/CfZtBNHIRDF/4S+ 99ZGcYa8nLxv0E+9/BGk1+srUPxS04U= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dd0ae66422fso199324276.0 for ; Wed, 06 Mar 2024 10:25:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749506; x=1710354306; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BNgaGcRDgPcaIXeO4Fkb/yIfx8W6judwG+Ary21N5qY=; b=EPLLtBxTZDQh+7u2NWS4F8BnKzxI9X3ndXDnE1JmS1/aJcDDZfQdb4yEDxDTrkaq9t 8PqVfrsP96ApGwycjFwGdwmT5cdbADW796jKJ5lKQC6thxyejg2XHzR4uJo6Cqo9Bhg6 6YR5bwFQOYoIjyeEnNBvFH90a4t/fzSzoHXYS1oqQSkKWnFm9rwVBsyTOCvBLENELztz NXw8Oa+3ADY6OIj0BXxh2lnerkVPOr+jKCxsZBhVECIEmm4RI2hcOLACwVn1Sjvr6Y1H tvS+tEjKbJUbas/yIrYX1mrfgQGCa69OZDxbR2k+waeuUUvt7AiV0V6DeTPvkoiDxrRD gCyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749506; x=1710354306; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BNgaGcRDgPcaIXeO4Fkb/yIfx8W6judwG+Ary21N5qY=; b=Ohkm8Cg/vQWgaf2xS6scqBDiOicrkG+SaVX834On5LhLRBTKaKhs2Gs2TasTSG1XZF BN2lm8WOZeX6OPUIORfw3ATgc0u/fFNYarSjH6AutCUAgst+2p6TtizWvKsz7IVIZxrv 7B71sVUoRy9IYJ/GmhzwflCpn6mhyU6jlcp+GVP+GvEkZnyN03lhI1x9kngrI5w2pU9n 7pYCy29XLwTcy+FfEv+Ddv5KBnPVXBi9nhCiaGEcclzMA3cUl1FZoLrEI17+4PIUF1F5 XlWrmZIvrUBQaRVuoK44pXnFDEZ0ibHEjEEy7LUsoriBLPwfxtpP0TIALN1Jj2qpojbe QIVA== X-Forwarded-Encrypted: i=1; AJvYcCV/mkydR2H2RXuh4cSzOeptMQxtixydnx2QvHZcu7fJ9QDMHFgftn0eloL2lZwzxmKaBgsIxexgUAqs6O0fFNvqyw0= X-Gm-Message-State: AOJu0YwBdLMuAPPJm2L3nwMCx7xbFFjyqPPPyU0q0XwqdP+AevrSbqWH kTamG32cF54obV/YzXZNHtfijji92yu4Y328V+r7v5ScKNWE/ZP5tm0NTDIbOjLMs8W+BTWvmQa qUQ== X-Google-Smtp-Source: AGHT+IHPRw/ef5CXgN/EcTIYcsNOxCQ2QBNqcSc4reoJ5he/nASVYkogu18nMMcl34PXaT57a/C89ka0n4s= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:1504:b0:dbd:b4e8:1565 with SMTP id q4-20020a056902150400b00dbdb4e81565mr1895059ybu.4.1709749506288; Wed, 06 Mar 2024 10:25:06 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:08 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-11-surenb@google.com> Subject: [PATCH v5 10/37] lib: code tagging framework From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 64CD1C0022 X-Stat-Signature: 6hmfkk9xhmsnr1e4yincb9xjjc7a5xq6 X-Rspam-User: X-HE-Tag: 1709749507-989459 X-HE-Meta: U2FsdGVkX18uiiiK9Ghm8NYzIG5IP+Q2uFHyl1puQSN8yezgKb4sikfCTqm0s8KtzKuq7ML0gUa/U3S/4RbwndiOCA9D2d3yC4uiMdjDdxKIm44TEROEuxl1AagHxSzwVyMuVQMg1SzRrsnTogc4bGSaU0uPeZHSJZG+EZFVO1i85WwECHbzZVVePyXan9ZfPplRgiCP0TLpNwAfUG46avvpjkkX4x5al3QHU97ePuntx0JZYe8p12Dl6xarkSrrC5NGisMxnRohS6WkjJMr8z21gP+7pMnGftImOtcUELsmYHttQAmPTft1w5M/TwsT0TwkglMZem80Wrii76qmVayLfOYkpn3lqkRgtyo7Rb6KZKFvX73Sw6LGfMa0ZhuiSbqNwqa9sbsAIuJbZYtzTRLNi28X3hfirSu7BndEh1SWL2u8P1V5E5Gs1xnluWWtBfGeSSj0hbA+raoIMnAngfwtT+H63sXY35fpcXpBPrh0coKq6gFV6m2KphmCHt80SjyCOmbc8DE9WLM0PK+TN8v9HxtfyUANYr2eZlynAmEnYHXTd+7BRbw6MfN/rdPjHlPcqy9MqPYfvYkChuKLduPaIgClbw5XN4yuIfWBxkR/Flu1vd6raiAr7G58LV5FWwQvSnNhbT6BYl8RUrRGjd8WZLbCfX4FXVnRxpqwkpWkvuaTZo/ukHyAjyY1D3iyWnzbL4QxIvepbhh4xtt2oIzUGItHMLN/wA9tklnN5J4uaofhzsNgVtJ3kWwtBBfwKKJJ5uofYncCgw3i6rnVPkIk9f2IaOwfBqZwUd+xuzPnC0LHuUgTI/wbJntnZcudt6aU3Hemf5okzfIRGciaxd2ovBgjZ/DSTKbUPPQficN306ewNP8Mtmrc132DJvyfoUzswQfA7P1P6PVzGs0YweBkZrqlIVVrzJkz/h3u1L/oVvSH8xIryv0BJ/BJczfsYAmMF+3JgkNxoWApx5Q TlxpSM4f WyinfNuJjjdc1hos282yk+J5DwVEXGNfmSftnz3AXESO/BYf2jmfkjhrgNkdPtH2SC3QTYnXSW874PNLO40tvKO0R66lp8njgKOGHiu+ouioDJ3rBdTndSnRk9ts+uLE42dkOeMzp4mwwbUxIGabvytuA/kefpKD+eFqo/YV4aFXy7hMFnbuwvjq4dBNzU3+YRQRwVIgvor0/KWT3rCJkxpLz14CoyS1KhpLZBKwfV+LsCT4cAhpp/QN5kEz21nFKkMisexnYhTm39AHOaNz2jpEzjo+7MLum4Oi/p4JEmAFQjP2TuNPjdo8msZgPq1Ov4d330RQyofO6+QoweCBUAnRzgOAaP8Rcx4eXo53BG56ncdkS2ckbSxO3R3C3oaW4ijiHmM7EKZAk8t8fHYp9noVN+qPVR34GQYeNlZ1TTzjZ4Bw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add basic infrastructure to support code tagging which stores tag common information consisting of the module name, function, file name and line number. Provide functions to register a new code tag type and navigate between code tags. Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/codetag.h | 68 +++++++++++++ lib/Kconfig.debug | 4 + lib/Makefile | 1 + lib/codetag.c | 219 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 292 insertions(+) create mode 100644 include/linux/codetag.h create mode 100644 lib/codetag.c diff --git a/include/linux/codetag.h b/include/linux/codetag.h new file mode 100644 index 000000000000..7734269cdb63 --- /dev/null +++ b/include/linux/codetag.h @@ -0,0 +1,68 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * code tagging framework + */ +#ifndef _LINUX_CODETAG_H +#define _LINUX_CODETAG_H + +#include + +struct codetag_iterator; +struct codetag_type; +struct codetag_module; +struct seq_buf; +struct module; + +/* + * An instance of this structure is created in a special ELF section at every + * code location being tagged. At runtime, the special section is treated as + * an array of these. + */ +struct codetag { + unsigned int flags; /* used in later patches */ + unsigned int lineno; + const char *modname; + const char *function; + const char *filename; +} __aligned(8); + +union codetag_ref { + struct codetag *ct; +}; + +struct codetag_type_desc { + const char *section; + size_t tag_size; +}; + +struct codetag_iterator { + struct codetag_type *cttype; + struct codetag_module *cmod; + unsigned long mod_id; + struct codetag *ct; +}; + +#ifdef MODULE +#define CT_MODULE_NAME KBUILD_MODNAME +#else +#define CT_MODULE_NAME NULL +#endif + +#define CODE_TAG_INIT { \ + .modname = CT_MODULE_NAME, \ + .function = __func__, \ + .filename = __FILE__, \ + .lineno = __LINE__, \ + .flags = 0, \ +} + +void codetag_lock_module_list(struct codetag_type *cttype, bool lock); +struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype); +struct codetag *codetag_next_ct(struct codetag_iterator *iter); + +void codetag_to_text(struct seq_buf *out, struct codetag *ct); + +struct codetag_type * +codetag_register_type(const struct codetag_type_desc *desc); + +#endif /* _LINUX_CODETAG_H */ diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index ef36b829ae1f..5485a5780fa7 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -968,6 +968,10 @@ config DEBUG_STACKOVERFLOW If in doubt, say "N". +config CODE_TAGGING + bool + select KALLSYMS + source "lib/Kconfig.kasan" source "lib/Kconfig.kfence" source "lib/Kconfig.kmsan" diff --git a/lib/Makefile b/lib/Makefile index 6b09731d8e61..6b48b22fdfac 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -235,6 +235,7 @@ obj-$(CONFIG_OF_RECONFIG_NOTIFIER_ERROR_INJECT) += \ of-reconfig-notifier-error-inject.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o +obj-$(CONFIG_CODE_TAGGING) += codetag.o lib-$(CONFIG_GENERIC_BUG) += bug.o obj-$(CONFIG_HAVE_ARCH_TRACEHOOK) += syscall.o diff --git a/lib/codetag.c b/lib/codetag.c new file mode 100644 index 000000000000..8b5b89ad508d --- /dev/null +++ b/lib/codetag.c @@ -0,0 +1,219 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include +#include +#include +#include + +struct codetag_type { + struct list_head link; + unsigned int count; + struct idr mod_idr; + struct rw_semaphore mod_lock; /* protects mod_idr */ + struct codetag_type_desc desc; +}; + +struct codetag_range { + struct codetag *start; + struct codetag *stop; +}; + +struct codetag_module { + struct module *mod; + struct codetag_range range; +}; + +static DEFINE_MUTEX(codetag_lock); +static LIST_HEAD(codetag_types); + +void codetag_lock_module_list(struct codetag_type *cttype, bool lock) +{ + if (lock) + down_read(&cttype->mod_lock); + else + up_read(&cttype->mod_lock); +} + +struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype) +{ + struct codetag_iterator iter = { + .cttype = cttype, + .cmod = NULL, + .mod_id = 0, + .ct = NULL, + }; + + return iter; +} + +static inline struct codetag *get_first_module_ct(struct codetag_module *cmod) +{ + return cmod->range.start < cmod->range.stop ? cmod->range.start : NULL; +} + +static inline +struct codetag *get_next_module_ct(struct codetag_iterator *iter) +{ + struct codetag *res = (struct codetag *) + ((char *)iter->ct + iter->cttype->desc.tag_size); + + return res < iter->cmod->range.stop ? res : NULL; +} + +struct codetag *codetag_next_ct(struct codetag_iterator *iter) +{ + struct codetag_type *cttype = iter->cttype; + struct codetag_module *cmod; + struct codetag *ct; + + lockdep_assert_held(&cttype->mod_lock); + + if (unlikely(idr_is_empty(&cttype->mod_idr))) + return NULL; + + ct = NULL; + while (true) { + cmod = idr_find(&cttype->mod_idr, iter->mod_id); + + /* If module was removed move to the next one */ + if (!cmod) + cmod = idr_get_next_ul(&cttype->mod_idr, + &iter->mod_id); + + /* Exit if no more modules */ + if (!cmod) + break; + + if (cmod != iter->cmod) { + iter->cmod = cmod; + ct = get_first_module_ct(cmod); + } else + ct = get_next_module_ct(iter); + + if (ct) + break; + + iter->mod_id++; + } + + iter->ct = ct; + return ct; +} + +void codetag_to_text(struct seq_buf *out, struct codetag *ct) +{ + if (ct->modname) + seq_buf_printf(out, "%s:%u [%s] func:%s", + ct->filename, ct->lineno, + ct->modname, ct->function); + else + seq_buf_printf(out, "%s:%u func:%s", + ct->filename, ct->lineno, ct->function); +} + +static inline size_t range_size(const struct codetag_type *cttype, + const struct codetag_range *range) +{ + return ((char *)range->stop - (char *)range->start) / + cttype->desc.tag_size; +} + +#ifdef CONFIG_MODULES +static void *get_symbol(struct module *mod, const char *prefix, const char *name) +{ + DECLARE_SEQ_BUF(sb, KSYM_NAME_LEN); + const char *buf; + + seq_buf_printf(&sb, "%s%s", prefix, name); + if (seq_buf_has_overflowed(&sb)) + return NULL; + + buf = seq_buf_str(&sb); + return mod ? + (void *)find_kallsyms_symbol_value(mod, buf) : + (void *)kallsyms_lookup_name(buf); +} + +static struct codetag_range get_section_range(struct module *mod, + const char *section) +{ + return (struct codetag_range) { + get_symbol(mod, "__start_", section), + get_symbol(mod, "__stop_", section), + }; +} + +static int codetag_module_init(struct codetag_type *cttype, struct module *mod) +{ + struct codetag_range range; + struct codetag_module *cmod; + int err; + + range = get_section_range(mod, cttype->desc.section); + if (!range.start || !range.stop) { + pr_warn("Failed to load code tags of type %s from the module %s\n", + cttype->desc.section, + mod ? mod->name : "(built-in)"); + return -EINVAL; + } + + /* Ignore empty ranges */ + if (range.start == range.stop) + return 0; + + BUG_ON(range.start > range.stop); + + cmod = kmalloc(sizeof(*cmod), GFP_KERNEL); + if (unlikely(!cmod)) + return -ENOMEM; + + cmod->mod = mod; + cmod->range = range; + + down_write(&cttype->mod_lock); + err = idr_alloc(&cttype->mod_idr, cmod, 0, 0, GFP_KERNEL); + if (err >= 0) + cttype->count += range_size(cttype, &range); + up_write(&cttype->mod_lock); + + if (err < 0) { + kfree(cmod); + return err; + } + + return 0; +} + +#else /* CONFIG_MODULES */ +static int codetag_module_init(struct codetag_type *cttype, struct module *mod) { return 0; } +#endif /* CONFIG_MODULES */ + +struct codetag_type * +codetag_register_type(const struct codetag_type_desc *desc) +{ + struct codetag_type *cttype; + int err; + + BUG_ON(desc->tag_size <= 0); + + cttype = kzalloc(sizeof(*cttype), GFP_KERNEL); + if (unlikely(!cttype)) + return ERR_PTR(-ENOMEM); + + cttype->desc = *desc; + idr_init(&cttype->mod_idr); + init_rwsem(&cttype->mod_lock); + + err = codetag_module_init(cttype, NULL); + if (unlikely(err)) { + kfree(cttype); + return ERR_PTR(err); + } + + mutex_lock(&codetag_lock); + list_add_tail(&cttype->link, &codetag_types); + mutex_unlock(&codetag_lock); + + return cttype; +} From patchwork Wed Mar 6 18:24:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584421 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1FA14C5475B for ; Wed, 6 Mar 2024 18:25:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 420236B009C; Wed, 6 Mar 2024 13:25:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3CFC16B009D; Wed, 6 Mar 2024 13:25:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 224F16B009E; Wed, 6 Mar 2024 13:25:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0A1E86B009C for ; Wed, 6 Mar 2024 13:25:12 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id BDC11A0F58 for ; Wed, 6 Mar 2024 18:25:11 +0000 (UTC) X-FDA: 81867441222.27.FEE4F6F Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf07.hostedemail.com (Postfix) with ESMTP id 8740A4001C for ; Wed, 6 Mar 2024 18:25:09 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=h8JCds78; spf=pass (imf07.hostedemail.com: domain of 3BLXoZQYKCEc130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3BLXoZQYKCEc130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749509; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tPAQlIYBmu/nqjnWAR+Y3waoaH5kDJ/L5+WSbugHpuw=; b=cfPCiEZ7mqhPXguE24GG6rvzswzbJFx39SP9FoB6IuaAz3z53nI6v0mtsQ8QeN1XvN3Y93 5Xm8iOHJnfSZHtLq1CBLg6wtHQKP9HxnY1AIDzuBDoRVlVLDneUM1uH7IPN4iYg9LW3xuR DyyfcOg9Afhy4alhO/xEI0J+Zw8ve94= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749509; a=rsa-sha256; cv=none; b=G/foJh0sU0foTOmzmIKMjAstf9qq9T9BYIp6OZ3KL/edzVzSkrVD8hnUYvFH/DAwJ1dYGh 29jQEs/3FfpBUB+40bKqePSlrtKOLL1TbeVuFU5pw5XkFQ9446juEjUy75aXyzE0cKHIza CYg4cgVbMb52QvaYs37lE7EcjdJxH1Y= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=h8JCds78; spf=pass (imf07.hostedemail.com: domain of 3BLXoZQYKCEc130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3BLXoZQYKCEc130nwkpxxpun.lxvurw36-vvt4jlt.x0p@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60904453110so154477b3.2 for ; Wed, 06 Mar 2024 10:25:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749508; x=1710354308; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tPAQlIYBmu/nqjnWAR+Y3waoaH5kDJ/L5+WSbugHpuw=; b=h8JCds78nrW+R8/UG5WoYoQpHVH+EO+/dzcLCoYuRVDDdyJBT82Q/aOEQDNaeDm+TQ 4XWSxpLqsqABK8qoXvbnP6+dZAZr6tn7p5uXXSTZ7ZKJWumO7QfVhC0uzu9/UfmINuKk AZ/fUL3HozclASxZ2Sz+uDtvOSpyS6YkBdBJjh0SzknTMnrjAbU/LpW5Ng2I2F+HdZPr j6z5uvXZfCu79Y0Cmime9oSlm4P11A5udVKstye7UxfJU4AXeNpu+WlA03buTnbXVm3y QNm8gThD/plxf0+iavZ1kraqbOOxTp+7/nB1bi9/Iy9RxXflbCoQHB3yEkBE3O/rv+16 Ds2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749508; x=1710354308; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tPAQlIYBmu/nqjnWAR+Y3waoaH5kDJ/L5+WSbugHpuw=; b=VSuZNfIpO48GxlI6HWE+KKIQ0FYe9Vugf8J0m+Um4o1rZze5M7lwHfX964r0JJOedc 0yzF3J+HbcCLQt4iXFnmmHJsbGDM21njdVjKrvZY3I1YLWWEKur192gig1N9h7iZrL4p tuTgBJz18pZse89zylHQfvXGfulB+hO9e2utumNgN/CsAEv5D8H/kdDMwf0II0m/1nIn +dClxW+b+c9xzZeHPlV7z643iPTwdTiF6NGJBHLfOzxwb50DLINqumZNqk4keBeuJAt7 RR3v6AcD/MkNHn5e6ONjGLBYODhAx74CjsHA/Jf1gO8yXvRZtI3wEmHhIY0SwrW4o7Nn vDhg== X-Forwarded-Encrypted: i=1; AJvYcCXoUVVSf1ag+BII81ACkZNiiZa0Z5YqR9iv311mD3k1r+etl+69/EQujSPp8oSUWDlmi3IEMZ5T7kR0csnssH146Fc= X-Gm-Message-State: AOJu0YzBXANCIdPtdfzcp2irDhF/YdB5JkIbxWVWhEgs4V4zknNEiKZR Jk5ly+LH/WTQLjLI35AT6tpoLHLZbyx+55cwA++3S9oHitj1MQBwc1OIOYIxWQQatcxjbjaTZZL e0w== X-Google-Smtp-Source: AGHT+IEd0tsvecVu5KDk9xxf7Y3b9Rk1jgR8DS8Z815x0mHqtjUMO8iZmPCQu0I1T2SCEsTJC91g87PAOnY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:690c:ec5:b0:609:247a:fd3c with SMTP id cs5-20020a05690c0ec500b00609247afd3cmr4821873ywb.2.1709749508522; Wed, 06 Mar 2024 10:25:08 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:09 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-12-surenb@google.com> Subject: [PATCH v5 11/37] lib: code tagging module support From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 8740A4001C X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: y36qh1xr7sr9pm1nkz6pqmhwts7txy6i X-HE-Tag: 1709749509-850661 X-HE-Meta: U2FsdGVkX19OcytNDwvDl9oaQCa1wVZu0bSzsa584rIB3fQRzFqs6YtFxNHvGWGmt06WQFXWgDC3w1U2UHPRDCIYbW9+rubm4QTFm9Q/tBLCr3T92w7EoCol+7szvl9gyMwEFbFbpVX9rQ+LPSAhVlh+v/LKVm2DdmPQf4N1BiC9EWZ0Vd73rnsrZeK18mS6dQOt/Ei5ALHtw7d8Jli+ndC99Njz09vgtUCQfjH2No4kLNvIUc+Cdjmjyop0QAXBByD+QGe+NWgHtBfZmJdXCchtrKsafeXoXRuo2UFKw4IYDowXNXsVWnptJZ/AIHKauWhVvdlFKy3FjqXQS9eiGvqpDr3OSm2jF3X8apd/YsLmAsUYLxIW0A6G/K6ZUbwdoD8ImYelkZg7MFhawo4C8H7D4UJ/HjZMympvu0SEnmpZ4wWL6ylMo07ENIm3Hkul9zZmIq/cZ/0xn1/HMcD68dWiEfKRFdhPAQTuWQD74iSwOYSOvUA2+Qw6h2eyWprw40ZE5f+qxt2K49KIf1XjDoepw/bkeISMJ+NTGp8Wuz/7V73e48CmGjhVm/fcG4xpR/i0lWP+qnvk0zCnhE6HnWuL97SC4O3EirW2X4rG3NTynZboyeMnSkA6gDqHDeEL5T6vMWHZbDNQP9g6FJdDx75HrF278YUPSjQgtdoGy+RVvCvW0OT0OREesW2sPuMZKZspw42A/VbUxiYq2r8W+CVHj6prWMRlUtWzAhr/vWCGRjmK/2fdqVYU8i9pCl76qA1WRj4DcXGCySAW0ByF8ZTz8O30oFW4BTU1KbZn332DSqW3cganzzruyTu5Qwdmo+TWhLm2LEz0TUiXCQQb86nfA4YhoMC0A0fgZShsTaKdv6/AymOHgX4gm5nIQ3rtD5hDAS/k+7T8/H2eJqKaKHhbllBtsyB0LmXNEhNC95+ZnR6raqaCEUZiE4WxagPkCfgygFIXhTWASwlXnpZ j62SOBO8 21YTzcTIDahlrPoxLZXfZYf+u43Uyn8dipUP+hKNILnMB30INp9Y5AC6hjWTnF5zq96PIHavkZULtjVi7M7fsrmHL3drvDc2YV+fc7YzHsk/V35hachhDn1GwfwUF2wS6T4a/yows6LLyUb6a1RFHjZCLZG4fZPgX0PL/+YRRkts1NYKhq/Ijpx9UnBn9Uy0eJM0bNtxsqLVJ7hOz1+M8YGL/7Wvypww4HkYN2A+j5EWJ8wDg9duH8+dNoIC34xY3tn6IQhkXIa6NU69+Hr+cB8NItWa9nHUqUQFOspaT+ccpDSfoOS+QjrlOSN7ielLV5iLrYGXthkW9e4MgDpJvxtgtzS0W/R2hyIt70RBkEwSQIvPgThaW0Bcm/TDIuMrdnPv4dyaIad+oc9VlcfHeX55dFZxAKUOYIQo765HoSXq0oddTIg5bqRfuBITYTldojutBVrIqeDuO4RN36JS33by6ZfWRmrheBwcBxH4c+DEmS+3NCKg3sIVTii5AmgIpPbwhsuBE1wrZ68uFbYgu3/iBrISZyBvBRNwNySrRUJ5VyTxixFhjLkP4JX2Gi3aJbfy06rXbSBklprzxHQGtcRrbYd+4nRGtO4Jhh3jTVKpQYo6EIBcolatqP0wEsjsO/MrKbhWJMLc1qYkrwkq83IGI01353XxZXQwbW+8sCY2I1JE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add support for code tagging from dynamically loaded modules. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/codetag.h | 12 +++++++++ kernel/module/main.c | 4 +++ lib/codetag.c | 58 +++++++++++++++++++++++++++++++++++++++-- 3 files changed, 72 insertions(+), 2 deletions(-) diff --git a/include/linux/codetag.h b/include/linux/codetag.h index 7734269cdb63..c44f5b83f24d 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -33,6 +33,10 @@ union codetag_ref { struct codetag_type_desc { const char *section; size_t tag_size; + void (*module_load)(struct codetag_type *cttype, + struct codetag_module *cmod); + void (*module_unload)(struct codetag_type *cttype, + struct codetag_module *cmod); }; struct codetag_iterator { @@ -65,4 +69,12 @@ void codetag_to_text(struct seq_buf *out, struct codetag *ct); struct codetag_type * codetag_register_type(const struct codetag_type_desc *desc); +#if defined(CONFIG_CODE_TAGGING) && defined(CONFIG_MODULES) +void codetag_load_module(struct module *mod); +void codetag_unload_module(struct module *mod); +#else +static inline void codetag_load_module(struct module *mod) {} +static inline void codetag_unload_module(struct module *mod) {} +#endif + #endif /* _LINUX_CODETAG_H */ diff --git a/kernel/module/main.c b/kernel/module/main.c index b0b99348e1a8..bf5a4afbe4c5 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -56,6 +56,7 @@ #include #include #include +#include #include #include #include "internal.h" @@ -1242,6 +1243,7 @@ static void free_module(struct module *mod) { trace_module_free(mod); + codetag_unload_module(mod); mod_sysfs_teardown(mod); /* @@ -2983,6 +2985,8 @@ static int load_module(struct load_info *info, const char __user *uargs, /* Get rid of temporary copy. */ free_copy(info, flags); + codetag_load_module(mod); + /* Done! */ trace_module_load(mod); diff --git a/lib/codetag.c b/lib/codetag.c index 8b5b89ad508d..54d2828eba25 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -124,15 +124,20 @@ static void *get_symbol(struct module *mod, const char *prefix, const char *name { DECLARE_SEQ_BUF(sb, KSYM_NAME_LEN); const char *buf; + void *ret; seq_buf_printf(&sb, "%s%s", prefix, name); if (seq_buf_has_overflowed(&sb)) return NULL; buf = seq_buf_str(&sb); - return mod ? + preempt_disable(); + ret = mod ? (void *)find_kallsyms_symbol_value(mod, buf) : (void *)kallsyms_lookup_name(buf); + preempt_enable(); + + return ret; } static struct codetag_range get_section_range(struct module *mod, @@ -173,8 +178,11 @@ static int codetag_module_init(struct codetag_type *cttype, struct module *mod) down_write(&cttype->mod_lock); err = idr_alloc(&cttype->mod_idr, cmod, 0, 0, GFP_KERNEL); - if (err >= 0) + if (err >= 0) { cttype->count += range_size(cttype, &range); + if (cttype->desc.module_load) + cttype->desc.module_load(cttype, cmod); + } up_write(&cttype->mod_lock); if (err < 0) { @@ -185,6 +193,52 @@ static int codetag_module_init(struct codetag_type *cttype, struct module *mod) return 0; } +void codetag_load_module(struct module *mod) +{ + struct codetag_type *cttype; + + if (!mod) + return; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) + codetag_module_init(cttype, mod); + mutex_unlock(&codetag_lock); +} + +void codetag_unload_module(struct module *mod) +{ + struct codetag_type *cttype; + + if (!mod) + return; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + struct codetag_module *found = NULL; + struct codetag_module *cmod; + unsigned long mod_id, tmp; + + down_write(&cttype->mod_lock); + idr_for_each_entry_ul(&cttype->mod_idr, cmod, tmp, mod_id) { + if (cmod->mod && cmod->mod == mod) { + found = cmod; + break; + } + } + if (found) { + if (cttype->desc.module_unload) + cttype->desc.module_unload(cttype, cmod); + + cttype->count -= range_size(cttype, &cmod->range); + idr_remove(&cttype->mod_idr, mod_id); + kfree(cmod); + } + up_write(&cttype->mod_lock); + } + mutex_unlock(&codetag_lock); +} + #else /* CONFIG_MODULES */ static int codetag_module_init(struct codetag_type *cttype, struct module *mod) { return 0; } #endif /* CONFIG_MODULES */ From patchwork Wed Mar 6 18:24:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584422 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 130F3C5475B for ; Wed, 6 Mar 2024 18:25:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D95DC6B009E; Wed, 6 Mar 2024 13:25:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D1E586B009F; Wed, 6 Mar 2024 13:25:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B4C086B00A0; Wed, 6 Mar 2024 13:25:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9FDE76B009E for ; Wed, 6 Mar 2024 13:25:14 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 793A1120CDC for ; Wed, 6 Mar 2024 18:25:13 +0000 (UTC) X-FDA: 81867441306.30.03B69A8 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf13.hostedemail.com (Postfix) with ESMTP id B585220006 for ; Wed, 6 Mar 2024 18:25:11 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ougjA2pD; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3BrXoZQYKCEk352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3BrXoZQYKCEk352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749511; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UUM81FzZaYB1+q8hTjh77vdQeiiRG/JwjOPLDZjum/8=; b=nuS3X/RaTWIbq3ULmZ4YnOhtj8J3hklyhacQOXGpJdmhtydJWFWGGeLRjgU7jSUO+J+aAU rCmh3Oc1Ar5FWi25z0nuj7fUMwLkq9VP0i70lcXMevZT/JL5ky632rgkMe+G6DepZyszmG 4b0TpUXTD1lIAagdhdMTY88EoL1J5NQ= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ougjA2pD; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3BrXoZQYKCEk352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3BrXoZQYKCEk352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749511; a=rsa-sha256; cv=none; b=AHKPlmF5Oo/agNzuw2DWyz5GNXbIZIskPkPZx45W/PKmFYZSOCIsOUBiifVQQ/nhHt3DQ4 bNr+fC7DxfS2LxsCG/tgDtxBn/0bN8FwM6HPYeGu4gpOfIKuQyLQZwmu/W3MEtv7cCWia7 2wRQSUv88zC+DlEb319q6mWsoNtYlSQ= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dcd1779adbeso4114555276.3 for ; Wed, 06 Mar 2024 10:25:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749511; x=1710354311; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UUM81FzZaYB1+q8hTjh77vdQeiiRG/JwjOPLDZjum/8=; b=ougjA2pDPDSDJpvx0+LQfy/5xLtYCGuj3VlcNqFS98lCkoRItnBDFpbmyVyVf+Vy0J StP3oHDBANY7HjE7GP6+xt4aaBk4qCoVJ6AEJl6uYy6yC52PBkoXBEhLCpA2cllfGGVK XxxtdeuyDavWX46yCNKCUX9C6NfREWaq1ll1mGab4poDJZfWRAxiprlxHXXoZxx8tI56 0buEhS5b8msJJooXHYO86lBFCyOWQ8bTgRImu2qBostnrpGT1xLUblYyphCKrenik91p pmcjLxrSDgnZKNbXgnup0w/Zzbbhbb/4UvGl4KHn+F+26MQ1K39xVu7fX95gnV6kwf2X QmXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749511; x=1710354311; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UUM81FzZaYB1+q8hTjh77vdQeiiRG/JwjOPLDZjum/8=; b=SVVVIONU65qjCj76gSdBRFZhvpz4AfhCVP24DCETryumHiT7v4V/ql7Y6vCdjyEbgm Jdnq6jkQmhVJbXiwt7zoTCibhhaGSvAkH8oKmRtdsFtKcgjXwPos0Zhw4zMFK5K+ST8N GCnKMFCDdRcTIepO9F43Yhc/ceZDA4vbOFBMBOTtC43trD6m9Wl+F3glyCdpDc9uRtr2 5UptsHaI6b5EbcuzCpj6TER7O5lZN2rfdLS/mzy3W/8Ss1jjB3ltLAWLHQLmW5Ced493 qKsaU632PnIigiVNnrurIlLzFrc1OYZ1N4hnK8bCKRJVHsid9duH1sMQxTIgAHTNLvE2 IyYA== X-Forwarded-Encrypted: i=1; AJvYcCU/EvruljEvOocpvKohiYORD9aSOSRD9ridkjJnlGob3qTJvbqp959f+o2KhocShF2vPTHvli9pAfwLKP2E1MSCZLo= X-Gm-Message-State: AOJu0YyrRndqk5370jyD+uAPg5jy56r7rSA2mUkY7Ji8dZ2vj3/dyTwp lewJjiqPSMHCnhVdV3zIJmhzL4Vms6ER/EqPpWobyjmwdn893pdWCqtnFJ2DwO+rkjge50lATcX Frg== X-Google-Smtp-Source: AGHT+IGqHIJUWrUsfejQErup2Z224PwJZwn/RbIXxQkXJPLPBFMv4l1nRilm/FfAADknSVQZ33UMPS4jncc= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:2492:b0:dcb:b9d7:2760 with SMTP id ds18-20020a056902249200b00dcbb9d72760mr4211044ybb.13.1709749510558; Wed, 06 Mar 2024 10:25:10 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:10 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-13-surenb@google.com> Subject: [PATCH v5 12/37] lib: prevent module unloading if memory is not freed From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: B585220006 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 91zphrrqsgoy6fqfmq5ur6n83ieiz3wb X-HE-Tag: 1709749511-147601 X-HE-Meta: U2FsdGVkX1/DwyM+guwmYMTmmOKdUkI1G4/Zytupcy87run8xPkkvCPeonOZKHf2kxy2mHq5SMXvzgvENfOYPRgP/b2sc6uX3I9aUVpheS9YOZCkhheGhJNDRAUq2nb74VLflAbIFnCYOjS9bcye8/eJw/mcrBCt5NElE65YW8s2DCsq3JBKOfG6dAOg00SQNMnQ4BV1krWYqY8QiKm8WFK4mGdTQ/w+7GB0DA///+kEwbB7PGJ6nF88DgbndCEIC1utxDAwyO3Wtjo2QKLKcfMiDI0yG+IPFG5xxk4PsGeOCXMIzsPRC4oyakLXUX5uwRpmb6eFU//GyTPl0VWeU4bQmrwTh2s0RbQiO9IiwYSCuWCYqw4m6raucoffNk/vUpgP6hpNlMRsdo2uvzOpJp8NE0xP5PfugeFE+o/oga9aBk/F/hqLw5XsCtepHDMCCdJae5o1dTQlKByS+ndKLKz/wAYrm0/UFg/rX8Ucwqju9eNkHemTAk5xTb/d2JBcaYDLIQazPMDjbfxJj2/l1yeBLnU/pXBZLDaEyuFDPYrglU81WpVRG1oevv6GT9FVILHPMiKpAOMsgRZrTsYM3u/J/AQfJUhRTYZHL6g5EQbORqAib3G6i2+qsi0ZDg78IC0hZhVKvTsAmF5KZsPmocjzwRmeDpTiLumq5BOYuVMRYQf5Kuk7cLrEaX4V0Ie2VyzR9lqOkUV3V5eo8ILAy9oNwyjUNIhTbBY41DLX3nHdPYnvowmfnMROWZs8oUWIo416jbJ5baISKzbDV3z4xgbGsXFXLCF7G1ipUhlh+T7fD8eNqaMy30hLHHkMWTp2rOnrUran5Lc5VXMKLmveCB6a8PA5QauFqJcR+AdFj1HiG4du3ntAHtptra00xbgFAqelwQ9JjOT3V0TP465Q2vs9ivPz+itkobOpt1Vv739hIGIx51lhNuOhNuTwTVf/MZi5XMHz9iGwstjx1oi OxEqCa/Q lWA6viYqPkBTZpjroM+AQ2z7a7o3CI3ZWJplnkr5K/yf5exPfgMgP/6ZY8pGcqY+NqvPuyYq/gKk78hOBBohcZL3EtX8EcF2iVoqsifkcQE40dIfIfwLl1OXzjZrbn4mquJfJzQDDG4nKImwDJUaejBH2NVE+BG9ppWwhUzldwz308XGKdbL0BTw8yAjFfgoxVwe4iPJDvMrj8nAniaFWMFEOkb78iQTljtezEVhCBzGEupAvZwiJLfrHRSa/E7ZTXQF+st+sosgIkFujKOsxjiQSUBx2K5Fv1GxLjq/8L4XlDj0qMQrTF10Aek1pk7/9O8sjw6I3ElYbwNtt8miDirzD34ETFZZtC/MsFKQqF/C6XnOKkXwXGV5Y03cmSk5XuODZQFex+NCUhyYDVbhTm8PG7/SkFaVSRwqN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Skip freeing module's data section if there are non-zero allocation tags because otherwise, once these allocations are freed, the access to their code tag would cause UAF. Signed-off-by: Suren Baghdasaryan --- include/linux/codetag.h | 6 +++--- kernel/module/main.c | 27 +++++++++++++++++++-------- lib/codetag.c | 11 ++++++++--- 3 files changed, 30 insertions(+), 14 deletions(-) diff --git a/include/linux/codetag.h b/include/linux/codetag.h index c44f5b83f24d..bfd0ba5c4185 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -35,7 +35,7 @@ struct codetag_type_desc { size_t tag_size; void (*module_load)(struct codetag_type *cttype, struct codetag_module *cmod); - void (*module_unload)(struct codetag_type *cttype, + bool (*module_unload)(struct codetag_type *cttype, struct codetag_module *cmod); }; @@ -71,10 +71,10 @@ codetag_register_type(const struct codetag_type_desc *desc); #if defined(CONFIG_CODE_TAGGING) && defined(CONFIG_MODULES) void codetag_load_module(struct module *mod); -void codetag_unload_module(struct module *mod); +bool codetag_unload_module(struct module *mod); #else static inline void codetag_load_module(struct module *mod) {} -static inline void codetag_unload_module(struct module *mod) {} +static inline bool codetag_unload_module(struct module *mod) { return true; } #endif #endif /* _LINUX_CODETAG_H */ diff --git a/kernel/module/main.c b/kernel/module/main.c index bf5a4afbe4c5..41c37ad3d16e 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -1211,15 +1211,19 @@ static void *module_memory_alloc(unsigned int size, enum mod_mem_type type) return module_alloc(size); } -static void module_memory_free(void *ptr, enum mod_mem_type type) +static void module_memory_free(void *ptr, enum mod_mem_type type, + bool unload_codetags) { + if (!unload_codetags && mod_mem_type_is_core_data(type)) + return; + if (mod_mem_use_vmalloc(type)) vfree(ptr); else module_memfree(ptr); } -static void free_mod_mem(struct module *mod) +static void free_mod_mem(struct module *mod, bool unload_codetags) { for_each_mod_mem_type(type) { struct module_memory *mod_mem = &mod->mem[type]; @@ -1230,20 +1234,27 @@ static void free_mod_mem(struct module *mod) /* Free lock-classes; relies on the preceding sync_rcu(). */ lockdep_free_key_range(mod_mem->base, mod_mem->size); if (mod_mem->size) - module_memory_free(mod_mem->base, type); + module_memory_free(mod_mem->base, type, + unload_codetags); } /* MOD_DATA hosts mod, so free it at last */ lockdep_free_key_range(mod->mem[MOD_DATA].base, mod->mem[MOD_DATA].size); - module_memory_free(mod->mem[MOD_DATA].base, MOD_DATA); + module_memory_free(mod->mem[MOD_DATA].base, MOD_DATA, unload_codetags); } /* Free a module, remove from lists, etc. */ static void free_module(struct module *mod) { + bool unload_codetags; + trace_module_free(mod); - codetag_unload_module(mod); + unload_codetags = codetag_unload_module(mod); + if (!unload_codetags) + pr_warn("%s: memory allocation(s) from the module still alive, cannot unload cleanly\n", + mod->name); + mod_sysfs_teardown(mod); /* @@ -1285,7 +1296,7 @@ static void free_module(struct module *mod) kfree(mod->args); percpu_modfree(mod); - free_mod_mem(mod); + free_mod_mem(mod, unload_codetags); } void *__symbol_get(const char *symbol) @@ -2298,7 +2309,7 @@ static int move_module(struct module *mod, struct load_info *info) return 0; out_enomem: for (t--; t >= 0; t--) - module_memory_free(mod->mem[t].base, t); + module_memory_free(mod->mem[t].base, t, true); return ret; } @@ -2428,7 +2439,7 @@ static void module_deallocate(struct module *mod, struct load_info *info) percpu_modfree(mod); module_arch_freeing_init(mod); - free_mod_mem(mod); + free_mod_mem(mod, true); } int __weak module_finalize(const Elf_Ehdr *hdr, diff --git a/lib/codetag.c b/lib/codetag.c index 54d2828eba25..408062f722ce 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -5,6 +5,7 @@ #include #include #include +#include struct codetag_type { struct list_head link; @@ -206,12 +207,13 @@ void codetag_load_module(struct module *mod) mutex_unlock(&codetag_lock); } -void codetag_unload_module(struct module *mod) +bool codetag_unload_module(struct module *mod) { struct codetag_type *cttype; + bool unload_ok = true; if (!mod) - return; + return true; mutex_lock(&codetag_lock); list_for_each_entry(cttype, &codetag_types, link) { @@ -228,7 +230,8 @@ void codetag_unload_module(struct module *mod) } if (found) { if (cttype->desc.module_unload) - cttype->desc.module_unload(cttype, cmod); + if (!cttype->desc.module_unload(cttype, cmod)) + unload_ok = false; cttype->count -= range_size(cttype, &cmod->range); idr_remove(&cttype->mod_idr, mod_id); @@ -237,6 +240,8 @@ void codetag_unload_module(struct module *mod) up_write(&cttype->mod_lock); } mutex_unlock(&codetag_lock); + + return unload_ok; } #else /* CONFIG_MODULES */ From patchwork Wed Mar 6 18:24:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584423 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00FEEC54E41 for ; Wed, 6 Mar 2024 18:25:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1AE866B009F; Wed, 6 Mar 2024 13:25:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 138376B00A0; Wed, 6 Mar 2024 13:25:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ECB4B6B00A1; Wed, 6 Mar 2024 13:25:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D0F036B009F for ; Wed, 6 Mar 2024 13:25:15 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id AB2231C11B5 for ; Wed, 6 Mar 2024 18:25:15 +0000 (UTC) X-FDA: 81867441390.07.80B29FC Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf04.hostedemail.com (Postfix) with ESMTP id E899D4002E for ; Wed, 6 Mar 2024 18:25:13 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=jbZQZs+8; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3CLXoZQYKCEs574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3CLXoZQYKCEs574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749514; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VnpH2fzc60uRea0gIp8vcDyDoI+2TUkhYU4AGh25C8A=; b=q3s03Bp7N5Mep57rXxjyjJPM4kAYehzVqHgHqqZBh5LnurQWUPi0FoiebBdrmRfDS8pGh3 zxy+nsEyGXXDnOgPtqzIwfH9GYEyHzmhSmxmoVkw5OLlpGSMOVObyKM/l0RpjGCNG/ypoB 1tad8Diwi+AhopRBg5s3WaDWOaYruVY= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=jbZQZs+8; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3CLXoZQYKCEs574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3CLXoZQYKCEs574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749514; a=rsa-sha256; cv=none; b=RrttKOLCVLZMIRFhQuD+q7OITVel15u8CxItkOvYCZkU0fGqIAU9MsFRz7AbwiqHeJA95B RiDeYDLvBq8qzzcuo7ej2YX+FNlxmDt76rRPgONQddDYCpQOFotHTYIH8tTlsC02C1PNG+ aoA2GeTorPdFTJU4us0udgx8MGazkjk= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6096c745c33so150267b3.3 for ; Wed, 06 Mar 2024 10:25:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749513; x=1710354313; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VnpH2fzc60uRea0gIp8vcDyDoI+2TUkhYU4AGh25C8A=; b=jbZQZs+8qv4mGwh1p5CY/eX2J11DWJayhWc8NK0VqD58j0bk+D0PpX646SBrqDJjpJ MI0k4DfKY2ECfXQWA0CiYwq0XElN/Hv9T35yo4DrwkorZW8NVfXHGlmM/hFV2A9B0EiT 8Vc15gqZCeEUWUHJGc1uE0IO5ugnJTHyvw9t0tJtb0LUcSg5AHJSlc7eoD7aGRjCtZE4 /QQ4tvawxuZhCA/UWc0gRa4lbI+AXK8tNQyxyAHWr/XIVABbLV2qDQzCSoQoY+dSvuWr z3S2OhYQxjCWDtR/sRRJhU4BT4EVAejm9wvX+S/JGn8VBjqb4VSnda1/gEasxmcFSGA7 w5Kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749513; x=1710354313; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VnpH2fzc60uRea0gIp8vcDyDoI+2TUkhYU4AGh25C8A=; b=fTPB6l7F5xVslu7GMcXvjpPTZKAxDnzUJp5vKEOvgEPvfZ52XSs1XZmrp/UrspUOsU NHDH6hbasZdq9zozg5NK4vWwUJDWDr/n0VwpenudgbGmDsKcMislRNY0dQHuP4PlABdz GkXsuFbPQRs7Kj/xLJFPKXpgqVLgTmHvL06Rwd+MPlgZk2/LX+hwrNzr6+I0RirmSRmd Ww2PEN7UVTOtsXHC8J8a70Hm6ynv9R7x3zERDfWm72YP7zjCP09si8xDPBxwriGokqVL wOJroFYe2h/iDO7AE0Wx/U9zm4GGM5a/JoQlQORnZ7CKcVWsWRYUEws5SSvDMf8jmBm9 p6nA== X-Forwarded-Encrypted: i=1; AJvYcCWtHO+cQFBax8U/8iUxvkE1cKYjuCDYUFr7kbFABDYFsU+Xm9I04wkK0kxS79CYUMBTrf8uW6xjAenc0hAfBB6TMtM= X-Gm-Message-State: AOJu0YxzZ06gtz2Su6dgqsE/8coxgN5kma2Q1pSb3+tKoPiVGD09sZbb 5ux3FoAO5P9HRabxEbZdntxl/IYB5TpNIwnpBb6UJaqwfXTnlVDBKoc0Ge4W9fWmt21fIvaIPLa SLw== X-Google-Smtp-Source: AGHT+IGX3juLGqkY9E1zWVL+Y8vMZ2gv0idtMMYFCn+Z2jw5L1fpCvfCW0xTdSo9nyCa2B1lhhZALB6C6dY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:1891:b0:dcc:54d0:85e0 with SMTP id cj17-20020a056902189100b00dcc54d085e0mr4037556ybb.11.1709749512799; Wed, 06 Mar 2024 10:25:12 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:11 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-14-surenb@google.com> Subject: [PATCH v5 13/37] lib: add allocation tagging support for memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: E899D4002E X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: jz6iwq8r3nto7bsc8kktnzhrdtxsornx X-HE-Tag: 1709749513-580350 X-HE-Meta: U2FsdGVkX1+5JpK/D35U3HepNklT5rL5ioxOsbWXedQ+drDYy/OR3TyqFPIA0PcQf6cU3QVMXj/c3UDYUARt6tPMVxSHUFn8g0bioB5x5v1YCE13EoOv7FkLzh40nruvzmSYXCYvVxuZ8iMsOZtC6fmHN6FWgt4PnI5ialNJBWdfPreaiYX36bOuMOPQhloZbgJ2KzNwv3BdD7Jl3/0pwK6M00WO+fmNh6JR70yLFcNrnKYe54rB3HQ9VStSdDx3wX83xP6voX5ln+2PXkcXNt0oqjobfUrD70msXkEclha1CYAisYohPhD29SrSjx4ih+Qtjg9SgrL5lOVoVDg8KM3sRv/TjYNukF9jy2X/wAL4y26mrZLXqROl2j3eXQGOtCT190r8vcaS+AfQ0HOYPGnMZZGinahbsFvo8Sr/Dcr+jiDe+yVi8c6QrexgDyT9jy2W/6O81ikiyQuNMX92GQno9U0eVOMqwRwzOncffZFVAldBkRlSxbfq5EfcVem+IGDdOrwPssM+UNJQdpoxzTdmenW0W6WScfNXAjNZB37rejQ3XXhtmckcsX2m+ABTbIcE45Bx85xWj6c6TxSXJ+Kjppyvz0/bHw7i6pQRKQ5SnVOCiLcs2wGMoJ+FwjPqLFLpB3cyBXUAfVadxAcvxiKlz45zY7vmnlrCmzo5roz4DkWx905oZJrA0bTKfwc2i5iJt3xMebCgraeMvDUi/t2tFZC/eK4K9FNplapGUdxeAjMhLoZGS93CBtBcD+VDXG0SlSGWFDUrT9Bq+13eTkn+f7R9j4W2a1xtNSJMYB2TIoUKw8Z7epmlps4B0Z9s9FWswFm1tojMKPdt44awFPT83YrPpulnGFQgfA/tDkIXmqk0r6gPp2iiTixP51mfH9JC5u1wu39YATbUoLAa1Fbnn3l23BSq0dezcDSWxwQ52Juz/XqZxFzq3nS1iLWsaIpSPYERZuweoNytIYR /iJgo2Eg Ns5rd4WNP13E9eIrTxqWijUK9ebhmBP/WXi0RCGfcjJoSksXy5MSMu4WhqnNYv7mAkXB2IqyyUxyPXCPj++Ysmzi9vguIJSk4//ljzA9ywEdTt3qgvRwPTIpiTMSRC+LIHoe1kU3E7wrve3Y1qtdKCMSgmNYXRv/UtDThPkmyh4nL+xZBFdzQCRQM9Olmb9myjO7Ixf9ql8lhtiAzYTXqSo32NBuGq1UkZ+uszfW4tJsHytuGZ/O7Hz22cxgpuOicrXCZBNGoOuTJSdsX/aHaLWALtXHk2mXtdtOjMltFHco4mpHGY65ZrmC1wT2ZAE9ZHfuGz4EWUGk8ea62C1VMRZdj8jSdmphW2bLMEF00x3S8RcDo7X9SrTuy613dL3QMU7UsMIhHw+Jg46RJQIi4dpqJG3+b7cE69PLk X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce CONFIG_MEM_ALLOC_PROFILING which provides definitions to easily instrument memory allocators. It registers an "alloc_tags" codetag type with /proc/allocinfo interface to output allocation tag information when the feature is enabled. CONFIG_MEM_ALLOC_PROFILING_DEBUG is provided for debugging the memory allocation profiling instrumentation. Memory allocation profiling can be enabled or disabled at runtime using /proc/sys/vm/mem_profiling sysctl when CONFIG_MEM_ALLOC_PROFILING_DEBUG=n. CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT enables memory allocation profiling by default. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- Documentation/admin-guide/sysctl/vm.rst | 16 +++ Documentation/filesystems/proc.rst | 29 +++++ include/asm-generic/codetag.lds.h | 14 +++ include/asm-generic/vmlinux.lds.h | 3 + include/linux/alloc_tag.h | 145 +++++++++++++++++++++++ include/linux/sched.h | 24 ++++ lib/Kconfig.debug | 25 ++++ lib/Makefile | 2 + lib/alloc_tag.c | 149 ++++++++++++++++++++++++ scripts/module.lds.S | 7 ++ 10 files changed, 414 insertions(+) create mode 100644 include/asm-generic/codetag.lds.h create mode 100644 include/linux/alloc_tag.h create mode 100644 lib/alloc_tag.c diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst index c59889de122b..e86c968a7a0e 100644 --- a/Documentation/admin-guide/sysctl/vm.rst +++ b/Documentation/admin-guide/sysctl/vm.rst @@ -43,6 +43,7 @@ Currently, these files are in /proc/sys/vm: - legacy_va_layout - lowmem_reserve_ratio - max_map_count +- mem_profiling (only if CONFIG_MEM_ALLOC_PROFILING=y) - memory_failure_early_kill - memory_failure_recovery - min_free_kbytes @@ -425,6 +426,21 @@ e.g., up to one or two maps per allocation. The default value is 65530. +mem_profiling +============== + +Enable memory profiling (when CONFIG_MEM_ALLOC_PROFILING=y) + +1: Enable memory profiling. + +0: Disable memory profiling. + +Enabling memory profiling introduces a small performance overhead for all +memory allocations. + +The default value depends on CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT. + + memory_failure_early_kill: ========================== diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst index 104c6d047d9b..8150dc3d689c 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -688,6 +688,7 @@ files are there, and which are missing. ============ =============================================================== File Content ============ =============================================================== + allocinfo Memory allocations profiling information apm Advanced power management info bootconfig Kernel command line obtained from boot config, and, if there were kernel parameters from the @@ -953,6 +954,34 @@ also be allocatable although a lot of filesystem metadata may have to be reclaimed to achieve this. +allocinfo +~~~~~~~ + +Provides information about memory allocations at all locations in the code +base. Each allocation in the code is identified by its source file, line +number, module (if originates from a loadable module) and the function calling +the allocation. The number of bytes allocated and number of calls at each +location are reported. + +Example output. + +:: + + > sort -rn /proc/allocinfo + 127664128 31168 mm/page_ext.c:270 func:alloc_page_ext + 56373248 4737 mm/slub.c:2259 func:alloc_slab_page + 14880768 3633 mm/readahead.c:247 func:page_cache_ra_unbounded + 14417920 3520 mm/mm_init.c:2530 func:alloc_large_system_hash + 13377536 234 block/blk-mq.c:3421 func:blk_mq_alloc_rqs + 11718656 2861 mm/filemap.c:1919 func:__filemap_get_folio + 9192960 2800 kernel/fork.c:307 func:alloc_thread_stack_node + 4206592 4 net/netfilter/nf_conntrack_core.c:2567 func:nf_ct_alloc_hashtable + 4136960 1010 drivers/staging/ctagmod/ctagmod.c:20 [ctagmod] func:ctagmod_start + 3940352 962 mm/memory.c:4214 func:alloc_anon_folio + 2894464 22613 fs/kernfs/dir.c:615 func:__kernfs_new_node + ... + + meminfo ~~~~~~~ diff --git a/include/asm-generic/codetag.lds.h b/include/asm-generic/codetag.lds.h new file mode 100644 index 000000000000..64f536b80380 --- /dev/null +++ b/include/asm-generic/codetag.lds.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __ASM_GENERIC_CODETAG_LDS_H +#define __ASM_GENERIC_CODETAG_LDS_H + +#define SECTION_WITH_BOUNDARIES(_name) \ + . = ALIGN(8); \ + __start_##_name = .; \ + KEEP(*(_name)) \ + __stop_##_name = .; + +#define CODETAG_SECTIONS() \ + SECTION_WITH_BOUNDARIES(alloc_tags) + +#endif /* __ASM_GENERIC_CODETAG_LDS_H */ diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index 5dd3a61d673d..c9997dc50c50 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -50,6 +50,8 @@ * [__nosave_begin, __nosave_end] for the nosave data */ +#include + #ifndef LOAD_OFFSET #define LOAD_OFFSET 0 #endif @@ -366,6 +368,7 @@ . = ALIGN(8); \ BOUNDED_SECTION_BY(__dyndbg_classes, ___dyndbg_classes) \ BOUNDED_SECTION_BY(__dyndbg, ___dyndbg) \ + CODETAG_SECTIONS() \ LIKELY_PROFILE() \ BRANCH_PROFILE() \ TRACE_PRINTKS() \ diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h new file mode 100644 index 000000000000..b970ff1c80dc --- /dev/null +++ b/include/linux/alloc_tag.h @@ -0,0 +1,145 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * allocation tagging + */ +#ifndef _LINUX_ALLOC_TAG_H +#define _LINUX_ALLOC_TAG_H + +#include +#include +#include +#include +#include +#include +#include + +struct alloc_tag_counters { + u64 bytes; + u64 calls; +}; + +/* + * An instance of this structure is created in a special ELF section at every + * allocation callsite. At runtime, the special section is treated as + * an array of these. Embedded codetag utilizes codetag framework. + */ +struct alloc_tag { + struct codetag ct; + struct alloc_tag_counters __percpu *counters; +} __aligned(8); + +#ifdef CONFIG_MEM_ALLOC_PROFILING + +static inline struct alloc_tag *ct_to_alloc_tag(struct codetag *ct) +{ + return container_of(ct, struct alloc_tag, ct); +} + +#ifdef ARCH_NEEDS_WEAK_PER_CPU +/* + * When percpu variables are required to be defined as weak, static percpu + * variables can't be used inside a function (see comments for DECLARE_PER_CPU_SECTION). + */ +#error "Memory allocation profiling is incompatible with ARCH_NEEDS_WEAK_PER_CPU" +#endif + +#define DEFINE_ALLOC_TAG(_alloc_tag) \ + static DEFINE_PER_CPU(struct alloc_tag_counters, _alloc_tag_cntr); \ + static struct alloc_tag _alloc_tag __used __aligned(8) \ + __section("alloc_tags") = { \ + .ct = CODE_TAG_INIT, \ + .counters = &_alloc_tag_cntr }; + +DECLARE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + mem_alloc_profiling_key); + +static inline bool mem_alloc_profiling_enabled(void) +{ + return static_branch_maybe(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + &mem_alloc_profiling_key); +} + +static inline struct alloc_tag_counters alloc_tag_read(struct alloc_tag *tag) +{ + struct alloc_tag_counters v = { 0, 0 }; + struct alloc_tag_counters *counter; + int cpu; + + for_each_possible_cpu(cpu) { + counter = per_cpu_ptr(tag->counters, cpu); + v.bytes += counter->bytes; + v.calls += counter->calls; + } + + return v; +} + +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG +static inline void alloc_tag_add_check(union codetag_ref *ref, struct alloc_tag *tag) +{ + WARN_ONCE(ref && ref->ct, + "alloc_tag was not cleared (got tag for %s:%u)\n", + ref->ct->filename, ref->ct->lineno); + + WARN_ONCE(!tag, "current->alloc_tag not set"); +} + +static inline void alloc_tag_sub_check(union codetag_ref *ref) +{ + WARN_ONCE(ref && !ref->ct, "alloc_tag was not set\n"); +} +#else +static inline void alloc_tag_add_check(union codetag_ref *ref, struct alloc_tag *tag) {} +static inline void alloc_tag_sub_check(union codetag_ref *ref) {} +#endif + +/* Caller should verify both ref and tag to be valid */ +static inline void __alloc_tag_ref_set(union codetag_ref *ref, struct alloc_tag *tag) +{ + ref->ct = &tag->ct; + /* + * We need in increment the call counter every time we have a new + * allocation or when we split a large allocation into smaller ones. + * Each new reference for every sub-allocation needs to increment call + * counter because when we free each part the counter will be decremented. + */ + this_cpu_inc(tag->counters->calls); +} + +static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, size_t bytes) +{ + alloc_tag_add_check(ref, tag); + if (!ref || !tag) + return; + + __alloc_tag_ref_set(ref, tag); + this_cpu_add(tag->counters->bytes, bytes); +} + +static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) +{ + struct alloc_tag *tag; + + alloc_tag_sub_check(ref); + if (!ref || !ref->ct) + return; + + tag = ct_to_alloc_tag(ref->ct); + + this_cpu_sub(tag->counters->bytes, bytes); + this_cpu_dec(tag->counters->calls); + + ref->ct = NULL; +} + +#else /* CONFIG_MEM_ALLOC_PROFILING */ + +#define DEFINE_ALLOC_TAG(_alloc_tag) +static inline bool mem_alloc_profiling_enabled(void) { return false; } +static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, + size_t bytes) {} +static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING */ + +#endif /* _LINUX_ALLOC_TAG_H */ diff --git a/include/linux/sched.h b/include/linux/sched.h index 998861865b84..f85b58e385a3 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -770,6 +770,10 @@ struct task_struct { unsigned int flags; unsigned int ptrace; +#ifdef CONFIG_MEM_ALLOC_PROFILING + struct alloc_tag *alloc_tag; +#endif + #ifdef CONFIG_SMP int on_cpu; struct __call_single_node wake_entry; @@ -810,6 +814,7 @@ struct task_struct { struct task_group *sched_task_group; #endif + #ifdef CONFIG_UCLAMP_TASK /* * Clamp values requested for a scheduling entity. @@ -2185,4 +2190,23 @@ static inline int sched_core_idle_cpu(int cpu) { return idle_cpu(cpu); } extern void sched_set_stop_task(int cpu, struct task_struct *stop); +#ifdef CONFIG_MEM_ALLOC_PROFILING +static inline struct alloc_tag *alloc_tag_save(struct alloc_tag *tag) +{ + swap(current->alloc_tag, tag); + return tag; +} + +static inline void alloc_tag_restore(struct alloc_tag *tag, struct alloc_tag *old) +{ +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + WARN(current->alloc_tag != tag, "current->alloc_tag was changed:\n"); +#endif + current->alloc_tag = old; +} +#else +#define alloc_tag_save(_tag) NULL +#define alloc_tag_restore(_tag, _old) do {} while (0) +#endif + #endif diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 5485a5780fa7..0dd6ab986246 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -972,6 +972,31 @@ config CODE_TAGGING bool select KALLSYMS +config MEM_ALLOC_PROFILING + bool "Enable memory allocation profiling" + default n + depends on PROC_FS + depends on !DEBUG_FORCE_WEAK_PER_CPU + select CODE_TAGGING + help + Track allocation source code and record total allocation size + initiated at that code location. The mechanism can be used to track + memory leaks with a low performance and memory impact. + +config MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT + bool "Enable memory allocation profiling by default" + default y + depends on MEM_ALLOC_PROFILING + +config MEM_ALLOC_PROFILING_DEBUG + bool "Memory allocation profiler debugging" + default n + depends on MEM_ALLOC_PROFILING + select MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT + help + Adds warnings with helpful error messages for memory allocation + profiling. + source "lib/Kconfig.kasan" source "lib/Kconfig.kfence" source "lib/Kconfig.kmsan" diff --git a/lib/Makefile b/lib/Makefile index 6b48b22fdfac..859112f09bf5 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -236,6 +236,8 @@ obj-$(CONFIG_OF_RECONFIG_NOTIFIER_ERROR_INJECT) += \ obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o obj-$(CONFIG_CODE_TAGGING) += codetag.o +obj-$(CONFIG_MEM_ALLOC_PROFILING) += alloc_tag.o + lib-$(CONFIG_GENERIC_BUG) += bug.o obj-$(CONFIG_HAVE_ARCH_TRACEHOOK) += syscall.o diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c new file mode 100644 index 000000000000..f09c8a422bc2 --- /dev/null +++ b/lib/alloc_tag.c @@ -0,0 +1,149 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include +#include +#include +#include +#include + +static struct codetag_type *alloc_tag_cttype; + +DEFINE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + mem_alloc_profiling_key); + +static void *allocinfo_start(struct seq_file *m, loff_t *pos) +{ + struct codetag_iterator *iter; + struct codetag *ct; + loff_t node = *pos; + + iter = kzalloc(sizeof(*iter), GFP_KERNEL); + m->private = iter; + if (!iter) + return NULL; + + codetag_lock_module_list(alloc_tag_cttype, true); + *iter = codetag_get_ct_iter(alloc_tag_cttype); + while ((ct = codetag_next_ct(iter)) != NULL && node) + node--; + + return ct ? iter : NULL; +} + +static void *allocinfo_next(struct seq_file *m, void *arg, loff_t *pos) +{ + struct codetag_iterator *iter = (struct codetag_iterator *)arg; + struct codetag *ct = codetag_next_ct(iter); + + (*pos)++; + if (!ct) + return NULL; + + return iter; +} + +static void allocinfo_stop(struct seq_file *m, void *arg) +{ + struct codetag_iterator *iter = (struct codetag_iterator *)m->private; + + if (iter) { + codetag_lock_module_list(alloc_tag_cttype, false); + kfree(iter); + } +} + +static void alloc_tag_to_text(struct seq_buf *out, struct codetag *ct) +{ + struct alloc_tag *tag = ct_to_alloc_tag(ct); + struct alloc_tag_counters counter = alloc_tag_read(tag); + s64 bytes = counter.bytes; + + seq_buf_printf(out, "%12lli %8llu ", bytes, counter.calls); + codetag_to_text(out, ct); + seq_buf_putc(out, ' '); + seq_buf_putc(out, '\n'); +} + +static int allocinfo_show(struct seq_file *m, void *arg) +{ + struct codetag_iterator *iter = (struct codetag_iterator *)arg; + char *bufp; + size_t n = seq_get_buf(m, &bufp); + struct seq_buf buf; + + seq_buf_init(&buf, bufp, n); + alloc_tag_to_text(&buf, iter->ct); + seq_commit(m, seq_buf_used(&buf)); + return 0; +} + +static const struct seq_operations allocinfo_seq_op = { + .start = allocinfo_start, + .next = allocinfo_next, + .stop = allocinfo_stop, + .show = allocinfo_show, +}; + +static void __init procfs_init(void) +{ + proc_create_seq("allocinfo", 0444, NULL, &allocinfo_seq_op); +} + +static bool alloc_tag_module_unload(struct codetag_type *cttype, + struct codetag_module *cmod) +{ + struct codetag_iterator iter = codetag_get_ct_iter(cttype); + struct alloc_tag_counters counter; + bool module_unused = true; + struct alloc_tag *tag; + struct codetag *ct; + + for (ct = codetag_next_ct(&iter); ct; ct = codetag_next_ct(&iter)) { + if (iter.cmod != cmod) + continue; + + tag = ct_to_alloc_tag(ct); + counter = alloc_tag_read(tag); + + if (WARN(counter.bytes, + "%s:%u module %s func:%s has %llu allocated at module unload", + ct->filename, ct->lineno, ct->modname, ct->function, counter.bytes)) + module_unused = false; + } + + return module_unused; +} + +static struct ctl_table memory_allocation_profiling_sysctls[] = { + { + .procname = "mem_profiling", + .data = &mem_alloc_profiling_key, +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + .mode = 0444, +#else + .mode = 0644, +#endif + .proc_handler = proc_do_static_key, + }, + { } +}; + +static int __init alloc_tag_init(void) +{ + const struct codetag_type_desc desc = { + .section = "alloc_tags", + .tag_size = sizeof(struct alloc_tag), + .module_unload = alloc_tag_module_unload, + }; + + alloc_tag_cttype = codetag_register_type(&desc); + if (IS_ERR_OR_NULL(alloc_tag_cttype)) + return PTR_ERR(alloc_tag_cttype); + + register_sysctl_init("vm", memory_allocation_profiling_sysctls); + procfs_init(); + + return 0; +} +module_init(alloc_tag_init); diff --git a/scripts/module.lds.S b/scripts/module.lds.S index bf5bcf2836d8..45c67a0994f3 100644 --- a/scripts/module.lds.S +++ b/scripts/module.lds.S @@ -9,6 +9,8 @@ #define DISCARD_EH_FRAME *(.eh_frame) #endif +#include + SECTIONS { /DISCARD/ : { *(.discard) @@ -47,12 +49,17 @@ SECTIONS { .data : { *(.data .data.[0-9a-zA-Z_]*) *(.data..L*) + CODETAG_SECTIONS() } .rodata : { *(.rodata .rodata.[0-9a-zA-Z_]*) *(.rodata..L*) } +#else + .data : { + CODETAG_SECTIONS() + } #endif } From patchwork Wed Mar 6 18:24:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584424 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE973C54E49 for ; Wed, 6 Mar 2024 18:25:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 50D776B00A1; Wed, 6 Mar 2024 13:25:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 492E26B00A2; Wed, 6 Mar 2024 13:25:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2BF936B00A3; Wed, 6 Mar 2024 13:25:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 11DD26B00A1 for ; Wed, 6 Mar 2024 13:25:18 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D6EA4A0F49 for ; Wed, 6 Mar 2024 18:25:17 +0000 (UTC) X-FDA: 81867441474.10.B2D55A5 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf03.hostedemail.com (Postfix) with ESMTP id DFC8A20014 for ; Wed, 6 Mar 2024 18:25:15 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="TSI0uQ/p"; spf=pass (imf03.hostedemail.com: domain of 3CrXoZQYKCE0796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3CrXoZQYKCE0796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749515; a=rsa-sha256; cv=none; b=fX4Bkbok9fMl5AZSHLZvwbIgcJ1xljsPnrYvuPj9gKQsfhyVSNvOH57VqBB/zUxHUoYXn6 ycrsm959e6fjVcu4D6YGrS7nqBNPLRx02GzbqIzWa9i1rR04h2U31eMQtcMB41rnVyl66v j+5Gskk5dwtNR09TgETn3T83IdMwDYs= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="TSI0uQ/p"; spf=pass (imf03.hostedemail.com: domain of 3CrXoZQYKCE0796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3CrXoZQYKCE0796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749515; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nqAwBcStn3+iWVa0bV8AV7qbDJDX846MDYUJTJVUXBw=; b=5YgGSzDWLIOufIxn5TLM2NgtOlz4g5ePiFLY9st7RQL8YkqBs3S4psSdyGRfU3nRWct2Ko foiyHAok0ytC3LbTMd28Pk89xpqToHqjh/P1yXkvnvZZLL7kcCbefaa8UicnJKqtROGpYn vxz6HSOm4tG9TDdYipMR3XnFYbxec5s= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc6dbdcfd39so13633850276.2 for ; Wed, 06 Mar 2024 10:25:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749515; x=1710354315; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nqAwBcStn3+iWVa0bV8AV7qbDJDX846MDYUJTJVUXBw=; b=TSI0uQ/poSxJS/6F9T24CQ3Q5ErR1x5wwHaSOFjQ0++Y77tya9HKMSm1/sVCPIox31 9z9ZHX7jZbj2nGTyCBDJUr/lCWjQIunU6F8u+nQlDVIXyAn/BAQ+lCV0PjoKjuMz21EN 9iP01ZYqioMBGdx1pybPReLC5Jy2GNycWv8Y4ijWUGRZyF4hQFdLsfhhM6/gcTz5mp1e PZSme9Lbr3hjF0XM5N9FqHntQrXNSzB8jEvd/EWQUiIi1HRm0Ndl2+6meWmd/sNsy3rO 9ILCi37zmdvDfgfQIp/5/zszS9PlY+X1JUEk/Ka4Yjqa0t4HaOPhLXj8UDur39RmbNaj 2kfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749515; x=1710354315; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nqAwBcStn3+iWVa0bV8AV7qbDJDX846MDYUJTJVUXBw=; b=YiHaXu6X+p8HsneF0+qD6481u0rsuNqzfLSFAqr3WkbVwsx4Cs9btssNUvWRdS8Jno lDUWxlDQ6nbTUsWGuYKv9xaPa2PuIRjOdhha+3CjcXdKe80907H7ioOgDjbOaEDD9FM/ 982+kNxnizjBUTAno+vhux2ySoZjQxx3BDi+s05w7SVL+XWsLz3V5jXJGuami3xBYnOX 5NWeiNYp8xBpb1DVJHp0PnTB+YlPVuTJL7/8brSbTpxNtyOn2Nzw4h2+wBqG1YWSOfNj ItH028cOOziBeOA/tNOONm6hPi/gZJ6nhWh7lwRmNyCaLE9VdsRPzCQkTB/E/7t++th7 IaiA== X-Forwarded-Encrypted: i=1; AJvYcCU0xCkDxlUOP0ed4jl3mt7RprKjjVmJorB8jL/pF27fSuIceY+F+KRKTUi7AUS1bSONFUVzYUjgPgMAuRviZE6+DeM= X-Gm-Message-State: AOJu0YxXxCvo+LYfICLS4ysoFjJt1Bjc+0anDQnbFOdi12ZmBgHCQnZb eo198QR6kwtOiHxLmWK+dyrs6/wqhJ/vrSrvCJSki9+xzIXECY4COGoNSjFBDXXWeQOOtlE4Nsf hFQ== X-Google-Smtp-Source: AGHT+IHUeBGtAOO+/l11NzJyIxqA9udwD8w5OmuvXdYK78869+3pCDSzsZpyxOUQHcMci8NQK2ebtmqhDyo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:1744:b0:dcf:6b50:9bd7 with SMTP id bz4-20020a056902174400b00dcf6b509bd7mr3993743ybb.7.1709749514977; Wed, 06 Mar 2024 10:25:14 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:12 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-15-surenb@google.com> Subject: [PATCH v5 14/37] lib: introduce support for page allocation tagging From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: DFC8A20014 X-Stat-Signature: qu565ina1i7cuc6czy4zwiqq1tp4iz9q X-Rspam-User: X-HE-Tag: 1709749515-373180 X-HE-Meta: U2FsdGVkX196OdHerOT0ZYfRph28DOKTw2aarUmP4tPo6duA0t2WGzjBtsGDDlQSQtgu1ew2ez7JyTCwTXkaPLms/LqakBEJTNzYVwCqP1nTjbuzytVE7ni6i4EqMsF4WHfCw+wTXn8BXeCI37UUsPqbH2HZ0925kOcXwFIvsaniOf1j7lXEa2xxqagv5vEmnrSgfU9Di1hlyWLvuqjijd1A58KlUCMbJbGcyt9BWTBNNn2DyXcJzucbZARQgt/fCkgzcIEghaUlnGSuAAmpots5aeNlANe5qzPSFsACOkTgV22wZaFqvuwtFiI/we4jRj1ucM7W7kArwWwkE3qePLoSn/FvaMlLFwj9FLk/XGW4jQICWRI3nOnvL1+hRdV9H1ZBaC2HupKyYdcX93xzXgLtfc33IhhByet7lTE+eM7y2GUhZQ9noHFQdPJ7d5NESX8yMSnoNDG3VXGqzwauyM9RC6hbtRstCPhkmbKKfOC5hVCtUb3K89AZlC2NTYMadLiHmOJEYcAYngj/uFkc5X4pSQo+h25D0RKO7lccLR1bpsJO4mKURWaNmD4aGZ5B/fGp7tJ/+uQpvPPIREcuPfpkBHCVlGmbpld2iysvOc8YNNIV4zwqxgS2wL7h6um+N0yRe7mPryt+7MzvAiN+nxgbq7pli3LxVRXIdGdAZbO1ggzhk2gnpnJaVOtV6luvFYgLyATUB4zx3hfNcMxutyvf0b2dBdn8o+6WE6L/vaPRQBJwFs3qKBx06AaWLohbXEVoINdYAYGt8PK/zTgCXPbSf0oqBZHsQkVcVa4dKwoUFrXUDAUJ4k8ZQcs0lva5kgw3IpQPjiPWzsllMir8r067fs7GiB9EKNRGZOdos8zTJn1mJfh9x4KciFbmCbiEIJHiPxUQfjcHzqf4SCV3zg4atUBgk6Iljd9DWgS2UOZRYSjBJMO0YGNKUz0/LidPX128a9011Km7aPDmFe0 4xgSRXaH vhwZLXRMRgUyY2uKu38OA0eR89NYcrC0Y1ZYhkGHaR6OGn5yuYteVaC48TY2nVuFwNAzHrFZe3k39VdUxBcsSbciexLefPvHk3Us7wfspKNTEATWL24TXWhmxmgYaZvGozQ5OOQRoNaqZn5fAx3F/NvKVcxOjdjx76XNuES+EjQnnsrR9JFm2fC3qFDBbHkFiCH3Ui5AZ7BB3sH8Pl0+Pt5E/jdXYca3zE0dpRYTxnKQfO5YaDe42alVy4D2PXwQ4X/BMcnSWraxY81ZR3/O4INujJFgR/ntE3XbFVlBrgxNOuCHbNH/HejL7/4yEoCw8T9KFdtrw2rNeiZAlHsdfSCeG7FNwTfmc38rks1tBKUYghjAQrr/MEpUDHLiLciPCGrgi6kL+XNxsBrPk+bic0D7yRQanbsJrv75yO44gWeOk/PzfnkNw2/27LC8J4aZTvT2bXZ+/5kS974A9h+XRysN595f6WIQ/YLxYzMeHplj3D0UjbvrGPypJwXetX3mUtpmp+L8keC4/S4Ai8QwQu2mFt6wYDFHJkR7rAWrHJdREp0MDcYNpqGqgyETeM4JBRR0Cjmphec9vJUy/QlcNOSUxSO2XuiSbs68kHcPIMMKS7VJw1kSPk71Nw1rqitSRz52l7ewRh4iCGthoTDq7bfBIDjFHr2y0BY1fbS2v79lTpPA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce helper functions to easily instrument page allocators by storing a pointer to the allocation tag associated with the code that allocated the page in a page_ext field. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Reviewed-by: Vlastimil Babka --- include/linux/page_ext.h | 1 - include/linux/pgalloc_tag.h | 78 +++++++++++++++++++++++++++++++++++++ lib/Kconfig.debug | 1 + lib/alloc_tag.c | 17 ++++++++ mm/mm_init.c | 1 + mm/page_alloc.c | 4 ++ mm/page_ext.c | 4 ++ 7 files changed, 105 insertions(+), 1 deletion(-) create mode 100644 include/linux/pgalloc_tag.h diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h index be98564191e6..07e0656898f9 100644 --- a/include/linux/page_ext.h +++ b/include/linux/page_ext.h @@ -4,7 +4,6 @@ #include #include -#include struct pglist_data; diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h new file mode 100644 index 000000000000..b49ab955300f --- /dev/null +++ b/include/linux/pgalloc_tag.h @@ -0,0 +1,78 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * page allocation tagging + */ +#ifndef _LINUX_PGALLOC_TAG_H +#define _LINUX_PGALLOC_TAG_H + +#include + +#ifdef CONFIG_MEM_ALLOC_PROFILING + +#include + +extern struct page_ext_operations page_alloc_tagging_ops; +extern struct page_ext *page_ext_get(struct page *page); +extern void page_ext_put(struct page_ext *page_ext); + +static inline union codetag_ref *codetag_ref_from_page_ext(struct page_ext *page_ext) +{ + return (void *)page_ext + page_alloc_tagging_ops.offset; +} + +static inline struct page_ext *page_ext_from_codetag_ref(union codetag_ref *ref) +{ + return (void *)ref - page_alloc_tagging_ops.offset; +} + +/* Should be called only if mem_alloc_profiling_enabled() */ +static inline union codetag_ref *get_page_tag_ref(struct page *page) +{ + if (page) { + struct page_ext *page_ext = page_ext_get(page); + + if (page_ext) + return codetag_ref_from_page_ext(page_ext); + } + return NULL; +} + +static inline void put_page_tag_ref(union codetag_ref *ref) +{ + page_ext_put(page_ext_from_codetag_ref(ref)); +} + +static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, + unsigned int order) +{ + if (mem_alloc_profiling_enabled()) { + union codetag_ref *ref = get_page_tag_ref(page); + + if (ref) { + alloc_tag_add(ref, task->alloc_tag, PAGE_SIZE << order); + put_page_tag_ref(ref); + } + } +} + +static inline void pgalloc_tag_sub(struct page *page, unsigned int order) +{ + if (mem_alloc_profiling_enabled()) { + union codetag_ref *ref = get_page_tag_ref(page); + + if (ref) { + alloc_tag_sub(ref, PAGE_SIZE << order); + put_page_tag_ref(ref); + } + } +} + +#else /* CONFIG_MEM_ALLOC_PROFILING */ + +static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, + unsigned int order) {} +static inline void pgalloc_tag_sub(struct page *page, unsigned int order) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING */ + +#endif /* _LINUX_PGALLOC_TAG_H */ diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 0dd6ab986246..3e06320474d4 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -978,6 +978,7 @@ config MEM_ALLOC_PROFILING depends on PROC_FS depends on !DEBUG_FORCE_WEAK_PER_CPU select CODE_TAGGING + select PAGE_EXTENSION help Track allocation source code and record total allocation size initiated at that code location. The mechanism can be used to track diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index f09c8a422bc2..cb5adec4b2e2 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -115,6 +116,22 @@ static bool alloc_tag_module_unload(struct codetag_type *cttype, return module_unused; } +static __init bool need_page_alloc_tagging(void) +{ + return true; +} + +static __init void init_page_alloc_tagging(void) +{ +} + +struct page_ext_operations page_alloc_tagging_ops = { + .size = sizeof(union codetag_ref), + .need = need_page_alloc_tagging, + .init = init_page_alloc_tagging, +}; +EXPORT_SYMBOL(page_alloc_tagging_ops); + static struct ctl_table memory_allocation_profiling_sysctls[] = { { .procname = "mem_profiling", diff --git a/mm/mm_init.c b/mm/mm_init.c index 549e76af8f82..2fd9bf044a79 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 16241906a368..9a91c8074556 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -54,6 +54,7 @@ #include #include #include +#include #include #include "internal.h" #include "shuffle.h" @@ -1101,6 +1102,7 @@ __always_inline bool free_pages_prepare(struct page *page, /* Do not let hwpoison pages hit pcplists/buddy */ reset_page_owner(page, order); page_table_check_free(page, order); + pgalloc_tag_sub(page, order); return false; } @@ -1140,6 +1142,7 @@ __always_inline bool free_pages_prepare(struct page *page, page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP; reset_page_owner(page, order); page_table_check_free(page, order); + pgalloc_tag_sub(page, order); if (!PageHighMem(page)) { debug_check_no_locks_freed(page_address(page), @@ -1533,6 +1536,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order, set_page_owner(page, order, gfp_flags); page_table_check_alloc(page, order); + pgalloc_tag_add(page, current, order); } static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags, diff --git a/mm/page_ext.c b/mm/page_ext.c index 4548fcc66d74..3c58fe8a24df 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -10,6 +10,7 @@ #include #include #include +#include /* * struct page extension @@ -82,6 +83,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = { #if defined(CONFIG_PAGE_IDLE_FLAG) && !defined(CONFIG_64BIT) &page_idle_ops, #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + &page_alloc_tagging_ops, +#endif #ifdef CONFIG_PAGE_TABLE_CHECK &page_table_check_ops, #endif From patchwork Wed Mar 6 18:24:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7C04C54E41 for ; Wed, 6 Mar 2024 18:25:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 24BDB6B00A2; Wed, 6 Mar 2024 13:25:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1D59F6B00A3; Wed, 6 Mar 2024 13:25:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F1D1B6B00A4; Wed, 6 Mar 2024 13:25:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D9DAA6B00A2 for ; Wed, 6 Mar 2024 13:25:19 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id B95341A0176 for ; Wed, 6 Mar 2024 18:25:19 +0000 (UTC) X-FDA: 81867441558.19.0AAF6D6 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf22.hostedemail.com (Postfix) with ESMTP id 06246C001B for ; Wed, 6 Mar 2024 18:25:17 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=oOB+CKpg; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3DLXoZQYKCE89B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3DLXoZQYKCE89B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749518; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bHZwja06se4PI5NvmXw6g0w6FqOSRgAaf+Zh29a7Axs=; b=l8vXhiTNgzUxmVyGZB1DaL2sR3A9OAduGovdUgcIMEXME5Uy/KVqSZlX0ShFpTZfHONsJH Bm48jsrfx4FRORWlAEIUEovnbwHU+5Y71wpYaqnEsnECqlZ4NGfthOM2b2rWMbhV2GVc2x aIZKcD8iIOu4oaPlXD40EIRH97tKG4A= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=oOB+CKpg; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3DLXoZQYKCE89B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3DLXoZQYKCE89B8v4sx55x2v.t532z4BE-331Crt1.58x@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749518; a=rsa-sha256; cv=none; b=ZCP3hGx3NzeMNeouYTLigG44+szttr4BiXHBN9Y/Ia38msYrjMOZhDDNp0iukU5sph4XN0 aJcoQwdyVU4lbv9YVXxzjHDyqPevrhiaMNUz+pSbFktGxRTg5sF2UneOvDFgtKScsHgt4q XeoxtwXVM/CGbndv7rggpH3D4a7YLqo= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc647f65573so12656258276.2 for ; Wed, 06 Mar 2024 10:25:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749517; x=1710354317; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=bHZwja06se4PI5NvmXw6g0w6FqOSRgAaf+Zh29a7Axs=; b=oOB+CKpgL6QEJTstBWHNtCsNNENAERdxRn9E5dqhU6zuyLXXUWPpJZu9PyfqgaC/RV ZdV1pE3nyXOEssc03ZoKnBTAWFdLGkB3/0RW4jfDOeYxM38pcl0GoTmjrg5iGcdYHdhv zcApsFokU3yr6H3wdX1mZOSP1XRvkk+Zr5Kwt75TDdNgOCA40olBuheeUCHV+h8YaaU/ 30ZwS5TqGNy9SJhO0iLYhJjF0gz2encNdbOHAVn0BmYRLZCR8krM6IIHLKHBxa+qG0Za /JrcunloGn7LahxRVHaz1RRZmVlyIXUAA4uIKyz2xQLQ5NSFiC/uKQLcBVqOa+xP2dpM V5RA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749517; x=1710354317; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bHZwja06se4PI5NvmXw6g0w6FqOSRgAaf+Zh29a7Axs=; b=AZXh0VLe+yZDqa4IgCjuUWCNKDbbTH3W+T+vJvPTUfM86Wg/h6XzcODv3iNVkXQrwk pwA5ut3NyTa/ThcxRHwox7q8ZomH28LUgwi0LoAcFhuTLvdu2KHznBnEJx6ma5INT3QN zoyq29mbI1SjlkAabVsKZQXoqnl13tWE0/yEAmHxs+wY9c7xzzxA985FXkd9C6lolltK gm5/DpqzU+roozSTmEMpVutNooSLo6Byvj9Vnj0SeK9G+V9m4Sc99jNKm4pIyYx5D7HO XQrG+sCkhz3+dHbjVitZTXRgg99SZibC8aTcWswEzwHblFWBbTYEK9DKJtOopuh+Orhz dgHA== X-Forwarded-Encrypted: i=1; AJvYcCXKwcGXNSzAwmVRnkADZbSTUJMh2c2AYqHVVOeOnqM5JO4vgK9Qkj4H1JdfDkgzYVOMr7wBSP8lmminRO7blkirKRA= X-Gm-Message-State: AOJu0YwRha0LjkaMsVsRzl5e8/OmqSSxkDaHxI8zKf4fkZdWvu4sIDPP e4miNNKckj1IWlQS+FOQBEUFTlal9bHt4WqgFwtZeBB5oo17TQcAn3maOP4gPLQ7CNjeD7udXi1 B6w== X-Google-Smtp-Source: AGHT+IEY0dq37W/YL8K/UAYJglAS6s8iwOM7lJC17R20EwJUoGE14Vwyo3M1UA8Agcy0JGlj8awomXhRYvE= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:18c9:b0:dcd:5e5d:458b with SMTP id ck9-20020a05690218c900b00dcd5e5d458bmr4001383ybb.3.1709749516858; Wed, 06 Mar 2024 10:25:16 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:13 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-16-surenb@google.com> Subject: [PATCH v5 15/37] lib: introduce early boot parameter to avoid page_ext memory overhead From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 06246C001B X-Stat-Signature: 9czf3c46h5g3badw8f46s3cjswnftpwg X-Rspam-User: X-HE-Tag: 1709749517-980484 X-HE-Meta: U2FsdGVkX1+t+Lq1/60dz5dbJAyxFYF8fzk2WEIngZDhi/q50cVGA07wViNqGwc+Y4z/YRlRJlGr0X26/q8EBpEBIH7YB718Ftf02fLGvQb3bTKzZICa0SYOFg8mizcUB8ea7iuYaVwKcDvds0poyUbtUqemVN3bqzJYJ4P/RcBNWfbDBbomGrRjaRC2/v6LeCM3mx9LeaZt4otRmu1s0JseoepB0u0SB+XQ5DAnYTiAmcYRKYxkycOcTHp2szx+HjQrlM+mq2IM9AXF0KYTa0EonmPxv95t2/WSiBeX4WhW/nqB6XdScffCHf8++pb43lDqkNHFPUuMY3zalBT0UFzyHxLnvUpkE8to3DjrpmPLBSQDgoyyE8Go7amhwA6Xv9Ft/Y1NcXLwbbA2swz28BFc1yKHGziFM1OiYPmlXfabSlDDrYbluJA1K4YYn6RXE5OikKbBTqeGEUHEqbTiO8CmXWLVX7cj8++mE8mqiVLqJMDwJ3qvtsJUKtqIZBpaZCt0coE+T8oPxWGT/FQkLcVIEIQL6bZnHbp1JcGv+wh9wiE79xlLtb6eXj9JOiZZ63agHQMmG+QmjsOzPG39g8AIulUq3gOjL4elYA8ang+eBt8YySW5U9+LzdcbjrJPh/gG7Zmsd79hEL0NwJ+fa0QZGT8P0eNY+K0V857anmkefjI8IWq//CtXerLoCk4E8YH53UzO4sxsEbPO605buXT/bfUh58feOzwaKLAjQYIIUcBCaWWXidFlyqovMKmoPHp5tr4CatBYSsabcM+F7iDfdDhjKd9SL3TKofqIHL7SPNF+q2NqQMEImYyFBvqCDlmYf2U4cwKk3SjivEYNK21WnJ4qNmT8A16ZTUovej0plhBxb3sLI5/IeZmaf3yVRiryPrJv2QNvjj8x2SBZgibCnip3FrP86kIM9cntrPaXuiOyHf0R6wqBvgfwc2I7pS+NxFncouh6Yw42huZ UOt+b+1D vxgytwZBpldqupx6PtTzG/SYcDyJ8jXedPjmTWWN1IF1tYlDVgdhXl9ps02GEwmKsYEbdi7o8npgRkkbZKJBF27TSpGDmFqdrQxVzq8AHBTQUmaSnwgQwh0h7L5ZYL1lI6tuOLCX0K0pIX9jq2e4pkEKhNLmBIJ16yILKnhYK6Ju9qapeUGQgzu45bTf/ESyRCjWjOn7z2ubL0igQ1/5gwuaGeXI8+/bs91edVATg41Hk2d6skveYW0OluadTPJlNPG2wFL9gxlM539tFYhtOsUJNE2tlM3BWSAxVQk8dz0mwZVC3JeuQhSYT54wj+28iwF/RXNg6A8CdoUWiT0SQbGAZPuxt2HN6hAjpCtfO4/hXk4fG2rPcrnQE/Z7Sr07S2zysk00L2smmq1DRzs4YK2s4C3BG2b4Rm7sV9J+TNzDPV4kiqgjfCQNl+9GcYMvvLqhd2BLap6bbjyucddIgvHGZfg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The highest memory overhead from memory allocation profiling comes from page_ext objects. This overhead exists even if the feature is disabled but compiled-in. To avoid it, introduce an early boot parameter that prevents page_ext object creation. The new boot parameter is a tri-state with possible values of 0|1|never. When it is set to "never" the memory allocation profiling support is disabled, and overhead is minimized (currently no page_ext objects are allocated, in the future more overhead might be eliminated). As a result we also lose ability to enable memory allocation profiling at runtime (because there is no space to store alloctag references). Runtime sysctrl becomes read-only if the early boot parameter was set to "never". Note that the default value of this boot parameter depends on the CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT configuration. When CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n the boot parameter is set to "never", therefore eliminating any overhead. CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y results in boot parameter being set to 1 (enabled). This allows distributions to avoid any overhead by setting CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n config and with no changes to the kernel command line. We reuse sysctl.vm.mem_profiling boot parameter name in order to avoid introducing yet another control. This change turns it into a tri-state early boot parameter. Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka --- lib/alloc_tag.c | 41 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index cb5adec4b2e2..617c2fbb6673 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -116,9 +116,46 @@ static bool alloc_tag_module_unload(struct codetag_type *cttype, return module_unused; } +#ifdef CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT +static bool mem_profiling_support __meminitdata = true; +#else +static bool mem_profiling_support __meminitdata; +#endif + +static int __init setup_early_mem_profiling(char *str) +{ + bool enable; + + if (!str || !str[0]) + return -EINVAL; + + if (!strncmp(str, "never", 5)) { + enable = false; + mem_profiling_support = false; + } else { + int res; + + res = kstrtobool(str, &enable); + if (res) + return res; + + mem_profiling_support = true; + } + + if (enable != static_key_enabled(&mem_alloc_profiling_key)) { + if (enable) + static_branch_enable(&mem_alloc_profiling_key); + else + static_branch_disable(&mem_alloc_profiling_key); + } + + return 0; +} +early_param("sysctl.vm.mem_profiling", setup_early_mem_profiling); + static __init bool need_page_alloc_tagging(void) { - return true; + return mem_profiling_support; } static __init void init_page_alloc_tagging(void) @@ -158,6 +195,8 @@ static int __init alloc_tag_init(void) if (IS_ERR_OR_NULL(alloc_tag_cttype)) return PTR_ERR(alloc_tag_cttype); + if (!mem_profiling_support) + memory_allocation_profiling_sysctls[0].mode = 0444; register_sysctl_init("vm", memory_allocation_profiling_sysctls); procfs_init(); From patchwork Wed Mar 6 18:24:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584426 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A065C54E41 for ; Wed, 6 Mar 2024 18:25:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5CDA86B00A4; Wed, 6 Mar 2024 13:25:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 57E006B00A5; Wed, 6 Mar 2024 13:25:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3D0A16B00A6; Wed, 6 Mar 2024 13:25:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 2968C6B00A4 for ; Wed, 6 Mar 2024 13:25:22 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E7CCF1C119E for ; Wed, 6 Mar 2024 18:25:21 +0000 (UTC) X-FDA: 81867441642.29.5C19B23 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf21.hostedemail.com (Postfix) with ESMTP id 1AA311C000B for ; Wed, 6 Mar 2024 18:25:19 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=R9UljYLM; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of 3D7XoZQYKCFICEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3D7XoZQYKCFICEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749520; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OEAYBW3Z4VXc6aDoxC6XGC2RO1GGkHEsC5P6UG5rRTs=; b=AsqDS2UP5WZZJEyijlT0Yx01oaN6Hz+eYvJkmM+f25Ck3A6w6TDMhjJhAsWgY2RXlverT2 6DEgu3qYRmi7aKx5RI9xFudzaS3o2ZcMoCIBM7Wf/oJgBV8omjet5Mr46PzK0waqOSNQ45 Oj89Dz5zS727WlX+3Rq+OAk6ogyvdf8= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=R9UljYLM; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of 3D7XoZQYKCFICEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3D7XoZQYKCFICEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749520; a=rsa-sha256; cv=none; b=tzluGLJYlFEH3jpQos9vUPyyQz4eBEXtZkpdowbEdu/r69Pcp3qWeNbPc2b2m1Ylswf1/X 2KDSz588Il6WuTkHERYiZnwr1tpC9DEx3n+N47QqZ8CuFEFKiXiCqjE5SbMAoPs+5Kxs4I MeqFi0xvfAxYp1RCAo1GXzIZsbKcJQs= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-609ccbcee7bso716917b3.0 for ; Wed, 06 Mar 2024 10:25:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749519; x=1710354319; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OEAYBW3Z4VXc6aDoxC6XGC2RO1GGkHEsC5P6UG5rRTs=; b=R9UljYLM5l70PSONsgpHKc2uICbxqezAqSUBoPMarneY8y43fwMMXZ86NPDeMt2PxK 85pNBKrw9bRlpqCd2d3lT9eSPdnWOdmzUv8yPhN6pjTythZ1LIvht7I/idItK4TKWFR2 XEYqzwolK/Su31dAKR+FhgyntVG4AeiUQQvhs20FQgiz2UXovyqon6WWOtRu9Q93EjG4 LzfsCJP1ja63Ek5S6858cuR2rELkF11K/4LvGhjES+Jb5PcAk1doDRmvnSpfZGA0msyL I44O9AxVfVQEmyUUyHmElKN6F9yvoVwbzY5pfWUmSX4w/i66UPWfd/c1qTSV9t2r44mc 08fA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749519; x=1710354319; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OEAYBW3Z4VXc6aDoxC6XGC2RO1GGkHEsC5P6UG5rRTs=; b=meU7cPMGetFQr6cmyTRGBOEt6FFv6EjV+LtHuozwHEemyvrfQwxvQDAtK4bimOUzRW bzecIuUf2NKlJabZrLcCtyxx/gmKsIhlvfrzvxHBLhoRnaxEhQsiAZM35WgH6afgNllR eLKNzvjP+s26HheO6n5VX9EtRnFAXIHJBCj9myyybhgXwfTDYGRJts9dJ26xGgvsTkgz G5wf1jR/ZDAkEGu5/pvSHa0vjuo2/ENehUKGJRctTSVSvVcqFp5FIKUgYXo9Z4yAvRR6 G0mdycVyEdZC3f3+34E2p2v4hnlZIVwr0K78PKhyPUwIi6VQwxviuOincZRe2xFZKCDf PHHg== X-Forwarded-Encrypted: i=1; AJvYcCUm/v/ycmP868huMmEoF6wJuQ8wSIM0jnbH9WxBsI3PH2CTv0MxuNbsql/P4LtaBbwAkfHbFHl1sFORSyQKYqxFXhc= X-Gm-Message-State: AOJu0Yy2K0MohZNo79SWuwIdw2m+PenS652g6zVfYYNNN8TcT3nl2Z3W H6Xfaf1eIsHmhpjG72uYtusdMqIa0zEl8XuSY5zj/hHEJkrKQseWWLwgQfViNWgmteQykFA8Cnt YGw== X-Google-Smtp-Source: AGHT+IGRA5aw0Cx2t1JPgn1w/V3cVE9EShRxIkrJJQcyCjV/1XkZdkPnLkct9rPYqab5CU3NhvDSCALRADo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a81:9950:0:b0:609:4d6f:7c0b with SMTP id q77-20020a819950000000b006094d6f7c0bmr3566046ywg.4.1709749519078; Wed, 06 Mar 2024 10:25:19 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:14 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-17-surenb@google.com> Subject: [PATCH v5 16/37] mm: percpu: increase PERCPU_MODULE_RESERVE to accommodate allocation tags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 1AA311C000B X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: gks8cgjwetu6m8kxq8tcm5sswtfjts7q X-HE-Tag: 1709749519-434513 X-HE-Meta: U2FsdGVkX1+ZCVFze+hCzj1plGD9TqEuXRaZdfu6GyFC1RwdePLRkLWRDrDukYrQMSVygGwmNLj1X70lzRcGHo2Odi8vsyBQvTf5sHkaG26I7TNli4DrzoZmP1l6MK7ijLjdF728JnkL30Vuz5R34sdrjh6szoV9wScGIvd07Rwt/01d3B3t4c5iwqgfxG99xPAN1/MKv6YgL4ezNW2xmVYIx5Ech/g5nncMGfn2LfJXO9ldOkkcNfe7MYtDw6CDFDKmrVX0aOjd22Lpdg+5g3fWt50qh3IRLmoqTOgGr7+g/MmrCTHF9IyDnSHkmxedJAt6HF5ukHauTshk7mzYmejEEZcH4o3xuCfxTKZhDX5KAnBvk2OEG0aIkYPdp2IFuo758AiPUbyw9VISy0SIt1OdbROIDzxHpPXdmvTjJRlOYkGcoEujF2I5px1vDEZPSjw1s7ru/2Jg+YvsFfD4IgI2/sI7fT12LUTMtJR0LDqPQ2VHHNkS1wQ2333xrRdUEct3SKVgqbq6t1qhZWez2kzIU3aVhtz5P5JWYYFz4r6f7IpiiqWFnhiyjbbSybXWb2v6FM9aoEE/79j4lPMcZojpD/gt3iuvynKSzoOE7RlTh/UakrFev9oAW46xErOCJ5dBtCAheOEI7jwX2lSokkfGeDwK+seArAYavnFxP5dkWHlJnH2GK6QdeCIwkJNPTPwkoW4YkHo1COXx9NcYPh9T3bpZHMaf9wUsjxK51eafIqZqaXLedWMmRwIZuFexwFRHHA2g4t+LqO2IVsV/yf7CpAEO2JHPIGrDm/GqatawNcvGaEVZxScfxoa0dCd6p68Le7BIhb5fqxl/RNehsyYgWJBtbz+Rk8QPnSKssdBxyHlCJVAhEX8Its/YMVbTtpb2HQ4TdL0qCY9LmLpcKw5AAY01P9I9/VfQW80kBVSQ4x/80axNXpB8wNcu8Uyjk3ngzBn3ado69pkKZJ6 Rtgaalho dwDROv/1VDD+LqwQ2HFY2wlaGCchnpP4oJ7W32+asBYSiVPQoOIHtpFvXMFCQkh14qgiQwyvwM+aoVmG2KJhAztul/l7uvrNonVDYDPcYIostIDOBFfkY3m70ui9NwqigIu6VgSIPb/XCJluAdpl0OVv9WsN94b6l8dgRWtTjnKdHuQA42yQ3BsBjBs+4VkgXuPJpVNTb+F5gNTIXlXo0bhXMgkTaCRyDLZhVG5ui2b6tcVh78ArhtR0wdl24yIuwQY/H+D5NSPmr0ceNc0/LzFj1zOpG4K7SjNWK+u16Ss7BOpVFONWc5N/fu5rVH3PYgqUTi8iJRS0t2qfm/wLMfBQ/0bkhmEt5ZHPq2i8xtRjZXAKlSgmRz5g+cvont8Fv0214aimrM2En7Oe9AOY93zi2OpF4LVOSgjj8dQk2aiO6dwBtiK5L65Ek0Il2un1mNdRxIJ7xCkauh6244FOqB3wFFn6NQSiyomOzrVVqEP6E39gm5Zx85euWX1GDNIZciae6N84Z+wuJQOmHxSEngIJD34b5hmAwfWuoZ8Zj+2FY8jEqfnEIDYzyKJ4g+wfdIVLn/kBcJbd6ZL3VQBvSU+mkR1znZJhaHGuqBZzWoLrLg2jtLwTzKvQ+nySAqJxgLpzyhrCZdsKD0Z0GqJ4o2JBY7Jks9RDHMCQipct+2BMCViryyhvpUSL3H7J2vc++zKk4saX5dPLy+z6N68LIvpvYsCV6leFUSe0Q9OIF14fysC/UH4Iquy6deOBxaA1F1i8cB6I2obHrXIzIG0kCHgEXHWyENv128MH2YFoFsqjtn0k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: As each allocation tag generates a per-cpu variable, more space is required to store them. Increase PERCPU_MODULE_RESERVE to provide enough area. A better long-term solution would be to allocate this memory dynamically. Signed-off-by: Suren Baghdasaryan Signed-off-by: Kent Overstreet Cc: Peter Zijlstra Cc: Tejun Heo --- include/linux/percpu.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 8c677f185901..62b5eb45bd89 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -14,7 +14,11 @@ /* enough to cover all DEFINE_PER_CPUs in modules */ #ifdef CONFIG_MODULES +#ifdef CONFIG_MEM_ALLOC_PROFILING +#define PERCPU_MODULE_RESERVE (8 << 12) +#else #define PERCPU_MODULE_RESERVE (8 << 10) +#endif #else #define PERCPU_MODULE_RESERVE 0 #endif From patchwork Wed Mar 6 18:24:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23C62C5475B for ; Wed, 6 Mar 2024 18:25:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 77DF06B00A5; Wed, 6 Mar 2024 13:25:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7056B6B00A7; Wed, 6 Mar 2024 13:25:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E80C6B00A8; Wed, 6 Mar 2024 13:25:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 37D536B00A5 for ; Wed, 6 Mar 2024 13:25:24 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 0955140658 for ; Wed, 6 Mar 2024 18:25:23 +0000 (UTC) X-FDA: 81867441768.07.53571E9 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf07.hostedemail.com (Postfix) with ESMTP id 4A4FB40010 for ; Wed, 6 Mar 2024 18:25:22 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=yVtMgmnC; spf=pass (imf07.hostedemail.com: domain of 3EbXoZQYKCFQEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3EbXoZQYKCFQEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749522; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ce5q/5s2y8sTRUjXlJUoDQaDDC72MtCTDHZZLC1kTDU=; b=4XtkMhZrIwBdFkBoYD7mfuKoL2jamclidFiyNmyBDlKFV4CFcLP9/FgwRx2KG6sqZBw1mL gx+qvFliafC5CTnwXD3qoYBy+ZffiQA9Hg/SWmF4eRvvCE79KCe4SES8mWVU1b57f5OoSD RxemAYoQN7WWRdzTyFw1lYxfy3foQwI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749522; a=rsa-sha256; cv=none; b=llhcrMFo3AQZA44TzDewPMDPGRm7ATv1FkM23DRB8E1c10jWRN6Cz2+pe7SgrfuWVIwu2k +fGQtEqxpB0vCn87ub36ldfNwgHbhZJJs0F3oh3miSB5mBPENYEZ7Dir6QOnu37aZ8KFb3 JjeS7cKWDoEiCsVQ8npNi1SLPhrByxk= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=yVtMgmnC; spf=pass (imf07.hostedemail.com: domain of 3EbXoZQYKCFQEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3EbXoZQYKCFQEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60996cdc37cso160077b3.2 for ; Wed, 06 Mar 2024 10:25:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749521; x=1710354321; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ce5q/5s2y8sTRUjXlJUoDQaDDC72MtCTDHZZLC1kTDU=; b=yVtMgmnCgd1eIOobdDD/OR6+DGWufgkBLJEk+NHCYPfRGJ9D4Ft+cAcT9wUR2apsE2 7+wXnky0h+J6BQyyuJk4fcFg6sgAsZO//BVFb2q5JAv0Vf44n1zfU0rQAdhPuVEnuHxD Tbo+UovOnnVa8DB3zyHNUMc2pJHUaFzjLGQIRLcwJOeBpkNqzaEcN+nzF4MGWpf+tqhO cpv6C5abNmj41ZuiUrXSh/n7SNkNW5YCTflWQrRdd0zEWSz6+Nn1b+xkEWYNvDfeaZ1f JWlkugJh78kO/2AmLIMZnAgVBXY66Xiaek72jeJzio26B/FTl3WgRXF2wyagDUu9B8wW HMcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749521; x=1710354321; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ce5q/5s2y8sTRUjXlJUoDQaDDC72MtCTDHZZLC1kTDU=; b=KDDEzDHx/aQCnZ6FxEixA4AcIeXhKoUnGTweHrjebBZ/KxSVFjk0UQlDuz9ragBVlX VEtTEVTLep46cPbG0YZA0ri+QdUrvpZCIiFSlU1Vxy4CDt4gpV2PzsyTCed3Hg11nrZL T8mEPSyQZNQj1HxLXNnjIf1cvrc1nwv1EMMqJqKra4D61JSg7PMmXpPAU5t2XjOovxoJ CZE/2HGCdVQMDKW+7agD4nTiTsMBujtC1e5PIE8oHYwRJNM1ShzBrslumy7K/+9qmLVA wBeGwKgyz5OfsJeTJNpYSpB7V1283253zIy7ao36azZT+TkFwni/vcDBoRR6+jtCncGq G8/A== X-Forwarded-Encrypted: i=1; AJvYcCXouqLzonVsym8lDSgWhmX9MJmFZog8/8XDj2djfkO/dpd4lNS/T01f523cFPgxBoRsleQseihRxWifyWkY7pWeBZA= X-Gm-Message-State: AOJu0Yw/D4PPh/HJ16nZx8SBnfQ9iifDkxwfA0Dspdx6ernEgV9So+c+ IwQg7dPtj7GxCer7iZzttiPd8H7UyQ7FfTIDmpBzsPwRXupEkM6QOLP/dEBL1H0kLqt8ozqSGhl uRA== X-Google-Smtp-Source: AGHT+IH32/IZGhdtDhHfi0erFgA8zBoZwZ5Yl8o/YC76tmXsM0VTu/ocS/EURR3ecpb+NdvcUbRKq6AuK40= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:1821:b0:dc7:9218:df3b with SMTP id cf33-20020a056902182100b00dc79218df3bmr687403ybb.10.1709749521260; Wed, 06 Mar 2024 10:25:21 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:15 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-18-surenb@google.com> Subject: [PATCH v5 17/37] change alloc_pages name in dma_map_ops to avoid name conflicts From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: pat54ebm3upsgxgejokks8zpyqcra8hx X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 4A4FB40010 X-Rspam-User: X-HE-Tag: 1709749522-554255 X-HE-Meta: U2FsdGVkX1+V6nrRdHQDD6OrqOsY+lJtmnDbcOqqswuEc2778oPZdi+LBMXkPkGJ3iznZJtLHZyvaArvIAhGfeVYfCDD1l+/4Pn7z/czS56+NOw//QsQTRKt7/Y6YDxQrkuiAe1J1a80eiEQ9kouv+xvfBHZMQJ2kXRUL8N6sU3Kl8YNA4P7l4TP/cg3orqpLpxMYcuc/fNQA8bGMs1MOf87jgjC7VAY/GzOf7g2eRQ+CAKn/HsNbnBiJVLxMYc50h7fYGPNkG5f1uHMFtF3SXxvW+mud3dZ60jQuw7ehZSLa9laWWxqZDGHQrd7V9xltzLdKK9eBAbgy1GDsRQw3z0cFfG9QEHThZF8N6WY7S9tW/zdOwoBa/rSs/wwSD6/UyDOGVOFtTHR8P87nLs1IqexJjc8J3CrxT6nZkQ1H5jH847KppFUT3Y9G1UXjRryJnWawFICBlLYJV7J5/qGfbyPTgulPN8QchCjkFzE2ajMbIlAJey8YgZ4hDLq2pWKflDfo/HHGigX9u1f96hYRAP/UbHKSCCjr3s6CNXIab8q81l+ouo22CiqjZjQRc8S1AKvQQ8oh6PUcgZxAiyFPZTuSbcSe9sLbCAVICSLogXH5t+qO2y13QnFDqbwKsBBQtDo0r7uWe5ob1kkCmaKlIP1RW7II8YsHXKy7HRKI8v0JwDqHeKHvdj15MkP3UWea/x0soj8TA78NGiuwxu0YrNJy24HXo6jhGRVdswu858Mp3F2Ix+PVmo/S8r9ej+1G1/bO5asfAjXCriXqaj7e9jMmd/8+SZzy3J7PsjCxFrZoMvsSbCzkbrsv5sOYw9MDdaiJGq/c0Jc177Q18OoAwa6+7HhLhVUTc7475dZ7tCam14ej6GaA2aIbe1U5wjVpJ7VkrhQ+8oo4mMRZKUH96iOjHLp+rK+s30ZhpC2jnl1ILrVL0tEMGvukFW/2GtoHrwFwmlEFsULJzg8c0u +8Z022EJ xfbWQh6g/kK/lEych5flQzeaJ5y3Ea4pXCfFNcDPbGU/VOkLsPKtEirug3nYHGLTVTW/J/FQaTC2rM6qdi71iZKpa+VVT1roj6q9uz9Ntji/jjO9x4zxUPsNK8bhFP5d8F42D+c7Qs55ZrLbUidlPjJvu0c4C8X1elddBOkuJhMljJE0URl/tVYlJWi5GBfCARHTe/+qaCFhk0P5kgv+ypRTd7ifQtIl5VNpzjAqjBk5eBa6TIx3VYEHADOPtmTjNqjaCQXhwBG5ylaaMUgljJ4ev41dPO6BJgqFkIjP35pqQ0Z/+9a1JPcK7j2Ko/yx78zz3k0te7f0dibU+fiKYiUiuGNqa8qykVc8jO2NW4URgk34XadABWBRnBSmODIg6Q3PaDUXZz4GRGX1MlVXdFhaJTBjKmxS2H1aOFxh3q48A189ZgNs8VwNZ3olAdTZB845cjtsaWitcJyZgwj+pLEvTkQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: After redefining alloc_pages, all uses of that name are being replaced. Change the conflicting names to prevent preprocessor from replacing them when it's not intended. Signed-off-by: Suren Baghdasaryan --- arch/alpha/kernel/pci_iommu.c | 2 +- arch/mips/jazz/jazzdma.c | 2 +- arch/powerpc/kernel/dma-iommu.c | 2 +- arch/powerpc/platforms/ps3/system-bus.c | 4 ++-- arch/powerpc/platforms/pseries/vio.c | 2 +- arch/x86/kernel/amd_gart_64.c | 2 +- drivers/iommu/dma-iommu.c | 2 +- drivers/parisc/ccio-dma.c | 2 +- drivers/parisc/sba_iommu.c | 2 +- drivers/xen/grant-dma-ops.c | 2 +- drivers/xen/swiotlb-xen.c | 2 +- include/linux/dma-map-ops.h | 2 +- kernel/dma/mapping.c | 4 ++-- 13 files changed, 15 insertions(+), 15 deletions(-) diff --git a/arch/alpha/kernel/pci_iommu.c b/arch/alpha/kernel/pci_iommu.c index c81183935e97..7fcf3e9b7103 100644 --- a/arch/alpha/kernel/pci_iommu.c +++ b/arch/alpha/kernel/pci_iommu.c @@ -929,7 +929,7 @@ const struct dma_map_ops alpha_pci_ops = { .dma_supported = alpha_pci_supported, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; EXPORT_SYMBOL(alpha_pci_ops); diff --git a/arch/mips/jazz/jazzdma.c b/arch/mips/jazz/jazzdma.c index eabddb89d221..c97b089b9902 100644 --- a/arch/mips/jazz/jazzdma.c +++ b/arch/mips/jazz/jazzdma.c @@ -617,7 +617,7 @@ const struct dma_map_ops jazz_dma_ops = { .sync_sg_for_device = jazz_dma_sync_sg_for_device, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; EXPORT_SYMBOL(jazz_dma_ops); diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c index 8920862ffd79..f0ae39e77e37 100644 --- a/arch/powerpc/kernel/dma-iommu.c +++ b/arch/powerpc/kernel/dma-iommu.c @@ -216,6 +216,6 @@ const struct dma_map_ops dma_iommu_ops = { .get_required_mask = dma_iommu_get_required_mask, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/arch/powerpc/platforms/ps3/system-bus.c b/arch/powerpc/platforms/ps3/system-bus.c index d6b5f5ecd515..56dc6b29a3e7 100644 --- a/arch/powerpc/platforms/ps3/system-bus.c +++ b/arch/powerpc/platforms/ps3/system-bus.c @@ -695,7 +695,7 @@ static const struct dma_map_ops ps3_sb_dma_ops = { .unmap_page = ps3_unmap_page, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; @@ -709,7 +709,7 @@ static const struct dma_map_ops ps3_ioc0_dma_ops = { .unmap_page = ps3_unmap_page, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/arch/powerpc/platforms/pseries/vio.c b/arch/powerpc/platforms/pseries/vio.c index 2dc9cbc4bcd8..0c90fc4c3796 100644 --- a/arch/powerpc/platforms/pseries/vio.c +++ b/arch/powerpc/platforms/pseries/vio.c @@ -611,7 +611,7 @@ static const struct dma_map_ops vio_dma_mapping_ops = { .get_required_mask = dma_iommu_get_required_mask, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/arch/x86/kernel/amd_gart_64.c b/arch/x86/kernel/amd_gart_64.c index 2ae98f754e59..c884deca839b 100644 --- a/arch/x86/kernel/amd_gart_64.c +++ b/arch/x86/kernel/amd_gart_64.c @@ -676,7 +676,7 @@ static const struct dma_map_ops gart_dma_ops = { .get_sgtable = dma_common_get_sgtable, .dma_supported = dma_direct_supported, .get_required_mask = dma_direct_get_required_mask, - .alloc_pages = dma_direct_alloc_pages, + .alloc_pages_op = dma_direct_alloc_pages, .free_pages = dma_direct_free_pages, }; diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 50ccc4f1ef81..8a1f7f5d1bca 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1710,7 +1710,7 @@ static const struct dma_map_ops iommu_dma_ops = { .flags = DMA_F_PCI_P2PDMA_SUPPORTED, .alloc = iommu_dma_alloc, .free = iommu_dma_free, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, .alloc_noncontiguous = iommu_dma_alloc_noncontiguous, .free_noncontiguous = iommu_dma_free_noncontiguous, diff --git a/drivers/parisc/ccio-dma.c b/drivers/parisc/ccio-dma.c index 9ce0d20a6c58..feef537257d0 100644 --- a/drivers/parisc/ccio-dma.c +++ b/drivers/parisc/ccio-dma.c @@ -1022,7 +1022,7 @@ static const struct dma_map_ops ccio_ops = { .map_sg = ccio_map_sg, .unmap_sg = ccio_unmap_sg, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/drivers/parisc/sba_iommu.c b/drivers/parisc/sba_iommu.c index 784037837f65..fc3863c09f83 100644 --- a/drivers/parisc/sba_iommu.c +++ b/drivers/parisc/sba_iommu.c @@ -1090,7 +1090,7 @@ static const struct dma_map_ops sba_ops = { .map_sg = sba_map_sg, .unmap_sg = sba_unmap_sg, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/drivers/xen/grant-dma-ops.c b/drivers/xen/grant-dma-ops.c index 76f6f26265a3..29257d2639db 100644 --- a/drivers/xen/grant-dma-ops.c +++ b/drivers/xen/grant-dma-ops.c @@ -282,7 +282,7 @@ static int xen_grant_dma_supported(struct device *dev, u64 mask) static const struct dma_map_ops xen_grant_dma_ops = { .alloc = xen_grant_dma_alloc, .free = xen_grant_dma_free, - .alloc_pages = xen_grant_dma_alloc_pages, + .alloc_pages_op = xen_grant_dma_alloc_pages, .free_pages = xen_grant_dma_free_pages, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c index 0e6c6c25d154..1c4ef5111651 100644 --- a/drivers/xen/swiotlb-xen.c +++ b/drivers/xen/swiotlb-xen.c @@ -403,7 +403,7 @@ const struct dma_map_ops xen_swiotlb_dma_ops = { .dma_supported = xen_swiotlb_dma_supported, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, .max_mapping_size = swiotlb_max_mapping_size, }; diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index 4abc60f04209..9ee319851b5f 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -29,7 +29,7 @@ struct dma_map_ops { unsigned long attrs); void (*free)(struct device *dev, size_t size, void *vaddr, dma_addr_t dma_handle, unsigned long attrs); - struct page *(*alloc_pages)(struct device *dev, size_t size, + struct page *(*alloc_pages_op)(struct device *dev, size_t size, dma_addr_t *dma_handle, enum dma_data_direction dir, gfp_t gfp); void (*free_pages)(struct device *dev, size_t size, struct page *vaddr, diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 58db8fd70471..5e2d51e1cdf6 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -570,9 +570,9 @@ static struct page *__dma_alloc_pages(struct device *dev, size_t size, size = PAGE_ALIGN(size); if (dma_alloc_direct(dev, ops)) return dma_direct_alloc_pages(dev, size, dma_handle, dir, gfp); - if (!ops->alloc_pages) + if (!ops->alloc_pages_op) return NULL; - return ops->alloc_pages(dev, size, dma_handle, dir, gfp); + return ops->alloc_pages_op(dev, size, dma_handle, dir, gfp); } struct page *dma_alloc_pages(struct device *dev, size_t size, From patchwork Wed Mar 6 18:24:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584428 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF6C7C54E49 for ; Wed, 6 Mar 2024 18:25:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BECAE6B00A7; Wed, 6 Mar 2024 13:25:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B73826B00A8; Wed, 6 Mar 2024 13:25:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 900816B00A9; Wed, 6 Mar 2024 13:25:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 773B96B00A7 for ; Wed, 6 Mar 2024 13:25:26 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 535F740B88 for ; Wed, 6 Mar 2024 18:25:26 +0000 (UTC) X-FDA: 81867441852.15.2D4D951 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf20.hostedemail.com (Postfix) with ESMTP id AE6031C0003 for ; Wed, 6 Mar 2024 18:25:24 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="T3/Vw5Ri"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3E7XoZQYKCFYGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3E7XoZQYKCFYGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749524; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jy6lo0otI0AMc2+YLcMgiqcHNvl++C5nY3G+RyA08IY=; b=H0HpllFyOmsjeu3KnrD80J12p5FwUzkdYaW10SY2xUb1o5k5vg55mWpho5lgToiDCCn7iF ga+4BE1s7BUExFiVzrbLTNe3LYYPFhJriwP0CGoILDAiFzjnq95Kneu5Y2fYNyocBxDuJf cmiENYdtevRzWkXEmuZH2H+ZschywL4= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="T3/Vw5Ri"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3E7XoZQYKCFYGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3E7XoZQYKCFYGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749524; a=rsa-sha256; cv=none; b=GpFJZ71vQDoEg2fvijxBT5pXZITbl1h3tWGWLF6YU3ZxdGDi90o6TAC5zTEDQ1BeyNv5ZU wwn5zJjL6clvL8+T329Q86deGJZx6wDEu2eHnE0/QUck4HHImMu+srMVuPlHY9uaZx1XfX 1+x+O+JyXPd2ftRgz0kK1dRkB1DR0ho= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dbf618042daso10844543276.0 for ; Wed, 06 Mar 2024 10:25:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749524; x=1710354324; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jy6lo0otI0AMc2+YLcMgiqcHNvl++C5nY3G+RyA08IY=; b=T3/Vw5Ri9BUe9QEPydXdXPoF8wAzDJzai2GlEM1cvjINlWySwy2JgLahF0x3l4GE4B Qh0r5kSM50ppTX5EcOWooU7QMWeSDhGtwC1t00RRnI77HRLCcG5D9ik5QFVmYouLnWBR iVBlTdOZPOvTPaj+TEmY/ZcfSoPBQTScZtzvd1kYWdYljYsyBO01wZeOXSUp66zuY5Rf NUOpmyzNUfs/6TNXcrdDQnKDl19HfJpTuIb9zpeMp2cT90OPOwEScC1KSHFjVeCIVE2c TKX5OG/WW2hx6XJuFWLUKnoIGS1j2mAT6I47rmmIE52woxTeSsTNtC3oixc/e7KroLz2 EWkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749524; x=1710354324; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jy6lo0otI0AMc2+YLcMgiqcHNvl++C5nY3G+RyA08IY=; b=iK/2tcejBdivL7qN2TOyxDB1JYmTqvhMp1D4ojUSZKOWFGzoZK6f7u5tCIh/azaYJ+ adnyOucDObvFcr1H8VxIIgnCkVqLDLSDKvoX6VP3jDg60ZgXC+e+yxM+3QqHymNdxmJp P6sRJMJLuGo0q3gA0iswP37dJWF9rsFUPEsc+EW1CaRsC50EGbiog6j1GeUmjh+QwKPE JO+eyg85/mHfpC6gPsHXIsL+nQqBZX8fAZQlvnBT2aOCdhBPC7RMjUDk5a0SVG61DlT+ yOMCqi43SyV7BU1nkYVJ0epmJa5T7DX1s1lZ8ShrVY472WyxmGB4+zr32C3nfoZ+FjXU eGww== X-Forwarded-Encrypted: i=1; AJvYcCVUaWuFZCndsrwcU6s5ZzoiD0GP3q34fFFkNjDUBiipbsUGXUP8aSy9DEijKFpa03nkXAHIA9EjDsucDEevHq31BH0= X-Gm-Message-State: AOJu0YzpxBOBs1DpVg4JqYh1pnUJjRHiQ0Kv6vQIbPMpewnM4e4IWYBg DJNHsGkLPkUJmtirgoIJm3nxCkuKtnpdsDA0yQJcVnRatLxWOnsKYVQf29QQ0+yhYls4hcl1W/n 11w== X-Google-Smtp-Source: AGHT+IFx0TGC+DPelPIcyBwRbQiNpSikwI0J+nQ9dKGPU08L0NPImfYK2ZejJERDz2G53gW6MeWHhLzQMu4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:1249:b0:dbe:d0a9:2be8 with SMTP id t9-20020a056902124900b00dbed0a92be8mr644257ybu.0.1709749523611; Wed, 06 Mar 2024 10:25:23 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:16 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-19-surenb@google.com> Subject: [PATCH v5 18/37] mm: enable page allocation tagging From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Stat-Signature: 1sg7mneocun7n443pojj47i7xau3q4ya X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: AE6031C0003 X-HE-Tag: 1709749524-83838 X-HE-Meta: U2FsdGVkX18aZ1AlZR2bmyq/QesDKdMviGJaFUysBgLZwVfrcD1g67WqN5EkEKXtT+lgTb83BcmE67VYHQ2RuCxx94grT4xueFSNxiE9wdCCSB8DL91FlME5tO8YBl0/zYEim75cnjaqiPqw3iR+d/TvSADsN6hXqRMezfwQZb/ta28LLljIIYfCV73mOVAAXfwPKNN8hrEIFoSYiJ6EiXJk11cY/9FNz8fOi3rCKqhOrOsoQQDtKQfl5r0s9ppZq2f0qqhSJRDPWvCFQP5NvhYtSrcOzFmCPCJyu3s/AIAbqO/u0nrF2Onmic2Cg698PpAl1JLA41Q1DRotrar3Na/uWJBMPyYERcfqV46jUBqbFLLof6X2GznpTRgB6L05c6ML5gzmu4eN4YYp+tpfEYOdlr4DTzvRzuKy/O5TbUqLBR4i+EfTGxQgqaYILM5C7GbAZmSSH01AqFZNi+jtErj6sdpoL2zpLUvaAZfVSyx7mZN8zMs8AjfvOyUlquQwwkOBjBrA524/8BMOaSZU0hBPYP/r40+9fecSuGpsoSP2IHZ4rMO/eERSXeIgOq4KNrnJVPb/KsxBeMzVlxtDbAg/wZjlcWBWIOeQLDOdq4YirD6XipJlqALqtza1uf0nO0RSGsHepapOzFvv0/CNP4j9bP+WIzkqdtmPHvIjAkkOX1fR1E7Gxcvbjgnk1+tF8suzFEzswV7/QrKXoll9hMtVe6ahW6fkXc7wB/DmML35zOG9jCLdwQT88JlRKqtqwJci0d9tprIF9tu0G3y1u4YzTnkHBHcVFOWJBhy0mNt83zMptlfwdV+u5OV0+MZ4e2Z5Pfh9yLMQGHhvC1M0jF15MKPcrWwFtXpiNNTjxlvFvhsMJpRMqeL5483trccoLW+Z/Ex/iQ12wIJGO4i4D3ZA5oXB9DQ/gTKNP+rMXDECMvCIpnAqI47ZU5hV+BlPlI0rZuqpBxuOVi1zAsn jGBxn6uL IcFpgYEZAQVsyLqEdqQ4niHcOvPGD6UTtWpnIG/XH4g7km1/dJ0sG80cIcP90rDJ/wTZff9cXq/zrWfcXo+mDjpp1QTSXLZ9xF65/SmQs1NRlJQtz2oYOBanh3Akq1TV9wceRRBgfmktqPwnWo3uIzYafvD977pg4TzFChzOqnt3vPXCVqwtXeeDLBDSgNwvGz79Mkpg3xNTOkBaeoHwsxXdbRPKwlyNLvRlu6hPDeozq/Ck0CYym3z48H7yaX33VvW9muxGkQLT3VAHVmaie7OugIOB0s/YQKjPs9iepzXHuQQdZe6502P715vA/Acl6U6qYoWReGXzyMNF668mv2v6Ws8DdKZnLlU+rO4GOQQwOx2Aozg4XgNe1DfkX+rVzuLmgClRSDzbQ4Dx8SpqJLD+lcFezWI1kf4vf9wIB0ffbbCM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Redefine page allocators to record allocation tags upon their invocation. Instrument post_alloc_hook and free_pages_prepare to modify current allocation tag. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Reviewed-by: Kees Cook --- include/linux/alloc_tag.h | 14 +++++ include/linux/gfp.h | 126 ++++++++++++++++++++++++-------------- include/linux/pagemap.h | 9 ++- mm/compaction.c | 7 ++- mm/filemap.c | 6 +- mm/mempolicy.c | 52 ++++++++-------- mm/page_alloc.c | 60 +++++++++--------- 7 files changed, 164 insertions(+), 110 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index b970ff1c80dc..28c0005edae1 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -142,4 +142,18 @@ static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) {} #endif /* CONFIG_MEM_ALLOC_PROFILING */ +#define alloc_hooks_tag(_tag, _do_alloc) \ +({ \ + struct alloc_tag * __maybe_unused _old = alloc_tag_save(_tag); \ + typeof(_do_alloc) _res = _do_alloc; \ + alloc_tag_restore(_tag, _old); \ + _res; \ +}) + +#define alloc_hooks(_do_alloc) \ +({ \ + DEFINE_ALLOC_TAG(_alloc_tag); \ + alloc_hooks_tag(&_alloc_tag, _do_alloc); \ +}) + #endif /* _LINUX_ALLOC_TAG_H */ diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 3e4c0c536a3d..601a00e755f7 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -6,6 +6,8 @@ #include #include +#include +#include struct vm_area_struct; struct mempolicy; @@ -175,42 +177,46 @@ static inline void arch_free_page(struct page *page, int order) { } static inline void arch_alloc_page(struct page *page, int order) { } #endif -struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, +struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask); -struct folio *__folio_alloc(gfp_t gfp, unsigned int order, int preferred_nid, +#define __alloc_pages(...) alloc_hooks(__alloc_pages_noprof(__VA_ARGS__)) + +struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask); +#define __folio_alloc(...) alloc_hooks(__folio_alloc_noprof(__VA_ARGS__)) -unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, +unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, nodemask_t *nodemask, int nr_pages, struct list_head *page_list, struct page **page_array); +#define __alloc_pages_bulk(...) alloc_hooks(alloc_pages_bulk_noprof(__VA_ARGS__)) -unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, +unsigned long alloc_pages_bulk_array_mempolicy_noprof(gfp_t gfp, unsigned long nr_pages, struct page **page_array); +#define alloc_pages_bulk_array_mempolicy(...) \ + alloc_hooks(alloc_pages_bulk_array_mempolicy_noprof(__VA_ARGS__)) /* Bulk allocate order-0 pages */ -static inline unsigned long -alloc_pages_bulk_list(gfp_t gfp, unsigned long nr_pages, struct list_head *list) -{ - return __alloc_pages_bulk(gfp, numa_mem_id(), NULL, nr_pages, list, NULL); -} +#define alloc_pages_bulk_list(_gfp, _nr_pages, _list) \ + __alloc_pages_bulk(_gfp, numa_mem_id(), NULL, _nr_pages, _list, NULL) -static inline unsigned long -alloc_pages_bulk_array(gfp_t gfp, unsigned long nr_pages, struct page **page_array) -{ - return __alloc_pages_bulk(gfp, numa_mem_id(), NULL, nr_pages, NULL, page_array); -} +#define alloc_pages_bulk_array(_gfp, _nr_pages, _page_array) \ + __alloc_pages_bulk(_gfp, numa_mem_id(), NULL, _nr_pages, NULL, _page_array) static inline unsigned long -alloc_pages_bulk_array_node(gfp_t gfp, int nid, unsigned long nr_pages, struct page **page_array) +alloc_pages_bulk_array_node_noprof(gfp_t gfp, int nid, unsigned long nr_pages, + struct page **page_array) { if (nid == NUMA_NO_NODE) nid = numa_mem_id(); - return __alloc_pages_bulk(gfp, nid, NULL, nr_pages, NULL, page_array); + return alloc_pages_bulk_noprof(gfp, nid, NULL, nr_pages, NULL, page_array); } +#define alloc_pages_bulk_array_node(...) \ + alloc_hooks(alloc_pages_bulk_array_node_noprof(__VA_ARGS__)) + static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask) { gfp_t warn_gfp = gfp_mask & (__GFP_THISNODE|__GFP_NOWARN); @@ -230,82 +236,104 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask) * online. For more general interface, see alloc_pages_node(). */ static inline struct page * -__alloc_pages_node(int nid, gfp_t gfp_mask, unsigned int order) +__alloc_pages_node_noprof(int nid, gfp_t gfp_mask, unsigned int order) { VM_BUG_ON(nid < 0 || nid >= MAX_NUMNODES); warn_if_node_offline(nid, gfp_mask); - return __alloc_pages(gfp_mask, order, nid, NULL); + return __alloc_pages_noprof(gfp_mask, order, nid, NULL); } +#define __alloc_pages_node(...) alloc_hooks(__alloc_pages_node_noprof(__VA_ARGS__)) + static inline -struct folio *__folio_alloc_node(gfp_t gfp, unsigned int order, int nid) +struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid) { VM_BUG_ON(nid < 0 || nid >= MAX_NUMNODES); warn_if_node_offline(nid, gfp); - return __folio_alloc(gfp, order, nid, NULL); + return __folio_alloc_noprof(gfp, order, nid, NULL); } +#define __folio_alloc_node(...) alloc_hooks(__folio_alloc_node_noprof(__VA_ARGS__)) + /* * Allocate pages, preferring the node given as nid. When nid == NUMA_NO_NODE, * prefer the current CPU's closest node. Otherwise node must be valid and * online. */ -static inline struct page *alloc_pages_node(int nid, gfp_t gfp_mask, - unsigned int order) +static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask, + unsigned int order) { if (nid == NUMA_NO_NODE) nid = numa_mem_id(); - return __alloc_pages_node(nid, gfp_mask, order); + return __alloc_pages_node_noprof(nid, gfp_mask, order); } +#define alloc_pages_node(...) alloc_hooks(alloc_pages_node_noprof(__VA_ARGS__)) + #ifdef CONFIG_NUMA -struct page *alloc_pages(gfp_t gfp, unsigned int order); -struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, +struct page *alloc_pages_noprof(gfp_t gfp, unsigned int order); +struct page *alloc_pages_mpol_noprof(gfp_t gfp, unsigned int order, struct mempolicy *mpol, pgoff_t ilx, int nid); -struct folio *folio_alloc(gfp_t gfp, unsigned int order); -struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, +struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order); +struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order, struct vm_area_struct *vma, unsigned long addr, bool hugepage); #else -static inline struct page *alloc_pages(gfp_t gfp_mask, unsigned int order) +static inline struct page *alloc_pages_noprof(gfp_t gfp_mask, unsigned int order) { - return alloc_pages_node(numa_node_id(), gfp_mask, order); + return alloc_pages_node_noprof(numa_node_id(), gfp_mask, order); } -static inline struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, +static inline struct page *alloc_pages_mpol_noprof(gfp_t gfp, unsigned int order, struct mempolicy *mpol, pgoff_t ilx, int nid) { - return alloc_pages(gfp, order); + return alloc_pages_noprof(gfp, order); } -static inline struct folio *folio_alloc(gfp_t gfp, unsigned int order) +static inline struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order) { return __folio_alloc_node(gfp, order, numa_node_id()); } -#define vma_alloc_folio(gfp, order, vma, addr, hugepage) \ - folio_alloc(gfp, order) +#define vma_alloc_folio_noprof(gfp, order, vma, addr, hugepage) \ + folio_alloc_noprof(gfp, order) #endif + +#define alloc_pages(...) alloc_hooks(alloc_pages_noprof(__VA_ARGS__)) +#define alloc_pages_mpol(...) alloc_hooks(alloc_pages_mpol_noprof(__VA_ARGS__)) +#define folio_alloc(...) alloc_hooks(folio_alloc_noprof(__VA_ARGS__)) +#define vma_alloc_folio(...) alloc_hooks(vma_alloc_folio_noprof(__VA_ARGS__)) + #define alloc_page(gfp_mask) alloc_pages(gfp_mask, 0) -static inline struct page *alloc_page_vma(gfp_t gfp, + +static inline struct page *alloc_page_vma_noprof(gfp_t gfp, struct vm_area_struct *vma, unsigned long addr) { - struct folio *folio = vma_alloc_folio(gfp, 0, vma, addr, false); + struct folio *folio = vma_alloc_folio_noprof(gfp, 0, vma, addr, false); return &folio->page; } +#define alloc_page_vma(...) alloc_hooks(alloc_page_vma_noprof(__VA_ARGS__)) + +extern unsigned long get_free_pages_noprof(gfp_t gfp_mask, unsigned int order); +#define __get_free_pages(...) alloc_hooks(get_free_pages_noprof(__VA_ARGS__)) -extern unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order); -extern unsigned long get_zeroed_page(gfp_t gfp_mask); +extern unsigned long get_zeroed_page_noprof(gfp_t gfp_mask); +#define get_zeroed_page(...) alloc_hooks(get_zeroed_page_noprof(__VA_ARGS__)) + +void *alloc_pages_exact_noprof(size_t size, gfp_t gfp_mask) __alloc_size(1); +#define alloc_pages_exact(...) alloc_hooks(alloc_pages_exact_noprof(__VA_ARGS__)) -void *alloc_pages_exact(size_t size, gfp_t gfp_mask) __alloc_size(1); void free_pages_exact(void *virt, size_t size); -__meminit void *alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) __alloc_size(2); -#define __get_free_page(gfp_mask) \ - __get_free_pages((gfp_mask), 0) +__meminit void *alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_mask) __alloc_size(2); +#define alloc_pages_exact_nid(...) \ + alloc_hooks(alloc_pages_exact_nid_noprof(__VA_ARGS__)) + +#define __get_free_page(gfp_mask) \ + __get_free_pages((gfp_mask), 0) -#define __get_dma_pages(gfp_mask, order) \ - __get_free_pages((gfp_mask) | GFP_DMA, (order)) +#define __get_dma_pages(gfp_mask, order) \ + __get_free_pages((gfp_mask) | GFP_DMA, (order)) extern void __free_pages(struct page *page, unsigned int order); extern void free_pages(unsigned long addr, unsigned int order); @@ -366,10 +394,14 @@ extern gfp_t vma_thp_gfp_mask(struct vm_area_struct *vma); #ifdef CONFIG_CONTIG_ALLOC /* The below functions must be run on a range from a single zone. */ -extern int alloc_contig_range(unsigned long start, unsigned long end, +extern int alloc_contig_range_noprof(unsigned long start, unsigned long end, unsigned migratetype, gfp_t gfp_mask); -extern struct page *alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask, - int nid, nodemask_t *nodemask); +#define alloc_contig_range(...) alloc_hooks(alloc_contig_range_noprof(__VA_ARGS__)) + +extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask, + int nid, nodemask_t *nodemask); +#define alloc_contig_pages(...) alloc_hooks(alloc_contig_pages_noprof(__VA_ARGS__)) + #endif void free_contig_range(unsigned long pfn, unsigned long nr_pages); diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 2df35e65557d..35636e67e2e1 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -542,14 +542,17 @@ static inline void *detach_page_private(struct page *page) #endif #ifdef CONFIG_NUMA -struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order); +struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order); #else -static inline struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order) +static inline struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order) { - return folio_alloc(gfp, order); + return folio_alloc_noprof(gfp, order); } #endif +#define filemap_alloc_folio(...) \ + alloc_hooks(filemap_alloc_folio_noprof(__VA_ARGS__)) + static inline struct page *__page_cache_alloc(gfp_t gfp) { return &filemap_alloc_folio(gfp, 0)->page; diff --git a/mm/compaction.c b/mm/compaction.c index 807b58e6eb68..70b01190d2f3 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1851,7 +1851,7 @@ static void isolate_freepages(struct compact_control *cc) * This is a migrate-callback that "allocates" freepages by taking pages * from the isolated freelists in the block we are migrating to. */ -static struct folio *compaction_alloc(struct folio *src, unsigned long data) +static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long data) { struct compact_control *cc = (struct compact_control *)data; struct folio *dst; @@ -1898,6 +1898,11 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data) return page_rmappable_folio(&dst->page); } +static struct folio *compaction_alloc(struct folio *src, unsigned long data) +{ + return alloc_hooks(compaction_alloc_noprof(src, data)); +} + /* * This is a migrate-callback that "frees" freepages back to the isolated * freelist. All pages on the freelist are from the same zone, so there is no diff --git a/mm/filemap.c b/mm/filemap.c index b4858d89f1b1..40d8cae674f6 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -957,7 +957,7 @@ int filemap_add_folio(struct address_space *mapping, struct folio *folio, EXPORT_SYMBOL_GPL(filemap_add_folio); #ifdef CONFIG_NUMA -struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order) +struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order) { int n; struct folio *folio; @@ -972,9 +972,9 @@ struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order) return folio; } - return folio_alloc(gfp, order); + return folio_alloc_noprof(gfp, order); } -EXPORT_SYMBOL(filemap_alloc_folio); +EXPORT_SYMBOL(filemap_alloc_folio_noprof); #endif /* diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 0fe77738d971..736f77746b02 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2200,15 +2200,15 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, */ preferred_gfp = gfp | __GFP_NOWARN; preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - page = __alloc_pages(preferred_gfp, order, nid, nodemask); + page = __alloc_pages_noprof(preferred_gfp, order, nid, nodemask); if (!page) - page = __alloc_pages(gfp, order, nid, NULL); + page = __alloc_pages_noprof(gfp, order, nid, NULL); return page; } /** - * alloc_pages_mpol - Allocate pages according to NUMA mempolicy. + * alloc_pages_mpol_noprof - Allocate pages according to NUMA mempolicy. * @gfp: GFP flags. * @order: Order of the page allocation. * @pol: Pointer to the NUMA mempolicy. @@ -2217,7 +2217,7 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, * * Return: The page on success or NULL if allocation fails. */ -struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, +struct page *alloc_pages_mpol_noprof(gfp_t gfp, unsigned int order, struct mempolicy *pol, pgoff_t ilx, int nid) { nodemask_t *nodemask; @@ -2248,7 +2248,7 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, * First, try to allocate THP only on local node, but * don't reclaim unnecessarily, just compact. */ - page = __alloc_pages_node(nid, + page = __alloc_pages_node_noprof(nid, gfp | __GFP_THISNODE | __GFP_NORETRY, order); if (page || !(gfp & __GFP_DIRECT_RECLAIM)) return page; @@ -2261,7 +2261,7 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, } } - page = __alloc_pages(gfp, order, nid, nodemask); + page = __alloc_pages_noprof(gfp, order, nid, nodemask); if (unlikely(pol->mode == MPOL_INTERLEAVE) && page) { /* skip NUMA_INTERLEAVE_HIT update if numa stats is disabled */ @@ -2277,7 +2277,7 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, } /** - * vma_alloc_folio - Allocate a folio for a VMA. + * vma_alloc_folio_noprof - Allocate a folio for a VMA. * @gfp: GFP flags. * @order: Order of the folio. * @vma: Pointer to VMA. @@ -2292,7 +2292,7 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, * * Return: The folio on success or NULL if allocation fails. */ -struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, +struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order, struct vm_area_struct *vma, unsigned long addr, bool hugepage) { struct mempolicy *pol; @@ -2300,15 +2300,15 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, struct page *page; pol = get_vma_policy(vma, addr, order, &ilx); - page = alloc_pages_mpol(gfp | __GFP_COMP, order, - pol, ilx, numa_node_id()); + page = alloc_pages_mpol_noprof(gfp | __GFP_COMP, order, + pol, ilx, numa_node_id()); mpol_cond_put(pol); return page_rmappable_folio(page); } -EXPORT_SYMBOL(vma_alloc_folio); +EXPORT_SYMBOL(vma_alloc_folio_noprof); /** - * alloc_pages - Allocate pages. + * alloc_pages_noprof - Allocate pages. * @gfp: GFP flags. * @order: Power of two of number of pages to allocate. * @@ -2321,7 +2321,7 @@ EXPORT_SYMBOL(vma_alloc_folio); * flags are used. * Return: The page on success or NULL if allocation fails. */ -struct page *alloc_pages(gfp_t gfp, unsigned int order) +struct page *alloc_pages_noprof(gfp_t gfp, unsigned int order) { struct mempolicy *pol = &default_policy; @@ -2332,16 +2332,16 @@ struct page *alloc_pages(gfp_t gfp, unsigned int order) if (!in_interrupt() && !(gfp & __GFP_THISNODE)) pol = get_task_policy(current); - return alloc_pages_mpol(gfp, order, - pol, NO_INTERLEAVE_INDEX, numa_node_id()); + return alloc_pages_mpol_noprof(gfp, order, pol, NO_INTERLEAVE_INDEX, + numa_node_id()); } -EXPORT_SYMBOL(alloc_pages); +EXPORT_SYMBOL(alloc_pages_noprof); -struct folio *folio_alloc(gfp_t gfp, unsigned int order) +struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order) { - return page_rmappable_folio(alloc_pages(gfp | __GFP_COMP, order)); + return page_rmappable_folio(alloc_pages_noprof(gfp | __GFP_COMP, order)); } -EXPORT_SYMBOL(folio_alloc); +EXPORT_SYMBOL(folio_alloc_noprof); static unsigned long alloc_pages_bulk_array_interleave(gfp_t gfp, struct mempolicy *pol, unsigned long nr_pages, @@ -2360,13 +2360,13 @@ static unsigned long alloc_pages_bulk_array_interleave(gfp_t gfp, for (i = 0; i < nodes; i++) { if (delta) { - nr_allocated = __alloc_pages_bulk(gfp, + nr_allocated = alloc_pages_bulk_noprof(gfp, interleave_nodes(pol), NULL, nr_pages_per_node + 1, NULL, page_array); delta--; } else { - nr_allocated = __alloc_pages_bulk(gfp, + nr_allocated = alloc_pages_bulk_noprof(gfp, interleave_nodes(pol), NULL, nr_pages_per_node, NULL, page_array); } @@ -2503,11 +2503,11 @@ static unsigned long alloc_pages_bulk_array_preferred_many(gfp_t gfp, int nid, preferred_gfp = gfp | __GFP_NOWARN; preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - nr_allocated = __alloc_pages_bulk(preferred_gfp, nid, &pol->nodes, + nr_allocated = alloc_pages_bulk_noprof(preferred_gfp, nid, &pol->nodes, nr_pages, NULL, page_array); if (nr_allocated < nr_pages) - nr_allocated += __alloc_pages_bulk(gfp, numa_node_id(), NULL, + nr_allocated += alloc_pages_bulk_noprof(gfp, numa_node_id(), NULL, nr_pages - nr_allocated, NULL, page_array + nr_allocated); return nr_allocated; @@ -2519,7 +2519,7 @@ static unsigned long alloc_pages_bulk_array_preferred_many(gfp_t gfp, int nid, * It can accelerate memory allocation especially interleaving * allocate memory. */ -unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, +unsigned long alloc_pages_bulk_array_mempolicy_noprof(gfp_t gfp, unsigned long nr_pages, struct page **page_array) { struct mempolicy *pol = &default_policy; @@ -2543,8 +2543,8 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, nid = numa_node_id(); nodemask = policy_nodemask(gfp, pol, NO_INTERLEAVE_INDEX, &nid); - return __alloc_pages_bulk(gfp, nid, nodemask, - nr_pages, NULL, page_array); + return alloc_pages_bulk_noprof(gfp, nid, nodemask, + nr_pages, NULL, page_array); } int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 9a91c8074556..eb5cae9b967d 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4424,7 +4424,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, * * Returns the number of pages on the list or array. */ -unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, +unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, nodemask_t *nodemask, int nr_pages, struct list_head *page_list, struct page **page_array) @@ -4560,7 +4560,7 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, pcp_trylock_finish(UP_flags); failed: - page = __alloc_pages(gfp, 0, preferred_nid, nodemask); + page = __alloc_pages_noprof(gfp, 0, preferred_nid, nodemask); if (page) { if (page_list) list_add(&page->lru, page_list); @@ -4571,13 +4571,13 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, goto out; } -EXPORT_SYMBOL_GPL(__alloc_pages_bulk); +EXPORT_SYMBOL_GPL(alloc_pages_bulk_noprof); /* * This is the 'heart' of the zoned buddy allocator. */ -struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, - nodemask_t *nodemask) +struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, + int preferred_nid, nodemask_t *nodemask) { struct page *page; unsigned int alloc_flags = ALLOC_WMARK_LOW; @@ -4639,38 +4639,38 @@ struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, return page; } -EXPORT_SYMBOL(__alloc_pages); +EXPORT_SYMBOL(__alloc_pages_noprof); -struct folio *__folio_alloc(gfp_t gfp, unsigned int order, int preferred_nid, +struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask) { - struct page *page = __alloc_pages(gfp | __GFP_COMP, order, + struct page *page = __alloc_pages_noprof(gfp | __GFP_COMP, order, preferred_nid, nodemask); return page_rmappable_folio(page); } -EXPORT_SYMBOL(__folio_alloc); +EXPORT_SYMBOL(__folio_alloc_noprof); /* * Common helper functions. Never use with __GFP_HIGHMEM because the returned * address cannot represent highmem pages. Use alloc_pages and then kmap if * you need to access high mem. */ -unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order) +unsigned long get_free_pages_noprof(gfp_t gfp_mask, unsigned int order) { struct page *page; - page = alloc_pages(gfp_mask & ~__GFP_HIGHMEM, order); + page = alloc_pages_noprof(gfp_mask & ~__GFP_HIGHMEM, order); if (!page) return 0; return (unsigned long) page_address(page); } -EXPORT_SYMBOL(__get_free_pages); +EXPORT_SYMBOL(get_free_pages_noprof); -unsigned long get_zeroed_page(gfp_t gfp_mask) +unsigned long get_zeroed_page_noprof(gfp_t gfp_mask) { - return __get_free_page(gfp_mask | __GFP_ZERO); + return get_free_pages_noprof(gfp_mask | __GFP_ZERO, 0); } -EXPORT_SYMBOL(get_zeroed_page); +EXPORT_SYMBOL(get_zeroed_page_noprof); /** * __free_pages - Free pages allocated with alloc_pages(). @@ -4862,7 +4862,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, } /** - * alloc_pages_exact - allocate an exact number physically-contiguous pages. + * alloc_pages_exact_noprof - allocate an exact number physically-contiguous pages. * @size: the number of bytes to allocate * @gfp_mask: GFP flags for the allocation, must not contain __GFP_COMP * @@ -4876,7 +4876,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, * * Return: pointer to the allocated area or %NULL in case of error. */ -void *alloc_pages_exact(size_t size, gfp_t gfp_mask) +void *alloc_pages_exact_noprof(size_t size, gfp_t gfp_mask) { unsigned int order = get_order(size); unsigned long addr; @@ -4884,13 +4884,13 @@ void *alloc_pages_exact(size_t size, gfp_t gfp_mask) if (WARN_ON_ONCE(gfp_mask & (__GFP_COMP | __GFP_HIGHMEM))) gfp_mask &= ~(__GFP_COMP | __GFP_HIGHMEM); - addr = __get_free_pages(gfp_mask, order); + addr = get_free_pages_noprof(gfp_mask, order); return make_alloc_exact(addr, order, size); } -EXPORT_SYMBOL(alloc_pages_exact); +EXPORT_SYMBOL(alloc_pages_exact_noprof); /** - * alloc_pages_exact_nid - allocate an exact number of physically-contiguous + * alloc_pages_exact_nid_noprof - allocate an exact number of physically-contiguous * pages on a node. * @nid: the preferred node ID where memory should be allocated * @size: the number of bytes to allocate @@ -4901,7 +4901,7 @@ EXPORT_SYMBOL(alloc_pages_exact); * * Return: pointer to the allocated area or %NULL in case of error. */ -void * __meminit alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) +void * __meminit alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_mask) { unsigned int order = get_order(size); struct page *p; @@ -4909,7 +4909,7 @@ void * __meminit alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) if (WARN_ON_ONCE(gfp_mask & (__GFP_COMP | __GFP_HIGHMEM))) gfp_mask &= ~(__GFP_COMP | __GFP_HIGHMEM); - p = alloc_pages_node(nid, gfp_mask, order); + p = alloc_pages_node_noprof(nid, gfp_mask, order); if (!p) return NULL; return make_alloc_exact((unsigned long)page_address(p), order, size); @@ -6346,7 +6346,7 @@ int __alloc_contig_migrate_range(struct compact_control *cc, } /** - * alloc_contig_range() -- tries to allocate given range of pages + * alloc_contig_range_noprof() -- tries to allocate given range of pages * @start: start PFN to allocate * @end: one-past-the-last PFN to allocate * @migratetype: migratetype of the underlying pageblocks (either @@ -6366,7 +6366,7 @@ int __alloc_contig_migrate_range(struct compact_control *cc, * pages which PFN is in [start, end) are allocated for the caller and * need to be freed with free_contig_range(). */ -int alloc_contig_range(unsigned long start, unsigned long end, +int alloc_contig_range_noprof(unsigned long start, unsigned long end, unsigned migratetype, gfp_t gfp_mask) { unsigned long outer_start, outer_end; @@ -6490,15 +6490,15 @@ int alloc_contig_range(unsigned long start, unsigned long end, undo_isolate_page_range(start, end, migratetype); return ret; } -EXPORT_SYMBOL(alloc_contig_range); +EXPORT_SYMBOL(alloc_contig_range_noprof); static int __alloc_contig_pages(unsigned long start_pfn, unsigned long nr_pages, gfp_t gfp_mask) { unsigned long end_pfn = start_pfn + nr_pages; - return alloc_contig_range(start_pfn, end_pfn, MIGRATE_MOVABLE, - gfp_mask); + return alloc_contig_range_noprof(start_pfn, end_pfn, MIGRATE_MOVABLE, + gfp_mask); } static bool pfn_range_valid_contig(struct zone *z, unsigned long start_pfn, @@ -6533,7 +6533,7 @@ static bool zone_spans_last_pfn(const struct zone *zone, } /** - * alloc_contig_pages() -- tries to find and allocate contiguous range of pages + * alloc_contig_pages_noprof() -- tries to find and allocate contiguous range of pages * @nr_pages: Number of contiguous pages to allocate * @gfp_mask: GFP mask to limit search and used during compaction * @nid: Target node @@ -6553,8 +6553,8 @@ static bool zone_spans_last_pfn(const struct zone *zone, * * Return: pointer to contiguous pages on success, or NULL if not successful. */ -struct page *alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask, - int nid, nodemask_t *nodemask) +struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask, + int nid, nodemask_t *nodemask) { unsigned long ret, pfn, flags; struct zonelist *zonelist; From patchwork Wed Mar 6 18:24:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584429 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F20EDC5475B for ; Wed, 6 Mar 2024 18:25:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C104E6B00A9; Wed, 6 Mar 2024 13:25:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B97856B00AA; Wed, 6 Mar 2024 13:25:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9EBAF6B00AB; Wed, 6 Mar 2024 13:25:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8C0576B00A9 for ; Wed, 6 Mar 2024 13:25:28 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 70120140534 for ; Wed, 6 Mar 2024 18:25:28 +0000 (UTC) X-FDA: 81867441936.27.A882BD0 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf09.hostedemail.com (Postfix) with ESMTP id C8D8C140007 for ; Wed, 6 Mar 2024 18:25:26 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=v3XAv0Hw; spf=pass (imf09.hostedemail.com: domain of 3FbXoZQYKCFgIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3FbXoZQYKCFgIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749526; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UytzFxvo2MN54m2UV3o6+O9sIke62l96NyrXCwye1Yc=; b=RMb8CEz/guOZ19phwxZhKVaZdaFG1VzbaVPnSmvDGI1B8qjoUsjgJgiOTWHEJg6WfM2msn hDrPk/TBQ+SBQzGRa1cChhlwtsACDhp1C0EWiC5A9Ct3P0KqM7PT/888k4l9OrJ+LIPxWP nXK046yt+vbNsZSG45MLmKwHKqSOK8s= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749526; a=rsa-sha256; cv=none; b=TFxOiqkUuGiuK6hzgCbtRqy85yi3bZ+zM/uRNChLOULGP/AkLPTyc4DMHucsZbi5VxUx8S 2yFtwEDPV6dNV1IIYYconyjqfQzXHdO9o/yrXShrWwELTe2oe5Db5fGRz1d3IUn6Db/E0n WRb8PWEGIoReM9nQuLksfyLXWIgoasM= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=v3XAv0Hw; spf=pass (imf09.hostedemail.com: domain of 3FbXoZQYKCFgIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3FbXoZQYKCFgIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60802b0afd2so837997b3.1 for ; Wed, 06 Mar 2024 10:25:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749526; x=1710354326; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UytzFxvo2MN54m2UV3o6+O9sIke62l96NyrXCwye1Yc=; b=v3XAv0HwooKL09LfUVlYxFTeXst7s9zBnX3wEKoEO9NFUPiTiMKMuPj4sTON4TrSYf N9tKQ9Djr2ifsK3LoduK9rLcA9LZMCBxpHjuLowQO9SA3UGeCW29e5NSRTi++NAYoqFR Y+2lTksLcgLD30Dg5CKJF00RuNRh9djBHKbaBLMbE2B8ca0fB8aGfiHiq0i6f2tBLjCW HodMEQVlinqc4xJCQ0Cr4BAf1c78imIxvjV+5nmF+GgyL08qhYZnJM/RtGZDckwzFFGH CyjFY1up4Voy+e9DM/toD7K60ddvsLA+SdiOoi1x9yT+XJgGI3TJI9eYl5RSseJUDEh5 rxLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749526; x=1710354326; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UytzFxvo2MN54m2UV3o6+O9sIke62l96NyrXCwye1Yc=; b=DOcy75TASD7SOwY2Y5XBDQLoosNHOchdWyuieC4AqZm4RowGSLBBdkvvzhB2yMmRPd UqFu40Ss9I0NYGbULqNvOLPSOe+eSQsbCmfOrjxBKngslnjdmnkOG3AFR9n6XI4YeB58 BzqrjPFsfKq0SHneL5urYdi9EiIYnqWCbOSz5Y0nS57kfcKD5baE5zn6lnkk/qM6qMVE wFB3jw3YWFxAReSGGynRPGCuFS3RJqxaGVnADktQlJtP+yYWaN3XkRwCjtuzP8Uz1O+F E8xL9D5R/y5j8wN+to8QhFTeaiW4/Iu2d5Ll46XBcfcTr+apO5b6qx9Epj3DfyrVauBJ kweg== X-Forwarded-Encrypted: i=1; AJvYcCWJz1claTCBvwrg5Nnls3BGPRr81dA0M0Wn6P9bcviB5XaW266suZPp6WwV6fq7pBRiZnRV7BwBjfmdtSfsvZoWtjU= X-Gm-Message-State: AOJu0Yx0aDSBnISBR6VFbEc5gHpw/nQrcOOqAnblHV1tQKs5A4I5Q8UM EODmWtHFbRuCupM+AbeADn5Ir2gbp0XZk6gFvfpJJEO1R6RnPPYO7Sw7G47iREDMnpccxaRfdFz mEw== X-Google-Smtp-Source: AGHT+IELFceSPWFrUjgphXkqF3QF81hXppl+wz2Z5M52XUCNjOCO0HkPXUz6VhR3ZLtCxZs2Az+yJ39E678= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a81:7948:0:b0:609:5bd8:de84 with SMTP id u69-20020a817948000000b006095bd8de84mr1217843ywc.0.1709749525732; Wed, 06 Mar 2024 10:25:25 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:17 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-20-surenb@google.com> Subject: [PATCH v5 19/37] mm: create new codetag references during page splitting From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: C8D8C140007 X-Rspam-User: X-Stat-Signature: dz1ubf6ha9xnqht96wi7hepb6bmu5okx X-Rspamd-Server: rspam03 X-HE-Tag: 1709749526-122990 X-HE-Meta: U2FsdGVkX18R3rrcDgfapuFWC6OxCUMfXBIBvdKhJ4mxPeJt/wWPELtwVcnUBGs/lErWIVXTxi3fK/8I3K1A2p/o65TMDbhHpY7G6eDNFYG+GJ+Ja9kczT84FQLvrPMhRKQkkoOV1GU+S8VvaYPIPHw5HnTUNb4i+CDN8KpO3mMhJsMQIONihumDdGoxSkVf7ZVlRUV+x/dN/4rtqS3w7qeB9fh7Ozm/U1vrFoNSD6faR+2+UPllB17M877soFJVn7l+BneS2MQOzDLooGKKZGw3ayXJiBXszGmleFGKiXrZaj02R2zCWRfwhJv2xg0AOn4sT90HIdsk8Urpvt5IMB4GzcsaEqZyf/wgkmB6SgtiWp3s+AkHyo32UlYOnsP90J8MNBmJxdTCQ0qwNWlPzvx0FDySw1hJxpFMZv2o1yLGKXzAgx5qvk7hFGdEx0pK++BMFY2qVa4Y9ashM7aKIi6Amc6z2CUx97DP9S2jT5DithOtyFLjQC4zoUGiPSE5Wfv2MzNDa83LX4YfKi3xFg34v8MxE7Atko9tyCjezfkXFVRrXtxIfsUukvE7NPz8u/nbT/D2THVgcG0aGAXyUt6ue//9OH4UYv0qoNwy9EfzMB9nBYcUdUcS6BOgGnyjmeP0s7Hzs+9dz0P8wP2FARg8TKiyJImweZlNXvIWIsrplaXYZZbMgqfsYjYleEP29TFbE4d3hb2TAX6t3BPsflQ8Zw9J65GCaLKMORuDQGWYsKLIQuIVWvk9qdDP6bSgDjfm75bD0sFmAxjFcyENDIXrAtWIoX12mj5c7D/gPwh1M1h/5uNMrqzJjKJEK+xXUo6qwvTiiS59DvcXNadftII8+UtQ2ctsH4VPLf10czAsxpnAQpe2ubP7JjnG9m8x7aSM2hiiqOFylyrInCNK8BC21xHd+KJhYicYRoU2SPD2Ca7I8Qt4HJsOqB2FpWSICAcZUSgVRnPlo6G9AfR WS2QRnUp A86AqB6YHxFiYqrMEt5NdRPbaF6DfBIVb75dZ3V7hKFwGpIYZjTjDF7k+1SJ8vybuRngOB7AtnCbxkDdtr52a6WkqyGoqT2EVQG5jS+7X61g7YRhwyUjdxaPcVccNr1bZPg/1bYROP6n7/M8lhXRVWBCC3qB9rcrbyrFsmMfmPoSBQIfd1UFv7TJMGuOAM+4eOStf87KqaQdvJpJY4rt2SADi3OtsyH6I32xSxN1KfEKheml7QuUyUZzSvpcdTySKVzp9SWgiOSOzpqRSel6tx3BF3YkDaCFY/x/hbeT9C+Ifb7BAkZi0p+eEP8P9Bx2ui+dqfeIicuskzSbI4V5Gqki7rtmgCGmgavuYqtDq7fRQUkmhOcvFo5mW43OXSXkqDqe7GLKVkYiEKE7XIWlvymrPQYr9yLLGGjUD X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a high-order page is split into smaller ones, each newly split page should get its codetag. After the split each split page will be referencing the original codetag. The codetag's "bytes" counter remains the same because the amount of allocated memory has not changed, however the "calls" counter gets increased to keep the counter correct when these individual pages get freed. Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka --- include/linux/alloc_tag.h | 9 +++++++++ include/linux/pgalloc_tag.h | 30 ++++++++++++++++++++++++++++++ mm/huge_memory.c | 2 ++ mm/page_alloc.c | 2 ++ 4 files changed, 43 insertions(+) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 28c0005edae1..bc9b1b99a55b 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -106,6 +106,15 @@ static inline void __alloc_tag_ref_set(union codetag_ref *ref, struct alloc_tag this_cpu_inc(tag->counters->calls); } +static inline void alloc_tag_ref_set(union codetag_ref *ref, struct alloc_tag *tag) +{ + alloc_tag_add_check(ref, tag); + if (!ref || !tag) + return; + + __alloc_tag_ref_set(ref, tag); +} + static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, size_t bytes) { alloc_tag_add_check(ref, tag); diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index b49ab955300f..9e6ad8e0e4aa 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -67,11 +67,41 @@ static inline void pgalloc_tag_sub(struct page *page, unsigned int order) } } +static inline void pgalloc_tag_split(struct page *page, unsigned int nr) +{ + int i; + struct page_ext *page_ext; + union codetag_ref *ref; + struct alloc_tag *tag; + + if (!mem_alloc_profiling_enabled()) + return; + + page_ext = page_ext_get(page); + if (unlikely(!page_ext)) + return; + + ref = codetag_ref_from_page_ext(page_ext); + if (!ref->ct) + goto out; + + tag = ct_to_alloc_tag(ref->ct); + page_ext = page_ext_next(page_ext); + for (i = 1; i < nr; i++) { + /* Set new reference to point to the original tag */ + alloc_tag_ref_set(codetag_ref_from_page_ext(page_ext), tag); + page_ext = page_ext_next(page_ext); + } +out: + page_ext_put(page_ext); +} + #else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int order) {} static inline void pgalloc_tag_sub(struct page *page, unsigned int order) {} +static inline void pgalloc_tag_split(struct page *page, unsigned int nr) {} #endif /* CONFIG_MEM_ALLOC_PROFILING */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a81a09236c16..d596449b5bc8 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -38,6 +38,7 @@ #include #include #include +#include #include #include @@ -2946,6 +2947,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, /* Caller disabled irqs, so they are still disabled here */ split_page_owner(head, order, new_order); + pgalloc_tag_split(head, 1 << order); /* See comment in __split_huge_page_tail() */ if (folio_test_anon(folio)) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index eb5cae9b967d..39dc4dcf14f5 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2663,6 +2663,7 @@ void split_page(struct page *page, unsigned int order) for (i = 1; i < (1 << order); i++) set_page_refcounted(page + i); split_page_owner(page, order, 0); + pgalloc_tag_split(page, 1 << order); split_page_memcg(page, order, 0); } EXPORT_SYMBOL_GPL(split_page); @@ -4850,6 +4851,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, struct page *last = page + nr; split_page_owner(page, order, 0); + pgalloc_tag_split(page, 1 << order); split_page_memcg(page, order, 0); while (page < --last) set_page_refcounted(last); From patchwork Wed Mar 6 18:24:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584430 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC6B5C5475B for ; Wed, 6 Mar 2024 18:25:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 162DF6B00AB; Wed, 6 Mar 2024 13:25:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C2F96B00AC; Wed, 6 Mar 2024 13:25:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E0B1F6B00AD; Wed, 6 Mar 2024 13:25:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C7CF76B00AB for ; Wed, 6 Mar 2024 13:25:30 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A6295160A25 for ; Wed, 6 Mar 2024 18:25:30 +0000 (UTC) X-FDA: 81867442020.04.1AEB0F7 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf30.hostedemail.com (Postfix) with ESMTP id EB3E48001E for ; Wed, 6 Mar 2024 18:25:28 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=qtuYRcdA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of 3F7XoZQYKCFoKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3F7XoZQYKCFoKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749529; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EvAS1Tk9rG7Nm0pd+ehjWtNqiQusCGfZiqvBjL56bfE=; b=ypz553gZIy2HsnCPF3uHiwOB6nE5t54EflLPTnPhJutnwxNQO5aWWiIHBFmRjnac5xT9QM sns1g8GugsmijnC/gbKEjcysWzSdIrBL8fjMlegwanVy0NjgAm4KPKnlXOm3ezXwG9XcUn R8eNPNo/sSUzwedTCro4bnyM0p9/2qc= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=qtuYRcdA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of 3F7XoZQYKCFoKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3F7XoZQYKCFoKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749529; a=rsa-sha256; cv=none; b=r01+EGApfPPbdYgL+B/fRAGb/ZpI4O2WqN+F7rJ96X4VVC0eBgH2EuJ4+sBYPBUFC0Tnpm sMqoy10WoLtKGw/uR65JeVoKUat3QADanorjQgbl602nXkf/WgCNGq/zoqJDk/Ev5AYiSm /i/B5JcQQnKviNfj78hzTPZcinMgsfM= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-609db871b90so316697b3.1 for ; Wed, 06 Mar 2024 10:25:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749528; x=1710354328; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=EvAS1Tk9rG7Nm0pd+ehjWtNqiQusCGfZiqvBjL56bfE=; b=qtuYRcdA7jqpGk8UEpdcybZVQQB0+mVwLbMGWvh+9dmsrmOkpODSd8kpVTBAY8A1sC 5MHD5Ss2aZMXWain6LsBei7G27b9ELavncvpNBqmIlyXe5b3CMQsO/b4B7qzP2QGlO4A EPDOTo1R6n/PJ9iwAa2OVjC5WrAenRV1S3WbKKnJvBhPJ65Vk8xTa/Imk1byKafzbJ9o GTTzW4UpN11MFXFby1e5qHFvrPkb5fOIcnjq0D1FNYUZW3BhcClC3o8S83IlC0wecnui mJmFumIKXz2uLf7omjbDCJOQ2iTUMF3sBbA7JAP6/wH0B6A5AJ3PT+D5uOR9ShnSm8Uk fTQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749528; x=1710354328; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=EvAS1Tk9rG7Nm0pd+ehjWtNqiQusCGfZiqvBjL56bfE=; b=dqNDt5ypsMoYoqYW5KDrBMtDqspwt+3VwNxQJRi5ES2L+UPGwEO/HHwEa4d2m4Gh6Z eCfoVEtRwJxNMv02A4bbSghwJ0rjhGLlCcX0cfaBW94rzBc6ecM+cyreEB/r47+xae1i py4Q5qva7mfCuUCxZQQTf6Rfm5OGF0TRMYdxPs0Felp9IK8qv0W2Bd4zwr39Yt3ZG9Xa lgwLYDzzD/Wt72VA8oub8mbOJVknlkHwJmO06J8BLKSU+eBMd6h4s2EcMYK0l/OlaL3U KAvcCSlDu4XgDfu5Q3Ah53Wy2v1aFyXemVBj6slv4rIpA/7uhJIGotfBP0FBQZPrCqru 1fjw== X-Forwarded-Encrypted: i=1; AJvYcCX2nJM1gd6NCVCH3l5dv1w4jEslUjn1yym3vY/7wbGitP+ONOJVN1GVDP/BC66rAp43SyBIdwM/Jl033tGWzboV3fI= X-Gm-Message-State: AOJu0Ywuz3/fMhgSAW3sOP3m2cftpWuGRwJNpCihbWwVhCOo1v/owdFq p1NjXw+rjLEjln8hWvBsJX52G1i7YWnaZehldokhR6dxzeLFIbeyIC9FVzafs22+aCsSubTDIbD +xA== X-Google-Smtp-Source: AGHT+IFQIptCkWW78zp6nNtKkGuc5SIQRmDLhLOqGJHxbi7rcmsMoDf48+EiK0QB4NL+su2PCoHiqX08ucE= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:690c:fcd:b0:609:33af:cca8 with SMTP id dg13-20020a05690c0fcd00b0060933afcca8mr4422200ywb.2.1709749527811; Wed, 06 Mar 2024 10:25:27 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:18 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-21-surenb@google.com> Subject: [PATCH v5 20/37] mm: fix non-compound multi-order memory accounting in __free_pages From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: EB3E48001E X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: oc4d8fue4yh8kdkm64bzxdx9ouqnzrmt X-HE-Tag: 1709749528-980489 X-HE-Meta: U2FsdGVkX1/UuPniS8rW5x6B0N1b8yD/arzoDTKvc+HmClZ2bww3QSl5hzayW8nQCtqsx6aD6ErpULoD7CoQeQMRM6B+jOri+Pds8cHlZfYxxAox702levOnD7YtZDfAvbdwlphXaP+IFkMZzEAsZezEG4+LFU7i+tVTu5eboBZwHv4gDJZGVBX2FgShiGhP2zhlHpEvouJFIRVxqeWYzgsCjqHJepQ16r87yDBKbyWMR/InIDj6Dv3924mZ1FkEx0q1wLLi1RVgAEzM0bEh1BSHef/dg35vpnl2G6ufcwYuOGGkMtUI6AzIIcmfwxs0fLJ9S9is+h4n7Gy2pXFJFAb9JU2eZHuHfsjeeXuRq43cHSRlqM9iAFeAShykrLTG+S+2yFg3QWmNxPhiGhsf5SzdSuqeQWaBTP9si8wOX7mOjgTUcuR5UvkXhZ5YhiPuADiAI1othfRRvANbO59bGO3MtNZmO59QFlnOYd3HW6lha0/GoxvvTGw0gcad4oAPKYtzcV66LRWED9yTkTXeVpPUL/CmQG6GJYUxb46tBxWbjsg1SS56Ykl/b5g9wR0k5CzhsiFwgHTvc09rasPQePpDVsgGUCh+gEvuag4WzP5peoil1KcRyVjGFvQZjxzdRfJ3D+frubtuGlW3ai/gimPnE+5YDUg1TC7YOx5jZntlZN38ExXOwQdSoPdXwMbcFZyfHUJdBQnIAWucMr3+fNoutOixA4mKtTvIajXlTWg5laytc+7ySRvIBSnzrhZaJNg+eEH47NQRk7RRdd7VEWrItbuKFoDg+Xs0rwSB29ylg81yzsrKR17xFVo/jp5X+vGos1OmgvrSlPntV7sW/bPb0qf59wtEV1qDq2nR3DGBL4az+CT+VcbAew8b1Pb309aDvMACyWxSXWTKDciyN+SII2sOSLO06DDOHYDVtEz6Qm0edqKCRLovdlI5dS4SKW5jVDBQjuH494DnEiG Rp1BaIFP pPPwNMQ280zWUtS9qOa4BX60qaEPcxGOmD0vnbYVgfTb5mAXyh9vPu51pHcYYKVPkRmkievh69UVW1O78G112ageqCozFz0h/8+Uk7MHZz3gAZ2cNs9tnI/iaKRJ/HHwpnHBAXsoPhxae8co9WBwslczF4v9GZheLh0SAOU7edjj/H1bhsONOot5VUgsmoSG8gU43ysCuFgQrOvzFk+MTvOBWzncYWVdACX2Xrf3JxWpHqABOwdWjQnt2Vouj7dwOvQPX8G6tLDM6u1p+bDX3+1s5wKHREuBTzHnI2tPB9U8PJkR6yEnflB9ggKPvVWxlElzAy/P5IlGw628BRzt5Vyvux/WFE7923aWuv0FhPR3gJqfICsR0e231k+AAX9lSe1PtUmo4apyYDiLyTWUNm36I+iRvtsaGYPn8GsmY1QJeDjnLBaqOVSUL8QJDhR1Zxb5oLWZ/xjqO0OqxX2qbj2268w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a non-compound multi-order page is freed, it is possible that a speculative reference keeps the page pinned. In this case we free all pages except for the first page, which will be freed later by the last put_page(). However put_page() ignores the order of the page being freed, treating it as a 0-order page. This creates a memory accounting imbalance because the pages freed in __free_pages() do not have their own alloc_tag and their memory was accounted to the first page. To fix this the first page should adjust its allocation size counter when "tail" pages are freed. Reported-by: Vlastimil Babka Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka --- include/linux/pgalloc_tag.h | 24 ++++++++++++++++++++++++ mm/page_alloc.c | 11 ++++++++++- 2 files changed, 34 insertions(+), 1 deletion(-) diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 9e6ad8e0e4aa..59de43172cc2 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -96,12 +96,36 @@ static inline void pgalloc_tag_split(struct page *page, unsigned int nr) page_ext_put(page_ext); } +static inline struct alloc_tag *pgalloc_tag_get(struct page *page) +{ + struct alloc_tag *tag = NULL; + + if (mem_alloc_profiling_enabled()) { + union codetag_ref *ref = get_page_tag_ref(page); + + alloc_tag_sub_check(ref); + if (ref && ref->ct) + tag = ct_to_alloc_tag(ref->ct); + put_page_tag_ref(ref); + } + + return tag; +} + +static inline void pgalloc_tag_sub_bytes(struct alloc_tag *tag, unsigned int order) +{ + if (mem_alloc_profiling_enabled() && tag) + this_cpu_sub(tag->counters->bytes, PAGE_SIZE << order); +} + #else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int order) {} static inline void pgalloc_tag_sub(struct page *page, unsigned int order) {} static inline void pgalloc_tag_split(struct page *page, unsigned int nr) {} +static inline struct alloc_tag *pgalloc_tag_get(struct page *page) { return NULL; } +static inline void pgalloc_tag_sub_bytes(struct alloc_tag *tag, unsigned int order) {} #endif /* CONFIG_MEM_ALLOC_PROFILING */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 39dc4dcf14f5..b402149a795f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4697,12 +4697,21 @@ void __free_pages(struct page *page, unsigned int order) { /* get PageHead before we drop reference */ int head = PageHead(page); + struct alloc_tag *tag = pgalloc_tag_get(page); if (put_page_testzero(page)) free_the_page(page, order); else if (!head) - while (order-- > 0) + while (order-- > 0) { free_the_page(page + (1 << order), order); + /* + * non-compound multi-order page accounts all allocations + * to the first page (just like compound one), therefore + * we need to adjust the allocation size of the first + * page as its order is ignored when put_page() frees it. + */ + pgalloc_tag_sub_bytes(tag, order); + } } EXPORT_SYMBOL(__free_pages); From patchwork Wed Mar 6 18:24:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24599C54E49 for ; Wed, 6 Mar 2024 18:25:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 054BE6B00AD; Wed, 6 Mar 2024 13:25:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EA5DC6B00AF; Wed, 6 Mar 2024 13:25:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD3E86B00AE; Wed, 6 Mar 2024 13:25:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B457A6B00AC for ; Wed, 6 Mar 2024 13:25:32 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 8E3F0803CA for ; Wed, 6 Mar 2024 18:25:32 +0000 (UTC) X-FDA: 81867442104.19.77BB383 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf23.hostedemail.com (Postfix) with ESMTP id DC2CF140006 for ; Wed, 6 Mar 2024 18:25:30 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=QP8VrmB1; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3GbXoZQYKCFwMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3GbXoZQYKCFwMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749530; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=r962bvR5LCwoMgFXL0pngYYUaEUVgmlMd//tXW54EfM=; b=QoWTR6Ct5qeYbIzkLLEgJKyiSd6SYOdbYKl5Ucxp+A6mLGJwnK3zEtCVT2REFz9UoQ+zgR anL6L5MiVOfabHtYWzPOP1MUiqZjjG/sIfi0VRMToqTIb9L1qJoaMMD163mBsPnyV0Ycbm IU0UnNouJXhcdX/GAigJd5101WanrvE= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=QP8VrmB1; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3GbXoZQYKCFwMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3GbXoZQYKCFwMOL8H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749530; a=rsa-sha256; cv=none; b=fJOkza8f6+652wJ9Zrp/6cazmwFFYW92+A7R4JCb6xm5+6/67CKh2gk9La6hSjFScxRbiZ cVf9ZChPdTc+zFtiZW2QvfvJW+nDAageNBD5f/SLIQ3EsUzY4jkUIUhsPPBxuFovjod5w5 QuyDgGrjUQ6a4qQOhbgOVQdMI20ILcs= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60996cdc37cso163167b3.2 for ; Wed, 06 Mar 2024 10:25:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749530; x=1710354330; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=r962bvR5LCwoMgFXL0pngYYUaEUVgmlMd//tXW54EfM=; b=QP8VrmB1Zufw29Onkv3WirzTMPdwn6ABWys1E/Bd1UoPe9iyIwox0SFk09bbwKCAKt oRv/3nDjqJMFwRHFNHL32a1Mh7X1tzNqLpgVInxYJ89dxbEJc6Zt/fKHu0hIH/+CJrHi MI9IOR6aPuGVSMB/7Dt/zMv52WlYWMHBfgzpxVLYRnjqt9cmFGPrSpgQwu/YN6YU5Vm3 HhCmq8aHnGS5nT0i8ivRqAfzFULVxJJKCU9Ka3/d7yT+BRo//LxcyaMF2wZjF8/K8z+1 OMhi4yGSmy7rPGl/Xe1e678xQt2Xh8g5o78bTUdpnu9GgYRqJlu+zKlYCKcqxwMl4zFQ Qg6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749530; x=1710354330; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=r962bvR5LCwoMgFXL0pngYYUaEUVgmlMd//tXW54EfM=; b=Z3SjcejBnJfPzU/iXRBRvPFKzVRHTTpVSeQOvwunsQzYkrEQ1+iTbzER9xYF+mQs35 kuZDA0h9T2sRtHTI2CmNQJp78hKsdfWVl1tEp3jSJv4Vdyh5o5VMI157SHKqwvNMZaDR VGrZqiZvLDFWI521WHmf5Rj+6DRs0XKrTOutDOLPTR/FG7RnmVYGlGbUuGk+ODkuyVnd cPpOQ6/GQkRNesHC5oQu7gyCFZ3yqTjB8e6HPR724/m2UOdyF5Yhpd6g8GNBANV7WP08 SYScMtE3gY85gT2MJnT/gMRRLKQ/bwZokk/bnul6mDR9rQUKYMp1/MSWaJ2LJ4ZYfzRj BcZQ== X-Forwarded-Encrypted: i=1; AJvYcCWEP2SPeHdlarQ/dkCGDHXvLQzpqg8vEVIWsW7GmZruDaq3AXYvuBvE5zHzSzyHneGjUed3faX8Kklm8mc2SBiMS7k= X-Gm-Message-State: AOJu0Yw1K62JIQZb2l7r2hpatqW2K2TMRBbS9QCYBmolYBAlPRMjk0xd EO69ol+f8wUdYddMS0L5N8JHE+Y6inodegWuqEcnx4xip60TK8p9OcmYC+mlZ+L7vk1YmxV9Wqe K4A== X-Google-Smtp-Source: AGHT+IGe7t6LCx7ntWuuN0pG3BUD23g3ZHCgKcqafT1luEHOL8Lf4ydoGId/CWh1jbpa65530MO9Fdk68K8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:690c:82:b0:5e6:1e40:e2e3 with SMTP id be2-20020a05690c008200b005e61e40e2e3mr3383691ywb.5.1709749529862; Wed, 06 Mar 2024 10:25:29 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:19 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-22-surenb@google.com> Subject: [PATCH v5 21/37] mm/page_ext: enable early_page_ext when CONFIG_MEM_ALLOC_PROFILING_DEBUG=y From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: DC2CF140006 X-Stat-Signature: 4rex5hfkeiee5ki9e4s9r5thmy7j3sno X-Rspam-User: X-HE-Tag: 1709749530-273058 X-HE-Meta: U2FsdGVkX1+zqhIIYtOG8785d+ybwhRHocDyIIqBdcz3fkBp/8B2tmWJZT/cr11AhtuMF12eI0T849d7b0vAcC2Pbg/MkamZ3BakA3l5V0ZmolTKCkkMyKGYM5V+T0hycTzDZwYsVnpFz/s8whDDiak6Fx30m6OfZZu+Rk1eOiHWPQHiWf75MWzuRZtTj7i8wXyc4N4+B7EietXUHHn+VYX3TjTnc3cEgAyEWhjdHCVR8bpH1zbyslSXnMs7otGlR6UbuYurYMm4qOAgVnU6dPh4Jeq27PLs0mzhEO81aV0S3enXLPu+fQ1735Yfjv+ItQpH29ybHlHnqKb5LMBFC6TUaPrDR3I4yDPjLwH1fi1dDqKzewEu3NCHbG1MLkLToTPATEMb99FVzW/XiQMjhRI43/kV5EEXU5vK10ep8D2fu8acNq84sSuXOv8qg9Rvxv4n/mPavpMsw+faVhaoTpGju0vGt3GZgfbjG/ap4Q44D8TvbLoaXAtu7blPTSMAQPK/ueqHD6/jm2MtWXucVWMJVDYNfQ33T4c8QnEvU+y/KiovEBqgsh8DBCPdsZsB4/LTZCHNaKpPODzXjrmB6UWK8bLCU7SvHDdoI4ioANIgXVs9qznBHJ3oOVadzZM6rlv5JuWUcMkA5zpXNoDWMt8vdLfUrri+OVKSmVj0O+vt+M/UBVGCu/uH/zjmaNhF3G61AnW+XgvMeU0TS6VMIpsJ5suYcgVVOtBmoPFklEaqnBudH0K6W1Y9C4tgh4GF3uI8X8PBStEWyArr57NzjGnfy9XZ5Gqb/BZhsj2DZT53EzmtmpwMa8FjoO5PQqvoyzAJHD9ao09aKGqpVCd6e6qD0uLfHYePZ+vAlU//yfVj05cYh8qb2YA9LMV3Qu4Y6QC0rpghSD9d4uXELqGKY7REQCCRA85fUgMRqzDnw+nUrqdcnlvL+hWxrOAv7Ls8LwkAkKO47p2ZN7Uqo6g f6MFo5eZ VXZPh6MF/0qi8aBNn2dWIME46CTA9TVvpu2QCdizCozevVYEX65m7DYySkNv7f35g235Wz+LoMXsNNAbktGUMpB+matWElfhduZsrVCXxmHBit3P5xgXIlKXsIrKyJ3ikCYkQI3URn+cL3gS6DF8ge0OiQEmhjLnr4OY+nT1ll51BFto0S+ZpDMKA6ktEGQNNw2IJzIq3Jmo0vQ5aeufWz0qJA/OjsFYX/ukktuMtUC+5ajzChzJxNQY9RBy1vFRsapBnZwfiP7Wof/jmFPpEmEXhiSJUmyYAfABcj3ni0unuSbKDrVzaC84C9gKAZN8m60Z844+7dw6PYlW5hbfsLtHpjZPsf+woMHXVpT5qnHCYLq4UWWKws6E/UaSpPCGkmFYWQTMQ1CGn3d5snaxZWerBxTulhGL5kCwST1KWTUNJz5FJBSMSZ/Z14P4Qwih+k9HJ8+wIXtLsFYfLRxm6RNMhlw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For all page allocations to be tagged, page_ext has to be initialized before the first page allocation. Early tasks allocate their stacks using page allocator before alloc_node_page_ext() initializes page_ext area, unless early_page_ext is enabled. Therefore these allocations will generate a warning when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled. Enable early_page_ext whenever CONFIG_MEM_ALLOC_PROFILING_DEBUG=y to ensure page_ext initialization prior to any page allocation. This will have all the negative effects associated with early_page_ext, such as possible longer boot time, therefore we enable it only when debugging with CONFIG_MEM_ALLOC_PROFILING_DEBUG enabled and not universally for CONFIG_MEM_ALLOC_PROFILING. Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka --- mm/page_ext.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/page_ext.c b/mm/page_ext.c index 3c58fe8a24df..e7d8f1a5589e 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -95,7 +95,16 @@ unsigned long page_ext_size; static unsigned long total_usage; +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG +/* + * To ensure correct allocation tagging for pages, page_ext should be available + * before the first page allocation. Otherwise early task stacks will be + * allocated before page_ext initialization and missing tags will be flagged. + */ +bool early_page_ext __meminitdata = true; +#else bool early_page_ext __meminitdata; +#endif static int __init setup_early_page_ext(char *str) { early_page_ext = true; From patchwork Wed Mar 6 18:24:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584432 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C7B2C5475B for ; Wed, 6 Mar 2024 18:25:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2262C6B00AE; Wed, 6 Mar 2024 13:25:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 186C26B00AF; Wed, 6 Mar 2024 13:25:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ECD646B00B0; Wed, 6 Mar 2024 13:25:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D58D76B00AE for ; Wed, 6 Mar 2024 13:25:34 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B2F86120144 for ; Wed, 6 Mar 2024 18:25:34 +0000 (UTC) X-FDA: 81867442188.02.7A24E63 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf17.hostedemail.com (Postfix) with ESMTP id 16A6A4001F for ; Wed, 6 Mar 2024 18:25:32 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=acOId87S; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3HLXoZQYKCF8PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3HLXoZQYKCF8PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749533; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rg6IJ0Q4L7KBxRODnLb4yefimTum4BvWKaOTt5/YB2o=; b=aeo8EF3T+Ja9dxB2j+qZNvfDMOx+NjQr50P8D6yLkqcjr97cIsVIs3wxnbRspg0WioJdH2 SWID/taNEqc21QRZZI5pcBJ6u/HdclnwqzLBlitxlzeFdK6kSc7vltbe3MtD1B1mWbao00 7vI6q46EBMM8X8U1V1mJf2i00hgqUZw= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=acOId87S; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3HLXoZQYKCF8PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3HLXoZQYKCF8PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749533; a=rsa-sha256; cv=none; b=U5yo9UDshx0ntqS90DuOWEe35rIV73xRqf9T4D1rNtE/N4bbd7Qk98aM7PrnK3xFtC064p EpNfSJvARcGshrwrjyeR/KiU0ZOMsLUrO4GTD0H6hk8s4FYaB4It0TlXMMgLOMiDv+bZ70 3zMhwA9g9uEj6/mQb9QShP0qtmHCwz0= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60996cdc37cso164217b3.2 for ; Wed, 06 Mar 2024 10:25:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749532; x=1710354332; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rg6IJ0Q4L7KBxRODnLb4yefimTum4BvWKaOTt5/YB2o=; b=acOId87SQE/40NNvc1vLrh+faf4TReBgkFFCeva/2Kl9WjfZl8z0wu/cI51DxZovLY NsTNOwfE5BeBS5ToYhbafNpK05FPryqBOmGGfJ6/i0csCDp3HDxlBRbE+qkKkiYUu66P 4dB4X+Xd/V2vTzyjpDM5GbTKa9gQo5mSXZHEYwNJjmCCpK04b1gKB0RZjSMULuHgbfnC E5bADp3BHE77BnckiOgv+knnDziuAiRyG4TDgtkD0uiGVY2999rFro9J6SX4XcCGiPGM uVOL9TZj7dj7/AoChXjLEEvqJyDYHK0Clm13UGLjjXxBxUooB0BtlUyapau9I/aR58R+ jnHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749532; x=1710354332; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rg6IJ0Q4L7KBxRODnLb4yefimTum4BvWKaOTt5/YB2o=; b=R22pGkDVz96+xLPf6bnvSW74jVt6ZRp3x46HL4MB4wnFidywubnTIthiSVhmK4WTQi atzTW1CGTOc2LGrzVIdI8Gga37SYtMZ+0Mzmov5lp5ZEn7VpxRfiXr98l0/qzq+UKyKG xdrP7BhkpDIq5iGbwTx8fqpL7+XaWEMKSRfOwfY/FHMPGeZRdMEyMMk/R/QB+fpsT2dl 2CsxGyDpSFllEmrRns9UJGVkt5NT/s8w5OaYNk70oxuxKftAQVE2J0ejkwI4Pjnw0g5T Qqi8YKEf+NjpAO0qKqrpNCMVyMk4yB+Esw0q/4ZIBe6XL8BNHCEIARRY25Fkx4DifgWR rdpw== X-Forwarded-Encrypted: i=1; AJvYcCU6uEDnR8n4GRcFC9WX+kyz6xDuVh+i8YTOdYOaljYrJg4c9QdI+K777yErRbvze5DZWvJdePaD+Mim49QGL0MSMR8= X-Gm-Message-State: AOJu0Yy+kR89KE4VRGMlAMqH5AMmsioOudHE5H7+3gd6TcprbCoW16ql 3QmegqEsxMcvKuAGOw6skjkKoJ+LaL08DhLCeHmsnRjqpXrZhNL5K2J1SZMnlRUalgLTECzWvHf a2Q== X-Google-Smtp-Source: AGHT+IF+RFcX0NYxjT6uNvnW+K0lH94IF3O72VJ6PT7ea1CL+TwW9QtFKtQNO2MOoPPDxhg0sWNTYKHdXg0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a81:9945:0:b0:609:781f:a7ab with SMTP id q66-20020a819945000000b00609781fa7abmr3355582ywg.1.1709749532080; Wed, 06 Mar 2024 10:25:32 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:20 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-23-surenb@google.com> Subject: [PATCH v5 22/37] lib: add codetag reference into slabobj_ext From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 16A6A4001F X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: q8btowh1auiex9xf74wj4hb9w7nypx5x X-HE-Tag: 1709749532-154884 X-HE-Meta: U2FsdGVkX18bNrbfiuA9/dIO4xxAXstAlnvr2GpItvQuuMsul9usFyajzatWBo0nltP4MF0WgcNWRqcY9gyEZL/Do6UrieTQXHqPDo+RRwzdPuae8kHiZqz1wD3jqtBeucd+4WVOtQ5+3QnXq6iqyAecwm7F8W9CH0wpjY1N3Vj5DFjd6YoXRwHJanAc8OmqjzDx1GHRtP8/b7sk3mC5oFPMdfhuaNBF4+I7rScS3Wm4D3mntg7koLRD3090m9zkddxgC5dZM+m5x1Bd6EIJVtmH4K/DZp6+XzgRvs7fy9mu2/WHx/Xm5l90pqS9hVzr/lTuM8Ox9XPQBs4al42M6M05kDtapBbHGoSURReVb/Iic0EEKJYdjALs8jKE/NGLycshMpcRB5zqYmCGorBKXw/ZBMxjvMt2LywjmieSUYTdXuPtJwWpWvpznx/6UgS7ZE+DrHvvFsIBl//ZIaZvOdjiitD3DoN7jDv/y/OPwHs5MT7AdxZCwBEX+8WR7PQW+T/eo+YKYKuiaHS55vE6es6dNOBqe3zDjS+hsiqjnuImBkcxzIlN2ulG2gKWVK6HkhnfCFBwpFVXVN2QD1hrJxEGR+P6w/A/+8e/tIN2Xd4iyRXZDSEWh9QOkjwo/Fbxf1mLO5xJY17atpFRRgKufC22KSPu+RimCi0GwmsqTlBWI0lPkx243iK35U6hwZS5xu/JUamlWlbpety9huXpdR62nK/zfX854qV1jMKSvzG/MGtV8oLwvYhlV7lftfFtwgQVKRSP2a2r6kYd8u+eChm9lEdDCsc9IgD387qHT8PiKi6C/D15AtdGLnZMW8JFnw/2IdKQse3NFHeN7VBzl8Vo3Q9GSoX0OU7QsSVZq1tAYjqk7qownF7kWuKZAdR1/lzZyYEDCe7tpMjMFeadAzfPySzmJSCIV98ly2mEJ4QgfC8wRybafCDSWX6zVGr2wsYX0EPO4ianakWbbnz fCr4+FCA gPIq6JBWvSsE+sHyWZEdAzrjSuEpnv9Ex9rJDcnQcHstiRzty7MMg06bXXVb4kJ2UDcS8d7CjTs9do1sx2hN6PD1n5gNpzCv+bL9ccS+3xK4xZjwFgPt9Vq7BdY5zY4eoiCGn4yNa4KXGuKfqbEYKqhYlK/jfWN781RsD1EqUvwqHagsi+QgqzakKqU2iTKKN5d6ABCApqNdCfmfv/MXixgsk8zP8dU0xWtqoRj+06ZpEZhekLmbi5QX+0ZSvHX7KK/4uubJHBfpHSj7mTrjsvithC/KTDNxkvBNyNiK3VMxMKYwdxqcLTYcy7X6Y8aQk0R7Tgss6f1o+4a2bh2gIssIg5JWrl5wVkERxStM5onoLzGTh4WUgA8H/+gS5l/KwjLJ3L0UqV2sfzway924cNbWMvcWTnd2hX7deoKnD+Ob1Qf6QA1u4M4Hcl8o+45a/G2Us2No5ipMuDesc0rzBihaF9A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To store code tag for every slab object, a codetag reference is embedded into slabobj_ext when CONFIG_MEM_ALLOC_PROFILING=y. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Reviewed-by: Vlastimil Babka --- include/linux/memcontrol.h | 5 +++++ lib/Kconfig.debug | 1 + 2 files changed, 6 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 7709fc3f8f5f..33cdb995751e 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1652,7 +1652,12 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, * if MEMCG_DATA_OBJEXTS is set. */ struct slabobj_ext { +#ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *objcg; +#endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + union codetag_ref ref; +#endif } __aligned(8); static inline void __inc_lruvec_kmem_state(void *p, enum node_stat_item idx) diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 3e06320474d4..dfb5a03aa47b 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -979,6 +979,7 @@ config MEM_ALLOC_PROFILING depends on !DEBUG_FORCE_WEAK_PER_CPU select CODE_TAGGING select PAGE_EXTENSION + select SLAB_OBJ_EXT help Track allocation source code and record total allocation size initiated at that code location. The mechanism can be used to track From patchwork Wed Mar 6 18:24:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584433 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19137C5475B for ; Wed, 6 Mar 2024 18:25:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E359C6B00B0; Wed, 6 Mar 2024 13:25:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DBCE46B00B1; Wed, 6 Mar 2024 13:25:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B4C686B00B2; Wed, 6 Mar 2024 13:25:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9CB376B00B0 for ; Wed, 6 Mar 2024 13:25:37 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5B20441017 for ; Wed, 6 Mar 2024 18:25:37 +0000 (UTC) X-FDA: 81867442314.29.EFFC373 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf27.hostedemail.com (Postfix) with ESMTP id 5F75040016 for ; Wed, 6 Mar 2024 18:25:35 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=hP0D7H4Z; spf=pass (imf27.hostedemail.com: domain of 3HrXoZQYKCGERTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3HrXoZQYKCGERTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749535; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=s15jajKF49lJj6S4Z8eIgzY9HreX5MNZRA5AU2XXX3Y=; b=ByyIX4zP2FdquxbcYMNOjdm5er+qyupJ4DJOh9ipWkLToCe1C8i7B+vGP1RT8FaxKcpOCm 8DjZQ35eB7Q8ZgVSZVDxxtcQAMt3uYgTPXeFmmtbqgVdYlAOi/x+EaF+nA7ZWut8LdrWcB XgA1Y1CRuw4ZyAWZ6BTYwESEGo0YeFk= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=hP0D7H4Z; spf=pass (imf27.hostedemail.com: domain of 3HrXoZQYKCGERTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3HrXoZQYKCGERTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749535; a=rsa-sha256; cv=none; b=Nm6TtjR4yVBEbjYJEzoIHSctjcVhkBAHDXPxr0t/8zLqqft8nbuHMyKHw+8gi8WhtKc49A bhXFaeRhV6whNT3cAL2QKTxtgRQBtX6VSIuhX1+hYYgwDoHx1olhIOebIUPR8ysY+6cr58 558nLnTgM1ek2hbZZuzuqkcHAt3ZJEE= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5ee22efe5eeso74657b3.3 for ; Wed, 06 Mar 2024 10:25:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749534; x=1710354334; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=s15jajKF49lJj6S4Z8eIgzY9HreX5MNZRA5AU2XXX3Y=; b=hP0D7H4ZZ8o8J5TE3qGzLrdt5sK89y3w6Lx3+gm9CdXDp7O8McC8brB7LiIerUT6NG hNkHjt+XB1n8FbxkxGcFovtHBgsvXgZlozJnHuV8v7G71XMG2AgZ5nMvtX9GT7qnT4TJ zP15O1pnXw6Fwd5b6ec6jaj9mbh2UO3fCrVgD97ElUoVF2GJmw1AaKizrr6Fsh+zRnLS ygguwySijkfwcF0VHbnlhwf7VaV7l+GX1snJ6VAqi1sRJ42YOsGLU0D9Z2oNfSaNiN9U ac96b3AGK2LkQidzZ8wHU7N/AUIXtYvifDSldvEgcd0G+bt7clXAudNXO7TdqH3WfUbj X46w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749534; x=1710354334; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=s15jajKF49lJj6S4Z8eIgzY9HreX5MNZRA5AU2XXX3Y=; b=dfYHCNf/3TI+ievIHRedVS47YLnz3iTNY4NwykIyhwu8z1YNImNDdTHLQpBHdhxW81 KfToohxgLP7j9ClGU8lY5exv+1C4tLidIpx164awE8WsIiuHLuIZjOTparkUzCAQ/yGv F230+Nzd9ymRX7f9Usgndy6KJBjYgIbxtD3zxmL64UmL3jElrXH0hqYhseyrgaR+1IMq KWmbH2oXMT+rJg5pi3XsKa074LGg3R3lJaA38nnOUwzPKcD0ZBysDrvEcc1fAKQDBq7s lV/JLfrtOYsqN5Sj7E4j20QuDcKtJBpIXVSWiO1iRVNPgFHKBm7koP6eiYCB5adgBLAS SqtA== X-Forwarded-Encrypted: i=1; AJvYcCVEqGdqul+PAvXA8IaSfCPrPszC2KFpSYQsNGgC2NgDGNdODpTGirlFPdIobHcl7LA0NdtsNS2OtcfRRKpW7/iJpXY= X-Gm-Message-State: AOJu0Yzrz85p/R/1+tRCuB97rQS3VFG2rOwboPA+Ek2OkmUD6MdLiDWc 34KtgNzRBjGVSrmK7u9hpnZ5P5Av2XULzaKZM7hbufJXX2LLJDI2g6DftZDvMgbLu9Y6SfFrzAK qnA== X-Google-Smtp-Source: AGHT+IGmcq8HQqCEBWeZhNDcRf4OQCj0ptCPLBS+QbTgQQ1y6EL/L/eLL61CpiC1ccJurAx7LcZUvWTR4g4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:690c:82:b0:609:78d7:4e9 with SMTP id be2-20020a05690c008200b0060978d704e9mr3296801ywb.6.1709749534383; Wed, 06 Mar 2024 10:25:34 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:21 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-24-surenb@google.com> Subject: [PATCH v5 23/37] mm/slab: add allocation accounting into slab allocation and free paths From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 5F75040016 X-Rspam-User: X-Stat-Signature: wdifsa9f1tutoyekxm5x9m8oxsezmnq4 X-Rspamd-Server: rspam01 X-HE-Tag: 1709749535-648671 X-HE-Meta: U2FsdGVkX1/6fZ0NDXkvvWVSYfuoRBjTK9rJomkcHpfAdtQTG1AJLE4RSV31YqeLbfV17ZycnkJUB7en4zDCcA16VrPcq8EEhasKYepLPKDf6T0sXVh0NLQsVZq2ASXuAYUGq8uA5XUAbtAgmiNaKXPHFiLBgUbt5eRxybknAnBKW0YKolnNVAZ48EK+HYGDxyoqah1xfNuzzFGuPmfYA6FQYKiMzSJHiQDpRzNneEA4oyuWja8l3IS9dFqBcOEr/muxscEOjeo92A+5fajd6QjG6JwZqUMOjCVXBy0AKFaYUH6QdTidZ+ydmzqjhLsZtQsNe757dKabRpLm5KW3Qod8fhdvCmo8AvyD44CfNTfDZLuCKBLAMNp26gnjL5DozXXMTcZ//tiHskjvgay+nb1FJPIVAjVGzcaGkM/jIOaHVGfONrqB7YudqegvEzSTvGkWccrm6cNi6d9Ei/+1hvE3+2ntXmPTqKFdP0byzKtAMTakcX7HKz9U2m8U/u8j+NrfhmTGGG0lDEUWxshpOPTFILzkZWQJkRNYGexvLGksaGCdUQH5Xz2GnkfY252LvAHMZnbnZF4KofaPmw/X/29GVxBg5HOruSl9YxrPpAi5kvVjt0tGEt0bZt6KE1ckr3CBPzI5DUoB0sHAFbhnzhGwXh8LSJFDGeR5DxNF9QW4t7VMUdWpUrpUJo1QO9YhS6+empaGuhAenliOOHXvo++ajFcrGZsTsMBO6jHqWCZc0/8e5nKpOlupkRgGHoFLXQ+Uf0iC2TrOHQWFxgac2UDtwGBm9D/0PEMl+e1Z3KFB/Pz7X3Y4N5202AJlLP2dV4zbXFhNzp2rxx4fqw8WS2b6IlzPyb9/D7hBkmZmjh14oQRcK0MzI8CETGGRh9bY3QRZoG0Xt3Vr720XmWx4QCTM2KhZKSN4ehFE7SlzlHoQ8RKquVtwwRG6AW2ZSK2jTemSwRT5qDzVWLO0BDs exyNWdNA xEB+t8ZLFYmRoZyJSXSmOXJYncpRwES2GwrbgA3WOI3d/4jyyUo6MZwOfa9lWWCwBW68HgKUV+Z7HaDGY9QV8jWBnn9EUjmM+fWNzzDKedrlwtqZ5aLSdF0nSFEU/f8tok9xWoGKqxjy5xugttvwSV8ljVlgyp73hlsEE7vOo92PNfkNda+tl1KTU2EDdD7NDRrsorVbBfL8n3iXrhFXjEnYQYnSiMscC5SqLDypAiQPMVoQh11ow+kSDuOVVpbFWhIn6ihSFVEEO77X3PwAa/mpI9KqmHO7yr9D8G3rCoisEuaRx9cRI2Gb3+7BA4Y+k5fJd9gPiJf0SzRLEIMHztHnH0zPgFhNgVQE+0bTKkBJc6OP+TUIBMVnFu56xQLvNjSxkaCqgknKa5IwItm5IZt9umphX0xKqAhmJ6ViRpNIaEnT2PT0K5dZ/gxI+hGuEjQn+kQ48xly9WFvkQe9DdAUv02m+f0rdQsII9pgDrNFZU4f6BaYlXDrrgHPF/beTzLfb7B/Trfn4blhboqoZ5/5wzShdArhXLNGwiBHFiReBAPomUaaSvOEVtkPof33KcK1TfJmg14yFq26a++eB5T+cWN7ogvYj2yi+T/z6eYysviG172B/X7ZUlc4OMui1USqM9bFGjKwVTrQ3FDRA1OE+8sgVMUIuQeNlyNGQYvvrXmYHH3HxRXz0BdsSTtQXBJJPaS4/O1jIJREQujMT5OZM4En6cMLUvlKCIJruuSJkPBU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Account slab allocations using codetag reference embedded into slabobj_ext. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Reviewed-by: Kees Cook Reviewed-by: Vlastimil Babka --- mm/slub.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 90 insertions(+), 1 deletion(-) diff --git a/mm/slub.c b/mm/slub.c index e94d3cc1b270..ea122aeb89fc 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1942,7 +1942,69 @@ static inline void free_slab_obj_exts(struct slab *slab) kfree(obj_exts); slab->obj_exts = 0; } + +static inline bool need_slab_obj_ext(void) +{ + if (mem_alloc_profiling_enabled()) + return true; + + /* + * CONFIG_MEMCG_KMEM creates vector of obj_cgroup objects conditionally + * inside memcg_slab_post_alloc_hook. No other users for now. + */ + return false; +} + +static inline struct slabobj_ext * +prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) +{ + struct slab *slab; + + if (!need_slab_obj_ext()) + return NULL; + + if (!p) + return NULL; + + if (s->flags & SLAB_NO_OBJ_EXT) + return NULL; + + if (flags & __GFP_NO_OBJ_EXT) + return NULL; + + slab = virt_to_slab(p); + if (!slab_obj_exts(slab) && + WARN(alloc_slab_obj_exts(slab, s, flags, false), + "%s, %s: Failed to create slab extension vector!\n", + __func__, s->name)) + return NULL; + + return slab_obj_exts(slab) + obj_to_index(s, slab, p); +} + +static inline void +alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, + int objects) +{ + struct slabobj_ext *obj_exts; + int i; + + if (!mem_alloc_profiling_enabled()) + return; + + obj_exts = slab_obj_exts(slab); + if (!obj_exts) + return; + + for (i = 0; i < objects; i++) { + unsigned int off = obj_to_index(s, slab, p[i]); + + alloc_tag_sub(&obj_exts[off].ref, s->size); + } +} + #else /* CONFIG_SLAB_OBJ_EXT */ + static int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, gfp_t gfp, bool new_slab) { @@ -1952,6 +2014,24 @@ static int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, static inline void free_slab_obj_exts(struct slab *slab) { } + +static inline bool need_slab_obj_ext(void) +{ + return false; +} + +static inline struct slabobj_ext * +prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) +{ + return NULL; +} + +static inline void +alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, + int objects) +{ +} + #endif /* CONFIG_SLAB_OBJ_EXT */ #ifdef CONFIG_MEMCG_KMEM @@ -2381,7 +2461,7 @@ static __always_inline void account_slab(struct slab *slab, int order, static __always_inline void unaccount_slab(struct slab *slab, int order, struct kmem_cache *s) { - if (memcg_kmem_online()) + if (memcg_kmem_online() || need_slab_obj_ext()) free_slab_obj_exts(slab); mod_node_page_state(slab_pgdat(slab), cache_vmstat_idx(s), @@ -3833,6 +3913,7 @@ void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, unsigned int orig_size) { unsigned int zero_size = s->object_size; + struct slabobj_ext *obj_exts; bool kasan_init = init; size_t i; gfp_t init_flags = flags & gfp_allowed_mask; @@ -3875,6 +3956,12 @@ void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, kmemleak_alloc_recursive(p[i], s->object_size, 1, s->flags, init_flags); kmsan_slab_alloc(s, p[i], init_flags); + obj_exts = prepare_slab_obj_exts_hook(s, flags, p[i]); +#ifdef CONFIG_MEM_ALLOC_PROFILING + /* obj_exts can be allocated for other reasons */ + if (likely(obj_exts) && mem_alloc_profiling_enabled()) + alloc_tag_add(&obj_exts->ref, current->alloc_tag, s->size); +#endif } memcg_slab_post_alloc_hook(s, objcg, flags, size, p); @@ -4353,6 +4440,7 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object, unsigned long addr) { memcg_slab_free_hook(s, slab, &object, 1); + alloc_tagging_slab_free_hook(s, slab, &object, 1); if (likely(slab_free_hook(s, object, slab_want_init_on_free(s)))) do_slab_free(s, slab, object, object, 1, addr); @@ -4363,6 +4451,7 @@ void slab_free_bulk(struct kmem_cache *s, struct slab *slab, void *head, void *tail, void **p, int cnt, unsigned long addr) { memcg_slab_free_hook(s, slab, p, cnt); + alloc_tagging_slab_free_hook(s, slab, p, cnt); /* * With KASAN enabled slab_free_freelist_hook modifies the freelist * to remove objects, whose reuse must be delayed. From patchwork Wed Mar 6 18:24:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584434 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0908C54E41 for ; Wed, 6 Mar 2024 18:25:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D97696B00B2; Wed, 6 Mar 2024 13:25:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D4AAF6B00B3; Wed, 6 Mar 2024 13:25:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AFEEC6B00B4; Wed, 6 Mar 2024 13:25:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 93EC66B00B2 for ; Wed, 6 Mar 2024 13:25:39 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 73BF0A15C0 for ; Wed, 6 Mar 2024 18:25:39 +0000 (UTC) X-FDA: 81867442398.11.2FEDDF8 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf05.hostedemail.com (Postfix) with ESMTP id C34BD10000B for ; Wed, 6 Mar 2024 18:25:37 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=YdgrK0Pv; spf=pass (imf05.hostedemail.com: domain of 3ILXoZQYKCGMTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3ILXoZQYKCGMTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749537; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=K6AcyGgSKYmz5a5MAIaPTUUBsabHZwFmfRjLl2PG8hA=; b=utHmcz/yoabW/jsuCC/9Ih04kuof7iE0JgWI65qCnqLIGOIzlaOoBgParDZd9ThGTPfx7L Vj8wSMXEqsx7A7DxjhNyBdLDON6LyltzFvcb9k8+naYCRr3HRH8L8AK+405XxzwKrkE9O7 KzWEoFdHbc7DuZ08TDXMwQ8zIF/Udf4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749537; a=rsa-sha256; cv=none; b=YuXReRPpI2usybldqkuN3JSiqnho+Ed0e/Pqm9GsiGCw9xWVdKWW7ItsZGLEqDHe/zpf3I pGmmK/c8X74mcZ9qBzi5jQzr8/0kK6o47ii2lwwjrB63lFXSAPXYTnyCRbOIfzGiVBjLDb vsVGH2UZ0vGTiM7GX9UYpD1q8LoNGWY= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=YdgrK0Pv; spf=pass (imf05.hostedemail.com: domain of 3ILXoZQYKCGMTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3ILXoZQYKCGMTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-607e8e8c2f1so115507b3.3 for ; Wed, 06 Mar 2024 10:25:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749537; x=1710354337; darn=kvack.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=K6AcyGgSKYmz5a5MAIaPTUUBsabHZwFmfRjLl2PG8hA=; b=YdgrK0PvSreSID6ZJrCDSCbcyGXbetgrq7u3dN+2tUb8mC+zBsPjiY37cijFdy7/SJ WMYVCp/AkczSvHG7AJlwpCGsNSzA98M3E3j17HJMGWafb4hUp82IfEsVNs61QkgChedN o+7pZCt6dMWVZxptVQJ3p65KiNyB5Jhdc6AHTpfHGtvaR2ViV45wLWfGnBG5nX5ek8/E ABj3rRjO2H4cgJ0oS8EdL6rimAITAg+Yx/LFhqOQ7wU4Gfn9Ct/MphALxgq2x0F86SK8 EPAx9aSyKsp0niH0hu+9GEDrI38cwtarDFanB4p3AY2LH8EDNmORjHizpMAHoUn6IejP iK6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749537; x=1710354337; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=K6AcyGgSKYmz5a5MAIaPTUUBsabHZwFmfRjLl2PG8hA=; b=gBmf4dNltammWRululDaMn6yU4s793qt5uKnsW9n7c1hWCEp1WU2H0wTF41WClUEgP h/CgthWrjSTc+MLN/o/4VGcU6iOFlNBnZn5zPtJ5nONOAGdKYskhULFjjU6BdXa9T0c0 J9ya1ACdEPUGr8FK9J6NPP6aVUvPsiZ5oRgCBEn6O99EyXmFXEWwj4msUuh6FEc5kVra FjZI+WbBhBAHtjd7hGwUYEAwUtpRL8xh9iCmhVnLvac8iMV5r2EPBisCcCphijdYVHIv DRo4/LVA5oWJNgaZC1jOnBmG1GxEhx5SZA2aFPjhxfTizT1+ttap9B4tyLGkSNX1ILaa JYtA== X-Forwarded-Encrypted: i=1; AJvYcCWoO7/ku0ls3ve3+w4fvduENWSb5vMZovtdFEo5Bh2CmBF8JSThXNkwvxgPJC4qlbWf+FnpKMIiSFaIqn9nsQ24k0w= X-Gm-Message-State: AOJu0YxzbIrUNOlUKn3XEaqfUy+7mVQcNOa4ck2UkqRz2kNQVZjOSB7Q Qk4Uwte5lfCT4nwltQiuMfp+g11fDjlc2ED+f5Y/0zdlLe5ujHT1ZfryHtkCvCgDbQaH4dx7/RV JvA== X-Google-Smtp-Source: AGHT+IHr70OmG1R1IO1Fluk0v1eGaHNwt9Z96sNJit+XMVYfO4W641pUVkuKWr/RRoRl/U2gSXjA5z2Bwhw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:18d3:b0:dc7:865b:22c6 with SMTP id ck19-20020a05690218d300b00dc7865b22c6mr633022ybb.8.1709749536658; Wed, 06 Mar 2024 10:25:36 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:22 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-25-surenb@google.com> Subject: [PATCH v5 24/37] rust: Add a rust helper for krealloc() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org, Miguel Ojeda , Alex Gaynor , Wedson Almeida Filho , Boqun Feng , Gary Guo , " =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= " , Benno Lossin , Andreas Hindborg , rust-for-linux@vger.kernel.org X-Stat-Signature: 5odamp7rteakgw7iwx3rnea63ad7r7zg X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: C34BD10000B X-Rspam-User: X-HE-Tag: 1709749537-764873 X-HE-Meta: U2FsdGVkX1/mvd4B3oXf0AAJyarVOM+PNN46jofkR3WaOW/ykWuQzvDko95jqJDZMBlqUvnjoHEd/heG3aDfDheYGvT5qUNs+H8WbZddMqm6ChTO2dYkEp2u+2gISIR2iWCXkzxfr9rc8pQq9Gp9zFZBHB5bhhjI78z5fNK4NGXIx1qb+eCXhJW0qVng6+iKhl+CMsVWLYdhQSQio2x3XHA1N/hSEoW4X6rrR1Hmt3nTuyPemRPyAkK35UXbEXy0AapubnQfKg/UgOVx9D8JQGoX175UKDsTg+3Vp6tgDlHwP41/IuHjaTQmaHTZGhCL94pOQ/d4IUiHt/d+dW4Iadf8H1bYz6drlQNGKiqSATxknBwxStIOi0TsYr7re56ijG7IG7K82FBZ8hQKbqcjpvydhFUdLv9buB4HEli6cFl8WfKuy3FJ0DanPpT5HQamcNcqkjCJ/JMEwdn4jSfQdGYDBJ8yyPtT/lZiYONXYfdHaJG+BTVGYy2B34afLMPu8kdFHs/4twx+V+Y4pyEoN4iD4gIAt/B+6kn71x8hAVGsma0kJt7TltfzdIP6M9XnIt9GGCJSifRAUV6YDMiIQPHRigvF+H6olJYyRa1DLW/5KHu7YKVG8eyAscRJEgb58WewW7Wfikh29mD+RyBSRsUJQrN61Nr7QbdwXZKpE8sRKXIzTPV04hhEo/Ij7DgvyXP7LKzpV9st7PhLPtjNUkroppIjVUNrb1/2JpV34CekOkN7JE0I3Jl73zvqOUozBhxGe9JfeR2KxXsXYYfqV7Flsn7q+WLK3t17lQBluVFbonnT1S/SGcEPS//WNtfFO3ZhiEZ85KRQlufrlxzo9+RYF/w7QXL4Sn6NZ4EUvfhMFzJAFHgFo53xJsTKDVKkzOg3j2DtdaVuHjSI4OxEHKReo0p553AM2dDATwOMr8md7/9wRLyaWY1rmkolavbVW0+1FyzTCB9JL1KhwaP KteCIbr+ SEvoL57yCbuweqLEgp99Oy8mLYCdHNaEM9OOaExvYDXhQqOneNzMbZLjgspd8qUpYV0kAA2bRccY1eiER3BMcMoCTMtENS8zuGqQLEIR2sn0PnC2JwlEdSKmleYXm12ZloJA8oU0fnIaJXnyjL69edu84yUgghkaBIkNtczoalv9+BldCLoFvBnjUhscux6J1GXsJdDA2RWI6BPocqrnlx/hZG1ewu9Px5n+l9lOMLppBkAAzGIs2tyf3No6XVgG3UTWaJxH9Fj6Xd87iyvAKf7lrYEA8K8Usx0kbO2GdTE/ZqOxzL+4KlE9D38eZCxYLVgJ81IbGIBaOSb1fVdFMPipVF4d7ln/EeXnxV71LfjkXesehcvV+enDDg+7p1HIVKEbEtAUa8wdx7+6t7daiurGmhUIye/wJZCjD X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet Memory allocation profiling is turning krealloc() into a nontrivial macro - so for now, we need a helper for it. Until we have proper support on the rust side for memory allocation profiling this does mean that all Rust allocations will be accounted to the helper. Signed-off-by: Kent Overstreet Cc: Miguel Ojeda Cc: Alex Gaynor Cc: Wedson Almeida Filho Cc: Boqun Feng Cc: Gary Guo Cc: "Björn Roy Baron" Cc: Benno Lossin Cc: Andreas Hindborg Cc: Alice Ryhl Cc: rust-for-linux@vger.kernel.org Signed-off-by: Suren Baghdasaryan Reviewed-by: Alice Ryhl Acked-by: Miguel Ojeda --- rust/helpers.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/rust/helpers.c b/rust/helpers.c index 70e59efd92bc..ad62eaf604b3 100644 --- a/rust/helpers.c +++ b/rust/helpers.c @@ -28,6 +28,7 @@ #include #include #include +#include #include #include #include @@ -157,6 +158,13 @@ void rust_helper_init_work_with_key(struct work_struct *work, work_func_t func, } EXPORT_SYMBOL_GPL(rust_helper_init_work_with_key); +void * __must_check rust_helper_krealloc(const void *objp, size_t new_size, + gfp_t flags) __realloc_size(2) +{ + return krealloc(objp, new_size, flags); +} +EXPORT_SYMBOL_GPL(rust_helper_krealloc); + /* * `bindgen` binds the C `size_t` type as the Rust `usize` type, so we can * use it in contexts where Rust expects a `usize` like slice (array) indices. From patchwork Wed Mar 6 18:24:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584435 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B551BC5475B for ; Wed, 6 Mar 2024 18:26:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 274C56B00B4; Wed, 6 Mar 2024 13:25:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1FB566B00B5; Wed, 6 Mar 2024 13:25:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ECFDB6B00B6; Wed, 6 Mar 2024 13:25:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C98D66B00B4 for ; Wed, 6 Mar 2024 13:25:41 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A3F701C07CB for ; Wed, 6 Mar 2024 18:25:41 +0000 (UTC) X-FDA: 81867442482.20.AE93948 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf26.hostedemail.com (Postfix) with ESMTP id E7B3C14001E for ; Wed, 6 Mar 2024 18:25:39 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zAUiL+sQ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3IrXoZQYKCGUVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3IrXoZQYKCGUVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749540; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n5/OuSdWvIQFOpMCE75pS/uA6aDuAy2uY0scKcuOlxY=; b=F6c4SEdFn206UsZsChmR+/hZ3b3cmp+AfcDgfBwYOKxrv9fGG3Y6WjbHh7TjwcnmPMJc3G lqLtqlGOT8g9dpJst5fQGr0HEKJ7ua1GmGIYlLboifOqAcPnMEaTiyc/fITa7T47r6UOmc 4wsuyKtNsvJMOEDHvDwCVuT77mQd7oQ= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zAUiL+sQ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3IrXoZQYKCGUVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3IrXoZQYKCGUVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749540; a=rsa-sha256; cv=none; b=Yas5nss7NWZ3+YCqKCDr7nAzn5rADeN7Yao0oMMNlXFdkhCFa4yxFri4nK6eF/hN1McrzL MVKqP+yKKTZU4ORsYKgvDLSag5i0RsXecH0/zPVGVIjTAYndNytnhRYCW7mcFC2LKgRDAD mPNXaxY7Rf/G3m0ngz1z6C3qPhm+tfM= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dce775fa8adso12666965276.1 for ; Wed, 06 Mar 2024 10:25:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749539; x=1710354339; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=n5/OuSdWvIQFOpMCE75pS/uA6aDuAy2uY0scKcuOlxY=; b=zAUiL+sQ+Ju9QLoL6sl5dLNIiFwWhHm/2S5AF8YUPSMFRwJ1ckJdhywQ6CNVm5qlOS TvWCubp5tKMSjppSZhT4LCBwBJa93xTyh5HVYpskaoDwcF6+T0z032jWT/6N8CUuIpuc tYhpAxjnv1sjlewvBpMCgQzElVCAD4v1kYrSt/JIlHtiB+V98AiLmgqVJNuKjI7tJQ7X BsMj0U0iRjBl2STKqBY/ORmy58EI7BjsILvE8NJm1ZUst6gpl2oGPpS5kZ4QGzkAXw5w dLr3RYqG+zbidQBnBTZhKYQuj6Tq+pwhm/wb5WYA9grVYpcQb/5LEMIV2KJk+zf1PxtP wgNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749539; x=1710354339; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=n5/OuSdWvIQFOpMCE75pS/uA6aDuAy2uY0scKcuOlxY=; b=jXaxT0qHt3iwvp6h9wd/e2h0c1m1qONCp0MWO3Lu36K65a9encw5rwye9V6pK17PPk XS0JrytkH+Hk+d7DWtRR9PrqBsWjMXUQmalHdJ5NM0UKj97tNP+sDCRWNO4p4bba3Z6Z CVP6nUnUsQXNpBfH9TiYF2G/BUXrtpkLYDyboVNd7lHXMewAeBrCtS/HeLFfola+KL8G aBwFjyGrjNeYYE7sr2sALM9UsTHm4DUtrXl2hLyVSarQQ5jkpxUr9lu5iQMww0GUfPNm mj/WhO/+V3HWSGfuwS8CWdA51yKysaLNn7GvoWLynQTZMONkboKl1cLqnFnm9aXHGieT nrTw== X-Forwarded-Encrypted: i=1; AJvYcCVaf6WNM7VsW1XzeWp5Bb5zzRe2/Z81v9fZX5e6tCANmoPmf+hMEL4JL1colqA1WnKkeJgqmq7bBDpDWtz4/H1Zrhg= X-Gm-Message-State: AOJu0YzttM7Mi7gtxQ05ESr1Opv1XT87GDhthS9me7X1DhaETElvcDzO EhgnNgRP9DAvWRIK0rJuwaXjC3EuRP8upJ0qUnsWp6x3ppzylsreL6HIEacNJMkyeqsTaX7yNDO XdA== X-Google-Smtp-Source: AGHT+IGxg595J5w9MMZ3uBSKiZyqLHmDCVJREKgtrffJ2jIUhXGrpsFAD4x/4WZKbERL2ICXQ9+fFmzLpKc= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:1004:b0:dc2:3441:897f with SMTP id w4-20020a056902100400b00dc23441897fmr3843294ybt.6.1709749538789; Wed, 06 Mar 2024 10:25:38 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:23 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-26-surenb@google.com> Subject: [PATCH v5 25/37] mm/slab: enable slab allocation tagging for kmalloc and friends From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Stat-Signature: 6y5ysb5ce7bxubpnn3mjjuezzuprmmy4 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: E7B3C14001E X-HE-Tag: 1709749539-50738 X-HE-Meta: U2FsdGVkX1+WFMPMFa5kGXsldPq5XXNgsUYLNZug1WIyKP8RdfLmB5nadR3vp+p+SnQUJ1hD3LM67a5XfABRaaFW/4Op0QNrImTW9Y8THn6CR1w+a39a+R4j4lsrS5E1Ain7e1VmoOiKGyN+ZANuvkk+Fs+bXlcuJV4/ACrqZvE35knApT7CBdlDkuehIIMjya3/Npt50lOMoxQ3yjGtT6kqPp9MQ/n29x1rUBBukACkqnklmIaKndG2/cBnLpr/FWWa8jXdGN7Xb8gDtzhOybqXYWX1kWoPa4tyADxu10wcu4ZrN+xp9Kn36mRgiNQb2PyG6KMgEqSsa5DFJIJm1q54VPRxGSyN9T+Ct8ue+kPRUCIfymO59h10PgMJUCKGEc31my92ScaNntdng6RCr8Shd3Bzl6U6WW06RK11whmz3c4aNq22THXExAsUwUTlUT7La+iBO32dwyKkE80RXj+M6PKOXvcTbkszeeTxnwfHn3LMVKe9mp88bohKJfXq7vmc64buxKnwqL9NvUqFKJWgfIFfkl5NZCHu8b/gXEfj26HmFBLwMy1dDuCfYjxH7BGYLzRiY9GRdVAa4A1x/aI5GC4vvsW6bwoFyRmxwF6LuXw19VQo+RNKaiKEPX8U7oGKnv1ptoYuSqYlHaR4kUFexAFwG5p08oamiSH/5hBluYe5FQw7hc2ILtCj2Hjo7k1+bMkyAj0fWE3cBKzKnY4Sre7b/XodCJxwVIvEFamcDcglXwqOC63uoY5VoEEl7gza1XeJgDvlQTpZ7jeY192tdHv9DI8KJay6nH001blEGtABtGFLTVhEoHQSQb/7GovmJUTHOVF5WNUnDw1vnOa19VEUOBfvBCxpjxmbF7tO74ZvbkDZaLIUsk9+ITVkm6pF+X6q0CyuDiMZelHCoAdMokpJqe3yntVmFMz4GsLyUf3N/f7PVVaaBOZUWXxRQmIt1iqTAIa9gLkhs+6 6jE0vt4Q YUxIbPUFhIhzXVd5FenOIEZYAc8DQK/20vrYrNXHleZTS1UJ5usC7bFNE3d8RVTKC44nH5cIyhHQplsa3JfrsWp23I1PLywcIddzveErkK0iv7qOnlLcFU1ALvC7SXaBNBhl75lvL5JdOOjHwVtnUXmSL5Dp0CtJGvE6JN32t+ry7SdIaiKNKjfRCUX6A9icSplYcIo+GfYcY7xYI2r0ym151MdOIeER3ufICZ16BE+e4gNfiivoE7APLgn5KaDPrJubVFk3asUxZquRDQXeYXS+wmwK1kC0o/ie0p+AbYOtowLiIWkoU98iSRNYZMdq6HSp9G1Rt2ELr7HFVefmkUWUYCFLRE4anGb4gRU8YlJLonbz+kqBHHKDF1yWdRpe2NMSrPX46LF0UobBt58U4YICNrbfoDc6kMAw73g2Svik07B4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Redefine kmalloc, krealloc, kzalloc, kcalloc, etc. to record allocations and deallocations done by these functions. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Reviewed-by: Kees Cook --- include/linux/fortify-string.h | 5 +- include/linux/slab.h | 169 +++++++++++++++++---------------- include/linux/string.h | 4 +- mm/slab_common.c | 6 +- mm/slub.c | 52 +++++----- mm/util.c | 20 ++-- 6 files changed, 130 insertions(+), 126 deletions(-) diff --git a/include/linux/fortify-string.h b/include/linux/fortify-string.h index 89a6888f2f9e..55f66bd8a366 100644 --- a/include/linux/fortify-string.h +++ b/include/linux/fortify-string.h @@ -697,9 +697,9 @@ __FORTIFY_INLINE void *memchr_inv(const void * const POS0 p, int c, size_t size) return __real_memchr_inv(p, c, size); } -extern void *__real_kmemdup(const void *src, size_t len, gfp_t gfp) __RENAME(kmemdup) +extern void *__real_kmemdup(const void *src, size_t len, gfp_t gfp) __RENAME(kmemdup_noprof) __realloc_size(2); -__FORTIFY_INLINE void *kmemdup(const void * const POS0 p, size_t size, gfp_t gfp) +__FORTIFY_INLINE void *kmemdup_noprof(const void * const POS0 p, size_t size, gfp_t gfp) { const size_t p_size = __struct_size(p); @@ -709,6 +709,7 @@ __FORTIFY_INLINE void *kmemdup(const void * const POS0 p, size_t size, gfp_t gfp fortify_panic(__func__); return __real_kmemdup(p, size, gfp); } +#define kmemdup(...) alloc_hooks(kmemdup_noprof(__VA_ARGS__)) /** * strcpy - Copy a string into another string buffer diff --git a/include/linux/slab.h b/include/linux/slab.h index 58794043ab5b..61e2a486d529 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -229,7 +229,10 @@ int kmem_cache_shrink(struct kmem_cache *s); /* * Common kmalloc functions provided by all allocators */ -void * __must_check krealloc(const void *objp, size_t new_size, gfp_t flags) __realloc_size(2); +void * __must_check krealloc_noprof(const void *objp, size_t new_size, + gfp_t flags) __realloc_size(2); +#define krealloc(...) alloc_hooks(krealloc_noprof(__VA_ARGS__)) + void kfree(const void *objp); void kfree_sensitive(const void *objp); size_t __ksize(const void *objp); @@ -481,7 +484,10 @@ static __always_inline unsigned int __kmalloc_index(size_t size, static_assert(PAGE_SHIFT <= 20); #define kmalloc_index(s) __kmalloc_index(s, true) -void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_size(1); +#include + +void *__kmalloc_noprof(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_size(1); +#define __kmalloc(...) alloc_hooks(__kmalloc_noprof(__VA_ARGS__)) /** * kmem_cache_alloc - Allocate an object @@ -493,9 +499,14 @@ void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_siz * * Return: pointer to the new object or %NULL in case of error */ -void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags) __assume_slab_alignment __malloc; -void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, - gfp_t gfpflags) __assume_slab_alignment __malloc; +void *kmem_cache_alloc_noprof(struct kmem_cache *cachep, + gfp_t flags) __assume_slab_alignment __malloc; +#define kmem_cache_alloc(...) alloc_hooks(kmem_cache_alloc_noprof(__VA_ARGS__)) + +void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru, + gfp_t gfpflags) __assume_slab_alignment __malloc; +#define kmem_cache_alloc_lru(...) alloc_hooks(kmem_cache_alloc_lru_noprof(__VA_ARGS__)) + void kmem_cache_free(struct kmem_cache *s, void *objp); /* @@ -506,29 +517,40 @@ void kmem_cache_free(struct kmem_cache *s, void *objp); * Note that interrupts must be enabled when calling these functions. */ void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p); -int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p); + +int kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags, size_t size, void **p); +#define kmem_cache_alloc_bulk(...) alloc_hooks(kmem_cache_alloc_bulk_noprof(__VA_ARGS__)) static __always_inline void kfree_bulk(size_t size, void **p) { kmem_cache_free_bulk(NULL, size, p); } -void *__kmalloc_node(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment +void *__kmalloc_node_noprof(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment __alloc_size(1); -void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t flags, int node) __assume_slab_alignment - __malloc; +#define __kmalloc_node(...) alloc_hooks(__kmalloc_node_noprof(__VA_ARGS__)) + +void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t flags, + int node) __assume_slab_alignment __malloc; +#define kmem_cache_alloc_node(...) alloc_hooks(kmem_cache_alloc_node_noprof(__VA_ARGS__)) -void *kmalloc_trace(struct kmem_cache *s, gfp_t flags, size_t size) +void *kmalloc_trace_noprof(struct kmem_cache *s, gfp_t flags, size_t size) __assume_kmalloc_alignment __alloc_size(3); -void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, - int node, size_t size) __assume_kmalloc_alignment +void *kmalloc_node_trace_noprof(struct kmem_cache *s, gfp_t gfpflags, + int node, size_t size) __assume_kmalloc_alignment __alloc_size(4); -void *kmalloc_large(size_t size, gfp_t flags) __assume_page_alignment +#define kmalloc_trace(...) alloc_hooks(kmalloc_trace_noprof(__VA_ARGS__)) + +#define kmalloc_node_trace(...) alloc_hooks(kmalloc_node_trace_noprof(__VA_ARGS__)) + +void *kmalloc_large_noprof(size_t size, gfp_t flags) __assume_page_alignment __alloc_size(1); +#define kmalloc_large(...) alloc_hooks(kmalloc_large_noprof(__VA_ARGS__)) -void *kmalloc_large_node(size_t size, gfp_t flags, int node) __assume_page_alignment +void *kmalloc_large_node_noprof(size_t size, gfp_t flags, int node) __assume_page_alignment __alloc_size(1); +#define kmalloc_large_node(...) alloc_hooks(kmalloc_large_node_noprof(__VA_ARGS__)) /** * kmalloc - allocate kernel memory @@ -584,37 +606,39 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node) __assume_page_align * Try really hard to succeed the allocation but fail * eventually. */ -static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) +static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t flags) { if (__builtin_constant_p(size) && size) { unsigned int index; if (size > KMALLOC_MAX_CACHE_SIZE) - return kmalloc_large(size, flags); + return kmalloc_large_noprof(size, flags); index = kmalloc_index(size); - return kmalloc_trace( + return kmalloc_trace_noprof( kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], flags, size); } - return __kmalloc(size, flags); + return __kmalloc_noprof(size, flags); } +#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__)) -static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node) +static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gfp_t flags, int node) { if (__builtin_constant_p(size) && size) { unsigned int index; if (size > KMALLOC_MAX_CACHE_SIZE) - return kmalloc_large_node(size, flags, node); + return kmalloc_large_node_noprof(size, flags, node); index = kmalloc_index(size); - return kmalloc_node_trace( + return kmalloc_node_trace_noprof( kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], flags, node, size); } - return __kmalloc_node(size, flags, node); + return __kmalloc_node_noprof(size, flags, node); } +#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__)) /** * kmalloc_array - allocate memory for an array. @@ -622,16 +646,17 @@ static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t fla * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -static inline __alloc_size(1, 2) void *kmalloc_array(size_t n, size_t size, gfp_t flags) +static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t size, gfp_t flags) { size_t bytes; if (unlikely(check_mul_overflow(n, size, &bytes))) return NULL; if (__builtin_constant_p(n) && __builtin_constant_p(size)) - return kmalloc(bytes, flags); - return __kmalloc(bytes, flags); + return kmalloc_noprof(bytes, flags); + return kmalloc_noprof(bytes, flags); } +#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__)) /** * krealloc_array - reallocate memory for an array. @@ -640,18 +665,19 @@ static inline __alloc_size(1, 2) void *kmalloc_array(size_t n, size_t size, gfp_ * @new_size: new size of a single member of the array * @flags: the type of memory to allocate (see kmalloc) */ -static inline __realloc_size(2, 3) void * __must_check krealloc_array(void *p, - size_t new_n, - size_t new_size, - gfp_t flags) +static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(void *p, + size_t new_n, + size_t new_size, + gfp_t flags) { size_t bytes; if (unlikely(check_mul_overflow(new_n, new_size, &bytes))) return NULL; - return krealloc(p, bytes, flags); + return krealloc_noprof(p, bytes, flags); } +#define krealloc_array(...) alloc_hooks(krealloc_array_noprof(__VA_ARGS__)) /** * kcalloc - allocate memory for an array. The memory is set to zero. @@ -659,16 +685,12 @@ static inline __realloc_size(2, 3) void * __must_check krealloc_array(void *p, * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -static inline __alloc_size(1, 2) void *kcalloc(size_t n, size_t size, gfp_t flags) -{ - return kmalloc_array(n, size, flags | __GFP_ZERO); -} +#define kcalloc(_n, _size, _flags) kmalloc_array(_n, _size, (_flags) | __GFP_ZERO) -void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, +void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, int node, unsigned long caller) __alloc_size(1); -#define kmalloc_node_track_caller(size, flags, node) \ - __kmalloc_node_track_caller(size, flags, node, \ - _RET_IP_) +#define kmalloc_node_track_caller(...) \ + alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_)) /* * kmalloc_track_caller is a special version of kmalloc that records the @@ -678,11 +700,9 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, * allocator where we care about the real place the memory allocation * request comes from. */ -#define kmalloc_track_caller(size, flags) \ - __kmalloc_node_track_caller(size, flags, \ - NUMA_NO_NODE, _RET_IP_) +#define kmalloc_track_caller(...) kmalloc_node_track_caller(__VA_ARGS__, NUMA_NO_NODE) -static inline __alloc_size(1, 2) void *kmalloc_array_node(size_t n, size_t size, gfp_t flags, +static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node) { size_t bytes; @@ -690,75 +710,56 @@ static inline __alloc_size(1, 2) void *kmalloc_array_node(size_t n, size_t size, if (unlikely(check_mul_overflow(n, size, &bytes))) return NULL; if (__builtin_constant_p(n) && __builtin_constant_p(size)) - return kmalloc_node(bytes, flags, node); - return __kmalloc_node(bytes, flags, node); + return kmalloc_node_noprof(bytes, flags, node); + return __kmalloc_node_noprof(bytes, flags, node); } +#define kmalloc_array_node(...) alloc_hooks(kmalloc_array_node_noprof(__VA_ARGS__)) -static inline __alloc_size(1, 2) void *kcalloc_node(size_t n, size_t size, gfp_t flags, int node) -{ - return kmalloc_array_node(n, size, flags | __GFP_ZERO, node); -} +#define kcalloc_node(_n, _size, _flags, _node) \ + kmalloc_array_node(_n, _size, (_flags) | __GFP_ZERO, _node) /* * Shortcuts */ -static inline void *kmem_cache_zalloc(struct kmem_cache *k, gfp_t flags) -{ - return kmem_cache_alloc(k, flags | __GFP_ZERO); -} +#define kmem_cache_zalloc(_k, _flags) kmem_cache_alloc(_k, (_flags)|__GFP_ZERO) /** * kzalloc - allocate memory. The memory is set to zero. * @size: how many bytes of memory are required. * @flags: the type of memory to allocate (see kmalloc). */ -static inline __alloc_size(1) void *kzalloc(size_t size, gfp_t flags) +static inline __alloc_size(1) void *kzalloc_noprof(size_t size, gfp_t flags) { - return kmalloc(size, flags | __GFP_ZERO); + return kmalloc_noprof(size, flags | __GFP_ZERO); } +#define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__)) +#define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node) -/** - * kzalloc_node - allocate zeroed memory from a particular memory node. - * @size: how many bytes of memory are required. - * @flags: the type of memory to allocate (see kmalloc). - * @node: memory node from which to allocate - */ -static inline __alloc_size(1) void *kzalloc_node(size_t size, gfp_t flags, int node) -{ - return kmalloc_node(size, flags | __GFP_ZERO, node); -} +extern void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) __alloc_size(1); +#define kvmalloc_node(...) alloc_hooks(kvmalloc_node_noprof(__VA_ARGS__)) -extern void *kvmalloc_node(size_t size, gfp_t flags, int node) __alloc_size(1); -static inline __alloc_size(1) void *kvmalloc(size_t size, gfp_t flags) -{ - return kvmalloc_node(size, flags, NUMA_NO_NODE); -} -static inline __alloc_size(1) void *kvzalloc_node(size_t size, gfp_t flags, int node) -{ - return kvmalloc_node(size, flags | __GFP_ZERO, node); -} -static inline __alloc_size(1) void *kvzalloc(size_t size, gfp_t flags) -{ - return kvmalloc(size, flags | __GFP_ZERO); -} +#define kvmalloc(_size, _flags) kvmalloc_node(_size, _flags, NUMA_NO_NODE) +#define kvzalloc(_size, _flags) kvmalloc(_size, _flags|__GFP_ZERO) + +#define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, _flags|__GFP_ZERO, _node) -static inline __alloc_size(1, 2) void *kvmalloc_array(size_t n, size_t size, gfp_t flags) +static inline __alloc_size(1, 2) void *kvmalloc_array_noprof(size_t n, size_t size, gfp_t flags) { size_t bytes; if (unlikely(check_mul_overflow(n, size, &bytes))) return NULL; - return kvmalloc(bytes, flags); + return kvmalloc_node_noprof(bytes, flags, NUMA_NO_NODE); } -static inline __alloc_size(1, 2) void *kvcalloc(size_t n, size_t size, gfp_t flags) -{ - return kvmalloc_array(n, size, flags | __GFP_ZERO); -} +#define kvmalloc_array(...) alloc_hooks(kvmalloc_array_noprof(__VA_ARGS__)) +#define kvcalloc(_n, _size, _flags) kvmalloc_array(_n, _size, _flags|__GFP_ZERO) -extern void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) +extern void *kvrealloc_noprof(const void *p, size_t oldsize, size_t newsize, gfp_t flags) __realloc_size(3); +#define kvrealloc(...) alloc_hooks(kvrealloc_noprof(__VA_ARGS__)) + extern void kvfree(const void *addr); DEFINE_FREE(kvfree, void *, if (_T) kvfree(_T)) diff --git a/include/linux/string.h b/include/linux/string.h index ab148d8dbfc1..14e4fb4340f4 100644 --- a/include/linux/string.h +++ b/include/linux/string.h @@ -214,7 +214,9 @@ extern void kfree_const(const void *x); extern char *kstrdup(const char *s, gfp_t gfp) __malloc; extern const char *kstrdup_const(const char *s, gfp_t gfp); extern char *kstrndup(const char *s, size_t len, gfp_t gfp); -extern void *kmemdup(const void *src, size_t len, gfp_t gfp) __realloc_size(2); +extern void *kmemdup_noprof(const void *src, size_t len, gfp_t gfp) __realloc_size(2); +#define kmemdup(...) alloc_hooks(kmemdup_noprof(__VA_ARGS__)) + extern void *kvmemdup(const void *src, size_t len, gfp_t gfp) __realloc_size(2); extern char *kmemdup_nul(const char *s, size_t len, gfp_t gfp); diff --git a/mm/slab_common.c b/mm/slab_common.c index 238293b1dbe1..5f9e25626dc7 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1184,7 +1184,7 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags) return (void *)p; } - ret = kmalloc_track_caller(new_size, flags); + ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_); if (ret && p) { /* Disable KASAN checks as the object's redzone is accessed. */ kasan_disable_current(); @@ -1208,7 +1208,7 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags) * * Return: pointer to the allocated memory or %NULL in case of error */ -void *krealloc(const void *p, size_t new_size, gfp_t flags) +void *krealloc_noprof(const void *p, size_t new_size, gfp_t flags) { void *ret; @@ -1223,7 +1223,7 @@ void *krealloc(const void *p, size_t new_size, gfp_t flags) return ret; } -EXPORT_SYMBOL(krealloc); +EXPORT_SYMBOL(krealloc_noprof); /** * kfree_sensitive - Clear sensitive information in memory before freeing diff --git a/mm/slub.c b/mm/slub.c index ea122aeb89fc..5e6d68d05740 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4007,7 +4007,7 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list return object; } -void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags) +void *kmem_cache_alloc_noprof(struct kmem_cache *s, gfp_t gfpflags) { void *ret = slab_alloc_node(s, NULL, gfpflags, NUMA_NO_NODE, _RET_IP_, s->object_size); @@ -4016,9 +4016,9 @@ void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags) return ret; } -EXPORT_SYMBOL(kmem_cache_alloc); +EXPORT_SYMBOL(kmem_cache_alloc_noprof); -void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, +void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags) { void *ret = slab_alloc_node(s, lru, gfpflags, NUMA_NO_NODE, _RET_IP_, @@ -4028,10 +4028,10 @@ void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, return ret; } -EXPORT_SYMBOL(kmem_cache_alloc_lru); +EXPORT_SYMBOL(kmem_cache_alloc_lru_noprof); /** - * kmem_cache_alloc_node - Allocate an object on the specified node + * kmem_cache_alloc_node_noprof - Allocate an object on the specified node * @s: The cache to allocate from. * @gfpflags: See kmalloc(). * @node: node number of the target node. @@ -4043,7 +4043,7 @@ EXPORT_SYMBOL(kmem_cache_alloc_lru); * * Return: pointer to the new object or %NULL in case of error */ -void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node) +void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t gfpflags, int node) { void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, s->object_size); @@ -4051,7 +4051,7 @@ void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node) return ret; } -EXPORT_SYMBOL(kmem_cache_alloc_node); +EXPORT_SYMBOL(kmem_cache_alloc_node_noprof); /* * To avoid unnecessary overhead, we pass through large allocation requests @@ -4068,7 +4068,7 @@ static void *__kmalloc_large_node(size_t size, gfp_t flags, int node) flags = kmalloc_fix_flags(flags); flags |= __GFP_COMP; - folio = (struct folio *)alloc_pages_node(node, flags, order); + folio = (struct folio *)alloc_pages_node_noprof(node, flags, order); if (folio) { ptr = folio_address(folio); lruvec_stat_mod_folio(folio, NR_SLAB_UNRECLAIMABLE_B, @@ -4083,7 +4083,7 @@ static void *__kmalloc_large_node(size_t size, gfp_t flags, int node) return ptr; } -void *kmalloc_large(size_t size, gfp_t flags) +void *kmalloc_large_noprof(size_t size, gfp_t flags) { void *ret = __kmalloc_large_node(size, flags, NUMA_NO_NODE); @@ -4091,9 +4091,9 @@ void *kmalloc_large(size_t size, gfp_t flags) flags, NUMA_NO_NODE); return ret; } -EXPORT_SYMBOL(kmalloc_large); +EXPORT_SYMBOL(kmalloc_large_noprof); -void *kmalloc_large_node(size_t size, gfp_t flags, int node) +void *kmalloc_large_node_noprof(size_t size, gfp_t flags, int node) { void *ret = __kmalloc_large_node(size, flags, node); @@ -4101,7 +4101,7 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node) flags, node); return ret; } -EXPORT_SYMBOL(kmalloc_large_node); +EXPORT_SYMBOL(kmalloc_large_node_noprof); static __always_inline void *__do_kmalloc_node(size_t size, gfp_t flags, int node, @@ -4128,26 +4128,26 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, return ret; } -void *__kmalloc_node(size_t size, gfp_t flags, int node) +void *__kmalloc_node_noprof(size_t size, gfp_t flags, int node) { return __do_kmalloc_node(size, flags, node, _RET_IP_); } -EXPORT_SYMBOL(__kmalloc_node); +EXPORT_SYMBOL(__kmalloc_node_noprof); -void *__kmalloc(size_t size, gfp_t flags) +void *__kmalloc_noprof(size_t size, gfp_t flags) { return __do_kmalloc_node(size, flags, NUMA_NO_NODE, _RET_IP_); } -EXPORT_SYMBOL(__kmalloc); +EXPORT_SYMBOL(__kmalloc_noprof); -void *__kmalloc_node_track_caller(size_t size, gfp_t flags, - int node, unsigned long caller) +void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, + int node, unsigned long caller) { return __do_kmalloc_node(size, flags, node, caller); } -EXPORT_SYMBOL(__kmalloc_node_track_caller); +EXPORT_SYMBOL(kmalloc_node_track_caller_noprof); -void *kmalloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) +void *kmalloc_trace_noprof(struct kmem_cache *s, gfp_t gfpflags, size_t size) { void *ret = slab_alloc_node(s, NULL, gfpflags, NUMA_NO_NODE, _RET_IP_, size); @@ -4157,9 +4157,9 @@ void *kmalloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) ret = kasan_kmalloc(s, ret, size, gfpflags); return ret; } -EXPORT_SYMBOL(kmalloc_trace); +EXPORT_SYMBOL(kmalloc_trace_noprof); -void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, +void *kmalloc_node_trace_noprof(struct kmem_cache *s, gfp_t gfpflags, int node, size_t size) { void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, size); @@ -4169,7 +4169,7 @@ void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, ret = kasan_kmalloc(s, ret, size, gfpflags); return ret; } -EXPORT_SYMBOL(kmalloc_node_trace); +EXPORT_SYMBOL(kmalloc_node_trace_noprof); static noinline void free_to_partial_list( struct kmem_cache *s, struct slab *slab, @@ -4778,8 +4778,8 @@ static int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, #endif /* CONFIG_SLUB_TINY */ /* Note that interrupts must be enabled when calling this function. */ -int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, - void **p) +int kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags, size_t size, + void **p) { int i; struct obj_cgroup *objcg = NULL; @@ -4807,7 +4807,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, return i; } -EXPORT_SYMBOL(kmem_cache_alloc_bulk); +EXPORT_SYMBOL(kmem_cache_alloc_bulk_noprof); /* diff --git a/mm/util.c b/mm/util.c index 6710efaf1802..9b8774d8dabb 100644 --- a/mm/util.c +++ b/mm/util.c @@ -115,7 +115,7 @@ char *kstrndup(const char *s, size_t max, gfp_t gfp) EXPORT_SYMBOL(kstrndup); /** - * kmemdup - duplicate region of memory + * kmemdup_noprof - duplicate region of memory * * @src: memory region to duplicate * @len: memory region length @@ -124,16 +124,16 @@ EXPORT_SYMBOL(kstrndup); * Return: newly allocated copy of @src or %NULL in case of error, * result is physically contiguous. Use kfree() to free. */ -void *kmemdup(const void *src, size_t len, gfp_t gfp) +void *kmemdup_noprof(const void *src, size_t len, gfp_t gfp) { void *p; - p = kmalloc_track_caller(len, gfp); + p = kmalloc_node_track_caller_noprof(len, gfp, NUMA_NO_NODE, _RET_IP_); if (p) memcpy(p, src, len); return p; } -EXPORT_SYMBOL(kmemdup); +EXPORT_SYMBOL(kmemdup_noprof); /** * kvmemdup - duplicate region of memory @@ -577,7 +577,7 @@ unsigned long vm_mmap(struct file *file, unsigned long addr, EXPORT_SYMBOL(vm_mmap); /** - * kvmalloc_node - attempt to allocate physically contiguous memory, but upon + * kvmalloc_node_noprof - attempt to allocate physically contiguous memory, but upon * failure, fall back to non-contiguous (vmalloc) allocation. * @size: size of the request. * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL. @@ -592,7 +592,7 @@ EXPORT_SYMBOL(vm_mmap); * * Return: pointer to the allocated memory of %NULL in case of failure */ -void *kvmalloc_node(size_t size, gfp_t flags, int node) +void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) { gfp_t kmalloc_flags = flags; void *ret; @@ -614,7 +614,7 @@ void *kvmalloc_node(size_t size, gfp_t flags, int node) kmalloc_flags &= ~__GFP_NOFAIL; } - ret = kmalloc_node(size, kmalloc_flags, node); + ret = kmalloc_node_noprof(size, kmalloc_flags, node); /* * It doesn't really make sense to fallback to vmalloc for sub page @@ -643,7 +643,7 @@ void *kvmalloc_node(size_t size, gfp_t flags, int node) flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(kvmalloc_node); +EXPORT_SYMBOL(kvmalloc_node_noprof); /** * kvfree() - Free memory. @@ -682,7 +682,7 @@ void kvfree_sensitive(const void *addr, size_t len) } EXPORT_SYMBOL(kvfree_sensitive); -void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) +void *kvrealloc_noprof(const void *p, size_t oldsize, size_t newsize, gfp_t flags) { void *newp; @@ -695,7 +695,7 @@ void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) kvfree(p); return newp; } -EXPORT_SYMBOL(kvrealloc); +EXPORT_SYMBOL(kvrealloc_noprof); /** * __vmalloc_array - allocate memory for a virtually contiguous array. From patchwork Wed Mar 6 18:24:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584436 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE4C5C54E41 for ; Wed, 6 Mar 2024 18:26:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 644196B00B6; Wed, 6 Mar 2024 13:25:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5CC946B00B7; Wed, 6 Mar 2024 13:25:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3ABE66B00B8; Wed, 6 Mar 2024 13:25:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 208BD6B00B6 for ; Wed, 6 Mar 2024 13:25:44 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id EBFCC1204E2 for ; Wed, 6 Mar 2024 18:25:43 +0000 (UTC) X-FDA: 81867442566.07.50D163C Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf27.hostedemail.com (Postfix) with ESMTP id 26D3E40006 for ; Wed, 6 Mar 2024 18:25:41 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=vCzQiWmU; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of 3JbXoZQYKCGgYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3JbXoZQYKCGgYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749542; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iFNwLDi8a6FCybl7tYSum2P7lYcwvuLcMH/fhzJCtO0=; b=j13qV/8OXVWexcr98G+afTL+Tus5s/vgUxpxBJRy4qmSF9nJXgCFIiN4STauGv7XFWWhIy KubW/MIBY5/68SM7nzOyEkU+dScgipeBJ7xUXLduM1jMwLIITHVS8J++dJhVv6n5lkmt5B JlEf9A83CKTB9sUEGuO/Csy1pfXpXzc= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=vCzQiWmU; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of 3JbXoZQYKCGgYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3JbXoZQYKCGgYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749542; a=rsa-sha256; cv=none; b=oLp2SVnwpVy+vD5DC+zdXSPSFu/INfNE6eHQH9EFHYa/6HJlxWiPZBspb+TYH1h9vbeIDP iOJx18+eQwUMpbv3dY/Cood6/+XduLcQIalXUtQIcWZCovg5tCe3Zn4vKGVUYwcm2fANp4 Wr4XuvzSNcWqsbieEXT7mBMWpvnfZEo= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-609ea26233dso705687b3.0 for ; Wed, 06 Mar 2024 10:25:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749541; x=1710354341; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=iFNwLDi8a6FCybl7tYSum2P7lYcwvuLcMH/fhzJCtO0=; b=vCzQiWmUOtE+0wCJvtUyom7xzTVSzAWT4pZv+fY39ca3KdZDsd77FB440tgyCCdcF/ ICfX5iFJcYjSiJ3wKg/yFJ7ODk0naLJmPy0qvAqAp74aiYk2u3w0c+XniIj70uTjmUtN n7hTSNK/OU5UafqP5Kocne9V33YkJ8zjAZYo8vMuLcmJeP3XwAbY7F6ffmZPbiAxbDiw njG491GbjAZDPwMNkSPfcY2XQfhnprG/ed9eVwhxtF7D4/lVJBNXBXF23OR+TEpPwzwr iMyKsfX9N3HrV6VZ+v0FsbwerBrM0Rbc7021IS0uhqg3pMJnOFsnMqm1f0SfttHx1GeD RBgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749541; x=1710354341; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iFNwLDi8a6FCybl7tYSum2P7lYcwvuLcMH/fhzJCtO0=; b=VyMDnB26v+Od6UtfwUDEZsVHtjZvbfE6RshspcpgfLuCMfvmql06IBteRj3i/Ih0h0 aJ0iwzY14cZJPTHmDNeteodXFsWFXRWMh66NigzYox4PgdAuBxdp5+9v3ngk5huQ2a0o atV/Xfo93uqeJKtrIzcAAlGGMeyLmQB5DpTgkTT7+YC9oae2loXcd+kmWa7N0AzabmOG Vbxa7dQhEn/m4msWmNndkhfYweD0mNTu9Ga9zySmgXVsi/By41Z60OGjbrfwx17zxxJd ksB3YzLyhwHMfLntQXqCQ4f8II7NyxO6tcMn04RBdXC0nFmR9C3Tsoo4hfrLEbtaVWkp 56Pw== X-Forwarded-Encrypted: i=1; AJvYcCUxYWVIU17Hh4zHXR/RPZnBM7lhImAmBH+d1L3UsKa4xZ0PHnauf1eM1Xn7Pald6uuE2CNRHdAHuQgUpKUVASke8cY= X-Gm-Message-State: AOJu0Yw+JFPb9h81nATtEofKk9I825u2fdXXFN1WyMtD1M+0mkJ6Tfb9 tlXUz/qLa0VloF4QAhzEpwY0YMYpMFa1hiIgpbjucF1NOM39wahbvwk7VZ55Xq/Y1PHb2M/ofyv 0lQ== X-Google-Smtp-Source: AGHT+IGvnYLIRGywxW7lIY5Dwg73H8sGGkieLckhKDQfCaS4NbJhnd10raXpA2sI5FmqAhpFwomFM/K3FC0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:690c:3504:b0:608:769c:d72a with SMTP id fq4-20020a05690c350400b00608769cd72amr4800532ywb.5.1709749541184; Wed, 06 Mar 2024 10:25:41 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:24 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-27-surenb@google.com> Subject: [PATCH v5 26/37] mempool: Hook up to memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 26D3E40006 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: mgzypp93jhcbku7asp7ri6r9xbjicnzz X-HE-Tag: 1709749541-529287 X-HE-Meta: U2FsdGVkX18kO0iBH7eZW+8XNPh0WcbrmNNwaOex4ymxU4/EeZl3Cz3nc6fG/15j3sKwJWJKcaYb3H2z76WU0P2lnFem5xTKSXs3hvt9lpOqfIbDTc3EVI8aWsIoVUSkyQMH7Rae9heG5UydfMRmgauBSK29QL9ssZV7MVq2L0iAUmXu8XLZXKKXIEvtSVAzODBYIqQjK6A5xm2XxRforXJvl9Mh0fTzHVw6fi7nHK0aaw6bmX5/a1lef9wQ4sZi2SlWYEqhCdpt6bXwJHsrTxrM97P8IgvoAoR00l0apM0GpWoSZC4P3eZcBnkkRzFFnpTFg69urenlQgIvAV1fKJYJz8c/J1JVZP2HRaKXkog96N4HeJVauz6daKfCVOOcmhKWSjg9cq6cL77t8+r8eWsvRdXwkmdOOpuuOhvoESuz2j8t5dy8P/FbH96yKXWHbOnF8pH1d07ertgXO3DhE4/gOb6YN0MyialU6gKBG/XXywxif4XI05g6f/PL1GKlqotd46DuPpMnym+4STFMUh0ZgVa9BRHE7m8ifbeWYwaffaHGjnnMBTyqeN8cOllZXWny6fULAAf0ijaVPq7+wzTInJCVVvr4+ddU1Wi9gNdQwY7A6uYHkUrDszyVyAyc+4t59mwQT9fb85A4gf3x7RzLpRfzlbdZe1AnBrmG++HVWac/6ipWbZ5G1OwTmwKt1DhBCpJ42r/YlB02jiGzesjLIFagEOrQgC0d08uGGaJ06QEIgrn/iqq+8ro0PIwRsvCGmByypr5W8X+SUI7OhpAUASTMrd1+fnORSYABpsgtIwORmw55aAdevky1RPw11314/CqfZcW30WnyVPTr4VSzU16WcGjNAqEFRe9SHFy1YmO0Kp7ENdPKG+T3k1N4/dZoZ7WjQej167IkBkTbuzIQ+5kfGKUbk9/Ed4BkMchxQ8PUYo7tOOGh13r5YqBP3/sLWtuC95fSYO71u0R L8+VtXNP WBgW7fSzXGOpz6kcPh8JwzvL5O1mX4fJC8mPB08LcaT3iIDOLhU7COWGBcQvBagKUv6TE7F71Ik63PLmVZOBRRrZTOs3ciKhc0K8QJS/7lryJcE5M97UQlvIp0Jr7AlmBHSdC9AkmastbmK0nkA0KhngJ/4egTNWZXKsT7PikkOCgK1wz0UiC/SfPv8lRMm0PidlDQ/L44l9dRGfRpgXZbzWyAe4bGnDWx1xCZzHEBJlGNcy2oRBk/675ybl1kymxIEeResnMWe3oRe4Hj+YJEFujeLy97FgjBVVhoQgmq9ZRga0Ix2LAuZw4DEkh+F2fX9RiCSMeEKbew7Vpw/QbdwfA5OBsmyl2VQvLDNNyHgweYRooh21KjfwZvHRdQEuBJ+0h+2DxIDEQto1lGZC578nhEBU4R4aY8GWm/bC9z2e+UuF8kjJY1bcweYpnKh7Of3H06qv8TzC/Ov8Z/eYeVRENtXEPIFPapBK+fz6FJFnWaH52Y9pe7EVN+mMEqHviDEktA96/FeF5i60vLXIYEV35u3Q8ZMfEPkAUQ3HpZ0T5xMpUcRQExHZBfNNzuZcffS1MRoek4E93Ll5kdb5uPNiwukTzD5rF18PfTCZcIG+fsy8pcJSYnYhorbZHtDF/kVPM8GirxWw2PtPNOBmyC4GswdAzN6L5BctCmzKJ0/UJkBY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet This adds hooks to mempools for correctly annotating mempool-backed allocations at the correct source line, so they show up correctly in /sys/kernel/debug/allocations. Various inline functions are converted to wrappers so that we can invoke alloc_hooks() in fewer places. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/mempool.h | 73 ++++++++++++++++++++--------------------- mm/mempool.c | 36 ++++++++------------ 2 files changed, 49 insertions(+), 60 deletions(-) diff --git a/include/linux/mempool.h b/include/linux/mempool.h index 7be1e32e6d42..69e65ca515ee 100644 --- a/include/linux/mempool.h +++ b/include/linux/mempool.h @@ -5,6 +5,8 @@ #ifndef _LINUX_MEMPOOL_H #define _LINUX_MEMPOOL_H +#include +#include #include #include @@ -39,18 +41,32 @@ void mempool_exit(mempool_t *pool); int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, gfp_t gfp_mask, int node_id); -int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + +int mempool_init_noprof(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data); +#define mempool_init(...) \ + alloc_hooks(mempool_init_noprof(__VA_ARGS__)) extern mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data); -extern mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, + +extern mempool_t *mempool_create_node_noprof(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, gfp_t gfp_mask, int nid); +#define mempool_create_node(...) \ + alloc_hooks(mempool_create_node_noprof(__VA_ARGS__)) + +#define mempool_create(_min_nr, _alloc_fn, _free_fn, _pool_data) \ + mempool_create_node(_min_nr, _alloc_fn, _free_fn, _pool_data, \ + GFP_KERNEL, NUMA_NO_NODE) extern int mempool_resize(mempool_t *pool, int new_min_nr); extern void mempool_destroy(mempool_t *pool); -extern void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) __malloc; + +extern void *mempool_alloc_noprof(mempool_t *pool, gfp_t gfp_mask) __malloc; +#define mempool_alloc(...) \ + alloc_hooks(mempool_alloc_noprof(__VA_ARGS__)) + extern void *mempool_alloc_preallocated(mempool_t *pool) __malloc; extern void mempool_free(void *element, mempool_t *pool); @@ -62,19 +78,10 @@ extern void mempool_free(void *element, mempool_t *pool); void *mempool_alloc_slab(gfp_t gfp_mask, void *pool_data); void mempool_free_slab(void *element, void *pool_data); -static inline int -mempool_init_slab_pool(mempool_t *pool, int min_nr, struct kmem_cache *kc) -{ - return mempool_init(pool, min_nr, mempool_alloc_slab, - mempool_free_slab, (void *) kc); -} - -static inline mempool_t * -mempool_create_slab_pool(int min_nr, struct kmem_cache *kc) -{ - return mempool_create(min_nr, mempool_alloc_slab, mempool_free_slab, - (void *) kc); -} +#define mempool_init_slab_pool(_pool, _min_nr, _kc) \ + mempool_init(_pool, (_min_nr), mempool_alloc_slab, mempool_free_slab, (void *)(_kc)) +#define mempool_create_slab_pool(_min_nr, _kc) \ + mempool_create((_min_nr), mempool_alloc_slab, mempool_free_slab, (void *)(_kc)) /* * a mempool_alloc_t and a mempool_free_t to kmalloc and kfree the @@ -83,17 +90,12 @@ mempool_create_slab_pool(int min_nr, struct kmem_cache *kc) void *mempool_kmalloc(gfp_t gfp_mask, void *pool_data); void mempool_kfree(void *element, void *pool_data); -static inline int mempool_init_kmalloc_pool(mempool_t *pool, int min_nr, size_t size) -{ - return mempool_init(pool, min_nr, mempool_kmalloc, - mempool_kfree, (void *) size); -} - -static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size) -{ - return mempool_create(min_nr, mempool_kmalloc, mempool_kfree, - (void *) size); -} +#define mempool_init_kmalloc_pool(_pool, _min_nr, _size) \ + mempool_init(_pool, (_min_nr), mempool_kmalloc, mempool_kfree, \ + (void *)(unsigned long)(_size)) +#define mempool_create_kmalloc_pool(_min_nr, _size) \ + mempool_create((_min_nr), mempool_kmalloc, mempool_kfree, \ + (void *)(unsigned long)(_size)) /* * A mempool_alloc_t and mempool_free_t for a simple page allocator that @@ -102,16 +104,11 @@ static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size) void *mempool_alloc_pages(gfp_t gfp_mask, void *pool_data); void mempool_free_pages(void *element, void *pool_data); -static inline int mempool_init_page_pool(mempool_t *pool, int min_nr, int order) -{ - return mempool_init(pool, min_nr, mempool_alloc_pages, - mempool_free_pages, (void *)(long)order); -} - -static inline mempool_t *mempool_create_page_pool(int min_nr, int order) -{ - return mempool_create(min_nr, mempool_alloc_pages, mempool_free_pages, - (void *)(long)order); -} +#define mempool_init_page_pool(_pool, _min_nr, _order) \ + mempool_init(_pool, (_min_nr), mempool_alloc_pages, \ + mempool_free_pages, (void *)(long)(_order)) +#define mempool_create_page_pool(_min_nr, _order) \ + mempool_create((_min_nr), mempool_alloc_pages, \ + mempool_free_pages, (void *)(long)(_order)) #endif /* _LINUX_MEMPOOL_H */ diff --git a/mm/mempool.c b/mm/mempool.c index dbbf0e9fb424..c47ff883cf36 100644 --- a/mm/mempool.c +++ b/mm/mempool.c @@ -240,17 +240,17 @@ EXPORT_SYMBOL(mempool_init_node); * * Return: %0 on success, negative error code otherwise. */ -int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, - mempool_free_t *free_fn, void *pool_data) +int mempool_init_noprof(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data) { return mempool_init_node(pool, min_nr, alloc_fn, free_fn, pool_data, GFP_KERNEL, NUMA_NO_NODE); } -EXPORT_SYMBOL(mempool_init); +EXPORT_SYMBOL(mempool_init_noprof); /** - * mempool_create - create a memory pool + * mempool_create_node - create a memory pool * @min_nr: the minimum number of elements guaranteed to be * allocated for this pool. * @alloc_fn: user-defined element-allocation function. @@ -265,17 +265,9 @@ EXPORT_SYMBOL(mempool_init); * * Return: pointer to the created memory pool object or %NULL on error. */ -mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn, - mempool_free_t *free_fn, void *pool_data) -{ - return mempool_create_node(min_nr, alloc_fn, free_fn, pool_data, - GFP_KERNEL, NUMA_NO_NODE); -} -EXPORT_SYMBOL(mempool_create); - -mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, - mempool_free_t *free_fn, void *pool_data, - gfp_t gfp_mask, int node_id) +mempool_t *mempool_create_node_noprof(int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data, + gfp_t gfp_mask, int node_id) { mempool_t *pool; @@ -291,7 +283,7 @@ mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, return pool; } -EXPORT_SYMBOL(mempool_create_node); +EXPORT_SYMBOL(mempool_create_node_noprof); /** * mempool_resize - resize an existing memory pool @@ -374,7 +366,7 @@ int mempool_resize(mempool_t *pool, int new_min_nr) EXPORT_SYMBOL(mempool_resize); /** - * mempool_alloc - allocate an element from a specific memory pool + * mempool_alloc_noprof - allocate an element from a specific memory pool * @pool: pointer to the memory pool which was allocated via * mempool_create(). * @gfp_mask: the usual allocation bitmask. @@ -387,7 +379,7 @@ EXPORT_SYMBOL(mempool_resize); * * Return: pointer to the allocated element or %NULL on error. */ -void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) +void *mempool_alloc_noprof(mempool_t *pool, gfp_t gfp_mask) { void *element; unsigned long flags; @@ -454,7 +446,7 @@ void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) finish_wait(&pool->wait, &wait); goto repeat_alloc; } -EXPORT_SYMBOL(mempool_alloc); +EXPORT_SYMBOL(mempool_alloc_noprof); /** * mempool_alloc_preallocated - allocate an element from preallocated elements @@ -562,7 +554,7 @@ void *mempool_alloc_slab(gfp_t gfp_mask, void *pool_data) { struct kmem_cache *mem = pool_data; VM_BUG_ON(mem->ctor); - return kmem_cache_alloc(mem, gfp_mask); + return kmem_cache_alloc_noprof(mem, gfp_mask); } EXPORT_SYMBOL(mempool_alloc_slab); @@ -580,7 +572,7 @@ EXPORT_SYMBOL(mempool_free_slab); void *mempool_kmalloc(gfp_t gfp_mask, void *pool_data) { size_t size = (size_t)pool_data; - return kmalloc(size, gfp_mask); + return kmalloc_noprof(size, gfp_mask); } EXPORT_SYMBOL(mempool_kmalloc); @@ -597,7 +589,7 @@ EXPORT_SYMBOL(mempool_kfree); void *mempool_alloc_pages(gfp_t gfp_mask, void *pool_data) { int order = (int)(long)pool_data; - return alloc_pages(gfp_mask, order); + return alloc_pages_noprof(gfp_mask, order); } EXPORT_SYMBOL(mempool_alloc_pages); From patchwork Wed Mar 6 18:24:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584437 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31182C5475B for ; Wed, 6 Mar 2024 18:26:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5AA326B00B7; Wed, 6 Mar 2024 13:25:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 508C96B00B8; Wed, 6 Mar 2024 13:25:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 35D2F6B00B9; Wed, 6 Mar 2024 13:25:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 1CE5F6B00B7 for ; Wed, 6 Mar 2024 13:25:46 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D0D6C160C1E for ; Wed, 6 Mar 2024 18:25:45 +0000 (UTC) X-FDA: 81867442650.14.D585EE1 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf05.hostedemail.com (Postfix) with ESMTP id 288B8100010 for ; Wed, 6 Mar 2024 18:25:43 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=QS4ZNBQo; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3J7XoZQYKCGoacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3J7XoZQYKCGoacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749544; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vneHWg8klAgpjQ629hDbHaGhEsZ8DTv1BUkMhsSVPcA=; b=hXdJCii1FcUxOvtqvGDkXPmisDpplsZFAjoxG7ehweEouzJ9nBBoAK1YeHzOWpUp5KWYsT D/2El69cVkPG+3ONVKr0/xfSKXCpKvZ9KLVn0LMS+aqLW/vZkFwbylm/h+9XcFE+ihh+S1 SbPYAZxYgCNM+k8osVAWFjzLx+YdFhc= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=QS4ZNBQo; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3J7XoZQYKCGoacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3J7XoZQYKCGoacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749544; a=rsa-sha256; cv=none; b=o8ja5nzJXH04ldF8rl7Gx/ij0b2BkQH8by1AFTUEL6TjF1vSWo6divifo+9GtO3kBYytOo q3ocipiTdgFUG0Cr1PWC0XeNrf/nMBJ+kfzb7zgenIqJqgQIIT5rCd4N1FxbT8KChKejOD aIx2sgYdJX/cVICjHVJBgHQOVCCiwww= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-608e88ca078so345077b3.1 for ; Wed, 06 Mar 2024 10:25:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749543; x=1710354343; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=vneHWg8klAgpjQ629hDbHaGhEsZ8DTv1BUkMhsSVPcA=; b=QS4ZNBQo6AV1uVcgRbvPwrf7PveSTXOohh5Dnzlx5YeDeFVWY2WvJa1xQco2kd1xfY 4f8sIFTgzD0dfRZK58iHMMuejPWVwIV1DwkGaKLesV+kRgzY1WD82DBZ2CO1tKuLjNjO roGKtr23ergx3A4vOpRHpLeGHPdOzRtd+j0Lm2HrGGY5A/xKnJZRYnjfoKY5uYl6RTr9 l55HoTYRxocjG0sS59nQq7ONNZWGLOij4yiB6sSStKWmAZfI2vvwBEox69aq/31oMhyi /FgZHfGTAT6fl8G6MPQ54+DsC1kh+6gvnHcy580hPE/MysoEmM/2UbTtR/41aztfxqIe E3cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749543; x=1710354343; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=vneHWg8klAgpjQ629hDbHaGhEsZ8DTv1BUkMhsSVPcA=; b=YJ1Z58M6JngCiX6wJIp72yiemsLVMyciFFu3EMcA6jJXR87F+VxSx2byejO4u/QZbi 0tg2MDF3UvZjPkW/68FR1GG0r621BNb4YHzpxQYT4qxo9ufNtmB5alp3uUXygR1Og233 ZXBgL6W6CIY1uDvU2u+ch+2sdJUOjmRRISkuSwfDgQPVeLySQjfu6efcCa1ItfASVWG6 9vuJdjJ9p/7wioXSnnncIh5PmomD5dmYdLtin00kGcxeHKgNNixBcMwO+CXkg/Rkoemk yF8hmLSX0qyanUlnTejddE58I0A4dHgeiMt09u/vUj6pKDAMAtCZxdWSAqdhptMaPe90 rCCw== X-Forwarded-Encrypted: i=1; AJvYcCWGE/ENxq5/zNQWU9K2ySHXN8YW0+rYzKkzNeaGJirhTbByJZ5vKmX0ChmhOovGKrTSal/9fP8WqD0W4SjFKgUM07A= X-Gm-Message-State: AOJu0Yyd3PjYfd0dFVx8nwBPw/fT696/SQRHEWSj68aQVI+v7uFUZ0b7 MKQIIv25sj4skvv4YaX+WTLjEoD1dzsdPao8CEMEKr7dAed9Tz6gC//8nn2uSzKVb1BajfOP4Bl yUA== X-Google-Smtp-Source: AGHT+IG4AC9xYftpcvIviDBOUGTau4RR94IarG/5rX8rXPHiGDJpcdBjqZ0HayshSXEQB5VUXXEBNI+ragU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:b1c:b0:dcc:e1a6:aca9 with SMTP id ch28-20020a0569020b1c00b00dcce1a6aca9mr3819417ybb.9.1709749543321; Wed, 06 Mar 2024 10:25:43 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:25 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-28-surenb@google.com> Subject: [PATCH v5 27/37] mm: percpu: Introduce pcpuobj_ext From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 288B8100010 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: zn6z1df7b4emeofibebb5d4gs5oxtrfs X-HE-Tag: 1709749543-17214 X-HE-Meta: U2FsdGVkX1/R4qIdo2UuM7H8OcwDJvVev3sOtSqi3o4l2nHbQyTsXPJu16dZI9lNw4Y4uA6ZT2WsJO9aafwjPbvnU7wIJOKno+4s5yLUZ0wVldBfjEKffSHV62v5IPGyOI5psj9zNyuUBseuhU3fFKEbQ1ejVTfbPiUjINFnQT9LbevELdjZOhlHXR+esFkPy42/qXjRq0yu3gPohIeovRBPRKQLZ6trhLWShWvN0EoYQf/jMpczCB0c3qTgqlRGD0qB6yfwJprcD/9g4K8s0z8ahjZNlqU8CgleaTFChTAS1BTy1NiUQmXCTUka5c2SGYa1e9/ZABr4jxc+4TK+8qGIMSCLvCY/K/1JZd0tOI1tz8yc7/FHQQHBCLdzvB+YkAVUVKmn3zCN2uwK7SbZytq17uSKCn4tdMU2t2s841toWCAw9V/uPedgtg2zYH9UVCYowhsPrnABwSDP6nXhLpcLepzSSTJ81jtuYE/0A8wjVaYUDrZr9vM0bC3Xze/nVyzBT2zmHUYRG0m8nD/wKO/c6CI+JbZXfgnn7lUELmwMFZmyj9jYr3EhMgw/6wVn83Wg6uLtlwWyzEIynTM6vTiq6pLsRlPMMF31tJtgBcMNJAEz8sWfbPt/PZZnH2hZF8hCbNvMJZ7NtkPXoCh8uc4AzIkTbivmmopWqzT9M5lC5SopURC0VQMWT90dUuFi85VVsUEbJD6fGVi8KH1oEYhADJp9vDdky0xnTnCZm+7N8djWIg7omY1hVTfqbxExr+MgXfRat6hugN2EUaaz6SS1/XbCFnpCRhGFVsHEFCIbO3dXnQ3uHjZJcsO7rHkWn5Wj++UjJoGAOk+OPs9MQW6O9T84DrsMtLX17w57G/hTa3+cRhAF/hXLw5j9Ypny7LRTo9SlxWApHrjM9ZqTUuBqqNYPPLyeyjFaaxKlzdSJH6IhFxByNmSqI4+cqZ34WSM2kkjm99DCJqxNTMm N1hPfWAU WSi4WSdolGOg41i8k0d9OxOPhHa8xKAQKrCCsxFt6EofK/a2j86FX71MmecTUCHqOkTOpUqfYVna97DYzPFxTVs2ft84xyCi3vcRgdTWRsUniYviD3u26ryoiSXF5y/Lb7P0+AFJtLeCsD7ME1vVF/geHiBIHhAw3bSVK1HUyXDBOpcHP4J9TU2eQyL4JZm8940klHnT6NR3+HV++XgG0OWqH659LFJSioeGqKY9oe5p6u4iyDWPnkaiUmLfz1QPgP3FxwCCOnFuE/cV2RB8NyDaVwT3i/9oWDYZXakqgz0O2AhoUtSpyubOX95OPNjxrZ+lt3gHbd8nQOHWyoS9SYtJ4I0mVX4RRWUHx2qA4zddwH1bhJkQOwjFCWyEjmVF+Ou0SwwMri6XIYdyyrPVns0KKo33/qNnZfV5KiLEepOax5RYSqCyVOJvrH1Bvvbr0SUT8lKFd9m1KJZqsczwttGdKOg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet Upcoming alloc tagging patches require a place to stash per-allocation metadata. We already do this when memcg is enabled, so this patch generalizes the obj_cgroup * vector in struct pcpu_chunk by creating a pcpu_obj_ext type, which we will be adding to in an upcoming patch - similarly to the previous slabobj_ext patch. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Cc: Andrew Morton Cc: Dennis Zhou Cc: Tejun Heo Cc: Christoph Lameter Cc: linux-mm@kvack.org --- mm/percpu-internal.h | 19 +++++++++++++++++-- mm/percpu.c | 30 +++++++++++++++--------------- 2 files changed, 32 insertions(+), 17 deletions(-) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index cdd0aa597a81..e62d582f4bf3 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -32,6 +32,16 @@ struct pcpu_block_md { int nr_bits; /* total bits responsible for */ }; +struct pcpuobj_ext { +#ifdef CONFIG_MEMCG_KMEM + struct obj_cgroup *cgroup; +#endif +}; + +#ifdef CONFIG_MEMCG_KMEM +#define NEED_PCPUOBJ_EXT +#endif + struct pcpu_chunk { #ifdef CONFIG_PERCPU_STATS int nr_alloc; /* # of allocations */ @@ -64,8 +74,8 @@ struct pcpu_chunk { int end_offset; /* additional area required to have the region end page aligned */ -#ifdef CONFIG_MEMCG_KMEM - struct obj_cgroup **obj_cgroups; /* vector of object cgroups */ +#ifdef NEED_PCPUOBJ_EXT + struct pcpuobj_ext *obj_exts; /* vector of object cgroups */ #endif int nr_pages; /* # of pages served by this chunk */ @@ -74,6 +84,11 @@ struct pcpu_chunk { unsigned long populated[]; /* populated bitmap */ }; +static inline bool need_pcpuobj_ext(void) +{ + return !mem_cgroup_kmem_disabled(); +} + extern spinlock_t pcpu_lock; extern struct list_head *pcpu_chunk_lists; diff --git a/mm/percpu.c b/mm/percpu.c index 4e11fc1e6def..2e5edaad9cc3 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1392,9 +1392,9 @@ static struct pcpu_chunk * __init pcpu_alloc_first_chunk(unsigned long tmp_addr, panic("%s: Failed to allocate %zu bytes\n", __func__, alloc_size); -#ifdef CONFIG_MEMCG_KMEM +#ifdef NEED_PCPUOBJ_EXT /* first chunk is free to use */ - chunk->obj_cgroups = NULL; + chunk->obj_exts = NULL; #endif pcpu_init_md_blocks(chunk); @@ -1463,12 +1463,12 @@ static struct pcpu_chunk *pcpu_alloc_chunk(gfp_t gfp) if (!chunk->md_blocks) goto md_blocks_fail; -#ifdef CONFIG_MEMCG_KMEM - if (!mem_cgroup_kmem_disabled()) { - chunk->obj_cgroups = +#ifdef NEED_PCPUOBJ_EXT + if (need_pcpuobj_ext()) { + chunk->obj_exts = pcpu_mem_zalloc(pcpu_chunk_map_bits(chunk) * - sizeof(struct obj_cgroup *), gfp); - if (!chunk->obj_cgroups) + sizeof(struct pcpuobj_ext), gfp); + if (!chunk->obj_exts) goto objcg_fail; } #endif @@ -1480,7 +1480,7 @@ static struct pcpu_chunk *pcpu_alloc_chunk(gfp_t gfp) return chunk; -#ifdef CONFIG_MEMCG_KMEM +#ifdef NEED_PCPUOBJ_EXT objcg_fail: pcpu_mem_free(chunk->md_blocks); #endif @@ -1498,8 +1498,8 @@ static void pcpu_free_chunk(struct pcpu_chunk *chunk) { if (!chunk) return; -#ifdef CONFIG_MEMCG_KMEM - pcpu_mem_free(chunk->obj_cgroups); +#ifdef NEED_PCPUOBJ_EXT + pcpu_mem_free(chunk->obj_exts); #endif pcpu_mem_free(chunk->md_blocks); pcpu_mem_free(chunk->bound_map); @@ -1646,9 +1646,9 @@ static void pcpu_memcg_post_alloc_hook(struct obj_cgroup *objcg, if (!objcg) return; - if (likely(chunk && chunk->obj_cgroups)) { + if (likely(chunk && chunk->obj_exts)) { obj_cgroup_get(objcg); - chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = objcg; + chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].cgroup = objcg; rcu_read_lock(); mod_memcg_state(obj_cgroup_memcg(objcg), MEMCG_PERCPU_B, @@ -1663,13 +1663,13 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *chunk, int off, size_t size) { struct obj_cgroup *objcg; - if (unlikely(!chunk->obj_cgroups)) + if (unlikely(!chunk->obj_exts)) return; - objcg = chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT]; + objcg = chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].cgroup; if (!objcg) return; - chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = NULL; + chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].cgroup = NULL; obj_cgroup_uncharge(objcg, pcpu_obj_full_size(size)); From patchwork Wed Mar 6 18:24:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584438 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6AD70C5475B for ; Wed, 6 Mar 2024 18:26:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 77D156B00B9; Wed, 6 Mar 2024 13:25:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 704A86B00BA; Wed, 6 Mar 2024 13:25:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 52FF86B00BB; Wed, 6 Mar 2024 13:25:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 3A2456B00B9 for ; Wed, 6 Mar 2024 13:25:48 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id EDE23803CA for ; Wed, 6 Mar 2024 18:25:47 +0000 (UTC) X-FDA: 81867442734.28.3644644 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf12.hostedemail.com (Postfix) with ESMTP id 4869940018 for ; Wed, 6 Mar 2024 18:25:46 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="suJlHS/f"; spf=pass (imf12.hostedemail.com: domain of 3KbXoZQYKCGwcebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3KbXoZQYKCGwcebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749546; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uTM4daMpjhNnHNsVheTGd2gBho3MSeHG2jrz4iVybwI=; b=BypyUVgBlKtdWVVYolzXiKNbwflnUxXfwqA9HAnp00/fBjD1nONG7nzsssShedTeOysHqg bKoqu2ZazHfr5TsfcGIKSF2ZPk137EcIlnAyNWZhOXPRsuuLUdEAPLOQ5fLluLBN/LOwiz fI6qCIGUDeUgyTHWU98pvwDFWuYF7eY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749546; a=rsa-sha256; cv=none; b=awypSPRe/84BKKXzhlNUXyiM6I1mxDBLluIzvAkK39H5Kfyi1VfvoQ7OBB7K2mauX+ob1E YoZjHHMXmIo993F+IjOG1eHax4pzqEwevbaflWR8RbxSX07z5VNvQYBXGDfucFSkGiuDfh y6YbZOtBD9Qx/dgby2JM4kmQtzCqJ/E= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="suJlHS/f"; spf=pass (imf12.hostedemail.com: domain of 3KbXoZQYKCGwcebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3KbXoZQYKCGwcebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-608dc99b401so725807b3.0 for ; Wed, 06 Mar 2024 10:25:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749545; x=1710354345; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=uTM4daMpjhNnHNsVheTGd2gBho3MSeHG2jrz4iVybwI=; b=suJlHS/f1ptV98PsbzkAbICfV7Q3x6YkTNboSLx30BFU5tmgKcg3Z35fNrI1b5tZHg T2igux5z/MZrKCW4R56VipuIjhIrrL+9MRnTiDZfh/p5bM0eqnMiFZEwdkwifBVDLkiN 4CunZAHx+XMDY8+Q49j9ajsOEOn5AhR7NUDcOJE+MgBNYj2YDFblONCDghMqxeSx9f57 2m4jOyjl4/xUXlCjtsOOABgYayOO/AJujfQrjp+srzsn0THbPyF80F6b1YRUuR9xch1l gbO2HVdNoTw3rXdQBJj0XJk+51cJBo/qjHSGb8+y7fJV4qsvZ1kkhNQ0F2oZSHsCwccv A7yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749545; x=1710354345; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uTM4daMpjhNnHNsVheTGd2gBho3MSeHG2jrz4iVybwI=; b=AVh8QTNt09TIeFpyNGGUrhDcVmOMD9r5y1/aFIQt+Z1RdRA2j2PRCBWe66B+76lWvh f3eRwatcLGEQjfqx6odgnjgOXnDpRedZVTNLEIJwceinzKCyNP6uM0ltN2BcQuvQmvd2 wLxr/8tfpo7+Ed4Vge8BQuEOF1PT1eIuds8TmYiTxyUzFZjiXe29s1k6/1pK5XONUHUU Yp9wMre0v1f+8m2/YTCUco24utApeJIfo0s4gMEDg0RPo58AhTen4HCOMrFKy5Ae3Vdd EcyBDfsWiA9gxtEBs4+7VVzssVFhKXcEzwJC+6yniV7ToOD6nltcCpOL1Z+IP9cNhxFC 355A== X-Forwarded-Encrypted: i=1; AJvYcCW1l6ApzCRXbKU4HIT9upZAgA8fceb+HHyPrbtr+p9IkvpE7J2pzKCzUdUEq0sXlV6EK1XK1sx5cbBDG7Do64EoVyc= X-Gm-Message-State: AOJu0Yx+u3OpL8LB2pHyKr1xBEQ6+VztVlUNtRaIhd3ep7VzALa8/ac6 I6y37EK+iw7l/pfO5Am/Bcbro5tMCc+l+MhRyoUFTiv/sUCzUs8AN6OmQJJGCr11wYK6MwnN6H8 WWg== X-Google-Smtp-Source: AGHT+IEnKzMil/vcPtNXvwckm3OECYjnTJYXETMu8tDMF6y1iGrklNs9lczJgJmwr4/pSL8XxNI2NQuejS8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:690c:e18:b0:609:247a:bdc5 with SMTP id cp24-20020a05690c0e1800b00609247abdc5mr4410842ywb.4.1709749545152; Wed, 06 Mar 2024 10:25:45 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:26 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-29-surenb@google.com> Subject: [PATCH v5 28/37] mm: percpu: Add codetag reference into pcpuobj_ext From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: p14w67sdrfk447mp4q59mstkcptnnqsu X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 4869940018 X-Rspam-User: X-HE-Tag: 1709749546-877340 X-HE-Meta: U2FsdGVkX19CQP3Iy4KqglKYlXht6DRi6ys4dl4w6eZ2gYnBM89q+/nIL6jQczLlEqWszpv2PRumxMJ7B9zCFGfjOaM9W3CMkse6aUDr+WNtlBRbBkuADd7vnYCdWxiJrNvvQ35kyJJnh+8AUes9W3udWVUUPe6b2U1mo7hTCyoCrTAOte7FQF/PQMzA56NReHuK4v4B2Up7XSDYWGIifaLoMi2pkqGnT8FAmYESj9WIsiwrP0M5dVXoLLcTOhzwWfJ2P34JrLYOCJ3nYBaMGZoebWn0pRjO49n1xEkiothoXUi5UFzdNoGzTPEHp57Z1nTaA3bTLCuLKZe5tJtxWGiSNgoLsiTzBris8zndFv0nuxAJpjS9S50A8R0VjIUdOANPmxb69k5AtbtagvtML30JobXoG+olmsUE6ovED9ju9Xc943kI7qdAx+FQWN8+BngCYA3iLj74G0utexmguf1LRNvfX0w+ldLKKxrL6dZEM011qHoax85rpW1z7/aUU4ACxf96Ay3bZoOZjOCzJeGvfxdP33CTmTJ5hezL0zDb6R4qK1lbEztlY49rXRKU3rXZq1QVe6eKrygf3EoPI/uxAloc7gEdch5/sc4H0l+fr1ZFmXLVItI6xa8eUdE64vySIOxVKsPcZurJMHUAtVn6qLbrc7SfKw9/lBko+O8VaZjadtFvZSExkovc0rnS1I2i7Z9X0W9zu+u2ECQZ43zBVm1BrGbQJMPlqgv8qRs1xUMd3uhQYkPc7MpeCCSTi0bklmhC4xSegN41eWoEgHHxgPqvGuUuJsSUxQpCjkub5nAzCnaGlPp5WLkJtEqZTceoAXr9rNWvM9trhQqfY1XNH5cgjsurUJHgNrNyYY2XV7QwrNx+8o8Q315S76jaGLAZxoI9hFDPBSjmeuNEXeRgXEgpCqgucNKyzFOVu9vIm5ZRmiqZmAGHy4GAmHvbaefPMTNq6w+Q6tawLmz N+ITxp9Y adGxop9/1Y7JceSHFu4kFZhfMQH+ZAa3Ba5Yj/x0CJ6Md95gnNOX8TXy4b7WYEvnmt5VjGxhykgKAE5KSeVtRF+8rU1eIgKfLqSeA1A1Yv9X0yjyGRVLKeIoDcC3AF4j9632c+0NhZe0DSwGTyPWAmCybroxysvU35K0F1qlz+6rmWoLnvqM7aSH93aw5v4gW3Z1rqX6MiITye0b58Lw3NplWGflYrlc+kWM3a9W1euDWYaAQ2zqCrfUuvGm3VoegE5/p9dvVK41DCCFQ/Ei/6cPbl1MeuOLdDGFvky74EtMBsWJF3oeJuE6lxlC1J51Da2tkftBfGsTBLQUhVpUPET5OTC+imgZ4V3qsSTkaG721o30YCkptxD+cbvxAzSX/tApFC28ANEl9qWm312mEgbn7hz/T0Q75DXJ9XygjO5lFwr7XUZMoK5m3EwL90IzS9M6b1NXb8qx+v22wwry0KHrYcg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet To store codetag for every per-cpu allocation, a codetag reference is embedded into pcpuobj_ext when CONFIG_MEM_ALLOC_PROFILING=y. Hooks to use the newly introduced codetag are added. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- mm/percpu-internal.h | 11 +++++++++-- mm/percpu.c | 26 ++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 2 deletions(-) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index e62d582f4bf3..7e42f0ca3b7b 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -36,9 +36,12 @@ struct pcpuobj_ext { #ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *cgroup; #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + union codetag_ref tag; +#endif }; -#ifdef CONFIG_MEMCG_KMEM +#if defined(CONFIG_MEMCG_KMEM) || defined(CONFIG_MEM_ALLOC_PROFILING) #define NEED_PCPUOBJ_EXT #endif @@ -86,7 +89,11 @@ struct pcpu_chunk { static inline bool need_pcpuobj_ext(void) { - return !mem_cgroup_kmem_disabled(); + if (IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING)) + return true; + if (!mem_cgroup_kmem_disabled()) + return true; + return false; } extern spinlock_t pcpu_lock; diff --git a/mm/percpu.c b/mm/percpu.c index 2e5edaad9cc3..90e9e4004ac9 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1699,6 +1699,32 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *chunk, int off, size_t size) } #endif /* CONFIG_MEMCG_KMEM */ +#ifdef CONFIG_MEM_ALLOC_PROFILING +static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, + size_t size) +{ + if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) { + alloc_tag_add(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, + current->alloc_tag, size); + } +} + +static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t size) +{ + if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) + alloc_tag_sub(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, size); +} +#else +static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, + size_t size) +{ +} + +static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t size) +{ +} +#endif + /** * pcpu_alloc - the percpu allocator * @size: size of area to allocate in bytes From patchwork Wed Mar 6 18:24:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584439 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62BC3C5475B for ; Wed, 6 Mar 2024 18:26:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D71146B00BB; Wed, 6 Mar 2024 13:25:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C89216B00BC; Wed, 6 Mar 2024 13:25:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92CC56B00BD; Wed, 6 Mar 2024 13:25:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7E7F06B00BB for ; Wed, 6 Mar 2024 13:25:50 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 58FF9A052B for ; Wed, 6 Mar 2024 18:25:50 +0000 (UTC) X-FDA: 81867442860.05.34584E5 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf22.hostedemail.com (Postfix) with ESMTP id BD9E1C001B for ; Wed, 6 Mar 2024 18:25:48 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=vAS4FeYZ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3K7XoZQYKCG4egdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3K7XoZQYKCG4egdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749548; a=rsa-sha256; cv=none; b=77+ROHsYjI4zdJtYAwSOd/qR1Zd8QRXL0Bi+F9V8talrJBiEklX+E83j31CKCLTB04FhnE 88P/n6y0PG1oNTjrRTI0Eut1Z0Nh1G6NNdkF5T8UxJQ3j1hBmdhXf8prGyoI+mfWiuhOn+ Y4Srpk+qXosCTA4imiNbg3ehaBoltXw= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=vAS4FeYZ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3K7XoZQYKCG4egdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3K7XoZQYKCG4egdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749548; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DxoUvKCzEXXEdeRbYveeEvTG3TXaR8hRTTgS3Q5TXls=; b=Iq1mKUCAE3tKVcEjgLFjh/a/zjHjGqGXKa85e6eRprdFEkT1CI2/HqACrurk5luWWjuJFO 5zbBAqNKyp6qM6SpJws7leG9S045bETT+qMmlwsad02r8GA9RjBzFvdlV0+7JmGNa00fvJ RehsmVwK53RsgToUev/j0Evnm2q/aVM= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcc05887ee9so10099840276.1 for ; Wed, 06 Mar 2024 10:25:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749548; x=1710354348; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=DxoUvKCzEXXEdeRbYveeEvTG3TXaR8hRTTgS3Q5TXls=; b=vAS4FeYZw5Hrbaqqr2ZEohfq3iV82GTfiMWUFql0KGlJS/lBguCKud9RPUldaktOyL pvbNJs/htZbLpSRB6cDW+X9SiWg5K9choR9PuZAI5rQ4FvTiNWkLowV82CsPWLkoemJM ur2U4lGrUXHhFWQqREC8XrMtgRiem+c3XSTBid4LDaQzamujAdEC9Y9DGFMDIZcQumwA 03JwNQrv6JgKYzFs9N8tGdqWnbZ/el35g4kbGAW6AvCn3fCPU+aVGNvgDbi8/seWLU/5 S+trWn75r+r8xv/JJHb16eM0ca3wHZqtclippLzuOyIQkUbttTtixHum1PEeKXebHIDa cKmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749548; x=1710354348; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DxoUvKCzEXXEdeRbYveeEvTG3TXaR8hRTTgS3Q5TXls=; b=hTFPAp23BGvudgt3JlYsVSIYGpnSprkMRqZV75sIR/LUEkDIxMypwgnnmFhajEdwlZ TVvflMjNHlVAVQv65t7uJpBChYrBFj1S4nv7lUax0RmZr4+C25ZbVGR+AA6olbZGX3eh bqKyMAyWT8P5/n5yKrFAYdRJ9LMfgFlxclWUzCqYUHAF6dnz4ScJvE2IFtTE2i3j1LTM rfXStxO7olSJsZQGNIXfW0Kcb3361PB5lAakNRC53+a6wH7bpv24NpO7Y6zYJvlxMScW W1+ohabLY+n5CzGoZDZskLtqIX26ez6VaAlpAXTI8+pio4Mc3OOUkhrQMzatmy3r6en+ OLGQ== X-Forwarded-Encrypted: i=1; AJvYcCVSeHd9XzHeN4vRTXTfDPUMEtFIj8DqWZNvIQxTeODMRMc1J0taklwkiakgM8xFHB2qmj2xwmCrba9yQPrQuhsw/w8= X-Gm-Message-State: AOJu0YzVI5j4/swrnjMzI1/pNOQ7mXFk6ug0JQ3fhJfouoG7iKwkEkNG M4a31/MhZ+/dS6LmsFUPqH3ukGCCx3Yq4lHwIiqFdfLDEmlWNShyF18ilHNQZSNPv8SOZelVGp7 3cA== X-Google-Smtp-Source: AGHT+IGiM8eS9ZI/wjAptyo7knt1Z9PGL0v/iP+d+T+GDYu7/A3uBJqs/ZKhwzkikTnjE1CsYg7wV6eZ1JE= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:10c1:b0:dcd:59a5:7545 with SMTP id w1-20020a05690210c100b00dcd59a57545mr531845ybu.10.1709749547368; Wed, 06 Mar 2024 10:25:47 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:27 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-30-surenb@google.com> Subject: [PATCH v5 29/37] mm: percpu: enable per-cpu allocation tagging From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: BD9E1C001B X-Stat-Signature: d69spwjuggqemg97rte49wewoibsymox X-HE-Tag: 1709749548-889167 X-HE-Meta: U2FsdGVkX1/bjR5knGxzmEH9+sP34XJ0FXhDUuQr8gHWiyKp4uIHSyTzmo0oGPfYAggG6Vuh7dETw20PuzRe1Oq50Z/WhdeM5MNFPu3oHQkgnTuqsknlu3Mv6mkR9X92a0UuvzbSYImHDgQuB0aCIRhqA1bFSbcs/EVobVpSibQRKzaau19wlawJEIrLuaLtyeg0+ARuwMirAqOVrSvsnVD8ZQqEBuOwHwzxwLvDzKqQlFahNpr5Hwj7YpmDinvU0hbMiXRj92+8xx3SKJqjYRHc6SskSDSgA/KqGrlaNVFUGOGMrsOsTSzztqcb1/T4MvSWpnpnj1PkWivQ/sSgGBgL6kGi3GqYYwVkB57MHtaNQyAp89QWTDOO/mEt52Ffr0udOQXn4RH1+WVk98pL9MKrdFueiGSaCey6pZlt+uSYJgpB5XynJ9IMecuEindgTFrWh9PWHfNmnCRYI2TZDOTuX7JizMxYEiRetOmpzsFeFpqwckWPVsyx2UF8VPVONjshbC+hcdfjJiVwEK0sYRK/djzbR6SV6kN6tUjmB1VHeIgTu6HfSZWnugXTBBURr6XdDYPT6kkMkpobiSmN82EKhdtCcMlRoWaetUp5bTMTwhjiUSzGciVRq/y/qMeU/ZYaoyqvfrA5UmP0eABi4SM6k9ZfLfpdW2oTTsdHL60tEHp9V2ht2uJRCiZkZG0BnK/WPf+ozAl19xc1k9liKRLV7j33pVQ1NF48LWsj2fcJy4WrN+lv9Zts9+yuWqYUTpQHW4FWYz/xf2u45fdNQ6cufxWlpT+nL5aPWXp14H1eI/gVNLk4/Z7yKIsKu81Z9xoKUV5A4MZK++tvtTqKrQpVCH7eLXChcXmq5m4mLjciIO06YM//edDP2tSNp4CISoNdT/dYU/v+C5ANqt4OKvvbcf4zTg4Yko8zuctSj7v/HQO8Hr5kVNLU0SNJLNuATM+APCZdOvllSnMi4lW sL1V3l7B BvLGVjisIZYVjNVWYOkFnC17ux6XE3tCdxzkfIwmDt8c6OXS2xM2Gd4ZGYiRQW11pr8MhToON1xrEJIDFklQI/wvR8koFy7D7dIWHljg4mJ2yey0ng9VSb14hZCM+/THO+k+OwlMVGU2Ni/AJPS6MCv/FgN1Zm4uL9j2bO1NPpCWcyBZArPj7JQ3WMvrKFRY6fre2J9cAdjPVzozbY69+4cQax7pGGQ6AoG7wFkyhbe6J4xLwjRRDF2k8EVeQWsHHii1SuprV1tigelJirFPLhAuTfEOAxiiuGcFsd3jODQNQVLmY3GTp03ojqSL5vU1GeTYqjCRnjLCI82Vu28TXxop7cObxx8N2rUaHJOXziFKuGWfW2wHXM6fHN91mzkwgJ7Ujc77WYzeILg8ClrKu2nnWInu7MOjnqOTB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Redefine __alloc_percpu, __alloc_percpu_gfp and __alloc_reserved_percpu to record allocations and deallocations done by these functions. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/percpu.h | 23 ++++++++++----- mm/percpu.c | 64 +++++------------------------------------- 2 files changed, 23 insertions(+), 64 deletions(-) diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 62b5eb45bd89..e54921c79c9a 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -2,6 +2,7 @@ #ifndef __LINUX_PERCPU_H #define __LINUX_PERCPU_H +#include #include #include #include @@ -9,6 +10,7 @@ #include #include #include +#include #include @@ -125,7 +127,6 @@ extern int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn); #endif -extern void __percpu *__alloc_reserved_percpu(size_t size, size_t align) __alloc_size(1); extern bool __is_kernel_percpu_address(unsigned long addr, unsigned long *can_addr); extern bool is_kernel_percpu_address(unsigned long addr); @@ -133,14 +134,16 @@ extern bool is_kernel_percpu_address(unsigned long addr); extern void __init setup_per_cpu_areas(void); #endif -extern void __percpu *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp) __alloc_size(1); -extern void __percpu *__alloc_percpu(size_t size, size_t align) __alloc_size(1); -extern void free_percpu(void __percpu *__pdata); +extern void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved, + gfp_t gfp) __alloc_size(1); extern size_t pcpu_alloc_size(void __percpu *__pdata); -DEFINE_FREE(free_percpu, void __percpu *, free_percpu(_T)) - -extern phys_addr_t per_cpu_ptr_to_phys(void *addr); +#define __alloc_percpu_gfp(_size, _align, _gfp) \ + alloc_hooks(pcpu_alloc_noprof(_size, _align, false, _gfp)) +#define __alloc_percpu(_size, _align) \ + alloc_hooks(pcpu_alloc_noprof(_size, _align, false, GFP_KERNEL)) +#define __alloc_reserved_percpu(_size, _align) \ + alloc_hooks(pcpu_alloc_noprof(_size, _align, true, GFP_KERNEL)) #define alloc_percpu_gfp(type, gfp) \ (typeof(type) __percpu *)__alloc_percpu_gfp(sizeof(type), \ @@ -149,6 +152,12 @@ extern phys_addr_t per_cpu_ptr_to_phys(void *addr); (typeof(type) __percpu *)__alloc_percpu(sizeof(type), \ __alignof__(type)) +extern void free_percpu(void __percpu *__pdata); + +DEFINE_FREE(free_percpu, void __percpu *, free_percpu(_T)) + +extern phys_addr_t per_cpu_ptr_to_phys(void *addr); + extern unsigned long pcpu_nr_pages(void); #endif /* __LINUX_PERCPU_H */ diff --git a/mm/percpu.c b/mm/percpu.c index 90e9e4004ac9..dd7eeb370134 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1726,7 +1726,7 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s #endif /** - * pcpu_alloc - the percpu allocator + * pcpu_alloc_noprof - the percpu allocator * @size: size of area to allocate in bytes * @align: alignment of area (max PAGE_SIZE) * @reserved: allocate from the reserved chunk if available @@ -1740,7 +1740,7 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s * RETURNS: * Percpu pointer to the allocated area on success, NULL on failure. */ -static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, +void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved, gfp_t gfp) { gfp_t pcpu_gfp; @@ -1907,6 +1907,8 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, pcpu_memcg_post_alloc_hook(objcg, chunk, off, size); + pcpu_alloc_tag_alloc_hook(chunk, off, size); + return ptr; fail_unlock: @@ -1935,61 +1937,7 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, return NULL; } - -/** - * __alloc_percpu_gfp - allocate dynamic percpu area - * @size: size of area to allocate in bytes - * @align: alignment of area (max PAGE_SIZE) - * @gfp: allocation flags - * - * Allocate zero-filled percpu area of @size bytes aligned at @align. If - * @gfp doesn't contain %GFP_KERNEL, the allocation doesn't block and can - * be called from any context but is a lot more likely to fail. If @gfp - * has __GFP_NOWARN then no warning will be triggered on invalid or failed - * allocation requests. - * - * RETURNS: - * Percpu pointer to the allocated area on success, NULL on failure. - */ -void __percpu *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp) -{ - return pcpu_alloc(size, align, false, gfp); -} -EXPORT_SYMBOL_GPL(__alloc_percpu_gfp); - -/** - * __alloc_percpu - allocate dynamic percpu area - * @size: size of area to allocate in bytes - * @align: alignment of area (max PAGE_SIZE) - * - * Equivalent to __alloc_percpu_gfp(size, align, %GFP_KERNEL). - */ -void __percpu *__alloc_percpu(size_t size, size_t align) -{ - return pcpu_alloc(size, align, false, GFP_KERNEL); -} -EXPORT_SYMBOL_GPL(__alloc_percpu); - -/** - * __alloc_reserved_percpu - allocate reserved percpu area - * @size: size of area to allocate in bytes - * @align: alignment of area (max PAGE_SIZE) - * - * Allocate zero-filled percpu area of @size bytes aligned at @align - * from reserved percpu area if arch has set it up; otherwise, - * allocation is served from the same dynamic area. Might sleep. - * Might trigger writeouts. - * - * CONTEXT: - * Does GFP_KERNEL allocation. - * - * RETURNS: - * Percpu pointer to the allocated area on success, NULL on failure. - */ -void __percpu *__alloc_reserved_percpu(size_t size, size_t align) -{ - return pcpu_alloc(size, align, true, GFP_KERNEL); -} +EXPORT_SYMBOL_GPL(pcpu_alloc_noprof); /** * pcpu_balance_free - manage the amount of free chunks @@ -2328,6 +2276,8 @@ void free_percpu(void __percpu *ptr) spin_lock_irqsave(&pcpu_lock, flags); size = pcpu_free_area(chunk, off); + pcpu_alloc_tag_free_hook(chunk, off, size); + pcpu_memcg_free_hook(chunk, off, size); /* From patchwork Wed Mar 6 18:24:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584440 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 767BAC54E41 for ; Wed, 6 Mar 2024 18:26:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B76396B00BD; Wed, 6 Mar 2024 13:25:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AFD016B00BE; Wed, 6 Mar 2024 13:25:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B2C26B00BF; Wed, 6 Mar 2024 13:25:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 738C56B00BD for ; Wed, 6 Mar 2024 13:25:52 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 468F71204E2 for ; Wed, 6 Mar 2024 18:25:52 +0000 (UTC) X-FDA: 81867442944.16.D3CBFF1 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf17.hostedemail.com (Postfix) with ESMTP id 8DCB74000E for ; Wed, 6 Mar 2024 18:25:50 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=CxMrkgEZ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3LbXoZQYKCHAgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3LbXoZQYKCHAgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749550; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YUUmukvCnHiMs6J38hCV2D1Gmer1PXmykJXdn9XXZgU=; b=xO/P+ZEGG74A1ycmfpXj+qNJalAWPyCnfR4aaUBk6ltt5s49pp8P5+CONmHhQyur+IIGbn nTgdEOt4tt6iYi/m9iwzRaIlafxYqJMgjTWFSV+2kPVxPWXuat/7tqoNzD7rsRKTQMr48J H05nkhHd1qMwq5ZD56jDcWSdchisglM= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=CxMrkgEZ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3LbXoZQYKCHAgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3LbXoZQYKCHAgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749550; a=rsa-sha256; cv=none; b=dU83MIB18Kwwz3Jxa+4fN7epD+h+RntBvi4i9DhNlhg/BjSHScwN5j8MgF+9wqmY8QL0e1 XwNADZrbMqq1eiX2AbOKLTD/0cy+NAEhFXLAyb4Ss+ofzZNL64vY9jGn+yLDICofL8bad9 3C0xsYtezu0hFbgRVSMAS5OmpiP3HDI= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc6dbdcfd39so13634945276.2 for ; Wed, 06 Mar 2024 10:25:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749549; x=1710354349; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YUUmukvCnHiMs6J38hCV2D1Gmer1PXmykJXdn9XXZgU=; b=CxMrkgEZOjzHLtjFzdU/uo6TBsUiBrBVQVO3gYKEpOKYTUJSJB6eVOI/z/2HYIQ2E+ ZbZDb0j0xU4807+75xP2jnbmHWvX0IyHsLGElkttph4G3q2XQswnpA7MYX+kPHMarDwh EaCbGnbqPi9KKVg4wNxLYbQhcSl5Yzpt4yHz1Qti8RZ7+MCJnL+2HFvbnWRChJmJS1Ct zviqpHJRuvmB0Y1R/zB6k8Vv6SuIeXJoTxfHyaDuOJ9Q3TBgwAxjME+JHVl+zDugnBMx 0jxHU6FoMtKDYxsRkq8bnzmS+5Rmmjfvkz1ISxq4gYfZ7R49GkgixgUQ0lJvEUjt//nz cLbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749549; x=1710354349; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YUUmukvCnHiMs6J38hCV2D1Gmer1PXmykJXdn9XXZgU=; b=Vo712kRnDHLrvjpW3ORLEEHYALkgixvHNuVEF0C+8fEDJNeyOo6YFjQXH84LnHMtwW 0ZOwjqmC8zI+USWlwFBMRDX/SdKeQCov/CMbIZvH6Jh9fhLO3tQtGK4xlBkL/jfLRczT yqFUlTDa4qHCrIIiFegE149EmfMoiCntRnDhWiC20JOOrOk5Joc4caG1foU8RS+vlH2N L3Zu0bfeB3j3WqFNId+T8XohK5Zt9IWrq0yFEF1AHto1MUa9zW75oiT2pBPMQLp864oj ydxyiPcAncn8bA2BrTkh6Xo1k/IYA3wpTaTU+XJAL/3jimK7ggDPKom17uzx0Ee7XZCO HQPA== X-Forwarded-Encrypted: i=1; AJvYcCVk/jV5L+bwjiTVxpqOsjJDovzzDovEviSklGAO+KXSAUMI7+x21UIGKU4xdBNLUNNc0dMYm6DEut1I+tPqCY/nEIk= X-Gm-Message-State: AOJu0YyCC+XuOojR+dseWuG5y6/yz5GcfSgyCnUfNjTpCvptds97uzim kA39cqGvuCHFWERbiSAv0/hb4SgFYbfAXuEYRuPVpOqHPFn5PR9m/Zx+eu87174B+YVva7mzIIF +0A== X-Google-Smtp-Source: AGHT+IEBo/NBBvAf4BoVrOGimhE8C4YGAKQr5UNfJ7TptyEuCaWEABum2H7J9Iawx0gApOTdXdiUztbnIiA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:1891:b0:dcc:54d0:85e0 with SMTP id cj17-20020a056902189100b00dcc54d085e0mr4037988ybb.11.1709749549455; Wed, 06 Mar 2024 10:25:49 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:28 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-31-surenb@google.com> Subject: [PATCH v5 30/37] mm: vmalloc: Enable memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 8DCB74000E X-Stat-Signature: t78ywxnw1813zxs55dbfhqekym3byewm X-Rspam-User: X-HE-Tag: 1709749550-228616 X-HE-Meta: U2FsdGVkX187ly6kKUd5SvkfTYEkVOvYX7YMSjMkwrlHPM3/TEp4MXU2mJXeGQjQbGXS2doYTOLQkjL9azJxfGEZ9fTv17xoJBoCMsIkQeyRZbJkFgiqGEB5fpLOBIoWZqVViRwsZQqnr0eHkkrSHo3UK2ZH+H0eDLx0OQLKvNJgF73+2h4ZjFAv4X9ockkH6b8xmYegLPWFXlvotAOVb8dQ4rsNX7QbwfKnSH43KV5WGE4ODnx6JPzlppiFIK/SK3+Z9WEzKywSIMwORp4BMtaxJ+RDl/oawUKX0qLNJUbWAVP7wYO5Dn8SNDi+Yn1TtvnwdxI1ENxtl4N8ZZ/RJ0OqHmCAIT02GAIxyH0100xbXaMPf3rKa24NzzaFWY325KlOfPBvylTtaF2AAolNiGBDsiwAPQNL6rNcYEJDaYTotVd9/9lGtrp3YZxMo5xUmdCZlddtq9P05EX6LIprGlBcPpovq8NudIqcywBhbuSXrKxVabZXlTOx3TXR9/QIXmmY87zkfg71QDk42SD7pggRTzfdEO1qq3GOdi2j3byl29fmDJUdTJcDWPiue8/ZlFmAAxpT+sG5IaRcQdZlWj9EOjnVbWlO+dX32d+2WbCegxUk8QTbDC3U+Vcciygy5wKqNiLt2CI1RIMsTRtxb2a32lcSaj/+F+9jh0qiNhHjQi65XFkJpC4h4dm/O/vcOWeOQ/6xgXq7SFPaKvfsneKCr/57lK243MLg6l6w+5ETkb6WhMPruIKowl/MZqm9r+RlnWZFNSdohzbfgCyaA4TTW+2ZevE8GDnOESgdT32hbQfLIj1lJ4tx9yXi+hYDeZk6vmJn9w3a9MOhrxBsjgV0ejOTdNj5vigrRteQoc5/6wvcLXO+8EgITk40fO+Nah/XmNyPWJP+4BQm8GcZp5BUUcJtaRmg3EAbDde0Hp2OKcdW+lemA8yhX0wa82vyvIAK0xSA4gXgKcBbTs4 pNlBQap3 0AXbiu9IOrX3reuOVLzLZNIEud+XZeTSG6qvA4OffKXqrWlw4q+WTELzbRE2ECR/KykBgvdV2xFtCTR6x8jw6Ya+Sm268t65Pc8mbce9I8SmaSRZlXFan8GzI6BTAYwpMu+WboB6y7Gww9ZWtN/sV4/LfLf0UgJ6B+ymaeODW4Gpr5a7aq8Tf//El6AbtUnlHP15NrS14GA/yXP/ioZ2STVF2gRbBS/kS+GuSS5v638hbIlGZ5KhGNR+iVG5n5aJ+LPwFs4+ldVZ+t1O7DScWckph7zDDdB2CNvXDL6PYjruTrCxq5CGwYfhpUn6w3k3tYdoSs9LPW/hJrRQHBANQsl92H5J87rrpg46Jjfw2zM1HHNrjhVCUV/Fh3PDMDfNpmOw04aeaC4zDaNPnza+/W1NxagfsozvKQl/7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet This wrapps all external vmalloc allocation functions with the alloc_hooks() wrapper, and switches internal allocations to _noprof variants where appropriate, for the new memory allocation profiling feature. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- drivers/staging/media/atomisp/pci/hmm/hmm.c | 2 +- include/linux/vmalloc.h | 60 ++++++++++---- kernel/kallsyms_selftest.c | 2 +- mm/nommu.c | 64 +++++++-------- mm/util.c | 24 +++--- mm/vmalloc.c | 88 ++++++++++----------- 6 files changed, 135 insertions(+), 105 deletions(-) diff --git a/drivers/staging/media/atomisp/pci/hmm/hmm.c b/drivers/staging/media/atomisp/pci/hmm/hmm.c index bb12644fd033..3e2899ad8517 100644 --- a/drivers/staging/media/atomisp/pci/hmm/hmm.c +++ b/drivers/staging/media/atomisp/pci/hmm/hmm.c @@ -205,7 +205,7 @@ static ia_css_ptr __hmm_alloc(size_t bytes, enum hmm_bo_type type, } dev_dbg(atomisp_dev, "pages: 0x%08x (%zu bytes), type: %d, vmalloc %p\n", - bo->start, bytes, type, vmalloc); + bo->start, bytes, type, vmalloc_noprof); return bo->start; diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 91810b4e9510..af97cacdab37 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -2,6 +2,8 @@ #ifndef _LINUX_VMALLOC_H #define _LINUX_VMALLOC_H +#include +#include #include #include #include @@ -137,26 +139,54 @@ extern unsigned long vmalloc_nr_pages(void); static inline unsigned long vmalloc_nr_pages(void) { return 0; } #endif -extern void *vmalloc(unsigned long size) __alloc_size(1); -extern void *vzalloc(unsigned long size) __alloc_size(1); -extern void *vmalloc_user(unsigned long size) __alloc_size(1); -extern void *vmalloc_node(unsigned long size, int node) __alloc_size(1); -extern void *vzalloc_node(unsigned long size, int node) __alloc_size(1); -extern void *vmalloc_32(unsigned long size) __alloc_size(1); -extern void *vmalloc_32_user(unsigned long size) __alloc_size(1); -extern void *__vmalloc(unsigned long size, gfp_t gfp_mask) __alloc_size(1); -extern void *__vmalloc_node_range(unsigned long size, unsigned long align, +extern void *vmalloc_noprof(unsigned long size) __alloc_size(1); +#define vmalloc(...) alloc_hooks(vmalloc_noprof(__VA_ARGS__)) + +extern void *vzalloc_noprof(unsigned long size) __alloc_size(1); +#define vzalloc(...) alloc_hooks(vzalloc_noprof(__VA_ARGS__)) + +extern void *vmalloc_user_noprof(unsigned long size) __alloc_size(1); +#define vmalloc_user(...) alloc_hooks(vmalloc_user_noprof(__VA_ARGS__)) + +extern void *vmalloc_node_noprof(unsigned long size, int node) __alloc_size(1); +#define vmalloc_node(...) alloc_hooks(vmalloc_node_noprof(__VA_ARGS__)) + +extern void *vzalloc_node_noprof(unsigned long size, int node) __alloc_size(1); +#define vzalloc_node(...) alloc_hooks(vzalloc_node_noprof(__VA_ARGS__)) + +extern void *vmalloc_32_noprof(unsigned long size) __alloc_size(1); +#define vmalloc_32(...) alloc_hooks(vmalloc_32_noprof(__VA_ARGS__)) + +extern void *vmalloc_32_user_noprof(unsigned long size) __alloc_size(1); +#define vmalloc_32_user(...) alloc_hooks(vmalloc_32_user_noprof(__VA_ARGS__)) + +extern void *__vmalloc_noprof(unsigned long size, gfp_t gfp_mask) __alloc_size(1); +#define __vmalloc(...) alloc_hooks(__vmalloc_noprof(__VA_ARGS__)) + +extern void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, unsigned long start, unsigned long end, gfp_t gfp_mask, pgprot_t prot, unsigned long vm_flags, int node, const void *caller) __alloc_size(1); -void *__vmalloc_node(unsigned long size, unsigned long align, gfp_t gfp_mask, +#define __vmalloc_node_range(...) alloc_hooks(__vmalloc_node_range_noprof(__VA_ARGS__)) + +void *__vmalloc_node_noprof(unsigned long size, unsigned long align, gfp_t gfp_mask, int node, const void *caller) __alloc_size(1); -void *vmalloc_huge(unsigned long size, gfp_t gfp_mask) __alloc_size(1); +#define __vmalloc_node(...) alloc_hooks(__vmalloc_node_noprof(__VA_ARGS__)) + +void *vmalloc_huge_noprof(unsigned long size, gfp_t gfp_mask) __alloc_size(1); +#define vmalloc_huge(...) alloc_hooks(vmalloc_huge_noprof(__VA_ARGS__)) + +extern void *__vmalloc_array_noprof(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); +#define __vmalloc_array(...) alloc_hooks(__vmalloc_array_noprof(__VA_ARGS__)) + +extern void *vmalloc_array_noprof(size_t n, size_t size) __alloc_size(1, 2); +#define vmalloc_array(...) alloc_hooks(vmalloc_array_noprof(__VA_ARGS__)) + +extern void *__vcalloc_noprof(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); +#define __vcalloc(...) alloc_hooks(__vcalloc_noprof(__VA_ARGS__)) -extern void *__vmalloc_array(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); -extern void *vmalloc_array(size_t n, size_t size) __alloc_size(1, 2); -extern void *__vcalloc(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); -extern void *vcalloc(size_t n, size_t size) __alloc_size(1, 2); +extern void *vcalloc_noprof(size_t n, size_t size) __alloc_size(1, 2); +#define vcalloc(...) alloc_hooks(vcalloc_noprof(__VA_ARGS__)) extern void vfree(const void *addr); extern void vfree_atomic(const void *addr); diff --git a/kernel/kallsyms_selftest.c b/kernel/kallsyms_selftest.c index 8a689b4ff4f9..2f84896a7bcb 100644 --- a/kernel/kallsyms_selftest.c +++ b/kernel/kallsyms_selftest.c @@ -82,7 +82,7 @@ static struct test_item test_items[] = { ITEM_FUNC(kallsyms_test_func_static), ITEM_FUNC(kallsyms_test_func), ITEM_FUNC(kallsyms_test_func_weak), - ITEM_FUNC(vmalloc), + ITEM_FUNC(vmalloc_noprof), ITEM_FUNC(vfree), #ifdef CONFIG_KALLSYMS_ALL ITEM_DATA(kallsyms_test_var_bss_static), diff --git a/mm/nommu.c b/mm/nommu.c index 5ec8f44e7ce9..69a6f3b4d156 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -137,28 +137,28 @@ void vfree(const void *addr) } EXPORT_SYMBOL(vfree); -void *__vmalloc(unsigned long size, gfp_t gfp_mask) +void *__vmalloc_noprof(unsigned long size, gfp_t gfp_mask) { /* * You can't specify __GFP_HIGHMEM with kmalloc() since kmalloc() * returns only a logical address. */ - return kmalloc(size, (gfp_mask | __GFP_COMP) & ~__GFP_HIGHMEM); + return kmalloc_noprof(size, (gfp_mask | __GFP_COMP) & ~__GFP_HIGHMEM); } -EXPORT_SYMBOL(__vmalloc); +EXPORT_SYMBOL(__vmalloc_noprof); -void *__vmalloc_node_range(unsigned long size, unsigned long align, +void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, unsigned long start, unsigned long end, gfp_t gfp_mask, pgprot_t prot, unsigned long vm_flags, int node, const void *caller) { - return __vmalloc(size, gfp_mask); + return __vmalloc_noprof(size, gfp_mask); } -void *__vmalloc_node(unsigned long size, unsigned long align, gfp_t gfp_mask, +void *__vmalloc_node_noprof(unsigned long size, unsigned long align, gfp_t gfp_mask, int node, const void *caller) { - return __vmalloc(size, gfp_mask); + return __vmalloc_noprof(size, gfp_mask); } static void *__vmalloc_user_flags(unsigned long size, gfp_t flags) @@ -179,11 +179,11 @@ static void *__vmalloc_user_flags(unsigned long size, gfp_t flags) return ret; } -void *vmalloc_user(unsigned long size) +void *vmalloc_user_noprof(unsigned long size) { return __vmalloc_user_flags(size, GFP_KERNEL | __GFP_ZERO); } -EXPORT_SYMBOL(vmalloc_user); +EXPORT_SYMBOL(vmalloc_user_noprof); struct page *vmalloc_to_page(const void *addr) { @@ -217,13 +217,13 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) * For tight control over page level allocator and protection flags * use __vmalloc() instead. */ -void *vmalloc(unsigned long size) +void *vmalloc_noprof(unsigned long size) { - return __vmalloc(size, GFP_KERNEL); + return __vmalloc_noprof(size, GFP_KERNEL); } -EXPORT_SYMBOL(vmalloc); +EXPORT_SYMBOL(vmalloc_noprof); -void *vmalloc_huge(unsigned long size, gfp_t gfp_mask) __weak __alias(__vmalloc); +void *vmalloc_huge_noprof(unsigned long size, gfp_t gfp_mask) __weak __alias(__vmalloc_noprof); /* * vzalloc - allocate virtually contiguous memory with zero fill @@ -237,14 +237,14 @@ void *vmalloc_huge(unsigned long size, gfp_t gfp_mask) __weak __alias(__vmalloc) * For tight control over page level allocator and protection flags * use __vmalloc() instead. */ -void *vzalloc(unsigned long size) +void *vzalloc_noprof(unsigned long size) { - return __vmalloc(size, GFP_KERNEL | __GFP_ZERO); + return __vmalloc_noprof(size, GFP_KERNEL | __GFP_ZERO); } -EXPORT_SYMBOL(vzalloc); +EXPORT_SYMBOL(vzalloc_noprof); /** - * vmalloc_node - allocate memory on a specific node + * vmalloc_node_noprof - allocate memory on a specific node * @size: allocation size * @node: numa node * @@ -254,14 +254,14 @@ EXPORT_SYMBOL(vzalloc); * For tight control over page level allocator and protection flags * use __vmalloc() instead. */ -void *vmalloc_node(unsigned long size, int node) +void *vmalloc_node_noprof(unsigned long size, int node) { - return vmalloc(size); + return vmalloc_noprof(size); } -EXPORT_SYMBOL(vmalloc_node); +EXPORT_SYMBOL(vmalloc_node_noprof); /** - * vzalloc_node - allocate memory on a specific node with zero fill + * vzalloc_node_noprof - allocate memory on a specific node with zero fill * @size: allocation size * @node: numa node * @@ -272,27 +272,27 @@ EXPORT_SYMBOL(vmalloc_node); * For tight control over page level allocator and protection flags * use __vmalloc() instead. */ -void *vzalloc_node(unsigned long size, int node) +void *vzalloc_node_noprof(unsigned long size, int node) { - return vzalloc(size); + return vzalloc_noprof(size); } -EXPORT_SYMBOL(vzalloc_node); +EXPORT_SYMBOL(vzalloc_node_noprof); /** - * vmalloc_32 - allocate virtually contiguous memory (32bit addressable) + * vmalloc_32_noprof - allocate virtually contiguous memory (32bit addressable) * @size: allocation size * * Allocate enough 32bit PA addressable pages to cover @size from the * page level allocator and map them into contiguous kernel virtual space. */ -void *vmalloc_32(unsigned long size) +void *vmalloc_32_noprof(unsigned long size) { - return __vmalloc(size, GFP_KERNEL); + return __vmalloc_noprof(size, GFP_KERNEL); } -EXPORT_SYMBOL(vmalloc_32); +EXPORT_SYMBOL(vmalloc_32_noprof); /** - * vmalloc_32_user - allocate zeroed virtually contiguous 32bit memory + * vmalloc_32_user_noprof - allocate zeroed virtually contiguous 32bit memory * @size: allocation size * * The resulting memory area is 32bit addressable and zeroed so it can be @@ -301,15 +301,15 @@ EXPORT_SYMBOL(vmalloc_32); * VM_USERMAP is set on the corresponding VMA so that subsequent calls to * remap_vmalloc_range() are permissible. */ -void *vmalloc_32_user(unsigned long size) +void *vmalloc_32_user_noprof(unsigned long size) { /* * We'll have to sort out the ZONE_DMA bits for 64-bit, * but for now this can simply use vmalloc_user() directly. */ - return vmalloc_user(size); + return vmalloc_user_noprof(size); } -EXPORT_SYMBOL(vmalloc_32_user); +EXPORT_SYMBOL(vmalloc_32_user_noprof); void *vmap(struct page **pages, unsigned int count, unsigned long flags, pgprot_t prot) { diff --git a/mm/util.c b/mm/util.c index 9b8774d8dabb..cb79cf4def6b 100644 --- a/mm/util.c +++ b/mm/util.c @@ -639,7 +639,7 @@ void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) * about the resulting pointer, and cannot play * protection games. */ - return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, 1, VMALLOC_START, VMALLOC_END, flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, node, __builtin_return_address(0)); } @@ -698,12 +698,12 @@ void *kvrealloc_noprof(const void *p, size_t oldsize, size_t newsize, gfp_t flag EXPORT_SYMBOL(kvrealloc_noprof); /** - * __vmalloc_array - allocate memory for a virtually contiguous array. + * __vmalloc_array_noprof - allocate memory for a virtually contiguous array. * @n: number of elements. * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -void *__vmalloc_array(size_t n, size_t size, gfp_t flags) +void *__vmalloc_array_noprof(size_t n, size_t size, gfp_t flags) { size_t bytes; @@ -711,18 +711,18 @@ void *__vmalloc_array(size_t n, size_t size, gfp_t flags) return NULL; return __vmalloc(bytes, flags); } -EXPORT_SYMBOL(__vmalloc_array); +EXPORT_SYMBOL(__vmalloc_array_noprof); /** - * vmalloc_array - allocate memory for a virtually contiguous array. + * vmalloc_array_noprof - allocate memory for a virtually contiguous array. * @n: number of elements. * @size: element size. */ -void *vmalloc_array(size_t n, size_t size) +void *vmalloc_array_noprof(size_t n, size_t size) { return __vmalloc_array(n, size, GFP_KERNEL); } -EXPORT_SYMBOL(vmalloc_array); +EXPORT_SYMBOL(vmalloc_array_noprof); /** * __vcalloc - allocate and zero memory for a virtually contiguous array. @@ -730,22 +730,22 @@ EXPORT_SYMBOL(vmalloc_array); * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -void *__vcalloc(size_t n, size_t size, gfp_t flags) +void *__vcalloc_noprof(size_t n, size_t size, gfp_t flags) { return __vmalloc_array(n, size, flags | __GFP_ZERO); } -EXPORT_SYMBOL(__vcalloc); +EXPORT_SYMBOL(__vcalloc_noprof); /** - * vcalloc - allocate and zero memory for a virtually contiguous array. + * vcalloc_noprof - allocate and zero memory for a virtually contiguous array. * @n: number of elements. * @size: element size. */ -void *vcalloc(size_t n, size_t size) +void *vcalloc_noprof(size_t n, size_t size) { return __vmalloc_array(n, size, GFP_KERNEL | __GFP_ZERO); } -EXPORT_SYMBOL(vcalloc); +EXPORT_SYMBOL(vcalloc_noprof); struct anon_vma *folio_anon_vma(struct folio *folio) { diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 25a8df497255..12caa794abd4 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3435,12 +3435,12 @@ vm_area_alloc_pages(gfp_t gfp, int nid, * but mempolicy wants to alloc memory by interleaving. */ if (IS_ENABLED(CONFIG_NUMA) && nid == NUMA_NO_NODE) - nr = alloc_pages_bulk_array_mempolicy(bulk_gfp, + nr = alloc_pages_bulk_array_mempolicy_noprof(bulk_gfp, nr_pages_request, pages + nr_allocated); else - nr = alloc_pages_bulk_array_node(bulk_gfp, nid, + nr = alloc_pages_bulk_array_node_noprof(bulk_gfp, nid, nr_pages_request, pages + nr_allocated); @@ -3470,9 +3470,9 @@ vm_area_alloc_pages(gfp_t gfp, int nid, break; if (nid == NUMA_NO_NODE) - page = alloc_pages(alloc_gfp, order); + page = alloc_pages_noprof(alloc_gfp, order); else - page = alloc_pages_node(nid, alloc_gfp, order); + page = alloc_pages_node_noprof(nid, alloc_gfp, order); if (unlikely(!page)) { if (!nofail) break; @@ -3529,10 +3529,10 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, /* Please note that the recursion is strictly bounded. */ if (array_size > PAGE_SIZE) { - area->pages = __vmalloc_node(array_size, 1, nested_gfp, node, + area->pages = __vmalloc_node_noprof(array_size, 1, nested_gfp, node, area->caller); } else { - area->pages = kmalloc_node(array_size, nested_gfp, node); + area->pages = kmalloc_node_noprof(array_size, nested_gfp, node); } if (!area->pages) { @@ -3615,7 +3615,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, } /** - * __vmalloc_node_range - allocate virtually contiguous memory + * __vmalloc_node_range_noprof - allocate virtually contiguous memory * @size: allocation size * @align: desired alignment * @start: vm area range start @@ -3642,7 +3642,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, * * Return: the address of the area or %NULL on failure */ -void *__vmalloc_node_range(unsigned long size, unsigned long align, +void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, unsigned long start, unsigned long end, gfp_t gfp_mask, pgprot_t prot, unsigned long vm_flags, int node, const void *caller) @@ -3771,7 +3771,7 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, } /** - * __vmalloc_node - allocate virtually contiguous memory + * __vmalloc_node_noprof - allocate virtually contiguous memory * @size: allocation size * @align: desired alignment * @gfp_mask: flags for the page level allocator @@ -3789,10 +3789,10 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, * * Return: pointer to the allocated memory or %NULL on error */ -void *__vmalloc_node(unsigned long size, unsigned long align, +void *__vmalloc_node_noprof(unsigned long size, unsigned long align, gfp_t gfp_mask, int node, const void *caller) { - return __vmalloc_node_range(size, align, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, align, VMALLOC_START, VMALLOC_END, gfp_mask, PAGE_KERNEL, 0, node, caller); } /* @@ -3801,15 +3801,15 @@ void *__vmalloc_node(unsigned long size, unsigned long align, * than that. */ #ifdef CONFIG_TEST_VMALLOC_MODULE -EXPORT_SYMBOL_GPL(__vmalloc_node); +EXPORT_SYMBOL_GPL(__vmalloc_node_noprof); #endif -void *__vmalloc(unsigned long size, gfp_t gfp_mask) +void *__vmalloc_noprof(unsigned long size, gfp_t gfp_mask) { - return __vmalloc_node(size, 1, gfp_mask, NUMA_NO_NODE, + return __vmalloc_node_noprof(size, 1, gfp_mask, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(__vmalloc); +EXPORT_SYMBOL(__vmalloc_noprof); /** * vmalloc - allocate virtually contiguous memory @@ -3823,12 +3823,12 @@ EXPORT_SYMBOL(__vmalloc); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc(unsigned long size) +void *vmalloc_noprof(unsigned long size) { - return __vmalloc_node(size, 1, GFP_KERNEL, NUMA_NO_NODE, + return __vmalloc_node_noprof(size, 1, GFP_KERNEL, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc); +EXPORT_SYMBOL(vmalloc_noprof); /** * vmalloc_huge - allocate virtually contiguous memory, allow huge pages @@ -3842,16 +3842,16 @@ EXPORT_SYMBOL(vmalloc); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_huge(unsigned long size, gfp_t gfp_mask) +void *vmalloc_huge_noprof(unsigned long size, gfp_t gfp_mask) { - return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, 1, VMALLOC_START, VMALLOC_END, gfp_mask, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL_GPL(vmalloc_huge); +EXPORT_SYMBOL_GPL(vmalloc_huge_noprof); /** - * vzalloc - allocate virtually contiguous memory with zero fill + * vzalloc_noprof - allocate virtually contiguous memory with zero fill * @size: allocation size * * Allocate enough pages to cover @size from the page level @@ -3863,12 +3863,12 @@ EXPORT_SYMBOL_GPL(vmalloc_huge); * * Return: pointer to the allocated memory or %NULL on error */ -void *vzalloc(unsigned long size) +void *vzalloc_noprof(unsigned long size) { - return __vmalloc_node(size, 1, GFP_KERNEL | __GFP_ZERO, NUMA_NO_NODE, + return __vmalloc_node_noprof(size, 1, GFP_KERNEL | __GFP_ZERO, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vzalloc); +EXPORT_SYMBOL(vzalloc_noprof); /** * vmalloc_user - allocate zeroed virtually contiguous memory for userspace @@ -3879,17 +3879,17 @@ EXPORT_SYMBOL(vzalloc); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_user(unsigned long size) +void *vmalloc_user_noprof(unsigned long size) { - return __vmalloc_node_range(size, SHMLBA, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, SHMLBA, VMALLOC_START, VMALLOC_END, GFP_KERNEL | __GFP_ZERO, PAGE_KERNEL, VM_USERMAP, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_user); +EXPORT_SYMBOL(vmalloc_user_noprof); /** - * vmalloc_node - allocate memory on a specific node + * vmalloc_node_noprof - allocate memory on a specific node * @size: allocation size * @node: numa node * @@ -3901,15 +3901,15 @@ EXPORT_SYMBOL(vmalloc_user); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_node(unsigned long size, int node) +void *vmalloc_node_noprof(unsigned long size, int node) { - return __vmalloc_node(size, 1, GFP_KERNEL, node, + return __vmalloc_node_noprof(size, 1, GFP_KERNEL, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_node); +EXPORT_SYMBOL(vmalloc_node_noprof); /** - * vzalloc_node - allocate memory on a specific node with zero fill + * vzalloc_node_noprof - allocate memory on a specific node with zero fill * @size: allocation size * @node: numa node * @@ -3919,12 +3919,12 @@ EXPORT_SYMBOL(vmalloc_node); * * Return: pointer to the allocated memory or %NULL on error */ -void *vzalloc_node(unsigned long size, int node) +void *vzalloc_node_noprof(unsigned long size, int node) { - return __vmalloc_node(size, 1, GFP_KERNEL | __GFP_ZERO, node, + return __vmalloc_node_noprof(size, 1, GFP_KERNEL | __GFP_ZERO, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(vzalloc_node); +EXPORT_SYMBOL(vzalloc_node_noprof); #if defined(CONFIG_64BIT) && defined(CONFIG_ZONE_DMA32) #define GFP_VMALLOC32 (GFP_DMA32 | GFP_KERNEL) @@ -3939,7 +3939,7 @@ EXPORT_SYMBOL(vzalloc_node); #endif /** - * vmalloc_32 - allocate virtually contiguous memory (32bit addressable) + * vmalloc_32_noprof - allocate virtually contiguous memory (32bit addressable) * @size: allocation size * * Allocate enough 32bit PA addressable pages to cover @size from the @@ -3947,15 +3947,15 @@ EXPORT_SYMBOL(vzalloc_node); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_32(unsigned long size) +void *vmalloc_32_noprof(unsigned long size) { - return __vmalloc_node(size, 1, GFP_VMALLOC32, NUMA_NO_NODE, + return __vmalloc_node_noprof(size, 1, GFP_VMALLOC32, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_32); +EXPORT_SYMBOL(vmalloc_32_noprof); /** - * vmalloc_32_user - allocate zeroed virtually contiguous 32bit memory + * vmalloc_32_user_noprof - allocate zeroed virtually contiguous 32bit memory * @size: allocation size * * The resulting memory area is 32bit addressable and zeroed so it can be @@ -3963,14 +3963,14 @@ EXPORT_SYMBOL(vmalloc_32); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_32_user(unsigned long size) +void *vmalloc_32_user_noprof(unsigned long size) { - return __vmalloc_node_range(size, SHMLBA, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, SHMLBA, VMALLOC_START, VMALLOC_END, GFP_VMALLOC32 | __GFP_ZERO, PAGE_KERNEL, VM_USERMAP, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_32_user); +EXPORT_SYMBOL(vmalloc_32_user_noprof); /* * Atomically zero bytes in the iterator. From patchwork Wed Mar 6 18:24:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584441 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7CBCC5475B for ; Wed, 6 Mar 2024 18:26:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1665A6B00BF; Wed, 6 Mar 2024 13:25:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0EB6B6B00C0; Wed, 6 Mar 2024 13:25:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E0B9A6B00C1; Wed, 6 Mar 2024 13:25:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CB8646B00BF for ; Wed, 6 Mar 2024 13:25:54 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A41198034A for ; Wed, 6 Mar 2024 18:25:54 +0000 (UTC) X-FDA: 81867443028.29.412433F Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf08.hostedemail.com (Postfix) with ESMTP id A8B4E160019 for ; Wed, 6 Mar 2024 18:25:52 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="hr/ZeLQa"; spf=pass (imf08.hostedemail.com: domain of 3L7XoZQYKCHIikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3L7XoZQYKCHIikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749552; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Fu+QokTCwH9vHgRIOuSmN9iHLTBbRC7AqqeSJz2bhTE=; b=I6Df4JcQ/bui/Ner+A5HuYysH5OkGkW6e6UJg1dSQMD3bf8Sy8UhirMwHf3w5E9Ve9ZRKi mEo/6KwkTxRnuAXI0EEpT4BRAbCipF9apEPDT/tRVCVJkQ5CLc4eknGq5n9lU4HVVNF9Mt 06gvC0avSSHbz00xQHqzR7ZcEs5Rk40= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749552; a=rsa-sha256; cv=none; b=lkkIZ3sXvmqwUymiIdZdCHX90ZjuuOgkJZbkIcWTVs5Lg2JVJfYxNnxnppc6ysxmOAgSD2 tW3A3ZH2VGgvr/jYG8HtubtKW/5esQDfbluHoHHZazN3wHUwrCSJxPpbucYTCFJpf40rts 9bFt7xrfi1CXdNQa8J3+vR4/5zLO8yg= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="hr/ZeLQa"; spf=pass (imf08.hostedemail.com: domain of 3L7XoZQYKCHIikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3L7XoZQYKCHIikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-609dc04bff0so209447b3.2 for ; Wed, 06 Mar 2024 10:25:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749552; x=1710354352; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Fu+QokTCwH9vHgRIOuSmN9iHLTBbRC7AqqeSJz2bhTE=; b=hr/ZeLQa/vSnFJKBBYMXXuDGBYJf47IIwLTkqHfrEprGm19UgmNv/JxP11kc+y/BUs VQFnV2JxgaN5upWc15abzaC+u411GaRQ9DKYOBG/nWox98PKPHTR5c7CvlKkTD3GNVjB A9QzfyD604/6mSvRk/3HKvfysCF00WARl7EC2ntzawMjeKurszrNkE4ldASlaVxhn6us 9CfEOuKiEcoa5o+Fs62vOzdi26UPU8aWXiLlgKbDEiWnlq5XUobXX2xP8HGKUa+r186r 31oWjnz0aboVDVhpC10mxsNnQcLET1BPZ9/jeIIoVFjJJSaR87/jO+eX2mOJalb1PKSg n9AQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749552; x=1710354352; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Fu+QokTCwH9vHgRIOuSmN9iHLTBbRC7AqqeSJz2bhTE=; b=IrDT1OvhTQz51PUz6bIHPKfWboQsU/vH+AxfqDHHIdJ1x33JMPYSy0VPukWfvbtKda 53gSwD4QK8qZTdfwsyAzLqyt+UrDtsjv011Wtda10pKE5qUvTLV6e2dghTwz13zBcGw1 0/3EXKUoGSP6Y16brf/2oMS8hc1pEnLOtXHAePWhHY7hGIcrK39DISt7SKj7agfevKr3 9UrtCqVaYQiIkBMeDIycWYQubn5g/qz8Jno2m+i8c99ukog9AwkqvTpLUaKsLjI8Sa76 nyhsRK/jLq5u2w/PvnyMl/iPv3InaGxZRzxD46t5zf33ad3zZCTQ95Pb3on8871MrrCa 0PMw== X-Forwarded-Encrypted: i=1; AJvYcCX3lCnhnG5JX/BZnmN7vh1yC0gargeOBlrjGKuX/eH9514q6v3q6Ms1J5ILERMPMrKPK8TrtwhmrBgUdVkREcrV+0w= X-Gm-Message-State: AOJu0YxJNJZ+y0ePzJFJF+FU/BhUwGFy3X3qeClgr3je9rmViTZmKTwf 17NNxVahuv8bAAhbihhOECEtBPs6pLB7pDljVdLsWC6h9x4fxePriDpTZ+LYg0vfXiWXV3E3mcV Amw== X-Google-Smtp-Source: AGHT+IH8uTiw8GF5vZKvpNdwUjiYw3f4d0sxZhsV7lTjdNn/jH/CQVsQaOx2OIyhhvAH+kWCwcf8uXZ95X4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:690c:c02:b0:608:66be:2f71 with SMTP id cl2-20020a05690c0c0200b0060866be2f71mr3512997ywb.9.1709749551693; Wed, 06 Mar 2024 10:25:51 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:29 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-32-surenb@google.com> Subject: [PATCH v5 31/37] rhashtable: Plumb through alloc tag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: A8B4E160019 X-Rspam-User: X-Stat-Signature: m59gh4yufder97ssxzdzjyb96eaxtp5m X-Rspamd-Server: rspam03 X-HE-Tag: 1709749552-312831 X-HE-Meta: U2FsdGVkX1/iYCHBEif3KjEAYOTf8tqvchV1ZtLPwCRlJmUR3umScYel2kt9LNowM4ggk2mzvIvdyUhsFEPsndBwVnJ2ZYuhnuLfy4cBLL6N7G8cdCkVuOdn1Ah3KzJLndVqCoRpoQ09BpARC8UFPyt3bg2XNDVRb74C6+hRVQivIvt3EtqGFMkUD3yst1tEjLdxEld/e2jQBIOThNlZOsrccYRrJJ8QsQIEov7tiFb+03Hl5bcHtTKRnx3x9qLYpq4mn8QaMb8kxQBVtbp66uN5kT9P3bbt0GJxnNOhImIGnheXlkVYsSHsDCgqaM9cCYNEPpiyMZ6wVzN6S8YFoP+NrxhEHR3qtHgwA1iZRmGeMqqWqB9lReNnq3a5ZUyhuedz5XjTXsixdBWnaQtfZ0duSdToKuW8DxOC7xlufn0ydIUvClnMdc+WwrUbJ/tNFVv5lKAlAd1Cnivk3p5VE/JwTNRdG2jpMju09jZXSgLzZgTd9JUcCCvcLLrjkw1fTdPN6d7/rs9lBC5j2JtCJXw7kgSBScelQCIHWQpSk9rCdHLi9lE/VJBXD8B3q+MnLz7uIe6w6o4ZDGSKTz6DuyG1D7WCoTY/cYJfwmMUWfmitnOoY77y8metLKE8bQgUJNHRZARvweynfBlAEkDk0KkhvBb1GBiChOCE05grRxW3Rwg/wZVtb0Io3lHqmPcVNr63H9W5NMbkjBXo1IBLLUuTQJ4VRUtKZda1hbjpOgCmojAytRH12j30klYwqkLL7EhmmUxZVfQPJslFMyBP53swb3xsExEPJgwSC9uGXWrcGzOBNwWEHK2pwC/M0y70YreYV3dOKCiSfjQ3F0hAC2/uQWH0duUBVgZKPQ2YCIJCdgmZl0QqNj1lUEOTjOvM71u83NYXQtnFbqnPTEKBnCUwqgFLjlOY24TH5D96CAOs5C/qQiciWtc0k/cxvlPcYCetwzcqIfADr25QmZm RLZPgjwI eYda9sprX97x0T+e7FZeONxZDClbERW1oSLbkEV4ZZXOd7I4tw1+anBtZcOiSh5vLdaMq6FRDmE6rZGlPBTo2j84Yw8gL1jcvLgVVg1KEjfblV36/FdNezVDRLsg+C2ArZEbsjIZWwesr2ZPib9EZa22Vkf5slYzhmuHqool+UgRwDzl2of6HCBsMlbNBU8K4SpCkdlTEhswd5fOSAuZOmUsCuIBVZyXzL0RpgTz+W0Yrm4ZOY0uEcotGQHyTPpy9w4luVhq3a+/TUIWabamhK3caGXFznL5IzJb4YeEObuN7CSAFS9eWIM0U/ZOPIbPdE0ak6ul4tDh463O93YcqTAGxZR7RxI3aRG+7rkWbSEijwl37BDB3e0kTwa3Xbl7mBFGYj4ToypJ3Fo4MGmp6lE7L4OUzWtKzwPZw X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet This gives better memory allocation profiling results; rhashtable allocations will be accounted to the code that initialized the rhashtable. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 3 +++ include/linux/rhashtable-types.h | 11 +++++++++-- lib/rhashtable.c | 28 +++++++++++++++++----------- 3 files changed, 29 insertions(+), 13 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index bc9b1b99a55b..cf69e037f645 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -141,6 +141,8 @@ static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) ref->ct = NULL; } +#define alloc_tag_record(p) ((p) = current->alloc_tag) + #else /* CONFIG_MEM_ALLOC_PROFILING */ #define DEFINE_ALLOC_TAG(_alloc_tag) @@ -148,6 +150,7 @@ static inline bool mem_alloc_profiling_enabled(void) { return false; } static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, size_t bytes) {} static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) {} +#define alloc_tag_record(p) do {} while (0) #endif /* CONFIG_MEM_ALLOC_PROFILING */ diff --git a/include/linux/rhashtable-types.h b/include/linux/rhashtable-types.h index b6f3797277ff..015c8298bebc 100644 --- a/include/linux/rhashtable-types.h +++ b/include/linux/rhashtable-types.h @@ -9,6 +9,7 @@ #ifndef _LINUX_RHASHTABLE_TYPES_H #define _LINUX_RHASHTABLE_TYPES_H +#include #include #include #include @@ -88,6 +89,9 @@ struct rhashtable { struct mutex mutex; spinlock_t lock; atomic_t nelems; +#ifdef CONFIG_MEM_ALLOC_PROFILING + struct alloc_tag *alloc_tag; +#endif }; /** @@ -127,9 +131,12 @@ struct rhashtable_iter { bool end_of_table; }; -int rhashtable_init(struct rhashtable *ht, +int rhashtable_init_noprof(struct rhashtable *ht, const struct rhashtable_params *params); -int rhltable_init(struct rhltable *hlt, +#define rhashtable_init(...) alloc_hooks(rhashtable_init_noprof(__VA_ARGS__)) + +int rhltable_init_noprof(struct rhltable *hlt, const struct rhashtable_params *params); +#define rhltable_init(...) alloc_hooks(rhltable_init_noprof(__VA_ARGS__)) #endif /* _LINUX_RHASHTABLE_TYPES_H */ diff --git a/lib/rhashtable.c b/lib/rhashtable.c index 6ae2ba8e06a2..35d841cf2b43 100644 --- a/lib/rhashtable.c +++ b/lib/rhashtable.c @@ -130,7 +130,8 @@ static union nested_table *nested_table_alloc(struct rhashtable *ht, if (ntbl) return ntbl; - ntbl = kzalloc(PAGE_SIZE, GFP_ATOMIC); + ntbl = alloc_hooks_tag(ht->alloc_tag, + kmalloc_noprof(PAGE_SIZE, GFP_ATOMIC|__GFP_ZERO)); if (ntbl && leaf) { for (i = 0; i < PAGE_SIZE / sizeof(ntbl[0]); i++) @@ -157,7 +158,8 @@ static struct bucket_table *nested_bucket_table_alloc(struct rhashtable *ht, size = sizeof(*tbl) + sizeof(tbl->buckets[0]); - tbl = kzalloc(size, gfp); + tbl = alloc_hooks_tag(ht->alloc_tag, + kmalloc_noprof(size, gfp|__GFP_ZERO)); if (!tbl) return NULL; @@ -181,7 +183,9 @@ static struct bucket_table *bucket_table_alloc(struct rhashtable *ht, int i; static struct lock_class_key __key; - tbl = kvzalloc(struct_size(tbl, buckets, nbuckets), gfp); + tbl = alloc_hooks_tag(ht->alloc_tag, + kvmalloc_node_noprof(struct_size(tbl, buckets, nbuckets), + gfp|__GFP_ZERO, NUMA_NO_NODE)); size = nbuckets; @@ -975,7 +979,7 @@ static u32 rhashtable_jhash2(const void *key, u32 length, u32 seed) } /** - * rhashtable_init - initialize a new hash table + * rhashtable_init_noprof - initialize a new hash table * @ht: hash table to be initialized * @params: configuration parameters * @@ -1016,7 +1020,7 @@ static u32 rhashtable_jhash2(const void *key, u32 length, u32 seed) * .obj_hashfn = my_hash_fn, * }; */ -int rhashtable_init(struct rhashtable *ht, +int rhashtable_init_noprof(struct rhashtable *ht, const struct rhashtable_params *params) { struct bucket_table *tbl; @@ -1031,6 +1035,8 @@ int rhashtable_init(struct rhashtable *ht, spin_lock_init(&ht->lock); memcpy(&ht->p, params, sizeof(*params)); + alloc_tag_record(ht->alloc_tag); + if (params->min_size) ht->p.min_size = roundup_pow_of_two(params->min_size); @@ -1076,26 +1082,26 @@ int rhashtable_init(struct rhashtable *ht, return 0; } -EXPORT_SYMBOL_GPL(rhashtable_init); +EXPORT_SYMBOL_GPL(rhashtable_init_noprof); /** - * rhltable_init - initialize a new hash list table + * rhltable_init_noprof - initialize a new hash list table * @hlt: hash list table to be initialized * @params: configuration parameters * * Initializes a new hash list table. * - * See documentation for rhashtable_init. + * See documentation for rhashtable_init_noprof. */ -int rhltable_init(struct rhltable *hlt, const struct rhashtable_params *params) +int rhltable_init_noprof(struct rhltable *hlt, const struct rhashtable_params *params) { int err; - err = rhashtable_init(&hlt->ht, params); + err = rhashtable_init_noprof(&hlt->ht, params); hlt->ht.rhlist = true; return err; } -EXPORT_SYMBOL_GPL(rhltable_init); +EXPORT_SYMBOL_GPL(rhltable_init_noprof); static void rhashtable_free_one(struct rhashtable *ht, struct rhash_head *obj, void (*free_fn)(void *ptr, void *arg), From patchwork Wed Mar 6 18:24:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584442 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BF6AC5475B for ; Wed, 6 Mar 2024 18:26:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E34D06B00C0; Wed, 6 Mar 2024 13:25:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DBC1D6B00C1; Wed, 6 Mar 2024 13:25:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BE7096B00C2; Wed, 6 Mar 2024 13:25:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AAE136B00C0 for ; Wed, 6 Mar 2024 13:25:56 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8CE42A065B for ; Wed, 6 Mar 2024 18:25:56 +0000 (UTC) X-FDA: 81867443112.07.07AE75E Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf18.hostedemail.com (Postfix) with ESMTP id DB3E01C001F for ; Wed, 6 Mar 2024 18:25:54 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=HJ0mz2Yd; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf18.hostedemail.com: domain of 3MbXoZQYKCHQkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3MbXoZQYKCHQkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749554; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+wmfS4WzHQ3O9sWr06d7OoGjn7qYNg0oNg7s1VmP4AA=; b=Pj4Id//vrxGX0jDjjgTDyLMikhN3z/Ex6jdY0+eo+9x+xhSILNa8JRGBMuVkzltXTv/LKv PTtnpoudfIgg761aCepWyELjmGNXpdJg0Y4usFPufg9YVDqbNu+6+72JeRNeQbGA64VFmH A3tIc7cC2Xhf1Wj9FT4p/s8BS2WxxF8= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=HJ0mz2Yd; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf18.hostedemail.com: domain of 3MbXoZQYKCHQkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3MbXoZQYKCHQkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749554; a=rsa-sha256; cv=none; b=3jBL/CK1B+IBEY23V++/WIjrmka8TGcMHHA2IAfe38PxMoAa3PRgQdjVc4UoM1GgoRqlP2 +MxPUaPj73Tv0gpwXnijuHViND3dUqV5nyx8Fr6yP2n2c4rpXwobMAv8Cb32lGQqrXlB/h gwyLAnbLbtgcnd7o7DyCMHsqkwFzsEQ= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc3645a6790so4024489276.0 for ; Wed, 06 Mar 2024 10:25:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749554; x=1710354354; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+wmfS4WzHQ3O9sWr06d7OoGjn7qYNg0oNg7s1VmP4AA=; b=HJ0mz2YdYOqaZ9Emxogc+EHgpSA/i3Vz+0gzeGSqk4gbOoY3Ec4JeyK/A19keUicbm KK1oY1wiPMovojCGdSFNYK5LEbnu2DeONwDT+FSTMx0bv6DywY91W/krSvTneB6hYu8T 6VHJFErbVnDSXfykggGhaeURpxMt38dWr4z2DatvEO91OQ8IaVBF/F6a3d+WG+JtEacm t5sh7Z12Qb9fhLgflmAVMy//vIpoazr9PY8Sm52a0CpWBIdBAcwIpOS8+Ln0ZXruvuFK 8qOH0IM9feODs4MmOjpINF8OWlryDx2tvCX0BzQF8iUN+xgWGGpj0ft3G3JP5R4Wl2Oo B0bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749554; x=1710354354; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+wmfS4WzHQ3O9sWr06d7OoGjn7qYNg0oNg7s1VmP4AA=; b=VwCbyBveBz/lR83kt9/MI89cd1WgJI/rVoqTVUwn0ycKmKqPwzHpyFbbC/bQDp1YRW jLrXFgIQ2759fvKPtGKEV3tmYoNiy9mzOsmjnW8hISusViQjwqUdvElGKutt5DT7bg9V pt+5+GYvn3FBDwObkcK43TeZLnVc4mj+sZjsk6m3JKXidHRNUGmht//IZtCBANrFDAyk A0nqfQsHUntpWUitQPR83JerCaDW51U7kQ6s+QCPt5u7kwN1OTq7jNq+10eydBK8qbnV rIVMKD3zNIevPbsEX/1rPU2kcgc2Apj8RGkEUYSAa58ZAT9TYj2HCwmCNBC+lI/FJENI kQJg== X-Forwarded-Encrypted: i=1; AJvYcCVYqfQMYO1RzbO+i00ApGef3PUNXOkdhhl86AjASuCap7+ORgMdz4Nj2tT020sNi0XauPHljRyexADhEPZ1aOrA0SY= X-Gm-Message-State: AOJu0Ywjtn8lmsXCGOdAlIYRia6Fgi/hDeBT+gOSu/40czD4SuHfgEVD iAMv84Pk9x+cUuSJGRy5IOdguKT1Ieh3ROGM5w/xrcNVaR+0dZWXcT7qYI2Lf4IdKhCn+69WIaV pPg== X-Google-Smtp-Source: AGHT+IF6YEyRmq18pPuKKpt59wbNGO1QMY+fEEbCCtJ5tUZChXHjWclQVIOgZ7rujqj4/I2Mqr5y06vHSeM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:150d:b0:dc6:fa35:b42 with SMTP id q13-20020a056902150d00b00dc6fa350b42mr4231906ybu.2.1709749553765; Wed, 06 Mar 2024 10:25:53 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:30 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-33-surenb@google.com> Subject: [PATCH v5 32/37] lib: add memory allocations report in show_mem() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: DB3E01C001F X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: zkg5fppgu5fecgc8tn8iru63hidssq6y X-HE-Tag: 1709749554-754731 X-HE-Meta: U2FsdGVkX19M+gvpn95aCY+xLIF5ZO+wLPGXcUj6jiDL5po3eeZwZrTIlop9IdEf+xvm3VlPvvjFy8GwPfgM1tP+yoIunQUbA004jrJaM3NhpOJmqJiInEioRe9+MnVEGG5eHtVFVTbV5DHeC8P/SiVJREk6C1lyiQBcVKAQ46P/4HgH8OoQ0PgG4Pm+jDA01vz+SzpomrknJCTz1ozpeCEEAgH3Ge1gJhe0u9RK66AUvVzA0KaLuS4ybk/MRjin9c/ldtf1MsHqk6axoKDIf1bXnemN7RV2YEvQS+PbzUJCPGaGDs64XeUwf/+JT27Tl47FS2gLQiFBtZZRfUtktgfVD0KLlTCYWb5jK3i1q/fcXR+BrlisvemPUTho1OM5mxCXO0sMd66/+UFwGAZZzAAz5SQZZllgHhhJ6bTp9nWqubFglmyavCNEH+kWUsasL+IuRjuWaznxLV/7CjWZSfAA5/l83frIN4SGfGbHFSILAfR1bq9qlmpqVKa9PHjjfF7+7IlSfVMgvtn8g+JfYa+vUg3KMatmbngZfNSFeuxFd68Gy3tScggpL2dP69LPkOnBfgWOuO2aUzonpGe2qs1hvOfjvqtvIPn4s6yyFJ27bOiTwuPL22qLvbhuf5EzpY263b1SNQ6XuAORw5uyuvRk45oPfYPAc8U6qHW7W9rb3/abg0Pz9PYIF5JG54TQtrmzCxjjsZYZZs7OmVQgmJu0M83g1k+wZp7TeZFiyC7mKorFvhhLpwqh669/tXKiWEFz01MPgyOd6ZA8LdD0+WsClT81DsGsnqXl0SKOLeaGLbkcX4PbNkNbq3MU4Pj0IO9B7BXUzLnSrWZ31itkavHpKmU0JcuK/EDFbODk32WhcXxHT3e22Mw+L69k4irwwqlAyL1yhKL+hKOqUcVGENfLkgHnIEhrHCJMaDTrKZwdEZawSR0JMbBbZoDlnRFgVn2hxnT194rRN4nP8nE RzEQyj6R AIdckOm/Efld3dkrmfcWfGY1S/wljExeA1yRHv2o0iunNFk/gF0p/VfqmUZ9LpU7NkG8y0SdjDIlc+tUtM6jsL182Jq9SnAkWIg970V4DAFcpZiPYmSwxDcDflCKts24H7Un+h5qsDCCt1mMaTns0voqnfTMQhI2ABPxyO6KrRfOhJoiv5EZ0WIhv6m5t3u8OC6kcLpFBDALbOqtXUDQoMzxb2KnCeKCbuB/TU+gsx6DHnatJYHIPAVqvuk2sNV1ikJ2jgLNQir6H58Szbbx1AggSv1nv77M5qgzdRUG+b6tSyOD6/yqAbU2rjVyBnXhF5zIcf5mDULY+/Aq3D0uU7GcQEleVFL78zDuFGy8NxXHXNMD08onuaFAAC5Xu+/80/hTYWqYQOlYzmijh7P/YwxnKIn03/v7VTuC+MgXGGDWEyAQAhW8oErYaxc7p3juFmBx/VqAUdibps58eTYwYszvGug== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Include allocations in show_mem reports. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka --- include/linux/alloc_tag.h | 7 +++++++ include/linux/codetag.h | 1 + lib/alloc_tag.c | 38 ++++++++++++++++++++++++++++++++++++++ lib/codetag.c | 5 +++++ mm/show_mem.c | 26 ++++++++++++++++++++++++++ 5 files changed, 77 insertions(+) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index cf69e037f645..aefe3c81a1e3 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -30,6 +30,13 @@ struct alloc_tag { #ifdef CONFIG_MEM_ALLOC_PROFILING +struct codetag_bytes { + struct codetag *ct; + s64 bytes; +}; + +size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sleep); + static inline struct alloc_tag *ct_to_alloc_tag(struct codetag *ct) { return container_of(ct, struct alloc_tag, ct); diff --git a/include/linux/codetag.h b/include/linux/codetag.h index bfd0ba5c4185..c2a579ccd455 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -61,6 +61,7 @@ struct codetag_iterator { } void codetag_lock_module_list(struct codetag_type *cttype, bool lock); +bool codetag_trylock_module_list(struct codetag_type *cttype); struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype); struct codetag *codetag_next_ct(struct codetag_iterator *iter); diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 617c2fbb6673..e24830c44783 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -86,6 +86,44 @@ static const struct seq_operations allocinfo_seq_op = { .show = allocinfo_show, }; +size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sleep) +{ + struct codetag_iterator iter; + struct codetag *ct; + struct codetag_bytes n; + unsigned int i, nr = 0; + + if (can_sleep) + codetag_lock_module_list(alloc_tag_cttype, true); + else if (!codetag_trylock_module_list(alloc_tag_cttype)) + return 0; + + iter = codetag_get_ct_iter(alloc_tag_cttype); + while ((ct = codetag_next_ct(&iter))) { + struct alloc_tag_counters counter = alloc_tag_read(ct_to_alloc_tag(ct)); + + n.ct = ct; + n.bytes = counter.bytes; + + for (i = 0; i < nr; i++) + if (n.bytes > tags[i].bytes) + break; + + if (i < count) { + nr -= nr == count; + memmove(&tags[i + 1], + &tags[i], + sizeof(tags[0]) * (nr - i)); + nr++; + tags[i] = n; + } + } + + codetag_lock_module_list(alloc_tag_cttype, false); + + return nr; +} + static void __init procfs_init(void) { proc_create_seq("allocinfo", 0444, NULL, &allocinfo_seq_op); diff --git a/lib/codetag.c b/lib/codetag.c index 408062f722ce..5ace625f2328 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -36,6 +36,11 @@ void codetag_lock_module_list(struct codetag_type *cttype, bool lock) up_read(&cttype->mod_lock); } +bool codetag_trylock_module_list(struct codetag_type *cttype) +{ + return down_read_trylock(&cttype->mod_lock) != 0; +} + struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype) { struct codetag_iterator iter = { diff --git a/mm/show_mem.c b/mm/show_mem.c index 8dcfafbd283c..bdb439551eef 100644 --- a/mm/show_mem.c +++ b/mm/show_mem.c @@ -423,4 +423,30 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx) #ifdef CONFIG_MEMORY_FAILURE printk("%lu pages hwpoisoned\n", atomic_long_read(&num_poisoned_pages)); #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + { + struct codetag_bytes tags[10]; + size_t i, nr; + + nr = alloc_tag_top_users(tags, ARRAY_SIZE(tags), false); + if (nr) { + pr_notice("Memory allocations:\n"); + for (i = 0; i < nr; i++) { + struct codetag *ct = tags[i].ct; + struct alloc_tag *tag = ct_to_alloc_tag(ct); + struct alloc_tag_counters counter = alloc_tag_read(tag); + + /* Same as alloc_tag_to_text() but w/o intermediate buffer */ + if (ct->modname) + pr_notice("%12lli %8llu %s:%u [%s] func:%s\n", + counter.bytes, counter.calls, ct->filename, + ct->lineno, ct->modname, ct->function); + else + pr_notice("%12lli %8llu %s:%u func:%s\n", + counter.bytes, counter.calls, ct->filename, + ct->lineno, ct->function); + } + } + } +#endif } From patchwork Wed Mar 6 18:24:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20D24C5475B for ; Wed, 6 Mar 2024 18:26:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C76B6B00C2; Wed, 6 Mar 2024 13:25:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 04E9D6B00C3; Wed, 6 Mar 2024 13:25:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D94A36B00C4; Wed, 6 Mar 2024 13:25:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C22246B00C2 for ; Wed, 6 Mar 2024 13:25:58 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8B492C047F for ; Wed, 6 Mar 2024 18:25:58 +0000 (UTC) X-FDA: 81867443196.12.4C289A1 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf28.hostedemail.com (Postfix) with ESMTP id B272EC0025 for ; Wed, 6 Mar 2024 18:25:56 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nYTTodnv; spf=pass (imf28.hostedemail.com: domain of 3M7XoZQYKCHYmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3M7XoZQYKCHYmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749556; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=O0frA1qINx6EEpp9W3MZZMQxDt0vbebTSbODwHd6D+c=; b=zOjt3NzaCYntCUpdJVi2vvOqwjKsZXWojhdYO3QhqNKkGAVIL1rMLbKzcAeLHEm2Rv1Quf VSoB/Hw4ml3T9ETq3vvzWAxYgSZpCktJwbxn3oL69GlD23b5IniiY2Ux39ElztUAU1VTur 7Ezfz7KnL5wNL4N5gPVsCzcy0TSQNV0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749556; a=rsa-sha256; cv=none; b=NXA418W0Ck+wFefopsNoIxwJAuK03Ikipx2vY2lowQsElaru82gTfrh6WJI//Yjd2HpJp2 IoX7GBBlDJ3ni2zOXhG5jQd/d1Z3g8cBkUzlWy23ZPtbj7r6/N8THUbZqcOznABIUOXB5J zRkhg5aNNO46J3leDl+SyWNWXkRM5jQ= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nYTTodnv; spf=pass (imf28.hostedemail.com: domain of 3M7XoZQYKCHYmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3M7XoZQYKCHYmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcdc3db67f0so1397213276.1 for ; Wed, 06 Mar 2024 10:25:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749556; x=1710354356; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=O0frA1qINx6EEpp9W3MZZMQxDt0vbebTSbODwHd6D+c=; b=nYTTodnvv3RLfsOXP1eptbUgLp23T+sZ8zRsZI9YvCfdfCyUbjVfxkxiY57jQlOteJ /HQvHtYXF/zgtTYkA+kzIg/O8OmbDVnd5UufsWrttJkK2kjA3q1EomQ6KKOHrGoqpB+Y ypcpdbpbwcrTjm2dnbMdkKnNdJvAbTLX9cXEkG3St3cJAkJH3sCuOPlPG7VfBnJRxi2a NDLmSg35nIaMObGElwzOUVX5vvNsJSxil5BmciGcoBnWHcQBZTho8RfzIPuQzp+MyROt R3FXQJUE3YFAzCQsRjAcOgVAaAspiAjHNT/DPvvo8taQUt594LfupkkKzRwLGjSRsQy8 Mriw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749556; x=1710354356; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=O0frA1qINx6EEpp9W3MZZMQxDt0vbebTSbODwHd6D+c=; b=jPQPVloFdd9x4glvrldFxRtqZt3jaGQ2tI4m/t0l7Bi5bZz2ZVYQa/dKjurQfT+9Hy Sb9OTbis6wqiczuFSB+rMnFbZMTUB/4am7ZcnsZIcgmlhDjweFLbfuNKRC1mwRxspR2O 98lSDRtLTOtIid1uBw6eeXj+EFqTb85FBCM+kLEQDb3ZmU9Aty2FNQo7OVDms+Y8txsb XKjsI1zfyJyTsIjOwtQ2P6c7H+MO8w6xshRQnHcSdGQv68pbdUD8Pp6f7Z7y9aZa8Q5I lEww3cTT6sUqYssmhAuAaEGua/prvT3i7M0K2jnDLE+3WRgBcbUq5Waps5l589QquLVF UXYw== X-Forwarded-Encrypted: i=1; AJvYcCXC/rC58CZBOBgzdTMpHHSDFZ2uQhuV3+kZ8oBMFazpnoVNQ0DH55Q1YaI+iUbTl+5kJBRCDtyKljvUN0jIfO/69vY= X-Gm-Message-State: AOJu0YxQ7gzKBn3urJV2dZt3B8CImfM0CgROFnOsdex8npGUaailCIff Dw+1SmC0+n+liPxoDjm3r9z+GwPGwIYaJBhvwvoDdTmmynRdr16iR2A8W4KTPuBjNrIxb7aMPAD T0w== X-Google-Smtp-Source: AGHT+IEXefXvPH1tjm5HxKdrHK6H7bskL9rQoKUPz6GiJtbM4dOshetWTVsxTxg9M/C0/zg3HXT9l9SvJvs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:6902:705:b0:dbe:d0a9:2be3 with SMTP id k5-20020a056902070500b00dbed0a92be3mr1901493ybt.3.1709749555637; Wed, 06 Mar 2024 10:25:55 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:31 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-34-surenb@google.com> Subject: [PATCH v5 33/37] codetag: debug: skip objext checking when it's for objext itself From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: B272EC0025 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 8hbde9s813qcmeqht4fpddj4ai9nfkjs X-HE-Tag: 1709749556-595199 X-HE-Meta: U2FsdGVkX189dXqg62biNkEPd7JDsUxNMOTCan/r8nv0TIT0qmQG+vImfhsur0ASDG+td4H9Jv4OSGvZDSZytv4mX6lUSr3aabrsPjOAaxToEZIDJ51y3//z9H9uKHIHH+4BgpViCuJ/njD32OP/fkZR6/+s75b5fA6Throfs02FoDq7sU00FQBsBnsOTVLEZ8/hlalwcB24zJXblGGHsEKhSVN07+hsPYQFIinS01JW84LDH7KxezFW+7KFTbCWjZbLkLm1GixWvGcqSQEacvd0VtaNc00FTno/fF52s0gj7t9DQby4mnlieVfVt/sFhF+6XZPVfu+u95YkTMMrg4AKodCJX6CNEcnTFEbyUBSk7OgANuR9HC/oMUoGZWrVCtpGEtRmD7PG9Vn4BmDvGZ0VKe0VGSllt4Y/OFcWL6jP8ZNjT8Q2gIqceG9hAJ0PW89RVoSTzvdB2n1sAXcUq6r0DkUbQ9mhqD6J4BeA0q9uaO3llM1LQ/iUgKU9zAK8qTP9L38M5MIDVsi+HHiNKsY+NKQqxmdntkyGEMcwHAIdIE1ExEnOJ71G4/pvspvaeukNhAdpngpmg0GbdwT7SegH2BOLwxQYJH+ir0TyxFXZ7XliI951UfuvM/vb+FmXBEVDJhwOAHm5mpUQyU8YVyKuaJXany/CYeByj68WZdOvo4guZDDjtLjXotRRBIxoLZboIxKdsSUQDAYqKFNmawhwe59/b1Qnd9rnxZAgvNXHo+jk/EJUMY4ueBYUgagwIwAbjLxf7s1SfFMvCsoxXGJdAW2d+bVzFwy21jkcObSGR6md6B41bn9D86uHCKIVsHmbH+CqoAXMf/7uUUNUn5T3TFCvTz3KDLPXJbSBvqf1n2FXaLkjyEM5hsTRLPTKIeecWz6fwSWZJBokSN6WvqxWqAH3dMKGWN/jdGAUdGaZMTRhVMfZXaJyD3bEvY6mNdwjc6S5lPr6D8/rjbD nIoZJ8GZ wZHU0PeCDZ3wrkOs+jl40BrDG5VAKIlSzNltsm/mx098w3vwyD11tlvUpSZNFbj8ix+zXzqe5Z64NUmX1mVVpZcEpjVCuwvww7y+9cfJOMfzsswgM8PWBd8m1fz8aJdVXYpGPzgi6ELqZ8at0XYf73oHxhtncNNtSagtl5HYf1EtpZhjFCBuw3dCJmmt4XfUqu5n7ey+R82q+3Jnf8FEf8Pq9eEelUrRWQnz7XGBtlAy16SJuAlMA7DxDG2xti1rRiO31tGDZIdpDUgQeWI6hxb5A/VLiLMPe7ZDKxBuTZ/5VvJH0ydx5DeoqNNbm0IijQMTzV0e4LXjOgquQCs1OOFq8zxJEK42XP/8krGBUAca2lLwrdfgaLduPVrEU8Oe78UxBPKKxQ38dWbgUcDRI3aTpWeVd10o52Br8cU/RmIgHcSwtavlRXF7SnwkMDL3Dq9WEfleyMWDrAX8rZlpKwfiF+0th8jURAZbFR9pv1EM8nH7X1CUs2uzxM0ieDBuB2jX4f1x5G9OgLTn9C0JZvb355DBm3gWF/B9672ujlW0w5+AOiKdK6gWG2hyw2MASPa09J5ez1ePoKP22FGla5WECRnBZyrxhMZ5bdq971YxXVMqwFdFaUARqx5O31G3/JpONV16yn6rlaDsz3XIb/MovxlpuCtyFutMLyfoy9aTuH2+Lg18acSmQl7zZ3Cozs/xcsibJJ5C3kSs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: objext objects are created with __GFP_NO_OBJ_EXT flag and therefore have no corresponding objext themselves (otherwise we would get an infinite recursion). When freeing these objects their codetag will be empty and when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled this will lead to false warnings. Introduce CODETAG_EMPTY special codetag value to mark allocations which intentionally lack codetag to avoid these warnings. Set objext codetags to CODETAG_EMPTY before freeing to indicate that the codetag is expected to be empty. Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 26 ++++++++++++++++++++++++++ mm/slub.c | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index aefe3c81a1e3..c30e6c944353 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -28,6 +28,27 @@ struct alloc_tag { struct alloc_tag_counters __percpu *counters; } __aligned(8); +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + +#define CODETAG_EMPTY ((void *)1) + +static inline bool is_codetag_empty(union codetag_ref *ref) +{ + return ref->ct == CODETAG_EMPTY; +} + +static inline void set_codetag_empty(union codetag_ref *ref) +{ + if (ref) + ref->ct = CODETAG_EMPTY; +} + +#else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + +static inline bool is_codetag_empty(union codetag_ref *ref) { return false; } + +#endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + #ifdef CONFIG_MEM_ALLOC_PROFILING struct codetag_bytes { @@ -140,6 +161,11 @@ static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) if (!ref || !ref->ct) return; + if (is_codetag_empty(ref)) { + ref->ct = NULL; + return; + } + tag = ct_to_alloc_tag(ref->ct); this_cpu_sub(tag->counters->bytes, bytes); diff --git a/mm/slub.c b/mm/slub.c index 5e6d68d05740..4a396e1315ae 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1883,6 +1883,30 @@ static inline enum node_stat_item cache_vmstat_idx(struct kmem_cache *s) #ifdef CONFIG_SLAB_OBJ_EXT +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + +static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) +{ + struct slabobj_ext *slab_exts; + struct slab *obj_exts_slab; + + obj_exts_slab = virt_to_slab(obj_exts); + slab_exts = slab_obj_exts(obj_exts_slab); + if (slab_exts) { + unsigned int offs = obj_to_index(obj_exts_slab->slab_cache, + obj_exts_slab, obj_exts); + /* codetag should be NULL */ + WARN_ON(slab_exts[offs].ref.ct); + set_codetag_empty(&slab_exts[offs].ref); + } +} + +#else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + +static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + /* * The allocated objcg pointers array is not accounted directly. * Moreover, it should not come from DMA buffer and is not readily @@ -1923,6 +1947,7 @@ static int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, * assign slabobj_exts in parallel. In this case the existing * objcg vector should be reused. */ + mark_objexts_empty(vec); kfree(vec); return 0; } @@ -1939,6 +1964,14 @@ static inline void free_slab_obj_exts(struct slab *slab) if (!obj_exts) return; + /* + * obj_exts was created with __GFP_NO_OBJ_EXT flag, therefore its + * corresponding extension will be NULL. alloc_tag_sub() will throw a + * warning if slab has extensions but the extension of an object is + * NULL, therefore replace NULL with CODETAG_EMPTY to indicate that + * the extension for obj_exts is expected to be NULL. + */ + mark_objexts_empty(obj_exts); kfree(obj_exts); slab->obj_exts = 0; } From patchwork Wed Mar 6 18:24:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584444 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7DAEBC5475B for ; Wed, 6 Mar 2024 18:26:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B005D6B00C4; Wed, 6 Mar 2024 13:26:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AB21F6B00C5; Wed, 6 Mar 2024 13:26:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 83EF66B00C6; Wed, 6 Mar 2024 13:26:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 71B066B00C4 for ; Wed, 6 Mar 2024 13:26:00 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 4B773A0499 for ; Wed, 6 Mar 2024 18:26:00 +0000 (UTC) X-FDA: 81867443280.29.BB0F478 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf30.hostedemail.com (Postfix) with ESMTP id A6EEA80010 for ; Wed, 6 Mar 2024 18:25:58 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JjO907E9; spf=pass (imf30.hostedemail.com: domain of 3NbXoZQYKCHgoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3NbXoZQYKCHgoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749558; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/rQHM1U0nLVwxoqb6VDJGJRpri0atHFBD/ukIXSlku4=; b=Hdeb2RW4p013s/weigOTvM/VYLy4KNoC9RdgdevcotWFZS2aQKAkNkVhZ55Ji0adEnpKg/ pMsykMUoem5DEngQ78J6gna86PwJj25fBIgew262TBOKOqQLwrjTunIA8B6RaAAI2NrzZv etYOhR52S5neukeA4hXydh4J8clubd8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749558; a=rsa-sha256; cv=none; b=wg6Y2lF4dfl143ahYbNoncL6uB/DQYx3jRT+J+qvbLyn3VAT9ks101XqMv9O+MtX919EOG 7IXhIF91ajTjxqNzKFJF8lP0mRgTHr7jEvXG/KLYqB7eZO3GWU95LXRAxg1REttqFW09XO 8odx6BFd64HwMV7yWshU9u6WhfQT2rk= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JjO907E9; spf=pass (imf30.hostedemail.com: domain of 3NbXoZQYKCHgoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3NbXoZQYKCHgoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-609a8fc232bso692297b3.0 for ; Wed, 06 Mar 2024 10:25:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749558; x=1710354358; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/rQHM1U0nLVwxoqb6VDJGJRpri0atHFBD/ukIXSlku4=; b=JjO907E9gCKQqk0dFWT6AzKXPHrtCYmYLJCjwznx0G5k5jgLySEVYfnMXyLQXDxW8+ 1lX/eHerccPDNU192DUKcqMVLMkxL8lVriVDJmUDv7SsWvfs7svQd7z4Jh8/WeZ7jcLh MtGxC07WVmYCJm+wDJJLa9CkvfJHdjc6I2OPt5bx9nqCV2A9c5VXdZ9eQYQzfyF0pbIp XZu78BR2pNXNZOTcCna6tA/sdQD3Pr4YlE8x/LcwjYRIldrGNoo++FueVaO6koUVTqJa O7XZWCP20FXRD4Y2uSeA5G+27D9ys9dH6XZ9M5XZoFcAQNYJbDyvfsCJs39qJ5HbsDlH rsqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749558; x=1710354358; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/rQHM1U0nLVwxoqb6VDJGJRpri0atHFBD/ukIXSlku4=; b=Z58imHpSERhjI3uQ925F4lbu0WVfZsw5lA53Ai1z9OI4A93ioKvxniCJZfKlL5sxw7 sCToVe+xfWVN+6gHF6MFhSHoKcS7r1iueDWbSKOB+LQRCMW9jfr8rNInLQBWigtSMXXj ooa/36Jd9BpRVXKIa2cwekH+NFzuDLVq4shpl6lMhONS97ASlrtkU+TJSA68S/jetNEF zYNK9jAddPpUDRFNpHqOugCGm62aFESeDnsslVSoxPXhNgdrkOEjaye+wK1AUD0aX5zB G1WABgilH4gUUcA0w01pN6kuwDOnUgyU4vB7WEnWKrJLcxakVfJpKVcbw3/ObM4Dqv2I Fiiw== X-Forwarded-Encrypted: i=1; AJvYcCXLbBrVAnTMYT521oIlsJjFUX3c5wiNJkHzqy8pcBIOArkdPgpXCqocnx+UMDrg6KdykyqyFqrrVGBK9Iwwx+c2NiQ= X-Gm-Message-State: AOJu0YwFODlmOKEpgXghYKLTGvT+JjRmXIg5G3XYRprkVwT1mghgwz8m tYmyFizKPo4IXsftMCthzmaF/aY59HOVHPsE/UfuPzTH30W1Fzrx9vNTudOnyFGfuXGVk4F1La8 Iww== X-Google-Smtp-Source: AGHT+IGylq2hXmjkk0zshnCen8YUvCWPBEs19UcrOnAI3PcEdGgYWrVUEFQJWagWIw3puXNNod8UwtNgR7M= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a81:9b97:0:b0:609:3c53:d489 with SMTP id s145-20020a819b97000000b006093c53d489mr3279719ywg.3.1709749557600; Wed, 06 Mar 2024 10:25:57 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:32 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-35-surenb@google.com> Subject: [PATCH v5 34/37] codetag: debug: mark codetags for reserved pages as empty From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: A6EEA80010 X-Rspam-User: X-Stat-Signature: mdjpctgbiquwn8xz53qnkrnznunxw19r X-Rspamd-Server: rspam03 X-HE-Tag: 1709749558-989022 X-HE-Meta: U2FsdGVkX181HrQqKeC52Vzdud2Fl6KEcl2w8+nlyuRmOHUyDvKNxUXOR6c+WCdhJkGpqL2rtNUamOldOKTC1ZSC5q+JgR7KdSBJRgAA4vWyWpmyhm6kNZfF7Jgq35B9vIStDbtk7nVX7RShuqCQxrR7lq7VUWmGawJhVEPs/kRysqSQrIhinhLKLkAu5+CTNWD1ShUDaBIhJgNv/vIxeOh6mQtthfhmb5Nh0nnVYScaIFacrkJqaXUvdRaq6VbDDy+tbe/upzfw0uCh8eJqo7nUZ0+g372HlgsckmSl3FbdIKMez1m+sCEilIW8JzXn2c3dn2d+sNUT157UWseDB+APuUojzzPsbQoyU45cAwrq2CXfBG2T7dK8vJOqNxYnwZK2QX6my0FmlMO7oxtn6OC+O8UkDa/HaajmeLip+Pe2YBPb+AIW2t3nQC86C5lM4xqG+4mVoaoUPqtcvAnPW58D1XnRNFCFsIF0eqE4G2c8k+BGFPm55uVnV20dNk4Fke7+lIFtcX9SviX0fzzCQai+V5Aw6Xw1qrIMLh/CpYAGP94k5pvwujf64/2J4Mlc6iQDKc+Q7cEgkHtziavA7g5aJWjNLr2GWjBaPtHRhm1mBooxKLZUlBuw8jgIs0fkH1hHaV0AX48fbEfb6sOjcMwPlgQhgSIA9Z0/UPIdoMor5Y0qU1spE6Jnb7GntIX1B5QZVs4RdY5P1KyqQW07kkqEK3/LACKtEGaidYdpPMiGQBXlDvzRqFXpyTdPLskpcI31IsP3eEl+QKfy4HBQzZE6azlf/bv21bO9GJcEhKVC6HQK47l8mGhdnB5HgLe5GQd2dB7F3+ybroeMv0A6MR1w0ubTVBEV90/97lvXv0wfCiofyGl/6GIFaVXBCV+Q1I4Iyft+glO87srUJ0DOLom7V90ewgd/GF6t5zdgP1hH+HhlzBn0oPS7amVlnKZ3Al6zl1NDSDj/GEyZy3w wmBOmVGw kfIILsHYtGMfGV6NZPs5V9CPrMMDS+jbriTmOxE3UX9XhuRkH4dTAjkCjMVlWJpm/eNroiVLJ8diT+EZmWMJPSQDEWiIe75LdlrWtr+P5nI5GUmqz3lZ4/qz72ownsyTQ1aqzCGpTB2MuCmXO+4tmzIaSgqN1ieIQRCy+ptW+phycb+F34hIcJrioiQOWgQFxgLpP0nsouvHvC7cWJdX1O+d4C5RuJGo/RBYR6ODozNIPR+ARH8yyLTWeoT7TpHc9I2isjGo/iVf6y+yM46Ax7Nn/SIoTamDHhJmGama9oDZ76k5yFnqhy5ZDvG8AnfT35bVz6zkRahcuW9lwkHFOxgM0bMxF8z9BRwuLUOjz9rv48DuOOkmz+czZ+hyCG9wYrqltgU1sq4NIHXZnQHUMoRakMC99igsYPw1smaXN4dNPWuo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To avoid debug warnings while freeing reserved pages which were not allocated with usual allocators, mark their codetags as empty before freeing. Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook --- include/linux/alloc_tag.h | 1 + include/linux/mm.h | 9 +++++++++ include/linux/pgalloc_tag.h | 2 ++ mm/mm_init.c | 12 +++++++++++- 4 files changed, 23 insertions(+), 1 deletion(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index c30e6c944353..100ddf66eb8e 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -46,6 +46,7 @@ static inline void set_codetag_empty(union codetag_ref *ref) #else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ static inline bool is_codetag_empty(union codetag_ref *ref) { return false; } +static inline void set_codetag_empty(union codetag_ref *ref) {} #endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ diff --git a/include/linux/mm.h b/include/linux/mm.h index 699e850d143c..9d25d449e512 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -3118,6 +3119,14 @@ extern void reserve_bootmem_region(phys_addr_t start, /* Free the reserved page into the buddy system, so it gets managed. */ static inline void free_reserved_page(struct page *page) { + if (mem_alloc_profiling_enabled()) { + union codetag_ref *ref = get_page_tag_ref(page); + + if (ref) { + set_codetag_empty(ref); + put_page_tag_ref(ref); + } + } ClearPageReserved(page); init_page_count(page); __free_page(page); diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 59de43172cc2..01f256234e60 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -120,6 +120,8 @@ static inline void pgalloc_tag_sub_bytes(struct alloc_tag *tag, unsigned int ord #else /* CONFIG_MEM_ALLOC_PROFILING */ +static inline union codetag_ref *get_page_tag_ref(struct page *page) { return NULL; } +static inline void put_page_tag_ref(union codetag_ref *ref) {} static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int order) {} static inline void pgalloc_tag_sub(struct page *page, unsigned int order) {} diff --git a/mm/mm_init.c b/mm/mm_init.c index 2fd9bf044a79..f45c2b32ba82 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2567,7 +2567,6 @@ void __init set_dma_reserve(unsigned long new_dma_reserve) void __init memblock_free_pages(struct page *page, unsigned long pfn, unsigned int order) { - if (IS_ENABLED(CONFIG_DEFERRED_STRUCT_PAGE_INIT)) { int nid = early_pfn_to_nid(pfn); @@ -2579,6 +2578,17 @@ void __init memblock_free_pages(struct page *page, unsigned long pfn, /* KMSAN will take care of these pages. */ return; } + + /* pages were reserved and not allocated */ + if (mem_alloc_profiling_enabled()) { + union codetag_ref *ref = get_page_tag_ref(page); + + if (ref) { + set_codetag_empty(ref); + put_page_tag_ref(ref); + } + } + __free_pages_core(page, order); } From patchwork Wed Mar 6 18:24:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584445 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2925C5475B for ; Wed, 6 Mar 2024 18:26:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 18B8A6B00C6; Wed, 6 Mar 2024 13:26:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 13A836B00C8; Wed, 6 Mar 2024 13:26:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA6966B00C7; Wed, 6 Mar 2024 13:26:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D76DA6B00C5 for ; Wed, 6 Mar 2024 13:26:02 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id AB313A0661 for ; Wed, 6 Mar 2024 18:26:02 +0000 (UTC) X-FDA: 81867443364.06.46F13C7 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf19.hostedemail.com (Postfix) with ESMTP id D78461A0021 for ; Wed, 6 Mar 2024 18:26:00 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=odrBIkVb; spf=pass (imf19.hostedemail.com: domain of 3N7XoZQYKCHoqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3N7XoZQYKCHoqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749560; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=j6JoLMysN5flVa9c5yM3pdaz8HR3taJXen1pd2wjpNc=; b=Hb/meiCVxQ/8blLB7QuIDbXuttAbwrVPo5DNkQFY7EtpVKU2eKmSnpC0oTRWahIHw90j14 CGmZp677Nuc4PQrUNq2TlNzcca/XIixKdNRPsXiJcuOQVyZQ/eehi6wQpiE9BoZkW2jmZb Vt2Qn+n7AZElJn9Idd46WdPOCuRmG74= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749560; a=rsa-sha256; cv=none; b=W6kXalqItvg84ogHFXloEW2KIwpZscpj9HHw63M2f2urKJohZo/MR9kx/NuDOy6EnwyYqd FZbKZ3nE93WzCS/e7vOUpbDh/VPVxfSGJKLoYu/AcmwahDvRVwZDG631pzGXrGs4A+I2xd 2iFeetumCqqJZ9c3OwSoSW8pAM9c/no= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=odrBIkVb; spf=pass (imf19.hostedemail.com: domain of 3N7XoZQYKCHoqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3N7XoZQYKCHoqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60802b0afd2so844927b3.1 for ; Wed, 06 Mar 2024 10:26:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749560; x=1710354360; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=j6JoLMysN5flVa9c5yM3pdaz8HR3taJXen1pd2wjpNc=; b=odrBIkVbHG2CUAdfektW16tuyF98w/bxYYTkgvkHfsyAeFndbSl9gMjgrHtXuWDjiu CzjbHSJY5uzTtuDOD2Rww8MSOiQHCBPppWiD+vCvJr+QZlp2yBahakhfqO10/KaU3IuS swJqqZSDJm/TByQaAF7XFJxdPgQIfrfgnVTSe6jdZ0w5epoeWrsbG9Dc3ZbaCM4qEIH8 n4vgD4Uqctoagn+UscPnBfKEyeuLBLKWQNdekSa9VlK0J4+JNhU6lv5m4xFex+ZoYagq spK1/Yx0srPMzG2HQtuovVgkJlQhrKUHw8cEH13NgEm4U62woJfCdvBG8Gba7r1Wg9Hz caOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749560; x=1710354360; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=j6JoLMysN5flVa9c5yM3pdaz8HR3taJXen1pd2wjpNc=; b=lNbMX+MekBIGeteOe2Ap9fCsORNvV4J+X1at26V/GG2UCMX5ch30djyr3uoxPzy1Yg VbhDhYU/s6IAbSPBx8DrLBSSuPflCDnX0Rw9+m9KqTmKQaci3d7YnQSRh4S/JTi5wWyk +CwOs16HwarnqtUSfXjwvIr9NtvSklqvufdHn2EeqmKJlUwZfwUtk/7WRNj7rexebq+V jRjrUvpRgpV/fLzEXKD+09bGAQJ9FWC9pu587Ddp/LjphbPYGRqz3bj4jdcKxcgZaX7W cmWXpKncJWxId1iIltfc4dl9qtAIO9dM5Etadr0TkJimiURpkWG2Pg5MX3+b9dEd4kkK B65A== X-Forwarded-Encrypted: i=1; AJvYcCVK0DVJue15b54Np0dhg7s/lp62TyTPP0N480G9DHFyGm+vFR77NvDVCu3ra/yIvuU1XpZyn0vcsjGPxtp0tvf0y3g= X-Gm-Message-State: AOJu0Yzsu1qJDT9TVZgke9xMwIgXnvu2EBD5FP5xp1vfE3xHf5qwXHZ/ pI0EWERlB6RSZyeIkmyscshWzGHwLtAdw5ouwxd0OahhLzPEHCP6wb5XM6dWJg3XXFdcvBVQv3O GMA== X-Google-Smtp-Source: AGHT+IFJf4+3zsOtaYj4w4sYtLT+urJK6k/XCbV3fma9m2tOepAucRt++i9xMDbmvInEP4W6X3kckDvXGNU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a81:ae60:0:b0:609:3c49:d77a with SMTP id g32-20020a81ae60000000b006093c49d77amr1209593ywk.5.1709749559768; Wed, 06 Mar 2024 10:25:59 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:33 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-36-surenb@google.com> Subject: [PATCH v5 35/37] codetag: debug: introduce OBJEXTS_ALLOC_FAIL to mark failed slab_ext allocations From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: D78461A0021 X-Rspam-User: X-Stat-Signature: to1418ohd1uhisfxnuj6npypf7daqn4g X-Rspamd-Server: rspam03 X-HE-Tag: 1709749560-878486 X-HE-Meta: U2FsdGVkX1+pXMALYEMN5gZnY8sj1nXkEartCuyQtP44egnnglkJLk2MQ9bhjQW70StenEwRjFts7GBp9ZCkI1n7mZ1E+9B3A13sWXSbw7BOc/8BCYfZVnsEWIt/sLDnbNYyVS11d//CPeQpAslagEc4/9rdmp9OzHa+tqXKgr2uwpHgiDTAinV2+sQUTP9te4PiiRn77ccW/TIJZp+pimos+QFE9oiqwSVaZXB9FoLvtyJ+OVjL7/79rL3icg1+76fSMIEcUNVtLJb5CjpIdWqop8R5WATHmOwDcYABNZVegIxpCY8CmsK8KpalGngC/Rtex4ZHlVOiaS335ke0zFIZjoVq7lG8QILYAXVQdgGVkvQY2gTCUZL440pDnOhGgW9HRxMPZ8AT39+02RHdUgYpsP8u5oItaDJ4HJXXgCTDI27feXi+gonpCJbML6djCDyI0kS10uGUdoaX58j9S197Cy4b5lwG2LBL0FfByj9+RVRZY6VUcMReaJwYGiqexPUM9EfFhNNVhc63sbEPwtOPK6Y4mRsqePr39++0wpA4M45xIuAycgSiRtDbg+Chf3qt4QCBohZeXs1k8dJ2QebJSSIqBcQiaoBGmgo9stF3u6h5FoMugWEFVwrJ75nCfWAOSlLpDVTF+x2IqtrXG2x+jONa8XK+I7HacmRTTIQmTL/LGDznyUfQB/QeVh2eWWjmUkczaHNNAIHFBQhENYZO9Dc8s5gtDt3eEJE5Ww7eXCiaDcD8yRDDCvgLKyzM2MdGtLpl/peBikQD4PkWJaRSvY8W0J7P02QSDNTECuGwGLW+gr0AS1UN/kZS9C5KKg3pFZfeQ0xNtnCBkOl3ScDC7fNFvJikE2yGv0weRaBz5SJb2hL7WCKoexYmYlcMQwA9/9CJQEOibTTY0YhjpvQACNnnH/QjoYU0RUxOPFb5BexYsoAgZYAZmmPCVNOEyekxCviK4MFVgXNGa0l NKrOKVHq pz4DdJ83agHRtHjCMfPYm2Cl9lJkL9VdaeVG8cQvQJbbRYp3IBYo9FRttv8ls0O6xNYTAmUs56bXfa+LSzRVGtHanI8R17SbFdHqQgihlDarfCQAquewjedlr4p2rTObwIaMAtue9y88/xCMLe/FtpXAdBJ/AyIoXOpdULJ4wPy8Q+DLsTBzbBLL5L5H7mBSlk73AW/TlhvYR19PtfrZWbO0l4u28HaX0FKWcKqfx6dE3LYJnbniIgShJ4rIBktbuKfiepWjvWc5OgHvAM/niirGdWQvSmZ/ZTcuG5nempn03oG3wN8QAWMEKfi5a7O92vgKIVouSCyJqbGhCu2I4np/kjjI4//D1TLUuU4j/bgULgL93qWNIDgtTg2ZbeIbf/tc8e6TXrtWaVggSvX9wMoQnM1rA6d19x+BB/ACynE5caAAgR1wv+hiJ72WzgxpsQUPh8UR4sN6PisjD7ivebF00lFUmhPKMtdQJnhin0Rtv1PSnL7KirairYV6NWVKrwO1t+WPkin4LfoW7VZDF//++YxpKkEiMetC3yXETH6ovr+QYlW+xpCU7Ahtz7UGb1hK5PZ2MnyRQPOa83OPqPapFIApF6TbKrNr9/41jzVt18EE8P+Kff8A/00fVSPuRNQ09HMHAG3o9U1PO+4/M92ubQTwGGd00CBedzuwE9wwmyxGLkLDeckqbczXp8CHrQyhrArPZGRM2ukl5Quc7q3fjiyOADCB+WL13 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If slabobj_ext vector allocation for a slab object fails and later on it succeeds for another object in the same slab, the slabobj_ext for the original object will be NULL and will be flagged in case when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled. Mark failed slabobj_ext vector allocations using a new objext_flags flag stored in the lower bits of slab->obj_exts. When new allocation succeeds it marks all tag references in the same slabobj_ext vector as empty to avoid warnings implemented by CONFIG_MEM_ALLOC_PROFILING_DEBUG checks. Signed-off-by: Suren Baghdasaryan --- include/linux/memcontrol.h | 4 +++- mm/slub.c | 46 ++++++++++++++++++++++++++++++++------ 2 files changed, 42 insertions(+), 8 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 33cdb995751e..3dfb69f97c67 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -366,8 +366,10 @@ enum page_memcg_data_flags { #endif /* CONFIG_MEMCG */ enum objext_flags { + /* slabobj_ext vector failed to allocate */ + OBJEXTS_ALLOC_FAIL = __FIRST_OBJEXT_FLAG, /* the next bit after the last actual flag */ - __NR_OBJEXTS_FLAGS = __FIRST_OBJEXT_FLAG, + __NR_OBJEXTS_FLAGS = (__FIRST_OBJEXT_FLAG << 1), }; #define OBJEXTS_FLAGS_MASK (__NR_OBJEXTS_FLAGS - 1) diff --git a/mm/slub.c b/mm/slub.c index 4a396e1315ae..d85fbf9019fa 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1901,9 +1901,33 @@ static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) } } +static inline void mark_failed_objexts_alloc(struct slab *slab) +{ + slab->obj_exts = OBJEXTS_ALLOC_FAIL; +} + +static inline void handle_failed_objexts_alloc(unsigned long obj_exts, + struct slabobj_ext *vec, unsigned int objects) +{ + /* + * If vector previously failed to allocate then we have live + * objects with no tag reference. Mark all references in this + * vector as empty to avoid warnings later on. + */ + if (obj_exts & OBJEXTS_ALLOC_FAIL) { + unsigned int i; + + for (i = 0; i < objects; i++) + set_codetag_empty(&vec[i].ref); + } +} + #else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) {} +static inline void mark_failed_objexts_alloc(struct slab *slab) {} +static inline void handle_failed_objexts_alloc(unsigned long obj_exts, + struct slabobj_ext *vec, unsigned int objects) {} #endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ @@ -1919,29 +1943,37 @@ static int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, gfp_t gfp, bool new_slab) { unsigned int objects = objs_per_slab(s, slab); - unsigned long obj_exts; - void *vec; + unsigned long new_exts; + unsigned long old_exts; + struct slabobj_ext *vec; gfp &= ~OBJCGS_CLEAR_MASK; /* Prevent recursive extension vector allocation */ gfp |= __GFP_NO_OBJ_EXT; vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, slab_nid(slab)); - if (!vec) + if (!vec) { + /* Mark vectors which failed to allocate */ + if (new_slab) + mark_failed_objexts_alloc(slab); + return -ENOMEM; + } - obj_exts = (unsigned long)vec; + new_exts = (unsigned long)vec; #ifdef CONFIG_MEMCG - obj_exts |= MEMCG_DATA_OBJEXTS; + new_exts |= MEMCG_DATA_OBJEXTS; #endif + old_exts = slab->obj_exts; + handle_failed_objexts_alloc(old_exts, vec, objects); if (new_slab) { /* * If the slab is brand new and nobody can yet access its * obj_exts, no synchronization is required and obj_exts can * be simply assigned. */ - slab->obj_exts = obj_exts; - } else if (cmpxchg(&slab->obj_exts, 0, obj_exts)) { + slab->obj_exts = new_exts; + } else if (cmpxchg(&slab->obj_exts, old_exts, new_exts) != old_exts) { /* * If the slab is already in use, somebody can allocate and * assign slabobj_exts in parallel. In this case the existing From patchwork Wed Mar 6 18:24:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584446 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 016A1C5475B for ; Wed, 6 Mar 2024 18:26:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 002416B00C7; Wed, 6 Mar 2024 13:26:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ECC426B00C8; Wed, 6 Mar 2024 13:26:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D200F6B00C9; Wed, 6 Mar 2024 13:26:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id BAD116B00C7 for ; Wed, 6 Mar 2024 13:26:04 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9AC668034A for ; Wed, 6 Mar 2024 18:26:04 +0000 (UTC) X-FDA: 81867443448.17.1122F91 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf29.hostedemail.com (Postfix) with ESMTP id D1AA412000B for ; Wed, 6 Mar 2024 18:26:02 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=hPE1QFk6; spf=pass (imf29.hostedemail.com: domain of 3ObXoZQYKCHwxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3ObXoZQYKCHwxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749562; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Jq5e0nRnwdhD4a9FMTFm9DydRhfgPVhlZe9ht+OK6P8=; b=QGpgIQFpbEy1rCrIywqQhUA7QyBzIAXjsCSVFC0II17w9jKXeWZzvNOS9nP4A/7c8gevzD ASdBYbUtiqrIBcAEUNPNmrskGaDHVZDFmRFY1vSog0Y50AJVEjQ+JTAnokuM2yvCtaEane HGi2+acP8azohwq5SN2/p43cx03FrjI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749562; a=rsa-sha256; cv=none; b=jgic+7KqEq14SH2GxwVSXzK8D8tn3L9zkABuGV+gVh0t4ymaEVsysbIzIfVOca0uf1SvD4 kUD+v9LWBRcqOZQ+HP/n/3T79IcFT6D5kPnbLrY2W94eugfBLuc9FgBwGCMSOLi5pK+oCo c3eKmD4kgylFkj293QPD6bCuQDdwG8g= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=hPE1QFk6; spf=pass (imf29.hostedemail.com: domain of 3ObXoZQYKCHwxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3ObXoZQYKCHwxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-607e8e8c2f1so123107b3.3 for ; Wed, 06 Mar 2024 10:26:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749562; x=1710354362; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Jq5e0nRnwdhD4a9FMTFm9DydRhfgPVhlZe9ht+OK6P8=; b=hPE1QFk6/XkyZZ8hkehhSZ2WZVrTIdVHnlNnrZOmMcJJfJSbQLXWKX+moF5HmK/RK3 5TgIrPQywn3L7zXdoYaAmjnltiHePrA8mftK8zaaHwFv07y33Cli50lwcAI7hheu9jAr wmWmpvzFlLMy5aYn/RdZPbH6OmhmIy5iI3uzgWvJ1GsPdDvALGBM3i9S6sNYmun6x/i7 hSv7XR/mvUswD+nD2OVoI/D+F3UtSqetzqTztMrfERen6IUoqqvQ2WtfVQV9U92uJ9OQ sJSPOJ/FoysqjM+3siRPjiXJlu8Rpqi34w7LE9TQmepQ/W7fEWydfz4I1v1P4ypXRkYL DhHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749562; x=1710354362; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Jq5e0nRnwdhD4a9FMTFm9DydRhfgPVhlZe9ht+OK6P8=; b=CzI/5TiWw4JfFiX7iuZ+bXD9hYKKP8EPBobivTc4hYDYcvzXhO66RXbh/WgVBbBLg8 YB1q0gEfkiLRRuf7G7PSxOigoa8vdMl/DxGwO5s0zhouoHA0I7uCBty0gDb4YKL7f3j2 15JzhlMxwGkFO6lIagOUUCO7H1YjVO/jeFnrvY+k9xJiL5TYR320tI8Pp24K2NGLnwwK mBbDMyiKXp+ejexCRMXOTNihzSM6Xc7LHMmcnahObaUiBonLfOdMNqBw5VxSDkmk2Vwq 8Hil0CqWVc/x2EKoxYXA9kMaVYcOdnpVU2XNKUDcdjY6y+YaJEtVfUgAKO4uZOWcQHDB 4fLw== X-Forwarded-Encrypted: i=1; AJvYcCXJgzm7RiSpXhBFfkRoYJZG38rBUs9I1jDRVuwRaFufQGIRUM8WzN6J6yOBk1eJTJwsmLqAzL0C0BuueBinsRta3ZI= X-Gm-Message-State: AOJu0YzKnMYOdnxvHvQzHM9kJAy2wGyYAIRKmXHbbKmyTGvOoRPB7575 YroImZcaggSbo8JysiScK9mMi/6BRmZ2f/iZgzrB48frrCh/K/5D+OWpsnNTswjY49WVFU4N1Lk LCg== X-Google-Smtp-Source: AGHT+IHvZ8kVm6l/VDxZg87uGkFZ91C+i4oKHcDrWRMYqfh/2LVOqc/iUX+s1MQiLrOuG91osjkHSkoZjwg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a05:690c:fce:b0:609:e1f:5a42 with SMTP id dg14-20020a05690c0fce00b006090e1f5a42mr3594072ywb.2.1709749561828; Wed, 06 Mar 2024 10:26:01 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:34 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-37-surenb@google.com> Subject: [PATCH v5 36/37] MAINTAINERS: Add entries for code tagging and memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: D1AA412000B X-Rspam-User: X-Stat-Signature: dnbq6ueceu5tunjxqe94h7etu6u6roeg X-Rspamd-Server: rspam03 X-HE-Tag: 1709749562-153536 X-HE-Meta: U2FsdGVkX19+Z49YAgTQE3WDzadJgr5sgmp0CZEhR0c7pYsexNxuvabLZkQmhc/YVQXkdLMS/V8THE2vJRFSGjPLAHKlJcBh0CSywSm68doggBFE+xdTnO2DPag7x3GsOS4F7uBd1BqgiN3X8XHisiNwwt5ifOQy+jKl73QzC7YL26uAcRRDTPIO9xv3S3kKufdywiCXUQZv74kxhiyJZkG7n7vaZ0MC2Lr6MrazI7W87MJiP3gI0OwLO103Ja6tTMnMWLNWpg6X9TcfmQUaFuZJ4+Igr04Cw/ZPMjSJ7x5SeUB85OZWENDmT29wJr4BUExIDbYbaS31+mgTq6CCcUxtWc5dhAOub7Wr7j0VOezkymlGRiE7POfigKXpc0i+4VctzVID8rxwavYzq6VjbZkp08tJjIK+xqeIe6WfwlOLc/2DI/K+oKb6n3HvVM4Y0o2pyAKoajSQuaqglP+LDrqU/lkIr/V+LS50xNc+vU2lP28YsUcAiF0ioZsdCTMroFcTCPjlG6Z7xy2WhqJADVrkVKIOE11dKRxaarrZFb3ZZg38VlkRsOWoASvL157KEa2AgQ68MUCZN/v+mdsuDtipBp6r6iRhW+pMOqO6L2BoBUZlgcQ1clOOrh4zWveYVBkvVUPzn3XOrMTVss1+uEhIfyCVuQW5v9XFnhxdrjiRvCQV7Rt89lH8IDbVGktTka572gjugEDHplVQCYg/4fQJcGA6c+FGI4fC/s+Onc5l9d5QZUV87Mus1MHzVH9e1kKk58OM7oJfg71LgEtUJBdx2fFlBJJtBgB2ZBPTg4gtMOH8ReyLNjFfNG0pPqlxuvBLrqKkizPCzc/rcgNxCU2rjm5ECM9I4rJJflFD4L+jFjOrxCOg2WvQEAJ7/7lRUpRcCmeDsZAewlyK6ccyxuNAwb6/LWYJk3Ky92ZDVakJrKcSRWQX3h6APqvCxDz8DXz6Cas4VhRaVzrOcnj ev/Ii8WR fCEa0/OHI88mnXHzAMW+vAfURIy8Rglr6B70mlYwG136fGwhYpRhVFZKDkJa5w3ii9ZQ4fI0W++/jiuW8yoZoqyO1/+utQR5pNQPW5q9qZBSRbeeSSOTpQE2FibpDcRc1jroEydLii0t0ghRn3wkylfIETtWFFTSIvZE8KjuxlrfAvv27hda7qdOytyxSqo8tYtnKKrhkrYZhmipOGw900otl3QCBh2OgtIYVEUTAfFtWGxEZ6xg6ENEd6eRSIwJliI8cCzQYJgemcyUvN63GzPfL1rGrYRrypWLyU5V0Jru1FzMK30FvQJ2HhIkSsdF8MtjReci2t3dVkf0uthajs92n+XLxUVNbzY8COqZ/99CmYBepEyccQhZuOj5YIylZfo8erPL4FXW1df2SmJ8WfveYdJj7Wa+yP0YW++K68f/wAUlemOUVThkgLQ/U33XLmAPFzpOcEJ+u7DwH7wcxvXwpprQQav3bjRJ4qI5ZoA0XXJ0GKrPKrBtuE9n3MRbVf7bl2zs8yD4wKTue9rTtqqCZoUH3qHoHViayHNItrsENnxV/qARX1frJDe+dc+Ak86zUEvMH2hn3/IGMUugH18QwfaNhage5kJGNpmF0lXooOebZAliJyBQZeXn6sHQhLrYXmBq+x5Nq0e5Gc2vIiX7Kct50K6chUbsCTKezYeUY+d8KEdwK+BH6Ad1fBTf1EYlZBVm3Wu2l2NrT4P5tUWLDdumITQcXe4EPc7NnDZU7dT2WKPTVCqGaKLjypMayZjUZziqCfFJ3yWUVqC9Z+sSCfT3kVw3fzR1FDlxVnSjn3zGttI4JftCrVLTk5r7FKKgMKJFzEa9iF1INA1aA8ckXt639Cc/3G7stHLZRSFdnBGi5d3TFUyxc0oENPmjMzztUmRHdEvriTbibCzeWCrd0LP5kOeCa2Sxeyuhd/9sH/kyCWyRxDIeY3Y/nVAN6Do+jaD9tg4ILJ20Z4OSkGlTsiLvv /tSDZVfq 7g3d0P2L+sGNz3u9WmOVJxtUaOSivch6bPgKb+tMCNNac3JZXn1iyQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet The new code & libraries added are being maintained - mark them as such. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook --- MAINTAINERS | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index f4ddbcdfb29a..03892d8032ca 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -5210,6 +5210,13 @@ S: Supported F: Documentation/process/code-of-conduct-interpretation.rst F: Documentation/process/code-of-conduct.rst +CODE TAGGING +M: Suren Baghdasaryan +M: Kent Overstreet +S: Maintained +F: include/linux/codetag.h +F: lib/codetag.c + COMEDI DRIVERS M: Ian Abbott M: H Hartley Sweeten @@ -14062,6 +14069,16 @@ F: mm/memblock.c F: mm/mm_init.c F: tools/testing/memblock/ +MEMORY ALLOCATION PROFILING +M: Suren Baghdasaryan +M: Kent Overstreet +L: linux-mm@kvack.org +S: Maintained +F: include/linux/alloc_tag.h +F: include/linux/codetag_ctx.h +F: lib/alloc_tag.c +F: lib/pgalloc_tag.c + MEMORY CONTROLLER DRIVERS M: Krzysztof Kozlowski L: linux-kernel@vger.kernel.org From patchwork Wed Mar 6 18:24:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE301C54E41 for ; Wed, 6 Mar 2024 18:26:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 497BC6B00C9; Wed, 6 Mar 2024 13:26:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F7EA6B00CA; Wed, 6 Mar 2024 13:26:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 24C046B00CB; Wed, 6 Mar 2024 13:26:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 113AD6B00C9 for ; Wed, 6 Mar 2024 13:26:07 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D7CA3120C7B for ; Wed, 6 Mar 2024 18:26:06 +0000 (UTC) X-FDA: 81867443532.30.0C601D5 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf07.hostedemail.com (Postfix) with ESMTP id 0C0F04000F for ; Wed, 6 Mar 2024 18:26:04 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=fmGvDDlk; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 3O7XoZQYKCH4uwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3O7XoZQYKCH4uwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749565; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yGhey761h+yk0goaaYaIkNpiVVUjQ3rocN1HojLD7sY=; b=IOUPZzzy+WIn9zruVx6MIqqA5v02qLK8SCg7wyUPEUBe78vLhUZYzxh4gni8WV8vxL1uvl 3hwhK6re1kMEu/dhloI3069976sX+qw+EiLQn1xSw5/1Dfwfmffj9YrD3aYHrkAzYiD6PO HCU8aHORFv/b3cD0OlK/fBqMT9D/CWc= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=fmGvDDlk; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 3O7XoZQYKCH4uwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3O7XoZQYKCH4uwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749565; a=rsa-sha256; cv=none; b=r6Oq+a1NW7kc/GzuXy8ySZH9PED/Pta/H4E2frsKkUnv3F9v9wJKA+5sdvDyYMBo+/iexO YDXP6BdOJxiAwRh3J216DM91lGFVPECxkrzfeTY4L3Be1BnGbx3VOlwQSAnsKpbLVelPf+ 5Dk5fjO3wbg9s2sgXJ3JcQ+UJiYUXwM= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcc05887ee9so10100151276.1 for ; Wed, 06 Mar 2024 10:26:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749564; x=1710354364; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=yGhey761h+yk0goaaYaIkNpiVVUjQ3rocN1HojLD7sY=; b=fmGvDDlkqKgqBfinoXO7ZcIZURnXuv+OgG1IZtbaMlrYOfVQuagjOS2mivOqKJektE yAAtpJs4uMJS5YlSV8TdECedusn6XLB6QyiIEt5c5S9GaT2LFdJPzjj8Nng7DhZ+f8OL dLPRaEiI8Ay1iKEj8+5BBvmn2UmaYcd+0XRCnAtjkhDZj92R0WJ02RPVMuh6w7azkZgx RDWZhtJ2XX/BKOby3X0gporbVV+duXIKQvRmBuOhSj/9lEFmo4xC071xHjfasSqrvO5e gocGInKUp3Qx9j3QDG+5iBC87FffnIn2AQCtELrJEYZRixGLTZ7szvH9GY398fh+Jd7d fS5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749564; x=1710354364; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=yGhey761h+yk0goaaYaIkNpiVVUjQ3rocN1HojLD7sY=; b=OdFKkvLtGpG6xwxBPcEpwVtg5WuHa/CYzS9OY/L2rQEW2Jq5T0wsuaW+K47b8+BNYY bvYRjRUqfeZ8zyqxYDsTmc87s+DsMj95ZIygUe+EWnAPMiBOawc84hxWlAiilrM0euXT OEFdwjEwjmC0EF29fhno3T6RG1ni0q1NrlysS5iufJhsASpMvzxXryaw+Huwwlt5mFuO EaYBpwBUAV4sq3LGRiy3NxhpUvwUl+HVabrFabEoRrHyLOyO8lrbYJwViPITOeWxWaEh rUjC+khqov/ZfLyn9XulDOQI7mgLcLB3HBYQhj9VeGRakX9sHXu4FRYoPXu3BI0wIETI 61Gw== X-Forwarded-Encrypted: i=1; AJvYcCUbD/n6jJ6LNMrVYDG8djKgXSyyWYyn+l7mnpCls7bQCpusV+9fNOQ9EuBQPRJv6WRgduxvaRiLxDEW4GgQWqHSccQ= X-Gm-Message-State: AOJu0YyVFA771Qj58qRLGUunb5+wLnKINRWefA0LmhRzF7uvtKpS7dRy s0iV7yNMGAxpxtewJGzbBhfWtIlBV/CGeu3c/8UkewjYbhuGx4pG8kt51BNxjqnf6qcJl5wn/G4 mKg== X-Google-Smtp-Source: AGHT+IGSwvnidbX0/rOv0INS3he6frJgJB+IsYvN52GAPjMVdVuwPoKCHiyeHTzcN2rL9OOwKUgSVjL4/4I= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a25:2fc1:0:b0:dc6:ebd4:cca2 with SMTP id v184-20020a252fc1000000b00dc6ebd4cca2mr528159ybv.11.1709749563843; Wed, 06 Mar 2024 10:26:03 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:35 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-38-surenb@google.com> Subject: [PATCH v5 37/37] memprofiling: Documentation From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Stat-Signature: d1s3k5c6yupe9o84eff51xitsqgg81jh X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 0C0F04000F X-HE-Tag: 1709749564-722991 X-HE-Meta: U2FsdGVkX1+eXOylHYgEEkcj94qRAl8m7VArospfPluBC65w72B8lYLKSfQ1UcZc8hE/cKWic0lrDg5dbBvLInsOL5IXiFyy6Z5iX0GMEWE4lxnotCjSmY5mUDsEpTPaox73SQsgSJAe9L4x7UB6PLNazEoO6ozFoj2CJMDECYjsWgyt3hrxerLcyM55reUDFy0LL7P4o/sFtxopnltaVav01VBdG6JtVVu14BW69jLmhWCa8N+NLI1lqjUjyw8oakH5ji0/MwIKg4kKPqQhHvb1dFWI1BbGmWglXtLbCH754rD96JxGwBNT+XL6m7bSkazHcsHbnusid91ElbV1qNk47UviSYd4haob13lFpD56xDtDKKohkTnVsraRmQAN8dlN4wb3yM/atmod18Ig1tVuKXHPAqUDYDwWOrnaHlXcrEO80mjjVnczS35up2RpMjzPnMgDidvAztTE7gHpbc8ThCQ+BoIuxwGm2bmpMd79Wi867PMbu+Agrj+TvDv2EroKELz+r+FFMoD6jpZ3DZF+z4ptLnYKiWoz9j2l5G+XaduAHJMZUtQvLPPgjRWzbbSVOVUU0bzhnQI1w3mtn0zeggzyacLKSB8OButVXsHc+bxnddrKhF/iOc2bLMv+3I3ZkggOJTn0MKYBhxbsUPoK06/61UQnjOluwDNsAMwJ6NZRWxnm6iC4YHgNFXV0Ti2JPk8BsVO39u06bxmT4JwlsVZSzkPrAzpyuNgEmvJ3p9LK4UDVXp4fF+svdyFBsR0OvurAFTZMzWcGnvewyM63h8ilNyXYfunxynA/gNbGaBuGDjfHu0GU5zxYs971WM7ph9fO5tiIXpdPR73hkwvSDkbFEHuz9S1NK0caD8dCgHFvhXIgL3f599K5jDr38k/JzCkxwnphDi5wI+L3P/jUsT3uhnO06TtikTtcz4Li6XDM84V7BK4Cps4WiN4g17oH0B+xgJIs8j75sjZ xnMgbLMt xmhYWcih5KAsDrYl4v8/e7sTCMxyMd4x83HcybVLoKP3mnoQcCzVwQlNosL35TXUzZbIjM6oHTJFHXg5dw6Gypii6/vfscIzO9tiByAt8SKXy4GKCesjbqTwDdgw4pWRG1tKAiy0cVE/24dPOAwrag6U0SuF99q5bgdLppumdk8GS3wdqbuqHtY9Z+dCjkBR8DA75j6CB8t4B8cIvjnP+lWM90b3RFno+QEFlCxopJ/R8f4E8KwpFKWGNVXHi9gjLUS8vTRube7M403e1FDZu0a0jzU6YdLeWVw7yz6AcLKG8h0SfCC9elBQXDnSTZLxYg26dqqVF7Y3HogTwGxifFvpLILX3ivUPQmQDJNb78n9HxUEq0f5JelNrHEHVN/Nr1372nk1YVSQlMEO86firjzzS5Eq2svaXr67N4AreTM3CVJYSsm1RlsJDppXk5TvuDXAA3qSqLyzdbW56ngGURjYGtFEb7nKIB1CvRfqH96fSpR1lql9P0Rp+pcOPGwymbrmn4oSghtJXxPdf/RpaavhIX6xVm2ggmbslwZ/42AaFIWOiNfAvOs4+2/5B5D6166EQ71ZEKWBWEppFfGIxQfTv3fiTegX7m2Gnk4avj3cmKs/npoMVDapcDdUsbSOTJShvV9eOcrYad7xyMId2cQPhjXQRkPxSrP2pWpDlIRTJ13ONCGE8poKpJQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet Provide documentation for memory allocation profiling. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- Documentation/mm/allocation-profiling.rst | 91 +++++++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100644 Documentation/mm/allocation-profiling.rst diff --git a/Documentation/mm/allocation-profiling.rst b/Documentation/mm/allocation-profiling.rst new file mode 100644 index 000000000000..8a862c7d3aab --- /dev/null +++ b/Documentation/mm/allocation-profiling.rst @@ -0,0 +1,91 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=========================== +MEMORY ALLOCATION PROFILING +=========================== + +Low overhead (suitable for production) accounting of all memory allocations, +tracked by file and line number. + +Usage: +kconfig options: + - CONFIG_MEM_ALLOC_PROFILING + - CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT + - CONFIG_MEM_ALLOC_PROFILING_DEBUG + adds warnings for allocations that weren't accounted because of a + missing annotation + +Boot parameter: + sysctl.vm.mem_profiling=0|1|never + + When set to "never", memory allocation profiling overheads is minimized and it + cannot be enabled at runtime (sysctl becomes read-only). + When CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y, default value is "1". + When CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n, default value is "never". + +sysctl: + /proc/sys/vm/mem_profiling + +Runtime info: + /proc/allocinfo + +Example output: + root@moria-kvm:~# sort -g /proc/allocinfo|tail|numfmt --to=iec + 2.8M 22648 fs/kernfs/dir.c:615 func:__kernfs_new_node + 3.8M 953 mm/memory.c:4214 func:alloc_anon_folio + 4.0M 1010 drivers/staging/ctagmod/ctagmod.c:20 [ctagmod] func:ctagmod_start + 4.1M 4 net/netfilter/nf_conntrack_core.c:2567 func:nf_ct_alloc_hashtable + 6.0M 1532 mm/filemap.c:1919 func:__filemap_get_folio + 8.8M 2785 kernel/fork.c:307 func:alloc_thread_stack_node + 13M 234 block/blk-mq.c:3421 func:blk_mq_alloc_rqs + 14M 3520 mm/mm_init.c:2530 func:alloc_large_system_hash + 15M 3656 mm/readahead.c:247 func:page_cache_ra_unbounded + 55M 4887 mm/slub.c:2259 func:alloc_slab_page + 122M 31168 mm/page_ext.c:270 func:alloc_page_ext +=================== +Theory of operation +=================== + +Memory allocation profiling builds off of code tagging, which is a library for +declaring static structs (that typcially describe a file and line number in +some way, hence code tagging) and then finding and operating on them at runtime +- i.e. iterating over them to print them in debugfs/procfs. + +To add accounting for an allocation call, we replace it with a macro +invocation, alloc_hooks(), that + - declares a code tag + - stashes a pointer to it in task_struct + - calls the real allocation function + - and finally, restores the task_struct alloc tag pointer to its previous value. + +This allows for alloc_hooks() calls to be nested, with the most recent one +taking effect. This is important for allocations internal to the mm/ code that +do not properly belong to the outer allocation context and should be counted +separately: for example, slab object extension vectors, or when the slab +allocates pages from the page allocator. + +Thus, proper usage requires determining which function in an allocation call +stack should be tagged. There are many helper functions that essentially wrap +e.g. kmalloc() and do a little more work, then are called in multiple places; +we'll generally want the accounting to happen in the callers of these helpers, +not in the helpers themselves. + +To fix up a given helper, for example foo(), do the following: + - switch its allocation call to the _noprof() version, e.g. kmalloc_noprof() + - rename it to foo_noprof() + - define a macro version of foo() like so: + #define foo(...) alloc_hooks(foo_noprof(__VA_ARGS__)) + +It's also possible to stash a pointer to an alloc tag in your own data structures. + +Do this when you're implementing a generic data structure that does allocations +"on behalf of" some other code - for example, the rhashtable code. This way, +instead of seeing a large line in /proc/allocinfo for rhashtable.c, we can +break it out by rhashtable type. + +To do so: + - Hook your data structure's init function, like any other allocation function + - Within your init function, use the convenience macro alloc_tag_record() to + record alloc tag in your data structure. + - Then, use the following form for your allocations: + alloc_hooks_tag(ht->your_saved_tag, kmalloc_noprof(...))