From patchwork Thu Mar 21 16:36:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599158 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CB3DCD11C2 for ; Thu, 21 Mar 2024 16:37:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC8286B008A; Thu, 21 Mar 2024 12:37:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C4C7F6B008C; Thu, 21 Mar 2024 12:37:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A2A686B0092; Thu, 21 Mar 2024 12:37:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 85FBE6B008A for ; Thu, 21 Mar 2024 12:37:16 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 02ACC1A0CBD for ; Thu, 21 Mar 2024 16:37:15 +0000 (UTC) X-FDA: 81921601272.29.9333E41 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf06.hostedemail.com (Postfix) with ESMTP id E9431180018 for ; Thu, 21 Mar 2024 16:37:13 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1wxBmp+4; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 3OGL8ZQYKCCUTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3OGL8ZQYKCCUTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039034; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yvHQT84Vpgs1c9BTtx/ZOKdQLP9IYCyPDqaPAHjwGfE=; b=eJrwcpLt9u609nD7AY8rjNUwavkTXVlVijKzpnGPkNhzc3mfwmLO/+I5OtDFJ6L1TLJvz3 H/Diz8K9w6ODQbWQRlDlQQJC4g9iVMJQa5nS3hEm6ry0sGkl/LXS1sHJxF5+B98f5E83SM hhjuP/fCYqGCPE6ZUaF2lLLFsxGAlhY= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1wxBmp+4; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 3OGL8ZQYKCCUTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3OGL8ZQYKCCUTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039034; a=rsa-sha256; cv=none; b=DmMBaMl9PKqEL0ERT9tY2xjVVaCLuA273OHcQh0lJTiSyI5C7sCpoc/nIB3GRAJcYi1lZp QXwAQ3Y5SJP79GOlXh6V9Lnk5zuyTTsZsCWw5fDtKXB4tj8W6e5phkZkbXX7JlRLRkMDcw fDCSgx4+3T0rd6PocSdccyqJtRSuWsg= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60a2386e932so22607347b3.1 for ; Thu, 21 Mar 2024 09:37:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039033; x=1711643833; darn=kvack.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=yvHQT84Vpgs1c9BTtx/ZOKdQLP9IYCyPDqaPAHjwGfE=; b=1wxBmp+449TLx7bExtG30Nrcf3ayTy9eaBq1xe++GgO8Q7fj/gdLausn/xF9CtNEkz oeCXvWbok/gNS1s+niFOOCNgwFTp8uCLNwKoRKvuIdo+Lt4Z2Igc/VdZOfjGsYpPJJUQ oLR4jseU9KH6bi6R1ZqwRdujfw4TWim3qd9gIV0NaV1xsikHa5l3IRDT3oqBaGpcbhLb keD/uKUjTdBM7bFd3HTMAsq6gZYAd2WwwKbVmIvAqekChz3+QGtYovOAczMgqZy2rbIa aJuHFj0CRN+kC/CRqUDQQZpNFSVyxPTs4tBpiGP5IAS1DSgWAEuK4FaqlrxszjDFUncx KZDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039033; x=1711643833; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=yvHQT84Vpgs1c9BTtx/ZOKdQLP9IYCyPDqaPAHjwGfE=; b=HbSSWJMoo5jBbbeCvWv/KZtWDnp02qzZbNRJUgf9rCoyBfDGc9g6kHGUkZKC/KRnVY 0OHNZw/0277l93gTHa/PllyDL/L2yE6B2ZpPjPQJGf0b+oxeo3jnncR2Pboa1nSXd1PI e+X7kcm1cFUP6D1nmJ/yPCS3NOXLGwAmE6ZIu45Lvx3GhNoeIqEum/YXGexODYYYQ+HY A750DYdosNYY4+kxYd5tMaDL/J24LU8rCnZNHfdJD4w7oezoqonG/Z3M0FA2QPebuSVX rxnB4ZDo8e71yk2LWom/7E6AnZZS0RErwIcUhP0dddvyLyqGCk2VpsKnbWJkmsm7gk48 aWtA== X-Forwarded-Encrypted: i=1; AJvYcCX0n/jl8sRR6na8441CTB9kEs6dHh7sIlEqwM3nRByShKPq0on9OfBj+eQ7znA+MtwVsWYbMZjrzqe1qpNWDYAfXEo= X-Gm-Message-State: AOJu0YwLSnz3S2/3wLn+DeD5KP/3mDWbNF7CPM/IWvSaO4A9+Y1RvMIj aNYv34hKUg276Ii7McNwcQXYPgEu8MBeV77unKvL6Sa3PqL3KLkx4VX/b1vlf/D5eQRRE2vhJwT 20A== X-Google-Smtp-Source: AGHT+IE0GrI48aMtLt86KJtvt2ABiO4vb9dUmX9df+RhMwB6AbI4ueJGxRCaSy1scrleMDbYTlfJxXp4A3s= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:18d1:b0:dc7:42:ecd with SMTP id ck17-20020a05690218d100b00dc700420ecdmr2405851ybb.6.1711039032771; Thu, 21 Mar 2024 09:37:12 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:23 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-2-surenb@google.com> Subject: [PATCH v6 01/37] fix missing vmalloc.h includes From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Stat-Signature: a34dygsaz7fd48c1673sfzd8mnb17gp5 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: E9431180018 X-HE-Tag: 1711039033-36214 X-HE-Meta: U2FsdGVkX198G/bd/QOz7uXZd/3kpH4D45nPjk4oV7C9n/GffFuYqMCF0xdZVFJ6VYdZQIZk3ktOHk+1DiDsFOapslsoUb3PiHgYksc052C8HCRT4x7B70liddlWgDrHm+cIVAlGCefayg+ptEqZNN5XPi6WJsC1oFEBsuJFz31aI08gOgVViGISEXs3kcdR0Xq5BhrRClWwKqTsL6HLS5NbfmpNxhFa0yHzbS1Q/oJK3VhAsjXCJnZlMV4AEpZfOppihgEZJbfV2cyiBRu+XDjcI2AupBnaMG2cepAdpWw2Jf8wVHXIti0/yaD+u5dlIdluMoQQdlgBDrssmeo7OmQwobUhRCpGe6Zv9LFcNEOr9H6YD8sMojvxt86w3m6rbazI+CC28My2OU0turBaCYSVIQoYWaLyFEAhUxLk8qpt8wMgSy8WvfHuBgcFrG9u60Y49lTff0YaM0+D1vWdQuFVYSRCBH9Z6kW9BNebSLTRL4w5EdfWl+2RF4B87Ht8uT8bOdXKn6JkOYeKxB8RAi3hLNlnFOLrrRtahS6tvznWQBLCOAcsQEqUZkwARe4k3KS55pMCav+iDAQAOHpLqeAd3+BUqx8N/KRe0ox27V9cq9HqX9yhnYwKbvXe6lprDSp9dwKn4bdGuscj3pLpgLlhV3auIym5AajDzK+7fUQR11ACzOmishHo8Obf8OIXEDYRnQRwHlfwXkilYh7yQE8BoArhB38WAHSwYlOVBTy5+I/oQuKWlTl26FSJkkfVLDEnHSYPvWD9SpcTdRjRWperPFQKeN5I1/L9t06R03/BkOgJoqvbpl0I3J1yQs6Bx6Fi+gvhKPe2LhkV0/15uO+eWON7aG80enD/9n5CBq/s4XlhsovLa4hp846YlnlOwZVNqvPOCN+cDzaWFbjKco6IEWWhWc+y9Q17NcLMfTXqHeFspRTE1yaS+6FWnvZTrf1pY84oNcQ9B2v2U1C 4Qaalosz k6kChRIi/NeagHnK3NqFF8JPZfLRDJDQXy6MSFYPC+EFAC3T31wpL9L6nqKluvxry1P1DamyQgYEvdRRXj9T39pf3K4ptKOu5c5/8Zr6EusfbiKuYXXY+/yNncb4/zdPQ4f4XqmlUTStg94QlDPRdXHfmT5sBBMhdNa90H7Iy5Ww7yvV1ZhuMeKjg0vKjv0Mbd46fkOUsh6yGijhYGt5T7qoUF+OQsnvb8HWKlNp3hIcwo8xJzPHUDjFZtCFdj8BVo/u6ADnbUd7DbtjGHHqMRm6bLRvwFe0nK0L8iVAcgr3E1rDAQ2SHu79p/cinGOsI9fVvXqwaOKh9fqcvisAFKwfgtfvH10NLYk5A/WFto0NtbyjiXTne9W+w3tfC2ie/AztYEozMg9d8AraDJni1wM1H8ZuchgZfDxNZsYPwQfCkodw/n5SBhUdY7sdnO8T7c+NgpC68TjwP6loeIAJ6QELN58KziSelhXoz+++FIE+zCh3E5YikH3z9/V3OJlaU97xvoyY0bOpWbiDiKcfzAPcNJwnIn+8kdo3s+dC+Zls7utQ/lQK8f/+mEsSfuDCV/xJiN1YE1m/A53lAw3B+XZ6f/DZYx7gtIcf+DmaJ4+i5VPJNo9qVvhgor4gAZbJUESHW+kmzPc+KzCid2VNUMzqPN0ZXJzuZtfaUq+eYv8aa9jqZ2M+WHL7kT95nL1kpFadTF/gwvXh9ut6DqXlvqE9pe1yG4pkGZYZMZ/X4bAHgxtQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet The next patch drops vmalloc.h from a system header in order to fix a circular dependency; this adds it to all the files that were pulling it in implicitly. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Reviewed-by: Pasha Tatashin --- arch/alpha/lib/checksum.c | 1 + arch/alpha/lib/fpreg.c | 1 + arch/alpha/lib/memcpy.c | 1 + arch/arm/kernel/irq.c | 1 + arch/arm/kernel/traps.c | 1 + arch/arm64/kernel/efi.c | 1 + arch/loongarch/include/asm/kfence.h | 1 + arch/powerpc/kernel/iommu.c | 1 + arch/powerpc/mm/mem.c | 1 + arch/riscv/kernel/elf_kexec.c | 1 + arch/riscv/kernel/probes/kprobes.c | 1 + arch/s390/kernel/cert_store.c | 1 + arch/s390/kernel/ipl.c | 1 + arch/x86/include/asm/io.h | 1 + arch/x86/kernel/cpu/sgx/main.c | 1 + arch/x86/kernel/irq_64.c | 1 + arch/x86/mm/fault.c | 1 + drivers/accel/ivpu/ivpu_mmu_context.c | 1 + drivers/gpu/drm/gma500/mmu.c | 1 + drivers/gpu/drm/i915/gem/i915_gem_pages.c | 1 + drivers/gpu/drm/i915/gem/selftests/mock_dmabuf.c | 1 + drivers/gpu/drm/i915/gt/shmem_utils.c | 1 + drivers/gpu/drm/i915/gvt/firmware.c | 1 + drivers/gpu/drm/i915/gvt/gtt.c | 1 + drivers/gpu/drm/i915/gvt/handlers.c | 1 + drivers/gpu/drm/i915/gvt/mmio.c | 1 + drivers/gpu/drm/i915/gvt/vgpu.c | 1 + drivers/gpu/drm/i915/intel_gvt.c | 1 + drivers/gpu/drm/imagination/pvr_vm_mips.c | 1 + drivers/gpu/drm/mediatek/mtk_drm_gem.c | 1 + drivers/gpu/drm/omapdrm/omap_gem.c | 1 + drivers/gpu/drm/v3d/v3d_bo.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_binding.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_cmd.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_devcaps.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_drv.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c | 1 + drivers/gpu/drm/vmwgfx/vmwgfx_ioctl.c | 1 + drivers/gpu/drm/xen/xen_drm_front_gem.c | 1 + drivers/hwtracing/coresight/coresight-trbe.c | 1 + drivers/net/ethernet/marvell/octeon_ep/octep_pfvf_mbox.c | 1 + drivers/net/ethernet/marvell/octeon_ep_vf/octep_vf_mbox.c | 1 + drivers/net/ethernet/microsoft/mana/hw_channel.c | 1 + drivers/platform/x86/uv_sysfs.c | 1 + drivers/scsi/mpi3mr/mpi3mr_transport.c | 2 ++ drivers/vfio/pci/pds/dirty.c | 1 + drivers/virt/acrn/mm.c | 1 + drivers/virtio/virtio_mem.c | 1 + include/linux/pds/pds_common.h | 2 ++ include/rdma/rdmavt_qp.h | 1 + mm/debug_vm_pgtable.c | 1 + sound/pci/hda/cs35l41_hda.c | 1 + 52 files changed, 54 insertions(+) diff --git a/arch/alpha/lib/checksum.c b/arch/alpha/lib/checksum.c index 3f35c3ed6948..c29b98ef9c82 100644 --- a/arch/alpha/lib/checksum.c +++ b/arch/alpha/lib/checksum.c @@ -14,6 +14,7 @@ #include #include +#include static inline unsigned short from64to16(unsigned long x) { diff --git a/arch/alpha/lib/fpreg.c b/arch/alpha/lib/fpreg.c index 7c08b225261c..3d32165043f8 100644 --- a/arch/alpha/lib/fpreg.c +++ b/arch/alpha/lib/fpreg.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #if defined(CONFIG_ALPHA_EV6) || defined(CONFIG_ALPHA_EV67) diff --git a/arch/alpha/lib/memcpy.c b/arch/alpha/lib/memcpy.c index cbac3dc6d963..0e536a1a39ff 100644 --- a/arch/alpha/lib/memcpy.c +++ b/arch/alpha/lib/memcpy.c @@ -18,6 +18,7 @@ #include #include +#include /* * This should be done in one go with ldq_u*2/mask/stq_u. Do it diff --git a/arch/arm/kernel/irq.c b/arch/arm/kernel/irq.c index fe28fc1f759d..dab42d066d06 100644 --- a/arch/arm/kernel/irq.c +++ b/arch/arm/kernel/irq.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c index 3bad79db5d6e..27addbf0f98c 100644 --- a/arch/arm/kernel/traps.c +++ b/arch/arm/kernel/traps.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include diff --git a/arch/arm64/kernel/efi.c b/arch/arm64/kernel/efi.c index 9afcc690fe73..4a92096db34e 100644 --- a/arch/arm64/kernel/efi.c +++ b/arch/arm64/kernel/efi.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include diff --git a/arch/loongarch/include/asm/kfence.h b/arch/loongarch/include/asm/kfence.h index 6c82aea1c993..54062656dc7b 100644 --- a/arch/loongarch/include/asm/kfence.h +++ b/arch/loongarch/include/asm/kfence.h @@ -10,6 +10,7 @@ #define _ASM_LOONGARCH_KFENCE_H #include +#include #include #include diff --git a/arch/powerpc/kernel/iommu.c b/arch/powerpc/kernel/iommu.c index 1185efebf032..65468d0829e1 100644 --- a/arch/powerpc/kernel/iommu.c +++ b/arch/powerpc/kernel/iommu.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include #include diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c index 3a440004b97d..a197d4c2244b 100644 --- a/arch/powerpc/mm/mem.c +++ b/arch/powerpc/mm/mem.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include diff --git a/arch/riscv/kernel/elf_kexec.c b/arch/riscv/kernel/elf_kexec.c index 54260c16f991..11c0d2e0becf 100644 --- a/arch/riscv/kernel/elf_kexec.c +++ b/arch/riscv/kernel/elf_kexec.c @@ -19,6 +19,7 @@ #include #include #include +#include #include int arch_kimage_file_post_load_cleanup(struct kimage *image) diff --git a/arch/riscv/kernel/probes/kprobes.c b/arch/riscv/kernel/probes/kprobes.c index 2f08c14a933d..71a8b8945b26 100644 --- a/arch/riscv/kernel/probes/kprobes.c +++ b/arch/riscv/kernel/probes/kprobes.c @@ -6,6 +6,7 @@ #include #include #include +#include #include #include #include diff --git a/arch/s390/kernel/cert_store.c b/arch/s390/kernel/cert_store.c index 554447768bdd..bf983513dd33 100644 --- a/arch/s390/kernel/cert_store.c +++ b/arch/s390/kernel/cert_store.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include diff --git a/arch/s390/kernel/ipl.c b/arch/s390/kernel/ipl.c index 1486350a4177..5d6d381aa0be 100644 --- a/arch/s390/kernel/ipl.c +++ b/arch/s390/kernel/ipl.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include diff --git a/arch/x86/include/asm/io.h b/arch/x86/include/asm/io.h index 294cd2a40818..7452fc193b4f 100644 --- a/arch/x86/include/asm/io.h +++ b/arch/x86/include/asm/io.h @@ -42,6 +42,7 @@ #include #include #include +#include #define build_mmio_read(name, size, type, reg, barrier) \ static inline type name(const volatile void __iomem *addr) \ diff --git a/arch/x86/kernel/cpu/sgx/main.c b/arch/x86/kernel/cpu/sgx/main.c index 166692f2d501..27892e57c4ef 100644 --- a/arch/x86/kernel/cpu/sgx/main.c +++ b/arch/x86/kernel/cpu/sgx/main.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include "driver.h" #include "encl.h" diff --git a/arch/x86/kernel/irq_64.c b/arch/x86/kernel/irq_64.c index fe0c859873d1..ade0043ce56e 100644 --- a/arch/x86/kernel/irq_64.c +++ b/arch/x86/kernel/irq_64.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 622d12ec7f08..a4cc20d0036d 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -20,6 +20,7 @@ #include /* efi_crash_gracefully_on_page_fault()*/ #include #include /* find_and_lock_vma() */ +#include #include /* boot_cpu_has, ... */ #include /* dotraplinkage, ... */ diff --git a/drivers/accel/ivpu/ivpu_mmu_context.c b/drivers/accel/ivpu/ivpu_mmu_context.c index fe6161299236..128aef8e5a19 100644 --- a/drivers/accel/ivpu/ivpu_mmu_context.c +++ b/drivers/accel/ivpu/ivpu_mmu_context.c @@ -6,6 +6,7 @@ #include #include #include +#include #include diff --git a/drivers/gpu/drm/gma500/mmu.c b/drivers/gpu/drm/gma500/mmu.c index a70b01ccdf70..4d78b33eaa82 100644 --- a/drivers/gpu/drm/gma500/mmu.c +++ b/drivers/gpu/drm/gma500/mmu.c @@ -5,6 +5,7 @@ **************************************************************************/ #include +#include #include "mmu.h" #include "psb_drv.h" diff --git a/drivers/gpu/drm/i915/gem/i915_gem_pages.c b/drivers/gpu/drm/i915/gem/i915_gem_pages.c index 0ba955611dfb..8780aa243105 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_pages.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_pages.c @@ -5,6 +5,7 @@ */ #include +#include #include "gt/intel_gt.h" #include "gt/intel_tlb.h" diff --git a/drivers/gpu/drm/i915/gem/selftests/mock_dmabuf.c b/drivers/gpu/drm/i915/gem/selftests/mock_dmabuf.c index b2a5882b8f81..075657018739 100644 --- a/drivers/gpu/drm/i915/gem/selftests/mock_dmabuf.c +++ b/drivers/gpu/drm/i915/gem/selftests/mock_dmabuf.c @@ -4,6 +4,7 @@ * Copyright © 2016 Intel Corporation */ +#include #include "mock_dmabuf.h" static struct sg_table *mock_map_dma_buf(struct dma_buf_attachment *attachment, diff --git a/drivers/gpu/drm/i915/gt/shmem_utils.c b/drivers/gpu/drm/i915/gt/shmem_utils.c index bccc3a1200bc..1fb6ff77fd89 100644 --- a/drivers/gpu/drm/i915/gt/shmem_utils.c +++ b/drivers/gpu/drm/i915/gt/shmem_utils.c @@ -7,6 +7,7 @@ #include #include #include +#include #include "i915_drv.h" #include "gem/i915_gem_object.h" diff --git a/drivers/gpu/drm/i915/gvt/firmware.c b/drivers/gpu/drm/i915/gvt/firmware.c index 4dd52ac2043e..d800d267f0e9 100644 --- a/drivers/gpu/drm/i915/gvt/firmware.c +++ b/drivers/gpu/drm/i915/gvt/firmware.c @@ -30,6 +30,7 @@ #include #include +#include #include "i915_drv.h" #include "gvt.h" diff --git a/drivers/gpu/drm/i915/gvt/gtt.c b/drivers/gpu/drm/i915/gvt/gtt.c index 094fca9b0e73..58cca4906f41 100644 --- a/drivers/gpu/drm/i915/gvt/gtt.c +++ b/drivers/gpu/drm/i915/gvt/gtt.c @@ -39,6 +39,7 @@ #include "trace.h" #include "gt/intel_gt_regs.h" +#include #if defined(VERBOSE_DEBUG) #define gvt_vdbg_mm(fmt, args...) gvt_dbg_mm(fmt, ##args) diff --git a/drivers/gpu/drm/i915/gvt/handlers.c b/drivers/gpu/drm/i915/gvt/handlers.c index efcb00472be2..ea9c30092767 100644 --- a/drivers/gpu/drm/i915/gvt/handlers.c +++ b/drivers/gpu/drm/i915/gvt/handlers.c @@ -52,6 +52,7 @@ #include "display/skl_watermark_regs.h" #include "display/vlv_dsi_pll_regs.h" #include "gt/intel_gt_regs.h" +#include /* XXX FIXME i915 has changed PP_XXX definition */ #define PCH_PP_STATUS _MMIO(0xc7200) diff --git a/drivers/gpu/drm/i915/gvt/mmio.c b/drivers/gpu/drm/i915/gvt/mmio.c index 5b5def6ddef7..780762f28aa4 100644 --- a/drivers/gpu/drm/i915/gvt/mmio.c +++ b/drivers/gpu/drm/i915/gvt/mmio.c @@ -33,6 +33,7 @@ * */ +#include #include "i915_drv.h" #include "i915_reg.h" #include "gvt.h" diff --git a/drivers/gpu/drm/i915/gvt/vgpu.c b/drivers/gpu/drm/i915/gvt/vgpu.c index 08ad1bd651f1..63c751ca4119 100644 --- a/drivers/gpu/drm/i915/gvt/vgpu.c +++ b/drivers/gpu/drm/i915/gvt/vgpu.c @@ -34,6 +34,7 @@ #include "i915_drv.h" #include "gvt.h" #include "i915_pvinfo.h" +#include void populate_pvinfo_page(struct intel_vgpu *vgpu) { diff --git a/drivers/gpu/drm/i915/intel_gvt.c b/drivers/gpu/drm/i915/intel_gvt.c index 9b6d87c8b583..5a01d60e5186 100644 --- a/drivers/gpu/drm/i915/intel_gvt.c +++ b/drivers/gpu/drm/i915/intel_gvt.c @@ -28,6 +28,7 @@ #include "gt/intel_context.h" #include "gt/intel_ring.h" #include "gt/shmem_utils.h" +#include /** * DOC: Intel GVT-g host support diff --git a/drivers/gpu/drm/imagination/pvr_vm_mips.c b/drivers/gpu/drm/imagination/pvr_vm_mips.c index b7fef3c797e6..6563dcde109c 100644 --- a/drivers/gpu/drm/imagination/pvr_vm_mips.c +++ b/drivers/gpu/drm/imagination/pvr_vm_mips.c @@ -14,6 +14,7 @@ #include #include #include +#include /** * pvr_vm_mips_init() - Initialise MIPS FW pagetable diff --git a/drivers/gpu/drm/mediatek/mtk_drm_gem.c b/drivers/gpu/drm/mediatek/mtk_drm_gem.c index 4f2e3feabc0f..3e519869b632 100644 --- a/drivers/gpu/drm/mediatek/mtk_drm_gem.c +++ b/drivers/gpu/drm/mediatek/mtk_drm_gem.c @@ -4,6 +4,7 @@ */ #include +#include #include #include diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c index 3421e8389222..9ea0c64c26b5 100644 --- a/drivers/gpu/drm/omapdrm/omap_gem.c +++ b/drivers/gpu/drm/omapdrm/omap_gem.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include diff --git a/drivers/gpu/drm/v3d/v3d_bo.c b/drivers/gpu/drm/v3d/v3d_bo.c index a07ede668cc1..a165cbcdd27b 100644 --- a/drivers/gpu/drm/v3d/v3d_bo.c +++ b/drivers/gpu/drm/v3d/v3d_bo.c @@ -21,6 +21,7 @@ #include #include +#include #include "v3d_drv.h" #include "uapi/drm/v3d_drm.h" diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_binding.c b/drivers/gpu/drm/vmwgfx/vmwgfx_binding.c index ae2de914eb89..2731f6ded1c2 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_binding.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_binding.c @@ -54,6 +54,7 @@ #include "vmwgfx_drv.h" #include "vmwgfx_binding.h" #include "device_include/svga3d_reg.h" +#include #define VMW_BINDING_RT_BIT 0 #define VMW_BINDING_PS_BIT 1 diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_cmd.c b/drivers/gpu/drm/vmwgfx/vmwgfx_cmd.c index 195ff8792e5a..dd4ca6a9c690 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_cmd.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_cmd.c @@ -31,6 +31,7 @@ #include #include +#include bool vmw_supports_3d(struct vmw_private *dev_priv) { diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_devcaps.c b/drivers/gpu/drm/vmwgfx/vmwgfx_devcaps.c index 829df395c2ed..6e6beff9e262 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_devcaps.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_devcaps.c @@ -25,6 +25,7 @@ * **************************************************************************/ +#include #include "vmwgfx_devcaps.h" #include "vmwgfx_drv.h" diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c index d3e308fdfd5b..7a451410ad77 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_drv.c @@ -53,6 +53,7 @@ #include #include #include +#include #define VMWGFX_DRIVER_DESC "Linux drm driver for VMware graphics devices" diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c b/drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c index cc3086e649eb..2e52d73eba48 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c @@ -35,6 +35,7 @@ #include #include +#include /* * Helper macro to get dx_ctx_node if available otherwise print an error diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_ioctl.c b/drivers/gpu/drm/vmwgfx/vmwgfx_ioctl.c index a1da5678c731..835d1eed8dd9 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_ioctl.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_ioctl.c @@ -31,6 +31,7 @@ #include #include +#include int vmw_getparam_ioctl(struct drm_device *dev, void *data, struct drm_file *file_priv) diff --git a/drivers/gpu/drm/xen/xen_drm_front_gem.c b/drivers/gpu/drm/xen/xen_drm_front_gem.c index 3ad2b4cfd1f0..63112ed975c4 100644 --- a/drivers/gpu/drm/xen/xen_drm_front_gem.c +++ b/drivers/gpu/drm/xen/xen_drm_front_gem.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c index 6136776482e6..96a32b213669 100644 --- a/drivers/hwtracing/coresight/coresight-trbe.c +++ b/drivers/hwtracing/coresight/coresight-trbe.c @@ -17,6 +17,7 @@ #include #include +#include #include "coresight-self-hosted-trace.h" #include "coresight-trbe.h" diff --git a/drivers/net/ethernet/marvell/octeon_ep/octep_pfvf_mbox.c b/drivers/net/ethernet/marvell/octeon_ep/octep_pfvf_mbox.c index 2e2c3be8a0b4..e6eb98d70f3c 100644 --- a/drivers/net/ethernet/marvell/octeon_ep/octep_pfvf_mbox.c +++ b/drivers/net/ethernet/marvell/octeon_ep/octep_pfvf_mbox.c @@ -15,6 +15,7 @@ #include #include #include +#include #include "octep_config.h" #include "octep_main.h" diff --git a/drivers/net/ethernet/marvell/octeon_ep_vf/octep_vf_mbox.c b/drivers/net/ethernet/marvell/octeon_ep_vf/octep_vf_mbox.c index 2eab21e43048..445b626efe11 100644 --- a/drivers/net/ethernet/marvell/octeon_ep_vf/octep_vf_mbox.c +++ b/drivers/net/ethernet/marvell/octeon_ep_vf/octep_vf_mbox.c @@ -7,6 +7,7 @@ #include #include #include +#include #include "octep_vf_config.h" #include "octep_vf_main.h" diff --git a/drivers/net/ethernet/microsoft/mana/hw_channel.c b/drivers/net/ethernet/microsoft/mana/hw_channel.c index 2729a2c5acf9..11021c34e47e 100644 --- a/drivers/net/ethernet/microsoft/mana/hw_channel.c +++ b/drivers/net/ethernet/microsoft/mana/hw_channel.c @@ -3,6 +3,7 @@ #include #include +#include static int mana_hwc_get_msg_index(struct hw_channel_context *hwc, u16 *msg_id) { diff --git a/drivers/platform/x86/uv_sysfs.c b/drivers/platform/x86/uv_sysfs.c index 38d1b692d3c0..40e010877189 100644 --- a/drivers/platform/x86/uv_sysfs.c +++ b/drivers/platform/x86/uv_sysfs.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include diff --git a/drivers/scsi/mpi3mr/mpi3mr_transport.c b/drivers/scsi/mpi3mr/mpi3mr_transport.c index d32ad46318cb..dabb91f0f75d 100644 --- a/drivers/scsi/mpi3mr/mpi3mr_transport.c +++ b/drivers/scsi/mpi3mr/mpi3mr_transport.c @@ -7,6 +7,8 @@ * */ +#include + #include "mpi3mr.h" /** diff --git a/drivers/vfio/pci/pds/dirty.c b/drivers/vfio/pci/pds/dirty.c index 68e8f006dfdb..c51f5e4c3dd6 100644 --- a/drivers/vfio/pci/pds/dirty.c +++ b/drivers/vfio/pci/pds/dirty.c @@ -3,6 +3,7 @@ #include #include +#include #include #include diff --git a/drivers/virt/acrn/mm.c b/drivers/virt/acrn/mm.c index fa5d9ca6be57..c088ee1f1180 100644 --- a/drivers/virt/acrn/mm.c +++ b/drivers/virt/acrn/mm.c @@ -12,6 +12,7 @@ #include #include #include +#include #include "acrn_drv.h" diff --git a/drivers/virtio/virtio_mem.c b/drivers/virtio/virtio_mem.c index 8e3223294442..e8355f55a8f7 100644 --- a/drivers/virtio/virtio_mem.c +++ b/drivers/virtio/virtio_mem.c @@ -21,6 +21,7 @@ #include #include #include +#include #include diff --git a/include/linux/pds/pds_common.h b/include/linux/pds/pds_common.h index 30581e2e04cc..5802e1deef24 100644 --- a/include/linux/pds/pds_common.h +++ b/include/linux/pds/pds_common.h @@ -4,6 +4,8 @@ #ifndef _PDS_COMMON_H_ #define _PDS_COMMON_H_ +#include + #define PDS_CORE_DRV_NAME "pds_core" /* the device's internal addressing uses up to 52 bits */ diff --git a/include/rdma/rdmavt_qp.h b/include/rdma/rdmavt_qp.h index 2e58d5e6ac0e..d67892944193 100644 --- a/include/rdma/rdmavt_qp.h +++ b/include/rdma/rdmavt_qp.h @@ -11,6 +11,7 @@ #include #include #include +#include /* * Atomic bit definitions for r_aflags. */ diff --git a/mm/debug_vm_pgtable.c b/mm/debug_vm_pgtable.c index 65c19025da3d..f1c9a2c5abc0 100644 --- a/mm/debug_vm_pgtable.c +++ b/mm/debug_vm_pgtable.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include diff --git a/sound/pci/hda/cs35l41_hda.c b/sound/pci/hda/cs35l41_hda.c index d3fa6e136744..990b5bd717a1 100644 --- a/sound/pci/hda/cs35l41_hda.c +++ b/sound/pci/hda/cs35l41_hda.c @@ -13,6 +13,7 @@ #include #include #include +#include #include "hda_local.h" #include "hda_auto_parser.h" #include "hda_jack.h" From patchwork Thu Mar 21 16:36:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599159 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7386EC6FD1F for ; Thu, 21 Mar 2024 16:37:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 12C136B008C; Thu, 21 Mar 2024 12:37:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B7706B0092; Thu, 21 Mar 2024 12:37:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD37F6B0093; Thu, 21 Mar 2024 12:37:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C394B6B008C for ; Thu, 21 Mar 2024 12:37:17 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 89CEA1C0D82 for ; Thu, 21 Mar 2024 16:37:17 +0000 (UTC) X-FDA: 81921601314.17.E1EC519 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf21.hostedemail.com (Postfix) with ESMTP id BD5B71C0019 for ; Thu, 21 Mar 2024 16:37:15 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=mEZhD7dN; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of 3OmL8ZQYKCCcVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3OmL8ZQYKCCcVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039035; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WfR76FD4Au5CxbM75qQNdquNFQiSYxURTySBYiFO3v0=; b=3P7qNcXXki11GeD4U0NGn+GTCYL/I9hAyk5uLdpdcOIyYXOm81rgD81VirEOpTWK77qtJ0 9yxJLllecccLapmAzEqQtHbZMoGOiz8o2ARnKhYJ0NfrDx9K7PgYrgbzSOIAeoMh5JbV4M lscNSP1gwyrT3u9gftD4OVOcAN0Yxmg= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=mEZhD7dN; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of 3OmL8ZQYKCCcVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3OmL8ZQYKCCcVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039035; a=rsa-sha256; cv=none; b=sdYtCUWnhjL9XpiUrxcf/2WxyBuWNX+3A+aZj/m/Ez+Yx4kGWkviHysiAjm0yNISG2d+19 bYNoTm6scQj4tOq4JFZD2vRkikOkf+c0Wd9T/fRETPEqdN5nY0gJAL0Rovfy7fm6/o/F04 J4LZfYdzQ36yzbXy9izydvLtUDSDdj4= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc743cc50a6so1520456276.2 for ; Thu, 21 Mar 2024 09:37:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039035; x=1711643835; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=WfR76FD4Au5CxbM75qQNdquNFQiSYxURTySBYiFO3v0=; b=mEZhD7dNv/Fiyo31h2tkpGYJY3yyE77gTic2qHEJ+m1u+L8w1PFBxIvv6hJlTYlb38 /XGeackbQRZjTfnrpJ+nJbjX8S4pzLoPI01i5ebfgjTpsnK26xrcnOXYP78zrZU1mVT9 ymsAppYFUUvLuCocqC5thBFmBIKLlAAXiftTINof6e98cLvajP2x6mbnLBi7Erw1hP7K ksSed5oJXAt5dL0V8KO0kHelljHZCsZm/in1D2yQwLzCtyjWTsVC0PHdhOt3yl1dQjq6 sAOEGv4xA8w2cZ0AG0aFTGOiIwB6QNWbkCZDaFV06xtD+xesG23jBvvudW+IPTIkz2dY UPzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039035; x=1711643835; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WfR76FD4Au5CxbM75qQNdquNFQiSYxURTySBYiFO3v0=; b=iBke1QpcOsoMs8OOku9m55sagvAvN50SLIWQ4zJ3udyHuHRaCn11KLdXVCY+TUmOY3 VqA4Y+IQIB+EvKETgCrspCaTEDUGkCKtDDOf4P0DHxV25Iqfp/SRpWGKlNKwrfJWjJmT BOVstPP+7rPGMB1plRU2Wkr7ZPOwKDSRFtCyLxi0a5qr53xB9jxoubiHW0khDwEnbivb kBUj+VqmELNMZk07H4uAIHpaW4Xf5PKymrOLnftvI4Y82iwoI7c7akVR6tOl4W/5cKBy TTE63AjbEXR1AxU1mU4WjzrgGT7grfemgM24vzXUcNjsoom252gvpdYq9WfoFNfbl3Qs PVVg== X-Forwarded-Encrypted: i=1; AJvYcCX9jETgEDD+uwCS9f8RS8Lf/ExPv2nOrgD+vjGlWRvEYZ+ywAqW/9XKNZhsDJxWFzvtDN6u4WQHSLT84U6FWZCFdJo= X-Gm-Message-State: AOJu0YwI3x3ql2vE6G5MDv7V4GdmtDvXIWrKCSqoxHTnbE0sUi9Wy0Lo 37f8ivUFg/fZigD4bLrkpGcXr6pqxRrkVjztVyg6F9yFWNXBVzect8+G7i3OnQwkxVVo5V5lUNK Ipw== X-Google-Smtp-Source: AGHT+IFhiFlsQoZCLjLg/oNv4a6rqVLDOqKxMR6InuX/fTP6XDr/Ywkc4XNBxFj0+HwVU6WiRQiaVUB3f4E= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:102a:b0:dc7:4af0:8c6c with SMTP id x10-20020a056902102a00b00dc74af08c6cmr1219995ybt.6.1711039034585; Thu, 21 Mar 2024 09:37:14 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:24 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-3-surenb@google.com> Subject: [PATCH v6 02/37] asm-generic/io.h: Kill vmalloc.h dependency From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: BD5B71C0019 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: kx3jeqjkuq57tao9q3dgn15biauijpr4 X-HE-Tag: 1711039035-674128 X-HE-Meta: U2FsdGVkX19VynE5b48igkLVt7P8NHNwygA0E/ntygwyaV2NIVI7x2o6YxUYz4nuBbwXjGa+77YrT+o3kppuJ9H9HLeLUXyuGVFz2Jq+Nh6rCmUDZO1zF7ZndKh7mq91H9HCCRyGvpJjNmm1u7XOTQMlEnaRVD8Lfge2FPfBL03usz1ICHQDiPVfksUXGAQn4cWGJ4aWzS9Cu6ICi1PBy9Ivm44xcM4LLNRSS1LVdKibzpRDQLhOUELZhW0+o2JlwXUKtEBiNs9uFiDz8/O/ZcYoXBy8umav9+pXT6YxikLZpZ/Jo2XzLOwOrJm4/seQnZDLuSSVOtarFBxolmxvC+MU09OoNEUOTKrljnh5zXzSIy/Yh5u8eIMwUpgKc+RWOCYURg/epqPKrxbVsPiaKZJaoS+QqSmZos6BT4BwoAIKdvFGIE1Mjl4q5Xty540qkMTMHo90J3Z8kIUsO0QvKsTMUg9Ms8l6ho+W2LLjkJJgRGiiXzD5g15PiHu/Nvlpi/fCWmujiX58aqOwSkyEQXZbTMz5MHdSMVXznRrA48JPc594YT723hOlKkBnecLoUdBJYx7AIRsyoIOaTFcPc8o12MMX3crv0F1I/6MwH/PwSiM6dsq2k0cGAq1WEjDXo4v9qiGQM190IU+RxWUdWUVL1kW9iPLoy1Exo4SegxHERpkoZUfplbNSjML0w2BaoysB4ReWyTX9KuELllrzvAxJ533gUp0XxWzKIRvbb5W0sLsnr26QfN+aKXLKsw1y+NLI0GOvPmvSooGpo5kPg884zQkGJBNRL8t/yrdZPXQJbgzImb1KDNie3Ko0NSvUy+4UW2qQbrvsoiS11T7gGj7a/VTqsPbO+IK185tC7hGUlzLh9eUVsAVgoCTKoNRTfIKD6fRedV+oExLgzG5dT0b9P8K3Fs3wJB/XI9qrBBBs+wRO75mBwsjGUXFBFXJMhjPmQ2P1ISSE9Cyp56w HZqqdLUR somkjaNekHqAWhpiFe5jXsG2K9CGtLTezJq+IilaK5Yy56hkHxAoZuRWl2aLsKQf5PljyfWVhHn4MFYt+AaGWeZPRqJ7tX+dVecunjttagR+eC7v1RhA2eZOgEFLEI6m+bIYK2scZGlnu14D3DYox6MemHNWsvLA3s67f3naJ65Bd69BGatyTNbCIIUuLbFp+H8WzNo8YGLQmqnWXc7H7Qz3TJlJT+vl0qXHsYXHSqOfQ/DPo5QmImCepL5ONKgbzuvWdqVrs57Sq+adYbHj0Zi7JvAWSMkk/pGy5G84sPBIN+8M2oXbVexWJF3ExfiKuhuz+DIM9nrlZLdKCJPGmbG78jlczr7DxtuL9HHt8fJUZkgShl3pY23SITeWjzCgptbdDUlA5Y6ekDtVOSTE/gXbOZJ4BELEjclhOjmvT0kc8G73NO4lGKrE8AsH0/GX7imN4Z+BEPPV9am0jeh+jYVXb0NGkvlPabw6c5L9FNfk81MsIte02/zXtYbMhFNcVUwb3XqwIPv084V226xrAjL/xhFqelj6YUqncOAbY8qV161ij4OS6xqWDs3BKZpsDkmFmOvvmTBZdJn9oO0I0t9Rahl3XP5xLZPBT4Si77jPoBKNsF/UAGzRhpX3KUUKs9YAVtLeawWWIalTS1f4d5nJUDHEPGdpX6sx6CmGp1MTbYtV4R7GYwrwm4Fx2ZzaT3kAoVG9/y5PwrbyAzxX4/51KkeyfD7AVzYmKl1CtkueQDwA+PUJ2rhrTniekIq+LPWYJ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000015, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet Needed to avoid a new circular dependency with the memory allocation profiling series. Naturally, a whole bunch of files needed to include vmalloc.h that were previously getting it implicitly. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Reviewed-by: Pasha Tatashin --- include/asm-generic/io.h | 1 - 1 file changed, 1 deletion(-) diff --git a/include/asm-generic/io.h b/include/asm-generic/io.h index bac63e874c7b..c27313414a82 100644 --- a/include/asm-generic/io.h +++ b/include/asm-generic/io.h @@ -991,7 +991,6 @@ static inline void iowrite64_rep(volatile void __iomem *addr, #ifdef __KERNEL__ -#include #define __io_virt(x) ((void __force *)(x)) /* From patchwork Thu Mar 21 16:36:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599160 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1776DC6FD1F for ; Thu, 21 Mar 2024 16:37:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3028F6B0092; Thu, 21 Mar 2024 12:37:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F1AB6B0093; Thu, 21 Mar 2024 12:37:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F34256B0095; Thu, 21 Mar 2024 12:37:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D3A766B0092 for ; Thu, 21 Mar 2024 12:37:19 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 7B4504026B for ; Thu, 21 Mar 2024 16:37:19 +0000 (UTC) X-FDA: 81921601398.23.921E6A2 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf16.hostedemail.com (Postfix) with ESMTP id A84FA180003 for ; Thu, 21 Mar 2024 16:37:17 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=P3DR2DT9; spf=pass (imf16.hostedemail.com: domain of 3PGL8ZQYKCCkXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3PGL8ZQYKCCkXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039037; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cRKfaiB6dOVHxpZUYVy57amQHTHiIgtHPVa28h/bxKY=; b=jlcAjePlVmNu2sF74bKAZe+q420lNFYnErZQSGMhpnOOpalSI0f0OqOWz7L33Ni+W+lPT1 hDlvl8lt0tTCsi5DYKCgvj6boX50MGibinbsNk2tnzQeevQEMxlAr0QjHCMQHOihH/t0Gx fX9mhNh43dMBvJUKHqoI71LLjZ4LBaM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039037; a=rsa-sha256; cv=none; b=rTVX/DiPRNUGNssftybMpWFShgBeNoeRYnkDVyRfBbndDQHtY5/vfKc5Xt0vvSKWm1Rnbx jLciSFGtoch9hIqYKEYX9ZAaRgFWWlAfsi0p/HAcGvMXonsYrgFcciFzzCr3pQ0qPiGOqH EuxW5j93Ikr0oIHOxZJxEqQ3bavfhpQ= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=P3DR2DT9; spf=pass (imf16.hostedemail.com: domain of 3PGL8ZQYKCCkXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3PGL8ZQYKCCkXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60a03635590so23057377b3.0 for ; Thu, 21 Mar 2024 09:37:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039037; x=1711643837; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=cRKfaiB6dOVHxpZUYVy57amQHTHiIgtHPVa28h/bxKY=; b=P3DR2DT9kochFauFsS9RyrwrM4WXR5Lao6YrMcEQVj4eG+289ROsaUhEQJ2JEPkQW5 anktoYANdHN9fzzLo9jG+1aGKBTJI/RKj0yvSPxsxFEdqXTao8g0JUniJWBD2fER+cIV dRt9eDtRNY35MjS3pV0b38Q3+DHyxgtPNxtjUR938NXrNO3Mv7H6J9nTt6YSmBmy0chD tUYKRJm0poSQ87CYhrFxxXgpdhsxF9yPsrqP3+Kb33xVnzVXURtI2qKsW2353HvafmR3 hcyRZa/WN2iwh2+0gUs+x6HsIBsr0LtwFxkB9yeYEHrA+U2V3ndrdYKHA80XKUSgwpSQ EThQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039037; x=1711643837; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=cRKfaiB6dOVHxpZUYVy57amQHTHiIgtHPVa28h/bxKY=; b=NkqEdxBx6wweAKQ5XWXW7gYzNQPiIM9pE3Yasv7wdkQeyqPHZIGdZlKw4/F1uuLltB S25+StXL37DToABFDX+9fIpumZuAY60B/VauNjbtfy3MAtvJNpbAbGTHpWXvHvQ+8CLG v69CvNMb9mD54eJ7jTKH5GSp/joUD4TtsXXvj5YMaKHkew9vp9xwFbCbigpf6jaGvIqK LFCyKx976nF+4EzEs5++i0Jr1+h+pYDOfVzjztAxU2fjKJjU+Xvqsz8RPFQxc9Itgna8 jekRn/3KrvfelKDNsQ3+DUNeAEVBJB4ocOge+RGUtKhDVgVm3XdSTa7T1ZWt/493l6Tk UZMQ== X-Forwarded-Encrypted: i=1; AJvYcCXHpxY6pB84kklF7VGPmLUR/wHs1Vz9gAHsUFIZHOivdsOEg17yqdu0tlhn+avPDK57x41xtGA7jdZ0oyyRcSnB3dA= X-Gm-Message-State: AOJu0YzXUxzF1gp0SAW13MdxYsLOGl986MrK1cPgZW+SGDtBdacOs0iH 0hf0XosVGRfd+n4xgZtuH+x+Jmq6udClJRIn7nfc0Ly989MhCL7I1pDYakwuns7PZv65jit3QMn 4+g== X-Google-Smtp-Source: AGHT+IFYgw5ohbM2qs9qWLSKEHVv/3K2+6lhzKzCxOpGXwJ9EG3Fd8M+K30yKZwcBlNXdCTg4+rc5Suy1A8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:690c:f88:b0:610:f11e:9d24 with SMTP id df8-20020a05690c0f8800b00610f11e9d24mr1686171ywb.4.1711039036599; Thu, 21 Mar 2024 09:37:16 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:25 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-4-surenb@google.com> Subject: [PATCH v6 03/37] mm/slub: Mark slab_free_freelist_hook() __always_inline From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: A84FA180003 X-Rspam-User: X-Stat-Signature: bsdabz1xfh1a37cbwn7gioewgjsdoj19 X-Rspamd-Server: rspam03 X-HE-Tag: 1711039037-751738 X-HE-Meta: U2FsdGVkX1/zltVuKZe3ynxQGVfRSkkkzwLGxN+TZAsJnZluyxwUsF/NUAuYi5BpAT2ML6PuXu+cfDctXVZc7S1a7iWGF1B848TApx8vofw5+gXduqHb++eWWUASmlftSTtHeWwatZTC5tLMLIiXIO7vR6IUmEPCPpm0QcESBmIAX9a+G4exNpVJRtvKR64/8kqWOBT0SVc/YczpSTPZ2h1zciztcclf5Hg23uZ+Rfyasi+zD6WEHWrK4vxWwhWiYU6UfdfbqJm9JP8my/n4Zopxzc2HpLV1ina70rKtkCKERnKdL6zN3HrZElalje2Fd2JInayvBIyha7zwvY+qSxnjudT133VjVn+zz5JI7CrHbJCfXwrgJEnH9tTFVXAmw1dVAD24G/3ivE11KZNFUPbWe1LGaTdHoJH/tH8xtqTvC822t30/VvWpa1Kntr+ik+DUFpJxdcAQtF40deU/d6kLHzeIICRHCeU5PQFrWQlE60zvTvQOkPDW5r09Uco0nWJcSLnKys2ElpYgfYfbYfD4dglHF3NDVE75ORREFBraIkZ+c0XZCHLDodyMH23bTU16JdyoGpugrrMCRSbX7YoGh/gEP+BhTVNWHrd8xSUP4C2txBQ4nB6eXX/bTwYPkZqrAk3SlEpqN9G2EWR0DHtYlBTZLG5tggYy796yAMwjRWSIwaDwUIiYAU8C01db/CXPBvbpbzINqHF4/hxERqd3FkkKLW3DNkP6H6zKxUeOLs7iu6SKxiVnJqVCTo7eWzA/5DON7s9zFs9JeMqX0eYFaLLMGSXiigPkvBFvrsZIZPzUOPAGgLeAAkjtbHuuqvjdDwun6U8hzNUspNPsy1hAfJBPZlwBVktnf6KA+ymFEsSTSUMCW510pZdz1SeL5N6oUJcs2hAPpxk2BF5IT7o8Y3j9/NiWrK1TEU3Lsy75PuUTuKLApItIK+Ql8HxibUG0n8LKXcpxqofjNlT HMhE2Qaq GSHm9YnPpr6u2YsTdx7c1O1GyYQi5MGF7dep+i68jqyrKPdyVGVJ0r9ELEHcWmFBt6kyqnMX3yrxWT8wwrm1UbUun24zre+uovfImIdqpk2bYJWMzgoO5shY9XesV71xjkUEujyr5Bmm0ssOHgRLE5jvTZB6BFkpv8aqAzhshfnvC6yOTdRydrProEjfhnN15G4kBrLDNHhY12Irl0XpzL2tfhxNU9is+OekJa18teE2ZK+0JweCapQfhDMrYvn+OHKaHqLbOjbZ12DOosZzqDK98ZbgGrD09xevKUDEZQ8NqczkLkoOr0ihb5js0pd0uXbhMvcEHMLNsspizu6WPkPzH21BJmkiYrEtRjYbJG4uATh8WX1nZFsU8yv9i9fIzR2MNZqTtOwk1Dvbr8L/qLM4ohtthWfKdHRXKqvcJUniRX3x22kDYEMoLYHDZjdYRmUrDCluMPhlEtpPdYxQtYx9I0sTs6D3Hsjr1Ui/5lFfUON4O6z0NMeeqsNZc/9wVXmfwJ5B7FDtHnxswNWC5uS1c4y13NeM5/d/jHDAG0ebW/VGIDyzJTQs2UZVuGFWKgZDtwy9CwYoIIBa66w8BIyyA3fhdjZtnJv3nbwYElJwgXHo5tWWpXRX9TTqDTT1c2/zocukOHJg2JC4vzIZ7buVu138vsPTrQHR3unG7HGr+ixXVltFkfCZunASMkZ3CmVnmPqN58VLvPaLAAEprkgDZQy+gszvU2OGrtmWcM0b7T5Een9LyEXVCCbD8CANDghpKzPv4qUjDHk9ipl3ZxMTqoAp9UxBuOLPzHd4ooiCyvA1Y9zx9wYDLKO/ROm9tXX2XiDWvqqGLhyhALMpOqkVHLw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet It seems we need to be more forceful with the compiler on this one. This is done for performance reasons only. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook Reviewed-by: Pasha Tatashin Reviewed-by: Vlastimil Babka --- mm/slub.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 1bb2a93cf7b6..bc9f40889834 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2106,9 +2106,9 @@ bool slab_free_hook(struct kmem_cache *s, void *x, bool init) return !kasan_slab_free(s, x, init); } -static inline bool slab_free_freelist_hook(struct kmem_cache *s, - void **head, void **tail, - int *cnt) +static __fastpath_inline +bool slab_free_freelist_hook(struct kmem_cache *s, void **head, void **tail, + int *cnt) { void *object; From patchwork Thu Mar 21 16:36:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599161 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C367AC6FD1F for ; Thu, 21 Mar 2024 16:37:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 502DB6B0095; Thu, 21 Mar 2024 12:37:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 486206B0096; Thu, 21 Mar 2024 12:37:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D8C26B0098; Thu, 21 Mar 2024 12:37:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0A4306B0095 for ; Thu, 21 Mar 2024 12:37:22 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id AAAFDA0278 for ; Thu, 21 Mar 2024 16:37:21 +0000 (UTC) X-FDA: 81921601482.02.202E32D Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf02.hostedemail.com (Postfix) with ESMTP id DD64280006 for ; Thu, 21 Mar 2024 16:37:19 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=HtXEH8wL; spf=pass (imf02.hostedemail.com: domain of 3PmL8ZQYKCCsZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3PmL8ZQYKCCsZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039039; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FcsF8xuW3B4rJ7bvWaC3URPqDGHFVTOJU/Md9c+vki8=; b=zCSyaaPxAr50JfbulOl8QgJZdYdZpQcBCIhDeb0hl8JaZfa1RHlbGoUYCTWUY+wvyLeSja 1MCdqNBe5oQmtC46KtUYi2EH8whzXUwMDmUN3i/reg891SXaA3+rgLXau7/iWdCKYmYolj N9Pdun3PAEir8lr4WKXIeMqfi2ZiRWs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039039; a=rsa-sha256; cv=none; b=Te3stHbH16sVO69D1TOlIrzcnggH6oSgrewKJExYITFlwW/Cnbl8WqhlO2/wN2l/ZKHTDe Ifavwu7wRlvsNL8jIFY98MY9MVSHFu5Nm9d23MhA8jeUbPbHhueSL24oUM06Is4/sdjxh7 e6AukjJS17mz2W4dDMsGRxU9SDr1csw= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=HtXEH8wL; spf=pass (imf02.hostedemail.com: domain of 3PmL8ZQYKCCsZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3PmL8ZQYKCCsZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60f9d800a29so20565417b3.0 for ; Thu, 21 Mar 2024 09:37:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039039; x=1711643839; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=FcsF8xuW3B4rJ7bvWaC3URPqDGHFVTOJU/Md9c+vki8=; b=HtXEH8wLLFqHHixt6NdQMhVLyEZOYK46YsnzUx6C21BdGG6wkA3pzN8potsjUe69u3 liNplLg6YTpP0MEX/Jw4lRX05AGm8GRw1qWAC4lg9aALwSPRPtfadgm4qKe0mUYbqTQR NE79YbBHX0vcL8N3fFITe/iW4u3Fm8snX9xlR0vAwbMkROM+H9PRcG8BI9GqMKKsLEwN g+mNhvBlOREuA4G7SuxT0UA4EldvP77i2N6ByoDV1ovAFcrdTs86adaGBuGXt4Jsf0Vk SpkHzYfFy5BjGdZngzn3izaCZUnvCRe7s4SFhecAOYb8RMmxjlMV7n03/oypF0z3gxMr 9FyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039039; x=1711643839; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FcsF8xuW3B4rJ7bvWaC3URPqDGHFVTOJU/Md9c+vki8=; b=mRUgCx07FE5JR5j1BbN3IiTifLEhDVnzJr6bGM9l8lCy4JpemF9WdNGedbgP/cg4SL M7Qj35e47oAnTbwGeVaIRMCqOKs1+KbviMl43o1EZcc+hCWOSEt7AZct298tCT68UpHD nGaGYtGsUmFnbcIlwP5PuIwepqKxA6BLxJRdTOOFWX4g9b8Iilbkrhpah54rSr1qhS4E FjoZfyyUZsBA/m8DszhhCpEUeZM0Tui6nBuRphiZ0VJ5Xs2H4iksjC3LNaLfRFkDcIqi SAAr4HYt/lovwMooidxot/BYurAaP7rCtTcjc3lnYXM0doaSSMmhmrh8Dq2Nv16tzPHy iZ7A== X-Forwarded-Encrypted: i=1; AJvYcCUCVjYHJpLulOo8Ab2KN5fRHsSHI1OaulZCs1DObIsF7bKjECDwz64/QKAymBgP6Ipgy4BouJOsDqqO+jgxfUVkKns= X-Gm-Message-State: AOJu0YwxdQHgV9SY50kIQurqyuTdx5wASpkKXyKE0fbYqqQv4KLVlqyG Av3d/PS6Z6eoUauDL4KuR7QoQW7CsUKEO69oy+Ag3gcM7P0pmxLhux4aqBCo28Ggo+O0J6aFo9R Nhg== X-Google-Smtp-Source: AGHT+IGrkFj5aE/qI6dA3cr3f2zIJD/myHIxBhs84G4EmxlJsslBgaCO9HAP/2XOVSrMpqDkAxdrZ+lOSbY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:1207:b0:dc6:b982:cfa2 with SMTP id s7-20020a056902120700b00dc6b982cfa2mr1182860ybu.8.1711039038760; Thu, 21 Mar 2024 09:37:18 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:26 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-5-surenb@google.com> Subject: [PATCH v6 04/37] scripts/kallysms: Always include __start and __stop symbols From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: DD64280006 X-Rspam-User: X-Stat-Signature: 9k9yjba14xe1yo1er5r6iy6j58reki4e X-Rspamd-Server: rspam03 X-HE-Tag: 1711039039-583487 X-HE-Meta: U2FsdGVkX1/CtBDMn3nSP0I0roWtAROwlqg+zRto2uoLqQ4Stki31JXEdveuvMppd8NRyZcvpZt322JgQszDQtVCOfHWjTruFHYh6QXesWnIqtq8LbvXbRWt/UvEy2iZo+7XErd39aDRy6qLvCQSA6onE/2o4471uYVSNqFRI357oWCVko7TRd7FQbpHMwHoRKObqgvdvl0mldrRXfiPChx24sLCSbDc8EidQt42nVbM6P7kNu2FsDaRaBQuY0CUuzMLEaeXnlyHE3XbWZ/32rn10HXP2f26iqcZF/Yb95Y99BFnzhNHst7lO9E7geuIFZSfgvMWp2YVGmfY6XFqvuFRu2KGFvGM0Gkk4fQrKoOXMCy9SqJA0QWEdGd6KJHfNtr/lP5C3oND8wtyM/nGZ9oD0HIP5tGgJu49oV0yz/cHc0dyXcG254d420cRCDDGL6iiQNi/Fkvwu5ajg4iOLdSCdIvY34RqwwVRkzzl9DBcdfhdpALcncIr3ZI+lGf+dUS7d6JThfkKdIw64bbR1jzlKlSnvR/TKZ7ofVJQMA/kA387txcT/iP1UVlu1E495a0E65VapxQvRlqvJBzWnnRlssMp04FszJHrReC94haprdanKseHgngMODlKE1O6IRIGGRIXt7/qHwlfHCyVqny0knLA5aKDaNPf2EKW2rK+EOEEe2Cm4cw9zvCtoTb9ltEtESdU8aQnhcadJyUtgtprWwAzONSGAsrVg3CMCMhL3pJ3BClNW5/wyfgGIRj/CY33VTQld0MOyxRZ8RxIqIBjsimL7gQYJmJXkhSjSGIdsqmcmVS8eQtBLhjafs54xlvs1UQ4jwLwmo6MCS5kYieIeeLCy1OAzsAa6X+bcx+c/hZY9NK/Igmiq3WYTsny5GUkSOy4Ee00GsJOp6aedeSzELOcxG6RNtNF/lrBs9kYfqVHao0lttR/kQn6R8zNqs1GnljzjvYUAouRa7p jrjYlDgk y5awDN0RRkjY5hI5g2KiXpgl1xzQQh+dksP5yCGB7PfzjArN1Fp5Z1/Gv53vWqSYnklIyBR7pczD7JzZyiKXysl6G6ruvK3Rrekg/sXdacnLU01fcUZ+KFoUGh+PJMrA04T1ir34s6byQUmU98ClSeEgvaDrhLP47MqlbeQ1SbospMSDs2n06EBVQXtZQtR4fWE08qWZDrS0REVeMu8uNIQB7gfKR+y/FS3MiMhzFGvS3DepNLAOD6Z6OXpVjlOW4n94vYS58tuhQmSKTY8FvzGt0v9v3P/5k9xwubymAFiI6KPVet3aTv6ZwLzm67Nb9Sb9qlbqlMBMvQPcpPT066lghzjwkFVPb+EBE3VT3pOqRW+J/2huYk5JuqOSrccul0MxScU8flcyiuDhpYzBIVj1Ri41DccaU06kzY3wW51rQYsvzNnmGCgxbWQ7cuUDKIfHHHxQeQjd/Nuz8kUU+4w8/5dtqd+dVuzm/52dvlwi6JVEksn+Ka/8ju9p2sL/eH1WTn5rFjw7gv48Ld+Viko7kqrKBM91iFIlID8dOniOp1e8jA9UVFH1PLkU1r+ltuWoq0HXKyqVzvQf2bFbUmZKf4wpx0xsCsAU6di3w2XYJyTidAeKNwtuMZeLr3AYc7XnV42bpV0Le2m3qWHckeY9Bz+c0orgW0sp9fWLdGW56rQeC/MOtLa+cAaf0fn0SypD4BoxwITc1JYxgb40RtaH0SN08pygH0X5I5IHjQoAVPVn82tmLoYSia77pViIpKJTGViGDbnb33rIaEc2JAQGXnYvqBM1GcWvh+w4pg7ZlSyoFZM4torzwg1IH2V+d/5b0NwYfZb4i9UWtvG4njR+Q+Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000338, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet These symbols are used to denote section boundaries: by always including them we can unify loading sections from modules with loading built-in sections, which leads to some significant cleanup. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook Reviewed-by: Pasha Tatashin --- scripts/kallsyms.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/scripts/kallsyms.c b/scripts/kallsyms.c index 653b92f6d4c8..47978efe4797 100644 --- a/scripts/kallsyms.c +++ b/scripts/kallsyms.c @@ -204,6 +204,11 @@ static int symbol_in_range(const struct sym_entry *s, return 0; } +static bool string_starts_with(const char *s, const char *prefix) +{ + return strncmp(s, prefix, strlen(prefix)) == 0; +} + static int symbol_valid(const struct sym_entry *s) { const char *name = sym_name(s); @@ -211,6 +216,14 @@ static int symbol_valid(const struct sym_entry *s) /* if --all-symbols is not specified, then symbols outside the text * and inittext sections are discarded */ if (!all_symbols) { + /* + * Symbols starting with __start and __stop are used to denote + * section boundaries, and should always be included: + */ + if (string_starts_with(name, "__start_") || + string_starts_with(name, "__stop_")) + return 1; + if (symbol_in_range(s, text_ranges, ARRAY_SIZE(text_ranges)) == 0) return 0; From patchwork Thu Mar 21 16:36:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599162 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DBA0C6FD1F for ; Thu, 21 Mar 2024 16:37:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 60F4F6B0098; Thu, 21 Mar 2024 12:37:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 597266B0099; Thu, 21 Mar 2024 12:37:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E9A26B009A; Thu, 21 Mar 2024 12:37:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 240516B0098 for ; Thu, 21 Mar 2024 12:37:24 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id EA2FA140A44 for ; Thu, 21 Mar 2024 16:37:23 +0000 (UTC) X-FDA: 81921601566.06.8413E68 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf03.hostedemail.com (Postfix) with ESMTP id 2454C20004 for ; Thu, 21 Mar 2024 16:37:21 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=4FHfZJIL; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3QGL8ZQYKCC0bdaNWKPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3QGL8ZQYKCC0bdaNWKPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039042; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jDSqL7VR8RZR1D2YCsFy3ymp62hocq5f4M4VfhBDRaA=; b=XsdZywSZBizRwDrFqMWdzwcLU/EEi4dJhfghEFiGIg/S6iJOKWXpREFDqg2SNeyj6d37+a G/brLNr7KRwiTcdS6mUkcCdIeG6wfwTQF0A2/UU9o4QFBdL7Tjs7kBsL9ahQAuRSksU+6c NgMoWzv0PNavTR1dVcCp1etzmDqvkrE= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=4FHfZJIL; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3QGL8ZQYKCC0bdaNWKPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3QGL8ZQYKCC0bdaNWKPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039042; a=rsa-sha256; cv=none; b=1A5FCNpBM8PNnC37SGmMG6ocb0yy4zBrfnbHGQxWrV+bC1wuFp4CYpITqreACBNnMqcdQ4 t6qEZHTRT8C6QMWrwCTfSgzW0aCBBeGzQRDjbqklcUjtIjRmRP3d/skrAf+YyJ2PfF10ld Mz7WtccgTKTik2mAnM7gqdhXn3/6zR0= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcc4563611cso1782259276.3 for ; Thu, 21 Mar 2024 09:37:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039041; x=1711643841; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jDSqL7VR8RZR1D2YCsFy3ymp62hocq5f4M4VfhBDRaA=; b=4FHfZJILYqjclvqmZVi8nVEzxRjUYxSMp3j5KKEHRoptCtct4pW+jsUCEAd4jOgp2O v7f897W8/Qhu3vPoLmD9YYTxIRzEHt38kHiOzi5F155BeKmGlrpM5pI1PppfxkfySw1h b4nnDeMJxc50/rRYi1bRTeIn0yzogEOm8N5JW4LFCEN8ACiV84pEBTQq+yvXK+IPpZh0 /C7spOuzAKvTNkn1cSTkPxlLO9kqvjrLq7b6Kn6bRbJE+DCIvtrsVqzXj30qxWd3bgUP OcIRjioSEhUZ2FN5VJUj2U34qKU39rkqRH7R+GmqkcFN6kZn/qp1elwN9yVBos7gjDWi eN6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039041; x=1711643841; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jDSqL7VR8RZR1D2YCsFy3ymp62hocq5f4M4VfhBDRaA=; b=aoFh/8UuLtxeflun1N9FegvEmCI+ExRrKzWwI4drKdHd+YbRXgubUHmtfsKeZbqLBi 5W37pVmOMNLnA1i+BTKhZocfEPKnuAALMY5alnAj9evi2qzmEWx4mvVxfxGSOaLY5hYU 6UsBq02z/2tvIHpjnSsiZQfF0lqslTB/OYQU4Gh0kSGwWoF37fl8PskIvxWlkl+IJN4U sybAehWztBzehkcbRhRiDAQYOGpmjq/rc16CrTfTqJQH/UxORPbafuLfmdVrj7XdOsyC J99IZ/AWIXaIwUc5R1w6CjLR//2M1jMCpAHnmkUmcUBViMO0rX+08H1AX7pRIpJrqmOQ Fhmw== X-Forwarded-Encrypted: i=1; AJvYcCUW3CRNLZO/RX4PNW/sWkGuCrGjxCjC5OpMuSNVclTqvEgwN/+f1IWJcqukwhIi2aQzd+HbsO8o/oSzpD9zmNr8HYE= X-Gm-Message-State: AOJu0YwebMPGmoXxTIn/NAul0uZ+TMMSPlzvoyNlzqT5tuN+FPrI/1JL lknni5cAhkBxiJc6iIrfubedydI1xLpZPcVdfyPsqJNEgcPfWnOLPOLk3EBs+/F5eNrznNZaYV/ pzA== X-Google-Smtp-Source: AGHT+IGraxwr8LJ1oKM13+l6GlQVbxX2plMlllQyHlZ1hTZW8w83otTqaTPIW9YmGt7R4YAvcGIgF556yuY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:703:b0:dc2:550b:a4f4 with SMTP id k3-20020a056902070300b00dc2550ba4f4mr5784120ybt.1.1711039040945; Thu, 21 Mar 2024 09:37:20 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:27 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-6-surenb@google.com> Subject: [PATCH v6 05/37] fs: Convert alloc_inode_sb() to a macro From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org, Alexander Viro X-Rspamd-Queue-Id: 2454C20004 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: x8gictyc3kzh4sz69m4trrkeqnumfiq6 X-HE-Tag: 1711039041-664976 X-HE-Meta: U2FsdGVkX1+iwKp0DNyrYAtTr/9JLMRvL6NwEmIg54vqLJV/cTq6ClYYfiEK0j6jnejtlccBKREnWlRSvtAyS04esESJemtPsm1OCENCTBsZrEojpxf/Qdf1Wby23fAtheGY/ZgtcmZ7kFgs3gAhDc1H2JcLcbVqWolwy0hcyAdn6yphwS1mMOtOzZNz41OR/NR5+3ZMAz+hSK6Zv8a0kk13t8NIxYCjUVi3zeSTWxooAaDaJSIiMIjJ8/EdoJdL57tSRzO3UmZubOyBdERdRIRVYzepNkD0Bpj/dAvN9XHeBJ0RvOBq8VOqq+9/RgDf4eVmpwwyCTkPPTeKqn/Ng6/VUZO6L+8eSVO4IK8b0DjUkAAW3P7FYMG7x+b+BNN/ThQ5r9HY3kUfm6bSN0pk6VF8e6U1znOP0OYz4zyucFmHIiIe7kNDzMpFqipCN+fUouDJiaDmb1pyiXXCsQ53hR7LeMISdanHWxzJryiv+SGFz44q5DnZB+gRRTUN0Bh6+GdLvqR8UH7m9yNiBAlTp/tjoiSyLc3Qva17X+UbPJDgHQcL2ExfQZYFw0KQ0Z/3yk/+ltVRJcx6yYsgN9c34Y9U4qjg5aBSdHDHzp6fYlmqxKr1syC8yd9rozcfujOTkBIIxJk1H14yq8AsVD7pIS3ich9Vqlk7MH20bLgCfhe0z+p8qPEc0gMhu83Y4wMrHbUScnquRer1LsaxiIhFw0rqzlpEuFgYSRvd21njVbEijKi8Gj+OWoGou1poIxc6CBGaUmguv6oiGzNDm3ngNyCwc/ep4rsEo1yn02PTgC3V1z1B4CfkDGC6fNzC6lSwkZVzChj1WGLuNdZQPEDFuep6WNH4bkqG1DDTKkFjmsjkW1Yiye615bfBLYY1rEQ06kwb2jtbQL6YvB008ZG92FEnHBR5UhvE2g+9jHScH4TB2Mi7lRWDfFHGk//pSgpuzewswdi4TAaOSlFWT/Y OxFs0iLu kBA+rqAGTN3PMIBXubXu2W/UvTiUR3c65JhfiUQKgmLtszfuxQ52Qd1l1wzPBJnCBkZfqQG7cO8lNwV9Qfd9EQDgABP10ssdtaRnS9R+Ir1puqvDHdPN3SlDTmUd0yq/viAkzSWmWiECuYhSuufQgPByTK0RF0o7AXM0wSfF8AOYmlpw0Knx1bT2hOsM9d18LtKWIfVHUd7L6ugpl15c48EYXWTYvTSFOSEwL5mWn/jdQmmgPAJjOPeBDdzi7YHEfIQO5NREq0bKiBaHwXEbym3MCHumvXMYU9Jr4SYnHChck/6NmEEpxInnXV/UM7tJ22TMXHcaRHRzyhxAzNuzOnEhXzP6We0ye/L0et7NUbAjLAZJ1/OLj3ci3ICdQ0wXB6TC5LN3mbAO0CgVVoczycJBAlS6efviY/4bh1p+Iz5zBriUy4CBU1KISbgKfvG7Jy1DRWdMOKZ55YoHGcmi1FniMlm7rkO/OagT3J472yUN4yucdQzU+B3FF6XOSOcMxxWUNjC857ZhPFWEaB3E5tQrTAbnvU5Iqef9Wdyvs8XQeqmSqQSqX3gkw58KUAZKATpLhge0bAKY14de+sNDGUvl/PIWjhPNKZlPg+GdTx4J7FB9wB3lhjoAMNPlf0M/tRRvhy4Ci9ndRN6xvDCLFzy07zCRm6q1nTzAnbwpfeL2W12nbUi0wN05eOjuyVPk4NbwOg1FtmsVvdr4vbfkCl+NytIUnISy4QlTS8/Lzdc9FNXspbt9cp9jj4F7kgdwW1Jd0jqBwbWCqX/JBQMWMSe/oZUP9xrhAfd/dTmkXbMIjXN/vuYDnt27QdOAhrvK8fVKM5RlCei5hVV6oy6KlmabpiIVlgwPbzdf5xxHERm8xxFf57pxv1DkJwHgBPRbhS8JvuByJ9u+Guoq95NyrZx2qYQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet We're introducing alloc tagging, which tracks memory allocations by callsite. Converting alloc_inode_sb() to a macro means allocations will be tracked by its caller, which is a bit more useful. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Cc: Alexander Viro Reviewed-by: Kees Cook Reviewed-by: Pasha Tatashin --- include/linux/fs.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index 00fc429b0af0..034f0c918eea 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -3083,11 +3083,7 @@ int setattr_should_drop_sgid(struct mnt_idmap *idmap, * This must be used for allocating filesystems specific inodes to set * up the inode reclaim context correctly. */ -static inline void * -alloc_inode_sb(struct super_block *sb, struct kmem_cache *cache, gfp_t gfp) -{ - return kmem_cache_alloc_lru(cache, &sb->s_inode_lru, gfp); -} +#define alloc_inode_sb(_sb, _cache, _gfp) kmem_cache_alloc_lru(_cache, &_sb->s_inode_lru, _gfp) extern void __insert_inode_hash(struct inode *, unsigned long hashval); static inline void insert_inode_hash(struct inode *inode) From patchwork Thu Mar 21 16:36:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599163 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3D85C6FD1F for ; Thu, 21 Mar 2024 16:37:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 549C76B009A; Thu, 21 Mar 2024 12:37:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D5416B009B; Thu, 21 Mar 2024 12:37:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E4996B009C; Thu, 21 Mar 2024 12:37:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 14E946B009A for ; Thu, 21 Mar 2024 12:37:26 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id DD1F2A0E21 for ; Thu, 21 Mar 2024 16:37:25 +0000 (UTC) X-FDA: 81921601650.21.C0A6F02 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf06.hostedemail.com (Postfix) with ESMTP id 11414180019 for ; Thu, 21 Mar 2024 16:37:23 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=IXvzWey+; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 3Q2L8ZQYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Q2L8ZQYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039044; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v0oqySW4AkJKGPTAgvOI/D0yl/u4uQhGUVWc4LnGMQI=; b=fIYEhSyzRki8NG4Q3KY+khDfb4X8chbzL6UQnlihPRRhIKLrkzacbGXe+1LZt+vElhlBo7 Pc+XjLyrTfJvVhcMnE01WuBBnGrzVzmOWABCGSpXdWBXlOt2P3DQ/N4n2qUvGlhW8vPc+n RidSPGu89iTXoqbCmY/LtR3kC6jWSIo= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=IXvzWey+; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 3Q2L8ZQYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Q2L8ZQYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039044; a=rsa-sha256; cv=none; b=wuekz6lcUpodRLE5ngUwmbJN/S0Q3Ql7OfxT2e7aRCaWGQ6vp0Jo6sEy8hf+HUOhJbspoA dhKGwVyul/xwkht3Tr1Z5dz7nqOU9gcjv9gDDSAX5XiNkp1N099iackKhWcSaTSbvpmEGj msde1t/xilY4mYI2fu6tEYjkqkC9LVI= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dd944567b6cso1544525276.2 for ; Thu, 21 Mar 2024 09:37:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039043; x=1711643843; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=v0oqySW4AkJKGPTAgvOI/D0yl/u4uQhGUVWc4LnGMQI=; b=IXvzWey+qLMGJyqSHuHkDUmRe4pi4Fws+NNKbRyk/LB/IwpIClkOd2KXB6jSn5bIci eZpn/ZPR2dr6HMIJWO5hojjCV0CzNz/z8JYCE5bwNTF731gAwZxz29uJb03MvZ0ZaQKH 5imKrWyKuTV81KEaazn8Pq+6u1d2ndrhZaEtlewI3Sef1kg57CGupCjv2roMWdUsr7fG rcnJHMPrADLW03CfsW2zCNL5tetQQ+icitqSiqcsy+LTk35/tmKMKLkY8PnPsbALd/Qb ZPKmvfMvcxGhbIGTvibjqKTa3gdU/GIaeB42CH0l+3/ALNVgpGB+cEcE0Gr9eVTgSbm+ y3/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039043; x=1711643843; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=v0oqySW4AkJKGPTAgvOI/D0yl/u4uQhGUVWc4LnGMQI=; b=okVCsmbEycACF6oyvEpFf/rXXG8wM2Twy0tjZo4twiDEk4EN07KpscetOShebPpofI JKrE1zy4nPTaiOhB21qMP2jlxRDpkXDFxbeJPqRaZJAjNRIEe5RI1MiO2BfoO0ozw6A4 dNXRxfq2JmidtG0UZgDGqM166pJlFH+sNF0sioL66K/9Y9b9aBW8fFnisq66iS0FvyvM 4xW5ws4F5eDLsYjWahjltUiAkTX9eTN0db8vIcxKPpqwCBz/vYBbEcwASslGfIlhzxAR 9/EQQkdl09SwNGiY28D3HmcMEWON8J0y/djHJWCjuTJIllGi6O6lyuDPFMOLUghMBg3p 0MUw== X-Forwarded-Encrypted: i=1; AJvYcCX92PS+wxT4wxXUXFsL3ltHJ01ezy99+HesEKnllHuZqzi80gkknDVFqhtg3qeDztDJYunHoWFZlmG+WaTJB+Gfv38= X-Gm-Message-State: AOJu0Yy8mbdd42Lo1dugGTyniwAElWa2LYIT/7e7jDNv+19lnGryvUPr 74Z/wOrlCGdMDTUAWcgbWppykgHtOjkOa0DxgfRG2KqYMNd7pNNVOByd7OYRqERzRawGLHD3bkx tgQ== X-Google-Smtp-Source: AGHT+IGpNpmorY30wsaOJMR82EZ2rL7WoOA3y/drFJPmdjdrt7V1GEnw6ljBDiIMdOLAVZgoGShZbFpFbCY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:118f:b0:dc2:26f6:fbc8 with SMTP id m15-20020a056902118f00b00dc226f6fbc8mr979928ybu.7.1711039043023; Thu, 21 Mar 2024 09:37:23 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:28 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-7-surenb@google.com> Subject: [PATCH v6 06/37] mm: introduce slabobj_ext to support slab object extensions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 11414180019 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: cn58mbtucu64ctf5tpaqj16tpuo8ub7m X-HE-Tag: 1711039043-617964 X-HE-Meta: U2FsdGVkX1+CJZmIykUR5zrK0raBvI+A0gNGW1HRulS3Jvzs4Q9asoc+9tx9H8Ge6TAIzQq8N7dzBZP6ergI7e38+URWy1g3ZHFmH2hXgTnQz9G/YcVgJ72xY84m3WFmalct43dz+cLLQAofFbePZzONTR6pTjqeQpnXGkysEUvZSGdhRAOeZ9Kd+tP4NXNqfB4lkAC17fLdNBmCgVXr3nsxY0HEhuGS0scNKSeX59n2UHO9CYYNgfyuP7gg7bHti8MWWEmYPnBEQCFymUYXYGp8Q9GE8fHQevOv6kYpYAq9Z7MxxOMXjI8eXtW9R72O7kZeYf/dYelEGaUAFkAUtJ8BBglfTK+R5z1UK0SWzDq6NtKL++nDzXbKKA83N/uFO46xCMn0t0u8Jk3/pB6rHeS04ak6Y9c+/cQUTaXqtXstzL7A6pT3CCSd/hXMSlQe0em5k0cZwYAtaqY4pnXm8m5AnsklagcOPJQJf3YjDiCJx9wdZycuj8GULzp5zVQWgmEupQdQmu/0eY9YK2TGIWTwO/713tuAERpEMs4G4auLP6DV3as+Sp83XeY4e3xrKfw5prrWoxbgl9Vz17eNlAxzjrtT5FdWhXRRNLTrOjRFUNfXolzQIoGYT0ul/qadJDV91gPg20gAYnjhyL2HpybfDQWmKETqAnlo0O2SmntWwCoMpB9w9xFv2ir/Uf2JwZHx99Rv6eVUkmYlh7xtcgXreBE0hC7GLW4QXirAvpwuzafuLxOk24bXITve25naoEB17KCJosj6oz6MSijcjvWJqE9p0sYcTUxFD1zpvYlYJzPbVYb9VOmtN/3Ki5BMIWUriYTXTRrtXvQYEIRlqu2yiVLXrWO0kqRRZOb0rOaENqOA2fgi8M4O8tDqbmW4MFRG4kA8J9FWfxLkcRY57KuX6plq3rmc+xX/nf2u17thlRIv2xe+m1OpDXq5YoQCO2J18RYKyipoWs+zh5y 2HSkj27X YZhngaRwMyMF/AAA2d6QEnEcSsHDHq2c67d/YIzyjbLk41wYZDp+5NgDcAQmb7YatMRXnD1j1EJjp5Q4uRSOto/La2Qkun3UKhzcQVEPSjbuVGl8iLSCNe3P4InW+oeZzxLc4mESZDdzMgh1eeOuS74ttFsJr8VBTvThINpH4D9wMHAoM1eBd0FrRG0EBnE7fA02cOPJONObyBDnb5Wu3iVlRf8x7JrBVxuKwpn1IQ/OzaKNJEsD1roLy2GTGEft9rakVOUTfxoHNW9Qlyj7pe6did3s0A92bU0x1zQgYk2lIqoHRMcAFJWkBJjdkM8yxjkomvbuOGq8W1PfKynWY+G5zRCTrqUSESlD4fsHpcrCk+RhBw0i5ogdnKT7gqHU7YuHbqmPkfwLx+qp/fC7E+1bBjJgWmYu58lfZ4LxAdHtSg5IlKilJg1UU+E5iGTxJBwiEA3Hc+HnAGkeVlv17bMsZvA+8bXOXtVFk6/LIfwa04nCQH3t6b7k7P0TOQ5ItBmxZz4Ka/KNpu3PReFVapUSsrvx0edHel50Da7GJCylOT9IeXRtdiwbdr9KGpSXw5ux2V/M3fCaOuHwN6H1Tm7xKsUC4D0Rq3uDA8WqK7Q58zLTRzVpospLBnGxfQ9FGucQdgW7NuLFtrKYEANeAFnjWofErRvM8BcpVJFQgr+2IrtMBP+d8n4V/PI9bkTbrWUvgeq0nyCOBpD9DbXhyFO6E2eHs2o0jtpEuslc6X9kMXr/xGip8hDkselnXWPlONKmhvVnb7QqVzZhzVLccrz4WHJzwbJoaEy2e X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently slab pages can store only vectors of obj_cgroup pointers in page->memcg_data. Introduce slabobj_ext structure to allow more data to be stored for each slab object. Wrap obj_cgroup into slabobj_ext to support current functionality while allowing to extend slabobj_ext in the future. Signed-off-by: Suren Baghdasaryan Reviewed-by: Pasha Tatashin Reviewed-by: Vlastimil Babka --- include/linux/memcontrol.h | 20 +++++--- include/linux/mm_types.h | 4 +- init/Kconfig | 4 ++ mm/kfence/core.c | 14 ++--- mm/kfence/kfence.h | 4 +- mm/memcontrol.c | 56 +++----------------- mm/page_owner.c | 2 +- mm/slab.h | 52 ++++++++++--------- mm/slub.c | 101 +++++++++++++++++++++++++++++-------- 9 files changed, 145 insertions(+), 112 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6264796815d..99f423742324 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -349,8 +349,8 @@ struct mem_cgroup { extern struct mem_cgroup *root_mem_cgroup; enum page_memcg_data_flags { - /* page->memcg_data is a pointer to an objcgs vector */ - MEMCG_DATA_OBJCGS = (1UL << 0), + /* page->memcg_data is a pointer to an slabobj_ext vector */ + MEMCG_DATA_OBJEXTS = (1UL << 0), /* page has been accounted as a non-slab kernel page */ MEMCG_DATA_KMEM = (1UL << 1), /* the next bit after the last actual flag */ @@ -388,7 +388,7 @@ static inline struct mem_cgroup *__folio_memcg(struct folio *folio) unsigned long memcg_data = folio->memcg_data; VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); - VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJCGS, folio); + VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_KMEM, folio); return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); @@ -409,7 +409,7 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio) unsigned long memcg_data = folio->memcg_data; VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); - VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJCGS, folio); + VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(!(memcg_data & MEMCG_DATA_KMEM), folio); return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); @@ -506,7 +506,7 @@ static inline struct mem_cgroup *folio_memcg_check(struct folio *folio) */ unsigned long memcg_data = READ_ONCE(folio->memcg_data); - if (memcg_data & MEMCG_DATA_OBJCGS) + if (memcg_data & MEMCG_DATA_OBJEXTS) return NULL; if (memcg_data & MEMCG_DATA_KMEM) { @@ -552,7 +552,7 @@ static inline struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgroup *ob static inline bool folio_memcg_kmem(struct folio *folio) { VM_BUG_ON_PGFLAGS(PageTail(&folio->page), &folio->page); - VM_BUG_ON_FOLIO(folio->memcg_data & MEMCG_DATA_OBJCGS, folio); + VM_BUG_ON_FOLIO(folio->memcg_data & MEMCG_DATA_OBJEXTS, folio); return folio->memcg_data & MEMCG_DATA_KMEM; } @@ -1633,6 +1633,14 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, } #endif /* CONFIG_MEMCG */ +/* + * Extended information for slab objects stored as an array in page->memcg_data + * if MEMCG_DATA_OBJEXTS is set. + */ +struct slabobj_ext { + struct obj_cgroup *objcg; +} __aligned(8); + static inline void __inc_lruvec_kmem_state(void *p, enum node_stat_item idx) { __mod_lruvec_kmem_state(p, idx, 1); diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 5240bd7bca33..4ae4684d1add 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -169,7 +169,7 @@ struct page { /* Usage count. *DO NOT USE DIRECTLY*. See page_ref.h */ atomic_t _refcount; -#ifdef CONFIG_MEMCG +#ifdef CONFIG_SLAB_OBJ_EXT unsigned long memcg_data; #endif @@ -331,7 +331,7 @@ struct folio { }; atomic_t _mapcount; atomic_t _refcount; -#ifdef CONFIG_MEMCG +#ifdef CONFIG_SLAB_OBJ_EXT unsigned long memcg_data; #endif #if defined(WANT_PAGE_VIRTUAL) diff --git a/init/Kconfig b/init/Kconfig index f3ea5dea9c85..7a20d713010c 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -929,6 +929,9 @@ config NUMA_BALANCING_DEFAULT_ENABLED If set, automatic NUMA balancing will be enabled if running on a NUMA machine. +config SLAB_OBJ_EXT + bool + menuconfig CGROUPS bool "Control Group support" select KERNFS @@ -962,6 +965,7 @@ config MEMCG bool "Memory controller" select PAGE_COUNTER select EVENTFD + select SLAB_OBJ_EXT help Provides control over the memory footprint of tasks in a cgroup. diff --git a/mm/kfence/core.c b/mm/kfence/core.c index 8350f5c06f2e..964b8482275b 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -595,9 +595,9 @@ static unsigned long kfence_init_pool(void) continue; __folio_set_slab(slab_folio(slab)); -#ifdef CONFIG_MEMCG - slab->memcg_data = (unsigned long)&kfence_metadata_init[i / 2 - 1].objcg | - MEMCG_DATA_OBJCGS; +#ifdef CONFIG_MEMCG_KMEM + slab->obj_exts = (unsigned long)&kfence_metadata_init[i / 2 - 1].obj_exts | + MEMCG_DATA_OBJEXTS; #endif } @@ -645,8 +645,8 @@ static unsigned long kfence_init_pool(void) if (!i || (i % 2)) continue; -#ifdef CONFIG_MEMCG - slab->memcg_data = 0; +#ifdef CONFIG_MEMCG_KMEM + slab->obj_exts = 0; #endif __folio_clear_slab(slab_folio(slab)); } @@ -1139,8 +1139,8 @@ void __kfence_free(void *addr) { struct kfence_metadata *meta = addr_to_metadata((unsigned long)addr); -#ifdef CONFIG_MEMCG - KFENCE_WARN_ON(meta->objcg); +#ifdef CONFIG_MEMCG_KMEM + KFENCE_WARN_ON(meta->obj_exts.objcg); #endif /* * If the objects of the cache are SLAB_TYPESAFE_BY_RCU, defer freeing diff --git a/mm/kfence/kfence.h b/mm/kfence/kfence.h index f46fbb03062b..084f5f36e8e7 100644 --- a/mm/kfence/kfence.h +++ b/mm/kfence/kfence.h @@ -97,8 +97,8 @@ struct kfence_metadata { struct kfence_track free_track; /* For updating alloc_covered on frees. */ u32 alloc_stack_hash; -#ifdef CONFIG_MEMCG - struct obj_cgroup *objcg; +#ifdef CONFIG_MEMCG_KMEM + struct slabobj_ext obj_exts; #endif }; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index aab3f5473203..8f9984bf6fd2 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2979,13 +2979,6 @@ void mem_cgroup_commit_charge(struct folio *folio, struct mem_cgroup *memcg) } #ifdef CONFIG_MEMCG_KMEM -/* - * The allocated objcg pointers array is not accounted directly. - * Moreover, it should not come from DMA buffer and is not readily - * reclaimable. So those GFP bits should be masked off. - */ -#define OBJCGS_CLEAR_MASK (__GFP_DMA | __GFP_RECLAIMABLE | \ - __GFP_ACCOUNT | __GFP_NOFAIL) /* * mod_objcg_mlstate() may be called with irq enabled, so @@ -3005,62 +2998,27 @@ static inline void mod_objcg_mlstate(struct obj_cgroup *objcg, rcu_read_unlock(); } -int memcg_alloc_slab_cgroups(struct slab *slab, struct kmem_cache *s, - gfp_t gfp, bool new_slab) -{ - unsigned int objects = objs_per_slab(s, slab); - unsigned long memcg_data; - void *vec; - - gfp &= ~OBJCGS_CLEAR_MASK; - vec = kcalloc_node(objects, sizeof(struct obj_cgroup *), gfp, - slab_nid(slab)); - if (!vec) - return -ENOMEM; - - memcg_data = (unsigned long) vec | MEMCG_DATA_OBJCGS; - if (new_slab) { - /* - * If the slab is brand new and nobody can yet access its - * memcg_data, no synchronization is required and memcg_data can - * be simply assigned. - */ - slab->memcg_data = memcg_data; - } else if (cmpxchg(&slab->memcg_data, 0, memcg_data)) { - /* - * If the slab is already in use, somebody can allocate and - * assign obj_cgroups in parallel. In this case the existing - * objcg vector should be reused. - */ - kfree(vec); - return 0; - } - - kmemleak_not_leak(vec); - return 0; -} - static __always_inline struct mem_cgroup *mem_cgroup_from_obj_folio(struct folio *folio, void *p) { /* * Slab objects are accounted individually, not per-page. * Memcg membership data for each individual object is saved in - * slab->memcg_data. + * slab->obj_exts. */ if (folio_test_slab(folio)) { - struct obj_cgroup **objcgs; + struct slabobj_ext *obj_exts; struct slab *slab; unsigned int off; slab = folio_slab(folio); - objcgs = slab_objcgs(slab); - if (!objcgs) + obj_exts = slab_obj_exts(slab); + if (!obj_exts) return NULL; off = obj_to_index(slab->slab_cache, slab, p); - if (objcgs[off]) - return obj_cgroup_memcg(objcgs[off]); + if (obj_exts[off].objcg) + return obj_cgroup_memcg(obj_exts[off].objcg); return NULL; } @@ -3068,7 +3026,7 @@ struct mem_cgroup *mem_cgroup_from_obj_folio(struct folio *folio, void *p) /* * folio_memcg_check() is used here, because in theory we can encounter * a folio where the slab flag has been cleared already, but - * slab->memcg_data has not been freed yet + * slab->obj_exts has not been freed yet * folio_memcg_check() will guarantee that a proper memory * cgroup pointer or NULL will be returned. */ diff --git a/mm/page_owner.c b/mm/page_owner.c index d17d1351ec84..5a77d792112a 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -484,7 +484,7 @@ static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret, if (!memcg_data) goto out_unlock; - if (memcg_data & MEMCG_DATA_OBJCGS) + if (memcg_data & MEMCG_DATA_OBJEXTS) ret += scnprintf(kbuf + ret, count - ret, "Slab cache page\n"); diff --git a/mm/slab.h b/mm/slab.h index d2bc9b191222..1c16dc8344fa 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -87,8 +87,8 @@ struct slab { unsigned int __unused; atomic_t __page_refcount; -#ifdef CONFIG_MEMCG - unsigned long memcg_data; +#ifdef CONFIG_SLAB_OBJ_EXT + unsigned long obj_exts; #endif }; @@ -97,8 +97,8 @@ struct slab { SLAB_MATCH(flags, __page_flags); SLAB_MATCH(compound_head, slab_cache); /* Ensure bit 0 is clear */ SLAB_MATCH(_refcount, __page_refcount); -#ifdef CONFIG_MEMCG -SLAB_MATCH(memcg_data, memcg_data); +#ifdef CONFIG_SLAB_OBJ_EXT +SLAB_MATCH(memcg_data, obj_exts); #endif #undef SLAB_MATCH static_assert(sizeof(struct slab) <= sizeof(struct page)); @@ -536,42 +536,44 @@ static inline bool kmem_cache_debug_flags(struct kmem_cache *s, slab_flags_t fla return false; } -#ifdef CONFIG_MEMCG_KMEM +#ifdef CONFIG_SLAB_OBJ_EXT + /* - * slab_objcgs - get the object cgroups vector associated with a slab + * slab_obj_exts - get the pointer to the slab object extension vector + * associated with a slab. * @slab: a pointer to the slab struct * - * Returns a pointer to the object cgroups vector associated with the slab, + * Returns a pointer to the object extension vector associated with the slab, * or NULL if no such vector has been associated yet. */ -static inline struct obj_cgroup **slab_objcgs(struct slab *slab) +static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) { - unsigned long memcg_data = READ_ONCE(slab->memcg_data); + unsigned long obj_exts = READ_ONCE(slab->obj_exts); - VM_BUG_ON_PAGE(memcg_data && !(memcg_data & MEMCG_DATA_OBJCGS), +#ifdef CONFIG_MEMCG + VM_BUG_ON_PAGE(obj_exts && !(obj_exts & MEMCG_DATA_OBJEXTS), slab_page(slab)); - VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_KMEM, slab_page(slab)); + VM_BUG_ON_PAGE(obj_exts & MEMCG_DATA_KMEM, slab_page(slab)); - return (struct obj_cgroup **)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct slabobj_ext *)(obj_exts & ~MEMCG_DATA_FLAGS_MASK); +#else + return (struct slabobj_ext *)obj_exts; +#endif } -int memcg_alloc_slab_cgroups(struct slab *slab, struct kmem_cache *s, - gfp_t gfp, bool new_slab); -void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, - enum node_stat_item idx, int nr); -#else /* CONFIG_MEMCG_KMEM */ -static inline struct obj_cgroup **slab_objcgs(struct slab *slab) +#else /* CONFIG_SLAB_OBJ_EXT */ + +static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) { return NULL; } -static inline int memcg_alloc_slab_cgroups(struct slab *slab, - struct kmem_cache *s, gfp_t gfp, - bool new_slab) -{ - return 0; -} -#endif /* CONFIG_MEMCG_KMEM */ +#endif /* CONFIG_SLAB_OBJ_EXT */ + +#ifdef CONFIG_MEMCG_KMEM +void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, + enum node_stat_item idx, int nr); +#endif size_t __ksize(const void *objp); diff --git a/mm/slub.c b/mm/slub.c index bc9f40889834..5c896c76812d 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1871,13 +1871,78 @@ static inline enum node_stat_item cache_vmstat_idx(struct kmem_cache *s) NR_SLAB_RECLAIMABLE_B : NR_SLAB_UNRECLAIMABLE_B; } -#ifdef CONFIG_MEMCG_KMEM -static inline void memcg_free_slab_cgroups(struct slab *slab) +#ifdef CONFIG_SLAB_OBJ_EXT + +/* + * The allocated objcg pointers array is not accounted directly. + * Moreover, it should not come from DMA buffer and is not readily + * reclaimable. So those GFP bits should be masked off. + */ +#define OBJCGS_CLEAR_MASK (__GFP_DMA | __GFP_RECLAIMABLE | \ + __GFP_ACCOUNT | __GFP_NOFAIL) + +static int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, + gfp_t gfp, bool new_slab) { - kfree(slab_objcgs(slab)); - slab->memcg_data = 0; + unsigned int objects = objs_per_slab(s, slab); + unsigned long obj_exts; + void *vec; + + gfp &= ~OBJCGS_CLEAR_MASK; + vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, + slab_nid(slab)); + if (!vec) + return -ENOMEM; + + obj_exts = (unsigned long)vec; +#ifdef CONFIG_MEMCG + obj_exts |= MEMCG_DATA_OBJEXTS; +#endif + if (new_slab) { + /* + * If the slab is brand new and nobody can yet access its + * obj_exts, no synchronization is required and obj_exts can + * be simply assigned. + */ + slab->obj_exts = obj_exts; + } else if (cmpxchg(&slab->obj_exts, 0, obj_exts)) { + /* + * If the slab is already in use, somebody can allocate and + * assign slabobj_exts in parallel. In this case the existing + * objcg vector should be reused. + */ + kfree(vec); + return 0; + } + + kmemleak_not_leak(vec); + return 0; } +static inline void free_slab_obj_exts(struct slab *slab) +{ + struct slabobj_ext *obj_exts; + + obj_exts = slab_obj_exts(slab); + if (!obj_exts) + return; + + kfree(obj_exts); + slab->obj_exts = 0; +} +#else /* CONFIG_SLAB_OBJ_EXT */ +static int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, + gfp_t gfp, bool new_slab) +{ + return 0; +} + +static inline void free_slab_obj_exts(struct slab *slab) +{ +} +#endif /* CONFIG_SLAB_OBJ_EXT */ + +#ifdef CONFIG_MEMCG_KMEM static inline size_t obj_full_size(struct kmem_cache *s) { /* @@ -1956,15 +2021,15 @@ static void __memcg_slab_post_alloc_hook(struct kmem_cache *s, if (likely(p[i])) { slab = virt_to_slab(p[i]); - if (!slab_objcgs(slab) && - memcg_alloc_slab_cgroups(slab, s, flags, false)) { + if (!slab_obj_exts(slab) && + alloc_slab_obj_exts(slab, s, flags, false)) { obj_cgroup_uncharge(objcg, obj_full_size(s)); continue; } off = obj_to_index(s, slab, p[i]); obj_cgroup_get(objcg); - slab_objcgs(slab)[off] = objcg; + slab_obj_exts(slab)[off].objcg = objcg; mod_objcg_state(objcg, slab_pgdat(slab), cache_vmstat_idx(s), obj_full_size(s)); } else { @@ -1985,18 +2050,18 @@ void memcg_slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, static void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects, - struct obj_cgroup **objcgs) + struct slabobj_ext *obj_exts) { for (int i = 0; i < objects; i++) { struct obj_cgroup *objcg; unsigned int off; off = obj_to_index(s, slab, p[i]); - objcg = objcgs[off]; + objcg = obj_exts[off].objcg; if (!objcg) continue; - objcgs[off] = NULL; + obj_exts[off].objcg = NULL; obj_cgroup_uncharge(objcg, obj_full_size(s)); mod_objcg_state(objcg, slab_pgdat(slab), cache_vmstat_idx(s), -obj_full_size(s)); @@ -2008,16 +2073,16 @@ static __fastpath_inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects) { - struct obj_cgroup **objcgs; + struct slabobj_ext *obj_exts; if (!memcg_kmem_online()) return; - objcgs = slab_objcgs(slab); - if (likely(!objcgs)) + obj_exts = slab_obj_exts(slab); + if (likely(!obj_exts)) return; - __memcg_slab_free_hook(s, slab, p, objects, objcgs); + __memcg_slab_free_hook(s, slab, p, objects, obj_exts); } static inline @@ -2028,10 +2093,6 @@ void memcg_slab_alloc_error_hook(struct kmem_cache *s, int objects, obj_cgroup_uncharge(objcg, objects * obj_full_size(s)); } #else /* CONFIG_MEMCG_KMEM */ -static inline void memcg_free_slab_cgroups(struct slab *slab) -{ -} - static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, struct list_lru *lru, struct obj_cgroup **objcgp, @@ -2298,7 +2359,7 @@ static __always_inline void account_slab(struct slab *slab, int order, struct kmem_cache *s, gfp_t gfp) { if (memcg_kmem_online() && (s->flags & SLAB_ACCOUNT)) - memcg_alloc_slab_cgroups(slab, s, gfp, true); + alloc_slab_obj_exts(slab, s, gfp, true); mod_node_page_state(slab_pgdat(slab), cache_vmstat_idx(s), PAGE_SIZE << order); @@ -2308,7 +2369,7 @@ static __always_inline void unaccount_slab(struct slab *slab, int order, struct kmem_cache *s) { if (memcg_kmem_online()) - memcg_free_slab_cgroups(slab); + free_slab_obj_exts(slab); mod_node_page_state(slab_pgdat(slab), cache_vmstat_idx(s), -(PAGE_SIZE << order)); From patchwork Thu Mar 21 16:36:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599164 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 963F9C54E58 for ; Thu, 21 Mar 2024 16:37:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9905B6B009B; Thu, 21 Mar 2024 12:37:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 919766B009C; Thu, 21 Mar 2024 12:37:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D0B86B009D; Thu, 21 Mar 2024 12:37:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4B9ED6B009B for ; Thu, 21 Mar 2024 12:37:28 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 2D9AB160C19 for ; Thu, 21 Mar 2024 16:37:28 +0000 (UTC) X-FDA: 81921601776.20.2800DEC Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf06.hostedemail.com (Postfix) with ESMTP id 5E8C3180028 for ; Thu, 21 Mar 2024 16:37:26 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=h1exedKV; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 3RWL8ZQYKCDIgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3RWL8ZQYKCDIgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039046; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WOYiJPoOo6EZ1v2aVMfBUZhNa7dlHF73f/fDRlekI3w=; b=KDSfEkeFjU5mq3gf0JHDHYZ+8NqBi8l4+XoJXUkn6teIK7/fa+YoQPkW9FYH8CTnUSZz7+ Cv2K7GTJk90z0m9Xyy+ZLyfotkKDnzSQTX2QW0vAyi95JpHeDHGyz2cwaUHtH8Gd3dOS59 4AsPvpTr1jpgGH/RzamArQHbYf6zXhs= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=h1exedKV; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 3RWL8ZQYKCDIgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3RWL8ZQYKCDIgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039046; a=rsa-sha256; cv=none; b=M0uDHbdcExjkPTA1PGkHErIRISQEOXeY8Tbt0S28oI0/D7zmE7vwrXrNBO+9ZVdRiRFdeo SKf5UdB8s7vXuLLMHgrkPIo4z6G9xU46yD6ZiU3sdgr9hQRKbsk5UqgT7gm10QgzZKc5cd ECmKghQFhwLjlE9XiZ8npS420RXkJ6A= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60a55cd262aso20235177b3.2 for ; Thu, 21 Mar 2024 09:37:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039045; x=1711643845; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=WOYiJPoOo6EZ1v2aVMfBUZhNa7dlHF73f/fDRlekI3w=; b=h1exedKV8O/r8toA1fNyRIcJnSIRcjVZLvboCDD7Am9b8RCRsoMOe1gHNQuIFtpVmo 6DdHJvgmGcLT2fCJ/6KW1vSJKca2ia0kzDJcNBAB5enNZapNgyp7RF27dsGIpUiQOQJC dko97ycVh6IEOQLIfUhgt3djTmtIysBCCB1JfdPZpwChzdBv8bJJyfJICcRh9QbNBs2v VeEy9l45SaeINwwblbJm74zBsBT+3cUyrE03f6/zAtrLi25n/yXAjCBG1tqWluvbeX78 OVOF0ql7yr2wJyzGRaQpJp5LvX8Uo4MwAADXNw5J4PHbqvTPfGMM2HQi2t+K0TmKywzX qULA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039045; x=1711643845; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WOYiJPoOo6EZ1v2aVMfBUZhNa7dlHF73f/fDRlekI3w=; b=VM+38m50HxBkOoBp5X313ZfLWTYdxSTxfiZ+g7ZzVQoaJDb+u4QN/Wi2Eq+TOzB6tx N0u2Mz0Ccuz2uiymu7vogCEMzrCicND9xcnsR8TM+zRDn6Fzbfn2f/MTg/1/ZuSWN1MM 3V2Gsfjz+IG4xdbKQQifGoai4lMFpS10IrzIsRFVX223zpSreQzYR6KPGwTSLJD4x20j BDogjqpO/OMmID4uAO744iy1nl786AtwDrd7G2NXX6BNOj1nARpE+XNRcyKTmBNIQh7q C/T8v3q7ZpSvkCAIL4gzLN1DX+mnyQ0r+88kVVhjUrC2yGLpcHK09sa9hueGNziNVuXW 2dSg== X-Forwarded-Encrypted: i=1; AJvYcCVOlT+cInpNo1jjBbb27bHFA+eB4QX2GQErERmPsB0zrgBd1zN8AFWOmC+4aiHWeqe9kpQcq8rYleQOU+Fq3ewehH4= X-Gm-Message-State: AOJu0YxygHpxCrc+LNuDro/KkxS8w/YoXweguM8q7MoFgohuVWehW+dn T3wJrJXJZNwtsOscavjvqgya60BcqF1Ley69EEl0BF6FBPuC0SkSqnJ+vD5ETZbZqLgsUZnMYfu sEA== X-Google-Smtp-Source: AGHT+IEAhjSJyNICBFt8fC07FMUXAIv3JTrEOHUKD0VuV0TBcrEbM/VY7SJ2rrR05OxGMObTDGwgaQeFx3s= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:72f:b0:dc6:eea0:1578 with SMTP id l15-20020a056902072f00b00dc6eea01578mr2412289ybt.13.1711039045315; Thu, 21 Mar 2024 09:37:25 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:29 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-8-surenb@google.com> Subject: [PATCH v6 07/37] mm: introduce __GFP_NO_OBJ_EXT flag to selectively prevent slabobj_ext creation From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 5E8C3180028 X-Stat-Signature: bix4nfb544ymftyjz3c7ec7z1z34py76 X-Rspam-User: X-HE-Tag: 1711039046-841309 X-HE-Meta: U2FsdGVkX19LDZE7jKMMzKdXE+HmFtkH4JyFjA/KY0OYtOkOwapL8pXsQSn1SAmrMXchPVX0RXUydjBMUt0lSPrbIYnOFzb0X9PbpZ02NrJBd2y+sGCF2Y1CxErYkpWMiWx/JaNsP0+aLGSk55PaEE0i47xTKJ7+01CNOOe3Ef0FL+KMuZhPA3qtJnbPKku6qixuFDm5aCoWGcNlCQHW28Z7b4cEcdxI3/Ah/tFqNkFzoh/U/PHbVmCgVEU4mgRJ8yRCnhI7vyTh8fiLT+e0jjBKfPmELOx0P5WcjuxslT2cgsCh3FCOa+ZjcQiCh2VWjf5CNGFa1jxPjmNC4Ru2YAbJNWO7KWMa/Ity4Iw7UFSEbYeTM0W6+sB8qRgOWuWfbUH451yniILCtuj0YT+MD2mqN1hjlO8lA2Z4v5b6b2WEPIExjRWRsY1idfhipzQbVsIa7UqAWmupyq6LQU7vDkPB/7Qnps+OrSFsbxzjk5P6kKiZosC49BUHch+GUZ1PgXikuyrlNbWZ5A0wMDSMGDSlJCSPx7/a2mX/v9tfjLi1ZwD2vhVhAwl1G8uIAIJkkrsQLOXxBmJMHebbI+7d6TmZUfn25L1eEzD79cou5zob1Uq09WuoYJx7y/SYBubSjf65vxEr1lnJm4PJKdblgRQ+/HCxP1b511vaxBlMWSpkIhkqKCBIoMkqUHOupvyeFgCsyeTxsrwL+wNzc386HYwgIHx9Upn1V9LByyQ0GjUx7WXJcPHyfL4wm+lGmAtPlgmhm7kKykQcAmVMxIqo2oFSqu/6t0MkwxHyiA62BJfoIiMeZdxmsT5ZnMXQ/4vZfThk/a+CMJyX0YDoAJFHOkXM65e4uMVjycCbUZbkSQqrKFZluvOCLsOxUXzo2NXXI6HGGkQP7HlAcNIeJUOmBgVLBuxnzTmd7ZZ1LKihXBrIfy7/Aru++WkzQSDC63r3+E9sC4LnG8PkLGYmlQN pqxjlPwf ElVmPSJ3aWSYRc/9IqqHjHnhXV33SuZjOaKqppuxOtOyM+OjCGcdszhTKZ2hIEq8vGtJDOtAWdJobxjcWZH7/BIK+oHT7ARjH8BTgGsOpMcsiWbyFZgHhcIRxXTYf0Gfm0j/17/FxmU1n9eN3EP8eOgJrn629KLN8gHXDdDP7fFeJq+HTf4x189AyRqFOf19asd3/xxFH/KQqZZDb6KrYmrM/Y9m7MqF06dHAEr9DbwLXL3idQJjHKVlJp0bzdrX3Z5vhvvZ732J52ww744RaZbjkKk2qu82H41cB2ygwWCx2nXrPGZ9psuHucEDrcUwxM+RYG72TPt+NjFQtlHfGETbC9Rhj5e/EMmQbcG77PNAfBUhF8M7PSjPMgX4uCm5xC5XLY9w9kMFE2MhxkYGXmkwJGG11YY/X+JS74XFVAacJE0A9MiRuWg6g/CC1h8z4b0aM7JDY1lJp74Ng5sHqA6ppaAetx7n3rofFOUlI2K/0vmYVMwCeW8eOhaCBMXz3ySxoryYTB9DTVSYRorY4bH9qzcvWiD7Vg/0NbFJcmzX2RU0m7O7i/4TyocxZPJho5RTiRO0twhY+aow6sMRvrGacajY0PtCsNeqEpgl+Z3rd0Wi43ewClHvZ6m+HHul6Lq+7vH+kWEfVoAJCrCg1ZuQw1AJL4TSBOrrS3xnT2w6+aEqAGfO0x3XYiZxETMxxce/CcrnabXi5/EaqIAAhrH3pts7eTSoTRiNfjmAfSY0Ta1EoFkidDWvyT+zsggmXMA0x1Jjr29xEyboBtZ96Q3IfEY3hfHBujnGZFCFCippooWXDxNMyFo6DEylWkf5sLLBxg5jiAPLLqMVUse7GeblryA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce __GFP_NO_OBJ_EXT flag in order to prevent recursive allocations when allocating slabobj_ext on a slab. Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook Reviewed-by: Pasha Tatashin Reviewed-by: Vlastimil Babka --- include/linux/gfp_types.h | 11 +++++++++++ mm/slub.c | 2 ++ 2 files changed, 13 insertions(+) diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 868c8fb1bbc1..e36e168d8cfd 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -52,6 +52,9 @@ enum { #endif #ifdef CONFIG_LOCKDEP ___GFP_NOLOCKDEP_BIT, +#endif +#ifdef CONFIG_SLAB_OBJ_EXT + ___GFP_NO_OBJ_EXT_BIT, #endif ___GFP_LAST_BIT }; @@ -93,6 +96,11 @@ enum { #else #define ___GFP_NOLOCKDEP 0 #endif +#ifdef CONFIG_SLAB_OBJ_EXT +#define ___GFP_NO_OBJ_EXT BIT(___GFP_NO_OBJ_EXT_BIT) +#else +#define ___GFP_NO_OBJ_EXT 0 +#endif /* * Physical address zone modifiers (see linux/mmzone.h - low four bits) @@ -133,12 +141,15 @@ enum { * node with no fallbacks or placement policy enforcements. * * %__GFP_ACCOUNT causes the allocation to be accounted to kmemcg. + * + * %__GFP_NO_OBJ_EXT causes slab allocation to have no object extension. */ #define __GFP_RECLAIMABLE ((__force gfp_t)___GFP_RECLAIMABLE) #define __GFP_WRITE ((__force gfp_t)___GFP_WRITE) #define __GFP_HARDWALL ((__force gfp_t)___GFP_HARDWALL) #define __GFP_THISNODE ((__force gfp_t)___GFP_THISNODE) #define __GFP_ACCOUNT ((__force gfp_t)___GFP_ACCOUNT) +#define __GFP_NO_OBJ_EXT ((__force gfp_t)___GFP_NO_OBJ_EXT) /** * DOC: Watermark modifiers diff --git a/mm/slub.c b/mm/slub.c index 5c896c76812d..2cb53642a091 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1889,6 +1889,8 @@ static int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, void *vec; gfp &= ~OBJCGS_CLEAR_MASK; + /* Prevent recursive extension vector allocation */ + gfp |= __GFP_NO_OBJ_EXT; vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, slab_nid(slab)); if (!vec) From patchwork Thu Mar 21 16:36:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599165 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B201C54E58 for ; Thu, 21 Mar 2024 16:37:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AF1976B009C; Thu, 21 Mar 2024 12:37:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A7DD26B009E; Thu, 21 Mar 2024 12:37:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 87D256B009F; Thu, 21 Mar 2024 12:37:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6C56B6B009C for ; Thu, 21 Mar 2024 12:37:30 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 022D51603E7 for ; Thu, 21 Mar 2024 16:37:29 +0000 (UTC) X-FDA: 81921601860.01.E515B89 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf01.hostedemail.com (Postfix) with ESMTP id 3548640005 for ; Thu, 21 Mar 2024 16:37:28 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=gTp2kMv5; spf=pass (imf01.hostedemail.com: domain of 3R2L8ZQYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3R2L8ZQYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039048; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SHSiwfbVG9SSjDRWdVOT6ZUhatpmkb8bIc+SfDPMGsQ=; b=QAkZpMr5nUqvirBrrL8zLBg0zNqLOVXIxC/XJ8ju3lvOPpJQCenZqP7yXbFQn0R6zKSRJE +/WH1F+md6AVAlaxqhfrJxWMQa38wy5Nd8X+Gxyr18V9nIpya/eiTlMQVQbq/hP8Wft+Or Oj2oUpuG2ME8B/HUWAEkmM0XPXZoVYg= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=gTp2kMv5; spf=pass (imf01.hostedemail.com: domain of 3R2L8ZQYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3R2L8ZQYKCDQikhUdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039048; a=rsa-sha256; cv=none; b=KH/RsTv746/kG9ZWw+HyUI9lT1jJLOXj7yCBYatYuKyo3TiMB4RiWPm65xilixIiZhQrQz qPVR43b8NzWJ6pdbyu51oEBM71vfaty1KUxjr4BBi/LOOEzt4n3CYelHJoA3h6BNQGqpBc lirGEi6pbb4UE1nUBKsXu8bXxdRJK9A= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60cd041665bso21140057b3.0 for ; Thu, 21 Mar 2024 09:37:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039047; x=1711643847; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SHSiwfbVG9SSjDRWdVOT6ZUhatpmkb8bIc+SfDPMGsQ=; b=gTp2kMv5/jsBRH54io4pMirGjgoI6BUTUOPj6HxuRmbJGSBVQD7+u7bbsHMQ+GtkmU rKHs5OiAHoRfa8wO1KK6YsAnIfNplx4hGyQ7VFwwGy1d0zh7fmyGl2VVICYMrF+IpChh yVeF7wrR4mfmMBGHWNOVfMzATsfA/YbsoCR+PI5ioxmSf2hKcbU+ROCvA0IoAmFiD54j Kb3IeSmbakBtWWdQc8zOEDgV/8GHu/IP0kiudH0bHEeT8Bm1bUwncvrud0sWD19XlGxt hIRravbGp9ktUBUfXvP5W91j2Wx8hWrkL8ESaulbZwn7Gm2iYyGtsRUGpqXmUHUTaO74 BzNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039047; x=1711643847; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SHSiwfbVG9SSjDRWdVOT6ZUhatpmkb8bIc+SfDPMGsQ=; b=nhc7CUjDDedQy381UAXAWU9Ihs/LCoeP8J+fmHwWm7WUrpnFTVu1ho1iuYK6imfOEv xzrgZLPwG6NwZTpY4hI3tVBhA41me9zojhu1cOxUFRS58YDqf2DvGHvCerqsvG4h6Lyk T9jjF8conLNRgUpUPhoZrW/ImEd8kbWUXqqR7CFndDmx1+S9IfIIyzMsVwbwy8M0Jixb wVk7hsxuYfjj/oexSZQQiidNCDLhiR3n/Y7WdEnrKLnzMPUZqzoteQXQu8oaTMXE3j5x y7olaS8fBTagirCNQj0nLVtP6nWWrOjfhEXoyvCVCkfPCW5OAqV957WlIPLGS6edYomB Y86A== X-Forwarded-Encrypted: i=1; AJvYcCUfrlabGE+G/sMvv7KWDT942oxXbwHWSSRxOa5HuhbZQUd/rwBx2hoWVkJIJpUa9YEX5lI0yzvbBH8F1l50H2pSY8w= X-Gm-Message-State: AOJu0YwIlXC3L/RtoaZxRmd7BvBN3l/5Iask3mZAUDv2zApRPnukfzH9 EKWVc1kXQ66lxt7yhC5Y/+OWu90HYB8f16ZV/QO3JDxL/EMlqAuSimvIU7W7tOvjm/yeyMLpYd4 Lnw== X-Google-Smtp-Source: AGHT+IHTxPHB7kx8Th/drFBQT8IBucAuzxxJkHgWcz7qLRX4uELPgEbRqEPHfHLIjKXlEPHR4aaEf+Jx60k= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:690c:d0b:b0:610:e0de:1387 with SMTP id cn11-20020a05690c0d0b00b00610e0de1387mr2235869ywb.2.1711039047349; Thu, 21 Mar 2024 09:37:27 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:30 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-9-surenb@google.com> Subject: [PATCH v6 08/37] mm/slab: introduce SLAB_NO_OBJ_EXT to avoid obj_ext creation From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 3548640005 X-Rspam-User: X-Stat-Signature: ifgswtswyu6sb3nunbqp7emd4thhcruz X-Rspamd-Server: rspam01 X-HE-Tag: 1711039048-487044 X-HE-Meta: U2FsdGVkX18Qn78Gth8Elj0zaGNCDNpKN3uH6NGz2aEv6IRTF9dmAeU71c1hcfnFMZ8g/js007ZvSvM4Q2n/1poI09sSPD7jwgEzH0ON0wQ1RcpivRPVFS2xVFpMb48S97qUyMvbv0hslCajJRdEhznZbQU34/CiDCqT5FXPw4Rxm4SwZl6pvMnJPO7CbnM6OWq5TbBmuqJsjcKoMJHllYxNs8t6V9QVndsuGkMWcGyPFcCLWmUrbMKo8CGKEessshtHLoEaIKAtGJ7uzuNP7WWsUUsE990Z0RXRMFnbOQqU0gjacIxSbi5Yk5QS76tPBqpV28AgSopyhopnZr9W+eKdNuGd4nlnRgfrEbKMwtfCYIcbIwFtQfacpZTIR2KfrGep5jraltXWv4cA0m7RavVf1AIiMTRf8dpapM/AwhejBZhZw6MI0xtBi5zMrrtI8cjHRUyr9T6EaKP6A7LwWFciHiyf+yRRsKVQUwsvdSJqlERxKd3kWFZL8ReJkBn6hdphRrQ1GhorvoRrDWiIZ36NkqyFjmFJxNmo5SEQC102nOWc7CjpnNr2zQ2szfhRbkqgSOvvueV1x1KN9Mr3nwSPUxysfWwc3Ce6CkcGL3IZYIFNSeMPX8d3o/iJeQpvqoSb4e0bXxobRuvh7V+GpsrhNJI4TG8FNY3oOFRv0HDWImK90h3dM/pLKrtVdKyTkNugsCqaTT/o8EYQrGs7D3ZC0xMPgr9E26p/Udf/+qs/GTuJjezmfAHz8c5/idJcXkJMOjKJTUEDYvG4xOhFrGWR0XtGxcWJqZXikx4bDS7zd4ZTPPje5ewZdu17+XDdL0j73/T7iZDhb85mN6idpR7/MDHvNaTaX/dpmK674hkx+oqRQtb+/DhEZHNafXPN/Z1krPqmGQRhTePfLiqskQQ5gPYKj+rfVN14FzYDv7bnQOE7EW+drlMpk1OiYG5bpkOgZBxTUQPKXA+DkV8 liMpqMzT zSRUTEfm6EjcVGRy3i3FvDN7cNIx489Gnu5s3Q8+3uRRyQZueEhs+sN2Jn4SwR23xm3tax2sm5FuwqrAlifkcdmNMTgPE2oVzjhqtb1CzPRpKmJFvfYsHN4+rwTXkfZQywjGBWFRdI4YZ5PUlhMXtfas5yuWBW0pss+lHKYNlmT9xyB++FsuFjBQA6bpldBEMMDszRBfm3xeaKFYq5Z+vrIUZeOo1t8VEW67/G/p2eRUndTKn46F6eN6q3vYHzsf/5IcG3pwS0hR5oJEySzuJhz5x2u5jYB1bvwdPOaSDRpwxcHaMT6VTHkTiU4lEO1YpBhOY+qPgArR5Kpso2YC6iNYe8F+XEmeBgeMghUm1Ge87hgMN/1bxej8QPTy7oRB1dofo4ALyQuuEf9Uqwp+OwnL6VtRZ11eNRWx+JLO7324xRkrEdmRVgzyNy06KhzYqcm58xner6g9ik9A8p2TXe+Ae4v33riMo8oR0cBatGSPoa6DObEW9n2bz4U5OQ6OcEofFXV+5yXoZnySfnGG1KyonTK7Auyp/JIxATj8rQuA0FHetW8zTe2JsI5pZfBxuWkN6q6ilTtlnFxfVJjvOdmytZgBwIRM9T1b91PcYGHzOhGawWWzdxA2zm3BI5pg4EYwGo1b1PJc7j9VqNnwDedTrN0Xn4GZE05gju8ZXxy18d1sVhxm3MywJ8SkVJJROzizD4pqvey1GqY4KG6VSd0Ptdimlm10kCNecLE/9Ay1+XsAxS7/kzOWKJaAjrmgjHXVLDuVUo5KYP5AaiOKULq2AtklElFlD9ixQBAIMXH1k8SaPx2NMg8+RidRU5x+DLP+P/3peo/CIqOf+kyws80R7/A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Slab extension objects can't be allocated before slab infrastructure is initialized. Some caches, like kmem_cache and kmem_cache_node, are created before slab infrastructure is initialized. Objects from these caches can't have extension objects. Introduce SLAB_NO_OBJ_EXT slab flag to mark these caches and avoid creating extensions for objects allocated from these slabs. Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook Reviewed-by: Pasha Tatashin Reviewed-by: Vlastimil Babka --- include/linux/slab.h | 10 ++++++++++ mm/slub.c | 5 +++-- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index e53cbfa18325..68ff754b85a4 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -56,6 +56,9 @@ enum _slab_flag_bits { #endif _SLAB_OBJECT_POISON, _SLAB_CMPXCHG_DOUBLE, +#ifdef CONFIG_SLAB_OBJ_EXT + _SLAB_NO_OBJ_EXT, +#endif _SLAB_FLAGS_LAST_BIT }; @@ -202,6 +205,13 @@ enum _slab_flag_bits { #endif #define SLAB_TEMPORARY SLAB_RECLAIM_ACCOUNT /* Objects are short-lived */ +/* Slab created using create_boot_cache */ +#ifdef CONFIG_SLAB_OBJ_EXT +#define SLAB_NO_OBJ_EXT __SLAB_FLAG_BIT(_SLAB_NO_OBJ_EXT) +#else +#define SLAB_NO_OBJ_EXT __SLAB_FLAG_UNUSED +#endif + /* * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests. * diff --git a/mm/slub.c b/mm/slub.c index 2cb53642a091..666dcc3b8a26 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -5693,7 +5693,8 @@ void __init kmem_cache_init(void) node_set(node, slab_nodes); create_boot_cache(kmem_cache_node, "kmem_cache_node", - sizeof(struct kmem_cache_node), SLAB_HWCACHE_ALIGN, 0, 0); + sizeof(struct kmem_cache_node), + SLAB_HWCACHE_ALIGN | SLAB_NO_OBJ_EXT, 0, 0); hotplug_memory_notifier(slab_memory_callback, SLAB_CALLBACK_PRI); @@ -5703,7 +5704,7 @@ void __init kmem_cache_init(void) create_boot_cache(kmem_cache, "kmem_cache", offsetof(struct kmem_cache, node) + nr_node_ids * sizeof(struct kmem_cache_node *), - SLAB_HWCACHE_ALIGN, 0, 0); + SLAB_HWCACHE_ALIGN | SLAB_NO_OBJ_EXT, 0, 0); kmem_cache = bootstrap(&boot_kmem_cache); kmem_cache_node = bootstrap(&boot_kmem_cache_node); From patchwork Thu Mar 21 16:36:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599166 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B470C54E58 for ; Thu, 21 Mar 2024 16:37:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC5D06B009E; Thu, 21 Mar 2024 12:37:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C74576B009F; Thu, 21 Mar 2024 12:37:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA22C6B00A0; Thu, 21 Mar 2024 12:37:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8C3746B009E for ; Thu, 21 Mar 2024 12:37:32 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 690A31C07D8 for ; Thu, 21 Mar 2024 16:37:32 +0000 (UTC) X-FDA: 81921601944.06.A0B7FC1 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf19.hostedemail.com (Postfix) with ESMTP id 8BF0A1A0011 for ; Thu, 21 Mar 2024 16:37:30 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TuhvuC3X; spf=pass (imf19.hostedemail.com: domain of 3SWL8ZQYKCDYkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3SWL8ZQYKCDYkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039050; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dkdT2DGL8QlDQh+VA6yRS9pNWPbAmDPXuEOr3/wpZDQ=; b=0apeidMRx5Eg4J4hUG3Mz5EH8ScQF+P8rS+bwd8cTkdaPRFWEcyoxri4X3E9EamehmGloq weNQDiOEZGMkKDdqj7p6TVN6T8B+XqBbO9RvvX3CDKNGA8mu+kbKWVMFy180MWqNPgGkcw p23UEqxs9YkVcKefmqw1iQnF5j8OjhM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039050; a=rsa-sha256; cv=none; b=X1wSAYqLKP1tTMKSE4pKNgnvGDnWw8vhmcmLTozn525iU7fOpVQtE19TXKLXLVnJ1olCV2 Yeuw0zjopmxzCMhOeGrkWuK1o8p3fZfrmZHozDtFUD9zq8gIwpku/8hYloa7T4M9ZxU9vX L0o4Etkdfgm3PEmefsB7tZ9+IVgNKbA= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TuhvuC3X; spf=pass (imf19.hostedemail.com: domain of 3SWL8ZQYKCDYkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3SWL8ZQYKCDYkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dcbee93a3e1so1860050276.3 for ; Thu, 21 Mar 2024 09:37:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039049; x=1711643849; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=dkdT2DGL8QlDQh+VA6yRS9pNWPbAmDPXuEOr3/wpZDQ=; b=TuhvuC3XC4+a6CPAxaPy7Mp+ZvcTcc1I7Wnwj3S6kyvjH4lJNBFYcxy8I2KLad25Ni IBcleImAblgn739OWXanqHpKXzoE+namnYw1a2gKVyQBesgP7acdA9RMKRIA3WxuDmoF 5+/JrgnFNI2uZ8heamTfvwJv1UlxHhD+zCIoULaKUsPka3EAVyBlDP5Uzpj/fjVmi3nV Ii6vWSA7LCxrkhN84wAVWpkGQoaa7aV3z68miSYxzpER4/sk/7V7SOffq6zOgPQvq0uP 84cJA5X9jASAIgE+6Ev66UIkwDPg/Ot4uAROqdOIpreyS4Jargp3uf6du5eJJcLIDc0F bdHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039049; x=1711643849; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dkdT2DGL8QlDQh+VA6yRS9pNWPbAmDPXuEOr3/wpZDQ=; b=I/mq4Wff4QUHqyl8NaVKganDxvcPusdM89QQB/1LLtesnl2jScbt1D6f5WvYCD+qjE qJb3rkN4ElbAPPAY/O2HnGW3pPYIYjjthL9Zv5C6OoesBu+fUsXpXL9M4MSW7UKs71FJ cA5exqpF87FO408Ny4XqAw52UF3yLu4YjSD5pMHHdWmVqr6tsjuFnVDvhxIz7QvUOcjU lEJfOLSmJSO2HXm9CydJ3VLf9lY3Z8wE9HOWsFw8dCnlDNAL2YSXn6r/dWeu2WB9xTi9 xgqfGw+e1tFxHftOr2VGiUGHpypEV+P3zW+HQDliPBK78BCF9N/VamFeOnvReX4VQTA9 rB7w== X-Forwarded-Encrypted: i=1; AJvYcCUtWHvDVhy2M2gelZO+FQUiXG0/RegQdJ1zazSmmDZsu5TYhMPSOGe83QGvzNb2Q8owJBt/MRLny0pxOoJuw6LQYGo= X-Gm-Message-State: AOJu0Yz92+s2AEY3DoKx+EZHbh4wVnXEDiINA+SM1O3o+AFygZYF1R34 fNHX/I/T5slqaSSA/Hcx3gbgHVFkarzgpge1SRxrYgCMxBotMjG7B2Eejn0og6Su8Lgny/PjSRE vcQ== X-Google-Smtp-Source: AGHT+IH5FL19hotlByvhDCzXe4teRPXoOwhKIdyfBan8YOb9XXKYaMuIFalLgecMbTUkzippyEaUxzmEK2E= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:188f:b0:dcf:f526:4cc6 with SMTP id cj15-20020a056902188f00b00dcff5264cc6mr1142116ybb.11.1711039049407; Thu, 21 Mar 2024 09:37:29 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:31 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-10-surenb@google.com> Subject: [PATCH v6 09/37] slab: objext: introduce objext_flags as extension to page_memcg_data_flags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 8BF0A1A0011 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: f664n3sb98z6swd3sdcm5s65b8yfqk3m X-HE-Tag: 1711039050-595418 X-HE-Meta: U2FsdGVkX1/K7JDNs1ch6SYiWsEpQ8tBz7uI5/lArNLBNJbLznqhKSOZvp4dd1U+SKCwwGH+2mbs91oS+DKqRRmh0WEZh+qQ3H/6luMZIMH+6RFxV1aGcgnWZOg6sWLrY1h9A/dk916M0FE4K1QC4a7z0cwK8NRkP8pl0Fa88xUzU2s6AiNcwaG8OvoQ2YPwPfFe/O+PrrzZ5FVtzg41QA1r/rGfr/BkorjfsJ9Pg4DMo0sVAaA16bDNjRZg9JrMbeJWyKRctNOqRjsFQ0W6ndsxpJrbVvnrIFivuxyGpCVrlAfM+Vep/u1/ERPVCIOyUf3GVf/wvH4Irn9BpU76j7HYytrvpqU/ZzljQLolRoje4fi3is+09cXWwfekOEXD93pm4gPrZN4p9uebjA0AZ8eM0NCv+CZFUT+rcXX1T2AOodUb3nmHexxMhMdqjkuAWNoL6IrWR0Sm7SXxCLEAKVqUHgcDW7l5EX+fUkcaY4jUbp4+jbFUJbUlagUEVhw8VFnMPqWioIrvZoyw+pFXyB+TDri5DIrcUhbGuos6nHaK5511deykvVPnRjBSCIPzDnNDZHVnL9sD3e0YQircUuI+D1CZH7raMstVdB11yDW1wFjqE2ftcCqb0hW8pvuHJmZJ7J1pQs7KccpjXpBO9g3xXTBlA35nI2oAt6Fl58QYxNsUIutHpPcyXM2KBE06fzfaTnVKmiPWCFzOSj4XZ4n+LnL4ynR/IkEHTQeWfYZYmOyrF5s/Rv4sm9LrgrX2t1zyob4EzJveViy4Jh+xAQm4r+0bRm44uWh2z2Jj9f145cUADdi40MLq3l+xL16OEG/3aOIt1exVtIx4h7G6aeyOUW1MipaH/LI0I9o/GspUYZhKP0OW2QX+09ifpiXFqZsoS0bgBhpyn0HfUeIJytnfnCudqmyvBEkAE9LhY1Wy95k5GsskYcQ65BxxPK4xXkxJU8tpjncfxK45OHf dM585gg+ gyVKsZERLOA1Yz154G5peoSyRJsQraZcViF79oC/zaKypV15gSAwt5zRSKxQyAkvHL8UAonYMGo8gSxtt1NWuEfipGB9lCbZMRzG9g/KT423n91TbriUVKzyM15mVF6XaXCBllRZ64f5h8ZCBPqkkFAqfGU+3K+Ca8M+1zJBIiMs9pFUh49DNI0qwmcJ3NfiEfCbMGbQL15u8BPSn5FL+6KgWP5htlKBf3XPHdWW5PKyCAXae6/IOQOsR+1h0W3FYmm3Kns5InpStBUOdsqiOtW1nMoQvq8RqlftBh7nflVSuIpS0/N4N9dlYO7pgJSq+cb7rZ2w8jBJGO/8v0pCn5EqPpC4l7Y3IjRshsr3/x2LZn+sBIfmOpRuxt/jm9zAIJUrGOvElXnsnHM95dK9ByhpbR9PcJt2PwxWL3fF0g71VZYDQkVH2Yy7U2Rm9ALCpkr4bZIFUJfqugEFzVhJUnL1Td0xcMrqMgqmmpn1/dWtk2J+rNszCYAkwZfJy+Z3NXL8Zev63FDTU1SdBjAgKnG3VC4EI/ut7VAW6RL9nT29js80Oz8ADpmffT87X1RBxvGyBbO3jtauHedaaCizgtmIPo+7B/xtwq5fluSul1+K/WGhRT3U7tSJI3vmZg26sxrcqPRmqglRk3/NMOI7jejaL9BrLC523IDAy1yx/dPFro9yxrBkv8vEUN6K9z3wRJ6jot0pn7FpzaxCVuUyqTeY8EoNhcref6T4PYBAFo13tFGeDN0SZKI39wcoPj7m/QPONZnk9oZksOrdtvVm3LO6S88sbDAI3slbx2tnXwp+x4Y7+ueg6mcVGY3yYkyukIYQcze0jp4gXvzcWuehhx3O/wg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce objext_flags to store additional objext flags unrelated to memcg. Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook Reviewed-by: Pasha Tatashin Reviewed-by: Vlastimil Babka --- include/linux/memcontrol.h | 29 ++++++++++++++++++++++------- mm/slab.h | 5 +---- 2 files changed, 23 insertions(+), 11 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 99f423742324..12afc2647cf0 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -357,7 +357,22 @@ enum page_memcg_data_flags { __NR_MEMCG_DATA_FLAGS = (1UL << 2), }; -#define MEMCG_DATA_FLAGS_MASK (__NR_MEMCG_DATA_FLAGS - 1) +#define __FIRST_OBJEXT_FLAG __NR_MEMCG_DATA_FLAGS + +#else /* CONFIG_MEMCG */ + +#define __FIRST_OBJEXT_FLAG (1UL << 0) + +#endif /* CONFIG_MEMCG */ + +enum objext_flags { + /* the next bit after the last actual flag */ + __NR_OBJEXTS_FLAGS = __FIRST_OBJEXT_FLAG, +}; + +#define OBJEXTS_FLAGS_MASK (__NR_OBJEXTS_FLAGS - 1) + +#ifdef CONFIG_MEMCG static inline bool folio_memcg_kmem(struct folio *folio); @@ -391,7 +406,7 @@ static inline struct mem_cgroup *__folio_memcg(struct folio *folio) VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_KMEM, folio); - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } /* @@ -412,7 +427,7 @@ static inline struct obj_cgroup *__folio_objcg(struct folio *folio) VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJEXTS, folio); VM_BUG_ON_FOLIO(!(memcg_data & MEMCG_DATA_KMEM), folio); - return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct obj_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } /* @@ -469,11 +484,11 @@ static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio) if (memcg_data & MEMCG_DATA_KMEM) { struct obj_cgroup *objcg; - objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK); return obj_cgroup_memcg(objcg); } - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } /* @@ -512,11 +527,11 @@ static inline struct mem_cgroup *folio_memcg_check(struct folio *folio) if (memcg_data & MEMCG_DATA_KMEM) { struct obj_cgroup *objcg; - objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + objcg = (void *)(memcg_data & ~OBJEXTS_FLAGS_MASK); return obj_cgroup_memcg(objcg); } - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return (struct mem_cgroup *)(memcg_data & ~OBJEXTS_FLAGS_MASK); } static inline struct mem_cgroup *page_memcg_check(struct page *page) diff --git a/mm/slab.h b/mm/slab.h index 1c16dc8344fa..65db525e93af 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -554,11 +554,8 @@ static inline struct slabobj_ext *slab_obj_exts(struct slab *slab) VM_BUG_ON_PAGE(obj_exts && !(obj_exts & MEMCG_DATA_OBJEXTS), slab_page(slab)); VM_BUG_ON_PAGE(obj_exts & MEMCG_DATA_KMEM, slab_page(slab)); - - return (struct slabobj_ext *)(obj_exts & ~MEMCG_DATA_FLAGS_MASK); -#else - return (struct slabobj_ext *)obj_exts; #endif + return (struct slabobj_ext *)(obj_exts & ~OBJEXTS_FLAGS_MASK); } #else /* CONFIG_SLAB_OBJ_EXT */ From patchwork Thu Mar 21 16:36:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599168 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84065C6FD1F for ; Thu, 21 Mar 2024 16:37:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AC95E6B00A1; Thu, 21 Mar 2024 12:37:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A79786B00A3; Thu, 21 Mar 2024 12:37:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D50F6B00A2; Thu, 21 Mar 2024 12:37:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 47CB46B00A1 for ; Thu, 21 Mar 2024 12:37:38 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 2268E141296 for ; Thu, 21 Mar 2024 16:37:38 +0000 (UTC) X-FDA: 81921602196.11.ECA71A9 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf16.hostedemail.com (Postfix) with ESMTP id BB57418001D for ; Thu, 21 Mar 2024 16:37:32 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=WGfJJ5uH; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 3S2L8ZQYKCDgmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3S2L8ZQYKCDgmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039052; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hrfC4K8F/Q+T2mI4qsQQV2fR1wUsQY/g8GL2B+37hNg=; b=IkWgXEacJpuNHrZMFBD35hMFegTS03hA1J8+gPsa1ZPmG/aEpXu/8H/9npJ7PIj/zEETJr 3yraIkN7pthuys8FzoL0nYgBUuKX+zNH8UWF8tzNGFLIvq5W5gHeDjpfagmmERU8NEX5ja COFDnTeIK2JxSIfNAj1YGtQQL7skHQ8= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=WGfJJ5uH; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 3S2L8ZQYKCDgmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3S2L8ZQYKCDgmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039052; a=rsa-sha256; cv=none; b=DpXfEoZPQthhdqfyVbBjQhXAMTKuf3A6tWaurMcXECnL4Rqdsnkwh20AJiyNjKjOASxvvY M6sRxmCpG3+9zRileXRNEcLMuNnNO4MUgWrRECRuM9F80fNPVyKsBm2k7TyM5WseSrkU1c zsHPxzSla9Sfh0yL7XGIGg5BFemNQ9Q= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60ccc3cfa39so16712997b3.2 for ; Thu, 21 Mar 2024 09:37:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039052; x=1711643852; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hrfC4K8F/Q+T2mI4qsQQV2fR1wUsQY/g8GL2B+37hNg=; b=WGfJJ5uHmHX1LUdRAj35iCrQhEvpIcd+MU0JhS8oAwG16Wl+J6SxrpKs08ylUojGd4 wy/gzYd2mUYPCv9+eGHY1FUE/9+jrvUpYyLJnuAHSaLn1hpM2hctU47vH579uEURCWrP X8wsxwXF5tYPtu0jqnDw2+QcgagK5pYd0kiyE1V+O3WxViRq2e4Inyv8NA24QTcKjIRj 49cQYjrxvQ2k0r9/a2HuDuFCtlaXtdCwh5g3PkH2LtEBgsCwOQGkFqqSItpKSz5le4lU WKFz03MpJKWXaRUpV2IMw3LPoNQehPmMoGFn+EJV/TVKEw8E3M8EPg1B5zxIljR5twBf zC1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039052; x=1711643852; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hrfC4K8F/Q+T2mI4qsQQV2fR1wUsQY/g8GL2B+37hNg=; b=hHaOmTLzfaz1AVBSqJN2rFLfqF12pKHqj+OC19YcTtafKI6T9bV5eXE21B+3uAuh2K 7FfJfJ49B5mwKhTsfLEOKm2L6wxHEMyScfGw8JkSSqnAgFN3L5Y93GbKetzr5VXJB/4Y bXbFbFcmXk2DiMh4TSiTvWaK65RI8gQbAjYEk1LYkm1j1A1qX+qh77RpYhIzrOgXZHBm ySLUMTLjPBNMqqIr0YCsgIn1TMUIQwAfI9bgdCA1nfalxSHUGWiNBAxUMUJdDJJPL/39 EF7kvsnAYSvra2bbiu838bqqRZZHAHxCKY8XnbPdq6tXQOPNI7BkBAA/rg4M4GnmNKPo l33w== X-Forwarded-Encrypted: i=1; AJvYcCW/32ffyU7Xq0r8Yeq3qtSYvytQJVpgRL8sBn0SqUnXoLRmU9r/FZ5EuZoIpU2HX7T25kUHVprOxg+36AzuQUAry8M= X-Gm-Message-State: AOJu0Yzc473sj1jF7BTCWcufBoGAeDkWsxNnEiKIxp5tQcZyK27TI3ZO /eief6EfMCS7bKRSjrZbditn178N3rt1+Qv8+LvWw3fbrNzQWYUq7EdF1kwAbrN/X27UnmlXMnt 0Ag== X-Google-Smtp-Source: AGHT+IFpSBGDOLAV/U358k2FNkIx88WUMnUVEq4fGBmj4tsKZJRhDdGZxZQUSD8tkAfYC1BnErXB1YJ5fBs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a0d:ca91:0:b0:610:fc58:5b83 with SMTP id m139-20020a0dca91000000b00610fc585b83mr1060972ywd.8.1711039051683; Thu, 21 Mar 2024 09:37:31 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:32 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-11-surenb@google.com> Subject: [PATCH v6 10/37] lib: code tagging framework From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Stat-Signature: sitf5zzrqce3p7d1w64csjsx777cb6ca X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: BB57418001D X-HE-Tag: 1711039052-205432 X-HE-Meta: U2FsdGVkX1/0bizppJPgxZ2g7gA0ZR7rtaj0q9sA/e9Z/vLIHA4RdUNvdq1i4ytzOq8xpJL+ox/8DVwmmFUTviUs8zJP6Ziu4ozJu7V45phJML6E83M/eCKQLyHnT31YQt5mevnz2ooPLGEgwhzjG8oyKz6Vj+yqglACxGPrq4g0J0BKSXzbroyuJYsVL+w+DUPaKSG1yKJFed9TlbH4+BNSEMT3W/PDjjs5az8VGfamF/TKliMthhPFJYOf5brk6ZqBuUJHFX6053pzeEnU3x4Gx8HHx8GA3HMer7DjWiSorm9RxQGR2mtDnbaxkHfHVDRADbCgT8fnA54a5/7kUGUofzn3MuvL1PmWYxcChd3B9eqddQmnh9LOQ5i5Bn0GpWK/QF4eSpD+AIY16CaQH+i3iJqnOocxE5TujA8/+3FLRxipuE2TvUpyKlJomd194L14n0kBDAji4K4Zi+zQCnRXq54BmUbqfM18M9EGYXKmpImvZ333HdxkfeKzkTHM9+A43vF4/EZxewFwiQ/IFyP6rtw4Gv7qn9bNvikPugw8t9WfxGmL9QWoF1kxLMN/9ZErm6F6HBNS2yyM4J+FlS3t9AiP8OMaUroXKWWX/p8n0bWErZnO5iZ4NxiPZyZCirpoLgw0OGGI7eV2h2gzK4haGNFLquVMVIVkSK5tAIKJ8U8wuu2oA5CTtWLZ1ZNFHx3DJ2LJFkqnT3XR2JuPmpli5uQHSd7En2nsrgrkDer2cxvKmqrY7gBogNweHuDbrVu0iiu4mHnwEpMIXIGIlVyeP1Fxb2cE7oygAP8XHC7BkHsuFKQSB6MJhyPjLhHIoXP4t7Wrm6rlMOELevNSmpSyh1qHDkrLr53sDWQkZNFs26TBulQ494NXeERHLzXr1Q6Ed+Fv/N6OIhSLhngbziWebQVGra6O7TYforj7R1wC/vbGmMM4rIVH+4uaTQca3Y+JwMpkZS4gpnqJW3S x6v29K71 D9YDM8UE68kJc/vlH7CeKATqfvBXHmkt0QCcniAj48hY0oe+6ybR2Hc6BbYhgdfDTmu8O8vFRdYvi+BQadukD8CYIkIemHN79iRSKjoHsh4qb40xduCOV8bC231/r94+ysq/vk4iuVbaHAMZZAmSoUGHtbKgSuQBvvR8fBTu+CUhuQd/BnVmkAhH71mRYhXVS/i1XOGoZwQxj9sKS+HTPvA+pDHq+FptZ6oFnBVeE+QeSbMI5jI8uGoQU846kNtL4Ipca8I43xWfHA1c87td6+hDnx+iHJKH0LOQTQCEXfpaz740jsCzXKAaDGC2v/11GuRN3vCV8wB1dRjDAtoZoXMTF4R4zFNCdCLpSgDXqTQ/AUtju5WaKF1Mh9m7cIlsHvx3IAb2JGeUt2WhXWTHJXHrnl6RyZcAvJoTC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add basic infrastructure to support code tagging which stores tag common information consisting of the module name, function, file name and line number. Provide functions to register a new code tag type and navigate between code tags. Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/codetag.h | 68 +++++++++++++ lib/Kconfig.debug | 4 + lib/Makefile | 1 + lib/codetag.c | 219 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 292 insertions(+) create mode 100644 include/linux/codetag.h create mode 100644 lib/codetag.c diff --git a/include/linux/codetag.h b/include/linux/codetag.h new file mode 100644 index 000000000000..7734269cdb63 --- /dev/null +++ b/include/linux/codetag.h @@ -0,0 +1,68 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * code tagging framework + */ +#ifndef _LINUX_CODETAG_H +#define _LINUX_CODETAG_H + +#include + +struct codetag_iterator; +struct codetag_type; +struct codetag_module; +struct seq_buf; +struct module; + +/* + * An instance of this structure is created in a special ELF section at every + * code location being tagged. At runtime, the special section is treated as + * an array of these. + */ +struct codetag { + unsigned int flags; /* used in later patches */ + unsigned int lineno; + const char *modname; + const char *function; + const char *filename; +} __aligned(8); + +union codetag_ref { + struct codetag *ct; +}; + +struct codetag_type_desc { + const char *section; + size_t tag_size; +}; + +struct codetag_iterator { + struct codetag_type *cttype; + struct codetag_module *cmod; + unsigned long mod_id; + struct codetag *ct; +}; + +#ifdef MODULE +#define CT_MODULE_NAME KBUILD_MODNAME +#else +#define CT_MODULE_NAME NULL +#endif + +#define CODE_TAG_INIT { \ + .modname = CT_MODULE_NAME, \ + .function = __func__, \ + .filename = __FILE__, \ + .lineno = __LINE__, \ + .flags = 0, \ +} + +void codetag_lock_module_list(struct codetag_type *cttype, bool lock); +struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype); +struct codetag *codetag_next_ct(struct codetag_iterator *iter); + +void codetag_to_text(struct seq_buf *out, struct codetag *ct); + +struct codetag_type * +codetag_register_type(const struct codetag_type_desc *desc); + +#endif /* _LINUX_CODETAG_H */ diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 733ee2ac0138..d2dbdd45fd9a 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -968,6 +968,10 @@ config DEBUG_STACKOVERFLOW If in doubt, say "N". +config CODE_TAGGING + bool + select KALLSYMS + source "lib/Kconfig.kasan" source "lib/Kconfig.kfence" source "lib/Kconfig.kmsan" diff --git a/lib/Makefile b/lib/Makefile index ffc6b2341b45..910335da8f13 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -233,6 +233,7 @@ obj-$(CONFIG_OF_RECONFIG_NOTIFIER_ERROR_INJECT) += \ of-reconfig-notifier-error-inject.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o +obj-$(CONFIG_CODE_TAGGING) += codetag.o lib-$(CONFIG_GENERIC_BUG) += bug.o obj-$(CONFIG_HAVE_ARCH_TRACEHOOK) += syscall.o diff --git a/lib/codetag.c b/lib/codetag.c new file mode 100644 index 000000000000..8b5b89ad508d --- /dev/null +++ b/lib/codetag.c @@ -0,0 +1,219 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include +#include +#include +#include + +struct codetag_type { + struct list_head link; + unsigned int count; + struct idr mod_idr; + struct rw_semaphore mod_lock; /* protects mod_idr */ + struct codetag_type_desc desc; +}; + +struct codetag_range { + struct codetag *start; + struct codetag *stop; +}; + +struct codetag_module { + struct module *mod; + struct codetag_range range; +}; + +static DEFINE_MUTEX(codetag_lock); +static LIST_HEAD(codetag_types); + +void codetag_lock_module_list(struct codetag_type *cttype, bool lock) +{ + if (lock) + down_read(&cttype->mod_lock); + else + up_read(&cttype->mod_lock); +} + +struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype) +{ + struct codetag_iterator iter = { + .cttype = cttype, + .cmod = NULL, + .mod_id = 0, + .ct = NULL, + }; + + return iter; +} + +static inline struct codetag *get_first_module_ct(struct codetag_module *cmod) +{ + return cmod->range.start < cmod->range.stop ? cmod->range.start : NULL; +} + +static inline +struct codetag *get_next_module_ct(struct codetag_iterator *iter) +{ + struct codetag *res = (struct codetag *) + ((char *)iter->ct + iter->cttype->desc.tag_size); + + return res < iter->cmod->range.stop ? res : NULL; +} + +struct codetag *codetag_next_ct(struct codetag_iterator *iter) +{ + struct codetag_type *cttype = iter->cttype; + struct codetag_module *cmod; + struct codetag *ct; + + lockdep_assert_held(&cttype->mod_lock); + + if (unlikely(idr_is_empty(&cttype->mod_idr))) + return NULL; + + ct = NULL; + while (true) { + cmod = idr_find(&cttype->mod_idr, iter->mod_id); + + /* If module was removed move to the next one */ + if (!cmod) + cmod = idr_get_next_ul(&cttype->mod_idr, + &iter->mod_id); + + /* Exit if no more modules */ + if (!cmod) + break; + + if (cmod != iter->cmod) { + iter->cmod = cmod; + ct = get_first_module_ct(cmod); + } else + ct = get_next_module_ct(iter); + + if (ct) + break; + + iter->mod_id++; + } + + iter->ct = ct; + return ct; +} + +void codetag_to_text(struct seq_buf *out, struct codetag *ct) +{ + if (ct->modname) + seq_buf_printf(out, "%s:%u [%s] func:%s", + ct->filename, ct->lineno, + ct->modname, ct->function); + else + seq_buf_printf(out, "%s:%u func:%s", + ct->filename, ct->lineno, ct->function); +} + +static inline size_t range_size(const struct codetag_type *cttype, + const struct codetag_range *range) +{ + return ((char *)range->stop - (char *)range->start) / + cttype->desc.tag_size; +} + +#ifdef CONFIG_MODULES +static void *get_symbol(struct module *mod, const char *prefix, const char *name) +{ + DECLARE_SEQ_BUF(sb, KSYM_NAME_LEN); + const char *buf; + + seq_buf_printf(&sb, "%s%s", prefix, name); + if (seq_buf_has_overflowed(&sb)) + return NULL; + + buf = seq_buf_str(&sb); + return mod ? + (void *)find_kallsyms_symbol_value(mod, buf) : + (void *)kallsyms_lookup_name(buf); +} + +static struct codetag_range get_section_range(struct module *mod, + const char *section) +{ + return (struct codetag_range) { + get_symbol(mod, "__start_", section), + get_symbol(mod, "__stop_", section), + }; +} + +static int codetag_module_init(struct codetag_type *cttype, struct module *mod) +{ + struct codetag_range range; + struct codetag_module *cmod; + int err; + + range = get_section_range(mod, cttype->desc.section); + if (!range.start || !range.stop) { + pr_warn("Failed to load code tags of type %s from the module %s\n", + cttype->desc.section, + mod ? mod->name : "(built-in)"); + return -EINVAL; + } + + /* Ignore empty ranges */ + if (range.start == range.stop) + return 0; + + BUG_ON(range.start > range.stop); + + cmod = kmalloc(sizeof(*cmod), GFP_KERNEL); + if (unlikely(!cmod)) + return -ENOMEM; + + cmod->mod = mod; + cmod->range = range; + + down_write(&cttype->mod_lock); + err = idr_alloc(&cttype->mod_idr, cmod, 0, 0, GFP_KERNEL); + if (err >= 0) + cttype->count += range_size(cttype, &range); + up_write(&cttype->mod_lock); + + if (err < 0) { + kfree(cmod); + return err; + } + + return 0; +} + +#else /* CONFIG_MODULES */ +static int codetag_module_init(struct codetag_type *cttype, struct module *mod) { return 0; } +#endif /* CONFIG_MODULES */ + +struct codetag_type * +codetag_register_type(const struct codetag_type_desc *desc) +{ + struct codetag_type *cttype; + int err; + + BUG_ON(desc->tag_size <= 0); + + cttype = kzalloc(sizeof(*cttype), GFP_KERNEL); + if (unlikely(!cttype)) + return ERR_PTR(-ENOMEM); + + cttype->desc = *desc; + idr_init(&cttype->mod_idr); + init_rwsem(&cttype->mod_lock); + + err = codetag_module_init(cttype, NULL); + if (unlikely(err)) { + kfree(cttype); + return ERR_PTR(err); + } + + mutex_lock(&codetag_lock); + list_add_tail(&cttype->link, &codetag_types); + mutex_unlock(&codetag_lock); + + return cttype; +} From patchwork Thu Mar 21 16:36:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599167 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1495C54E58 for ; Thu, 21 Mar 2024 16:37:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6E3E96B00A5; Thu, 21 Mar 2024 12:37:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C4EB6B00A4; Thu, 21 Mar 2024 12:37:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C1B56B00A3; Thu, 21 Mar 2024 12:37:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 222D26B00A1 for ; Thu, 21 Mar 2024 12:37:38 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 03A0B16015A for ; Thu, 21 Mar 2024 16:37:37 +0000 (UTC) X-FDA: 81921602196.11.E3C0534 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf04.hostedemail.com (Postfix) with ESMTP id B7C1E40020 for ; Thu, 21 Mar 2024 16:37:34 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=leKtAEWV; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3TWL8ZQYKCDooqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3TWL8ZQYKCDooqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039054; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BO7qlg8CrJrpGAZLx2tB5/z68XAeLcOjL+jbqSm1NrE=; b=n/ffmaKMG1VyQMHOOTLwySEtVdwxYzpMLxqlgfcOfH0153G66MeewVCbqyLFPjoCa7jhpu whEoL3Do2Ho2uWyjf6SISXoNh9Ti/d5rmZtgsDQB0r36C0s08Xly46F520enSZe9aBf6uQ jsBS3O3OGHQkpOuzOGGP8c0996mrzEw= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=leKtAEWV; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 3TWL8ZQYKCDooqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3TWL8ZQYKCDooqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039054; a=rsa-sha256; cv=none; b=xcGphInaacx5j4a6uPNGu3GZ3XURw/Qiy36IsSoFzRt00A4x6LMjJAqLqhkPUuAXEOy+OF reht+ZGpHruyT/P1Qrsa4rJTMH9lsgwSDcKDlY5sqkGLGenKWI/R1Ya/81zXhRp7y6uOIe DhX2WcUw4QqyFVfMfZCm2aJvZFnDQlY= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcc58cddb50so1750336276.0 for ; Thu, 21 Mar 2024 09:37:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039054; x=1711643854; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BO7qlg8CrJrpGAZLx2tB5/z68XAeLcOjL+jbqSm1NrE=; b=leKtAEWVFYpjMaRtNpM/O8TNQkAlb+t4qIwD/bxHFtPHF/I5dVt6dptvnW25De5XGv JCV5jubUf5tTD0MQExCvnIPQtWatL6zoDLgF3ARRtvclEFY9Mbu/kzEOcwVepjfOJuWU fRDlY8hrYXnNk5s3SJ1YAU9BAnnMusBdYlNfTOoSwnb4dFKhcwjreLSbArNPbh6admJL wLV8HojUrL/vArLkJEwQV7IWu8oa8eWu8vFPVTPo/v1pUr3MtrKmPqQjSeBBolv4tBRo cg6py5KLfgHsBP8q5uanvZpKL98IaSr4W+pHw0PCCSxcSxo/DAxZP7M84s5Mwb9XCbC2 ifBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039054; x=1711643854; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BO7qlg8CrJrpGAZLx2tB5/z68XAeLcOjL+jbqSm1NrE=; b=Rir/GHvpYJGclHnfmC/iDYeU/5xtu5OB5K3goT6aag81JhSxzkbX3M2nFB60rLB5Nx VJDDGbOoErl2P5Uk0wKS0ZAG01VyGvdvTlQRgEz/xE7+pdv9+UrSa4oqJasZxhrl20fr 37MUsUzUHdWAK6nlU8e/1fGWeokYCGt3gvJkTXCqczg6yGlP83F2JraJQg+iqoxst5GI feZe3whbiBPOPRA0zZZErRd1T7KGhWLCXMTpG95MJdnzBdR+GD9IsjiYtrHvb4S+8xvJ KdiQlP/Vl99xW5jXPZUpXzDDkrKQkHxupnTkLRWtlEF8CCcsINqWg72weWmfZyfxxICd 7JvQ== X-Forwarded-Encrypted: i=1; AJvYcCVgRJeKxErjmLAWDCIjG+iZDRaqJfRj0NuCe/viABZ4QjbinFBPNbm9vBQ4RiNp23Pa4Fnrt+Y9PckG0a4U2aL4ubM= X-Gm-Message-State: AOJu0Yz8bomlt8vt55xxzKgoj4baWKQyJtjUosJBIPhK3Lt+C4MBB5KK 8GtTOBTUA6E3d/xX+hWaimIp2e2A472+WEbmXTzgLTZevVWiIPZeE/g38LBxQ0T4vhwQF5+zJL7 hfg== X-Google-Smtp-Source: AGHT+IFS8l6862ma7ZQ5nEeHVOl9wKPYzuGCINkU9A5+oHuouEfxTwfUntJ3DBz0B7w6WbE929F2a5bqsSQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:2681:b0:dcb:bc80:8333 with SMTP id dx1-20020a056902268100b00dcbbc808333mr5470245ybb.13.1711039053606; Thu, 21 Mar 2024 09:37:33 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:33 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-12-surenb@google.com> Subject: [PATCH v6 11/37] lib: code tagging module support From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: B7C1E40020 X-Stat-Signature: 3junr3bf53hj3a9fr7oh6wbkg418qjpa X-Rspam-User: X-HE-Tag: 1711039054-530016 X-HE-Meta: U2FsdGVkX1+8PBGiXUbwgqyqBvV08w78+gm0KTn051qw0KeFkpLgKbDVr3m9SK0R/5i8JslkLaPxaYLQahNNf2vgl2sAlWAXMgLYkZGvC3aLkVnGS5N3JU/YQHW6NbY4ipIRfXCwC68zL3JpvkjLBLOgHWTFSFdyZOjoBn4NyreKkCLoIxnQGwFJIvneFNRRWCMm5grbyRaxjPu2Xdcb4k94RwXG2t0Qo89rTqWNM01gOIdNsdK0a+6C+/Devcxn8oJ52mUUeaqIogazDHYOgEV2tI7EIf4Yp2H2MhI4QBoNB2K9Kopw9O4vU7pKl6HhntFSzdDIiH5hotIPbCexKESBJ1DpOhCB9/VTCMk8xmt/7IaenM74fOLbd8LY7vtEZU+ZRK1u7HGVm6mGlTy435pFhnoYbryBHC2xJ5GgcBcOxunysntq05YfoDXcQYT+YW9PCXycPC1uvM2qLlKxte6p9lBJhKGmZIb/H2WTDW1lBh5+VdHVpT3lR8SV//DR8RZ1xRhRBQu4/5PioPRQnbkTARL5EuOulIz4murT0AzsZPXsn4a4gZ7sMNNaSLjOpbwEYWbkgis+FJw5fUom1Dla88yJvxBlc7Q3p8Q+VyAf14TJkAdpCRF5i45D2LFauv6Uy22CG9EZPcflFx7h0raP7orjqYEWXJ26C/bdSc+2CDWWVWj6kad5aCa1b+ubWMPueGxe/IjApS7huGVEnrJXHNedmrXVBDlBe/xgrCfWh6gCvkfwURr0aCGZ0JmUGfT9N8znZOgabzd4lvdJQXDXaQh1K6BUJsPOe3WuOh6T4S149D0n1BxH0Zk19zGnOEhsFzEorouBIzyTkEkArEDMmIazMAjC4kYgJAvw4EZFh37lepTQ6OaqKORQDMRR7IGbY2WRKcdQIxM5IMTqmD+H5QMF4mc6Nmb+ipi4P5keMr99h4fZ0sM4IgUHTAcBap1Yv2Chi/Gc/ny+6eD JMCGk0E0 JCj7WPXByxgkYc5FMOrI263w8vmKygPIJQzCDObUv4o2cUfPrY8qwf/0dCMfNVp3Tv0Gf2Of49X0km29qQkwVZ6RBona43wYxRqBX9z2S5RVvndC9IJbAVipd2ARednjVX485vkCmOnPrMaRVAoMLzjMvJt2oixAGn4oxpraE1goKqwTq4y5Ili3XKi7NW8IrQ+bDNSlgmpi+rTCQjUUHWXCp8z2V3gM3nWa/44HEKW3c9NWP1LzkkKrUrXmHNSkdVY50fw6qkwNNesOKmg2Y7LfLLVhbI5ULWyV92EvAB27CEe0zfIOzUnkjbLn+P4AV98zDGsD57PmoOagUr10vTzdFq9K+s8SMBmSpwm1PzMCtL/NkkHZCZ++e2hiBY3FLJJ3wldoqAdd03lWuHE8R3GEsV3EGICZ33KOWL7lYUVZiwNmvTMvAhLSbl2VD+DWTrfs0dXCKPLfyAPzz48cDtekM1dRk/MrmfJ6KUB9lUeGyuE6sTyGVLUbqkRftx9UuY7sgUUP/Z9G3EpVghNFDCO8U46vgyWf/Ye0Tu3SRGxftuNhZZYkK/QyCeDj6oWvCltkk8bMCOtzjWYrP2kdzs1wc1e80FNp3R/+aADjFk9WnLgnQJy01j2rSG5XIKugINP7cpOBQaya6/9Iv18HDjyDEA0ZCphfMhsFsaaEaDQz/E00= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add support for code tagging from dynamically loaded modules. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/codetag.h | 12 +++++++++ kernel/module/main.c | 4 +++ lib/codetag.c | 58 +++++++++++++++++++++++++++++++++++++++-- 3 files changed, 72 insertions(+), 2 deletions(-) diff --git a/include/linux/codetag.h b/include/linux/codetag.h index 7734269cdb63..c44f5b83f24d 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -33,6 +33,10 @@ union codetag_ref { struct codetag_type_desc { const char *section; size_t tag_size; + void (*module_load)(struct codetag_type *cttype, + struct codetag_module *cmod); + void (*module_unload)(struct codetag_type *cttype, + struct codetag_module *cmod); }; struct codetag_iterator { @@ -65,4 +69,12 @@ void codetag_to_text(struct seq_buf *out, struct codetag *ct); struct codetag_type * codetag_register_type(const struct codetag_type_desc *desc); +#if defined(CONFIG_CODE_TAGGING) && defined(CONFIG_MODULES) +void codetag_load_module(struct module *mod); +void codetag_unload_module(struct module *mod); +#else +static inline void codetag_load_module(struct module *mod) {} +static inline void codetag_unload_module(struct module *mod) {} +#endif + #endif /* _LINUX_CODETAG_H */ diff --git a/kernel/module/main.c b/kernel/module/main.c index e1e8a7a9d6c1..ffa6b3e9cb43 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -56,6 +56,7 @@ #include #include #include +#include #include #include #include "internal.h" @@ -1242,6 +1243,7 @@ static void free_module(struct module *mod) { trace_module_free(mod); + codetag_unload_module(mod); mod_sysfs_teardown(mod); /* @@ -2995,6 +2997,8 @@ static int load_module(struct load_info *info, const char __user *uargs, /* Get rid of temporary copy. */ free_copy(info, flags); + codetag_load_module(mod); + /* Done! */ trace_module_load(mod); diff --git a/lib/codetag.c b/lib/codetag.c index 8b5b89ad508d..54d2828eba25 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -124,15 +124,20 @@ static void *get_symbol(struct module *mod, const char *prefix, const char *name { DECLARE_SEQ_BUF(sb, KSYM_NAME_LEN); const char *buf; + void *ret; seq_buf_printf(&sb, "%s%s", prefix, name); if (seq_buf_has_overflowed(&sb)) return NULL; buf = seq_buf_str(&sb); - return mod ? + preempt_disable(); + ret = mod ? (void *)find_kallsyms_symbol_value(mod, buf) : (void *)kallsyms_lookup_name(buf); + preempt_enable(); + + return ret; } static struct codetag_range get_section_range(struct module *mod, @@ -173,8 +178,11 @@ static int codetag_module_init(struct codetag_type *cttype, struct module *mod) down_write(&cttype->mod_lock); err = idr_alloc(&cttype->mod_idr, cmod, 0, 0, GFP_KERNEL); - if (err >= 0) + if (err >= 0) { cttype->count += range_size(cttype, &range); + if (cttype->desc.module_load) + cttype->desc.module_load(cttype, cmod); + } up_write(&cttype->mod_lock); if (err < 0) { @@ -185,6 +193,52 @@ static int codetag_module_init(struct codetag_type *cttype, struct module *mod) return 0; } +void codetag_load_module(struct module *mod) +{ + struct codetag_type *cttype; + + if (!mod) + return; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) + codetag_module_init(cttype, mod); + mutex_unlock(&codetag_lock); +} + +void codetag_unload_module(struct module *mod) +{ + struct codetag_type *cttype; + + if (!mod) + return; + + mutex_lock(&codetag_lock); + list_for_each_entry(cttype, &codetag_types, link) { + struct codetag_module *found = NULL; + struct codetag_module *cmod; + unsigned long mod_id, tmp; + + down_write(&cttype->mod_lock); + idr_for_each_entry_ul(&cttype->mod_idr, cmod, tmp, mod_id) { + if (cmod->mod && cmod->mod == mod) { + found = cmod; + break; + } + } + if (found) { + if (cttype->desc.module_unload) + cttype->desc.module_unload(cttype, cmod); + + cttype->count -= range_size(cttype, &cmod->range); + idr_remove(&cttype->mod_idr, mod_id); + kfree(cmod); + } + up_write(&cttype->mod_lock); + } + mutex_unlock(&codetag_lock); +} + #else /* CONFIG_MODULES */ static int codetag_module_init(struct codetag_type *cttype, struct module *mod) { return 0; } #endif /* CONFIG_MODULES */ From patchwork Thu Mar 21 16:36:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599169 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 389BBC54E58 for ; Thu, 21 Mar 2024 16:37:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 23DD96B00A2; Thu, 21 Mar 2024 12:37:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1794F6B00A3; Thu, 21 Mar 2024 12:37:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EBF006B00A4; Thu, 21 Mar 2024 12:37:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D0F086B00A2 for ; Thu, 21 Mar 2024 12:37:38 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 914B9120BA1 for ; Thu, 21 Mar 2024 16:37:38 +0000 (UTC) X-FDA: 81921602196.08.C2943E3 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf15.hostedemail.com (Postfix) with ESMTP id DAF73A000D for ; Thu, 21 Mar 2024 16:37:36 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=SdYyRSmY; spf=pass (imf15.hostedemail.com: domain of 3T2L8ZQYKCDwqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3T2L8ZQYKCDwqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039056; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LQ90PfNulEINqNY+MaQbbDhVRZQhbHdS/nw/zX5Mh3o=; b=VEvl46TTwgoDyvTzoT9cPuJo0rbooebw9ToGRduz14g5akRbYPlk80yORu3Cjo32T4vUyg nSZ+HCJ1ZgkQbUfz9HFLaE3LEBkSzLIUBH4AaxwJZUHegKs7vplWpuHmBZmDu6D1u9INpv b/XMs87V/PAA3v5IDlfCPhHepAhFUuk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039056; a=rsa-sha256; cv=none; b=5oYMQ/JgPxeLJ6TQea19dh81RclqR/LFfjNRiDBKTOnayuiAPkfH0Kevb3wRP3/Y2pDWae o7fSGfCp2Ngr0phrbnrwER2qbb84axs7xFgdGrK1yLfeEMdZYvE2oCMGEeYiNRdF3Pwfau tD4aCz/oK6vxzuykEsTqp0btKHYKWsE= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=SdYyRSmY; spf=pass (imf15.hostedemail.com: domain of 3T2L8ZQYKCDwqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3T2L8ZQYKCDwqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc6ceade361so2069797276.0 for ; Thu, 21 Mar 2024 09:37:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039056; x=1711643856; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=LQ90PfNulEINqNY+MaQbbDhVRZQhbHdS/nw/zX5Mh3o=; b=SdYyRSmYOyZidIv/072r0L2PukE5CFm774UQuCUptF64PA22hZ5wtaeWOOphWVMj0P 8DsEpuOtN3l/OizjllEYO4nIF7WPN3FgcqywlPbzvxFoTZGblTbd+PpttqeLpBpULBDK VT03KHh8Zevhbc0kIIEyDHiTOtb6auyP5RHa3twK1QmczsjkSHPhCzyHYjsqnJeUOTq6 A/kzcSJgQ+to6V7DRAuxAaehQ8zzSayE+rcQTVZfVGMtM7xrxAOANvaadVFD/zB0xpPY OMsOtoXErMFJqrX5zGmjmyfh+ZVhIBpDiOgl/C8u8XJq/QZHbb5UAqnfwE3/m5t8deSk ySoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039056; x=1711643856; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LQ90PfNulEINqNY+MaQbbDhVRZQhbHdS/nw/zX5Mh3o=; b=mn47JodZgJspOvrSPzxc1dbdCS3y3KzOh/ncmXJqYedvNUSkE8C/BaZELnsQEJPeEK Fdm5w6OHBF0F0Jk38EC8bOwAipqBvxT0Hv4im6vJucLI17u92/PK47VyaZd4Fa+YR4/6 7YNfILXbxbNoGz8ZwjbkXBwCaRekQ8ngG4C4DnprGJ/Tm325bq+tpREhY2Q7MmO4OGi0 jJZZirXgnR0Fhh2OxVkKZPUJ5YiMzgw2NlGAgbLBHQSmtPJSM1qHxhwQeSU3yejpbsB8 6vhgJAMp0QgTag8o+EWlQMJkz69I1RDay8y/yF+STIu4GBX1kChkidcpCMUXknfBnQTX aD1g== X-Forwarded-Encrypted: i=1; AJvYcCV/49MN5RGWl4sd5Ydvh4zEDThh0ZmlxkigqBUFtMyDA6+HOjK0+34FKVRjlhTPcYNgj7TJOWxANB1PMFvzK3uIlC4= X-Gm-Message-State: AOJu0Yy5aiYOublZPGJqSBrsAHW0uIv5NgkWCsSh3RcnYZGXhsG3kVhO e3k0xyiGRU36eAl9wcKE/zRyFjupUL3Ua0JFvRGrA06LCfyFNAA0o6rSiLcp5Ifna9SWUm27V1g i+Q== X-Google-Smtp-Source: AGHT+IGn9vwdodSrX//TEYzDB0YX9rTZ3uTQLDwcOKBoMjo1PHAt30ZDw1z5T1FWhqViEm1/BwfsKtlP0DE= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:1144:b0:dcb:abcc:62be with SMTP id p4-20020a056902114400b00dcbabcc62bemr5818428ybu.6.1711039055775; Thu, 21 Mar 2024 09:37:35 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:34 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-13-surenb@google.com> Subject: [PATCH v6 12/37] lib: prevent module unloading if memory is not freed From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: bo148sewbh9zop7w9cxe8899bm6jdgzg X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: DAF73A000D X-Rspam-User: X-HE-Tag: 1711039056-814182 X-HE-Meta: U2FsdGVkX1/ijo02wDpxdtifv2xwfYjVeWD6jhWTevYcfqealkvgFawp0xHOB1GCbqHH9d3bTyWjwfz2+kvnbvTxJZPyvjvQR1Jz2TOzKG3+EahR1vqejhEP+l7+eQfNWzfpdW4AP9BWaxFCQ5h1hZIgmY9prtsPaFIq5hxp3UP1S/Q41Se9UjLvXmHYHye9OmX7owvl8JM3TSLbuVrfI+eBmGNzTj1YSs1OZv1ir0sNw1yVB6gK4ht4q7z5wXT+PywwH2zIts5Pa3MRWyGfgblkqAnG87WhJ1mCNYMZLG7OiTP6FWEIENUzLWXph7GJd29rK8IE9oLjV0SPaVY94i9znfH1KTjg5GWspcCiHavBFdw4LsRfj08Y8+WT29qaJEjCBnXOPUrQGLho1BxEk29eGTukbASEftpKHLA5Pk+cjR7RvppATA5LPXZZSgSVL0082n/+rhN9a8mq2Akb3YxU8Q96C8CDOg527gCbUD4Ve3pAIRtoTzqYUpwzzHzy/xF2xQqFYi7/+KlOzK6EopUGdkcP6033mc8t32k1GMT02eXrZUffd6v5dbHd7aLLItDrR/J0UYGL9P6QYNw36yFDPa3t2RZ8cksr+05RPYzTrxBV9xrKZqdn+QfzwEhJZVRrxPPcCO6kXscgazDBqFMhC4e5S9DQ3lidQw0ySoRxLWYZfHbyGhtS+8UsqfBSIFhD9To3PiNRxuxeS6ryiZeKmzYJXZ/zQvCKuhqU2Fqb68vKJ/KF9UKDlKv7/Y60eMw49n3ihU44ny5cUOKVZSZyR6U0fNIglQ8omeTM1j6LslR1iFHI2AaB++GBgJ7SeGTrUi191D94TuK5fMdEstvrs4iBP5U/LeenxxsECUrapnO7gVsVZBJDKfG575aq2ceJWHFVJ6eDZ74NgL3rNYCuaQb0YFLcuYpcdapqrS6uGc9JqklmRiOVkipHASuMB5/tAe8wKOhJd2PD80T kl5MopcU nqNe3ppF8UGGsJnIVlzbJPL1R51Rf2SDUP2cvywFKgLSmdyMKz5mYkgPPs6tR0/J69+dmhn0Z1oSMzFyvH7Kd6J+0qsUwQGPTaaPFd8jWa/hrVtAZebqJ0b8XDfgY6JfJfA/U3tdgecziWcbiz8aJF9uDxc9QmVgN0KLjYHShDrRewiRXtUPDy4uT3NogP+zMgLGrZdtVBAcUIUhZOHLBm1Rwc9VXT1q1mvC1RpEboPPX/hrd+QSeHIjv8cQTInCTWrczg7DFtTIjxdU20HzP/8JoCWzaOilbfft9uKgpUC2dLjDsSpgckz6GvgtNU0kzCGk/AGMctpzMt2SCTlMzhETjru71+m0Q9k3MlF451VWvREEtqwdLUopfyRF4xn8e1LnZtXrGIZQEqxbd05pdX3/PwHNKJMKHMmdY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Skip freeing module's data section if there are non-zero allocation tags because otherwise, once these allocations are freed, the access to their code tag would cause UAF. Signed-off-by: Suren Baghdasaryan --- include/linux/codetag.h | 6 +++--- kernel/module/main.c | 27 +++++++++++++++++++-------- lib/codetag.c | 11 ++++++++--- 3 files changed, 30 insertions(+), 14 deletions(-) diff --git a/include/linux/codetag.h b/include/linux/codetag.h index c44f5b83f24d..bfd0ba5c4185 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -35,7 +35,7 @@ struct codetag_type_desc { size_t tag_size; void (*module_load)(struct codetag_type *cttype, struct codetag_module *cmod); - void (*module_unload)(struct codetag_type *cttype, + bool (*module_unload)(struct codetag_type *cttype, struct codetag_module *cmod); }; @@ -71,10 +71,10 @@ codetag_register_type(const struct codetag_type_desc *desc); #if defined(CONFIG_CODE_TAGGING) && defined(CONFIG_MODULES) void codetag_load_module(struct module *mod); -void codetag_unload_module(struct module *mod); +bool codetag_unload_module(struct module *mod); #else static inline void codetag_load_module(struct module *mod) {} -static inline void codetag_unload_module(struct module *mod) {} +static inline bool codetag_unload_module(struct module *mod) { return true; } #endif #endif /* _LINUX_CODETAG_H */ diff --git a/kernel/module/main.c b/kernel/module/main.c index ffa6b3e9cb43..2d25eebc549d 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -1211,15 +1211,19 @@ static void *module_memory_alloc(unsigned int size, enum mod_mem_type type) return module_alloc(size); } -static void module_memory_free(void *ptr, enum mod_mem_type type) +static void module_memory_free(void *ptr, enum mod_mem_type type, + bool unload_codetags) { + if (!unload_codetags && mod_mem_type_is_core_data(type)) + return; + if (mod_mem_use_vmalloc(type)) vfree(ptr); else module_memfree(ptr); } -static void free_mod_mem(struct module *mod) +static void free_mod_mem(struct module *mod, bool unload_codetags) { for_each_mod_mem_type(type) { struct module_memory *mod_mem = &mod->mem[type]; @@ -1230,20 +1234,27 @@ static void free_mod_mem(struct module *mod) /* Free lock-classes; relies on the preceding sync_rcu(). */ lockdep_free_key_range(mod_mem->base, mod_mem->size); if (mod_mem->size) - module_memory_free(mod_mem->base, type); + module_memory_free(mod_mem->base, type, + unload_codetags); } /* MOD_DATA hosts mod, so free it at last */ lockdep_free_key_range(mod->mem[MOD_DATA].base, mod->mem[MOD_DATA].size); - module_memory_free(mod->mem[MOD_DATA].base, MOD_DATA); + module_memory_free(mod->mem[MOD_DATA].base, MOD_DATA, unload_codetags); } /* Free a module, remove from lists, etc. */ static void free_module(struct module *mod) { + bool unload_codetags; + trace_module_free(mod); - codetag_unload_module(mod); + unload_codetags = codetag_unload_module(mod); + if (!unload_codetags) + pr_warn("%s: memory allocation(s) from the module still alive, cannot unload cleanly\n", + mod->name); + mod_sysfs_teardown(mod); /* @@ -1285,7 +1296,7 @@ static void free_module(struct module *mod) kfree(mod->args); percpu_modfree(mod); - free_mod_mem(mod); + free_mod_mem(mod, unload_codetags); } void *__symbol_get(const char *symbol) @@ -2298,7 +2309,7 @@ static int move_module(struct module *mod, struct load_info *info) return 0; out_enomem: for (t--; t >= 0; t--) - module_memory_free(mod->mem[t].base, t); + module_memory_free(mod->mem[t].base, t, true); return ret; } @@ -2428,7 +2439,7 @@ static void module_deallocate(struct module *mod, struct load_info *info) percpu_modfree(mod); module_arch_freeing_init(mod); - free_mod_mem(mod); + free_mod_mem(mod, true); } int __weak module_finalize(const Elf_Ehdr *hdr, diff --git a/lib/codetag.c b/lib/codetag.c index 54d2828eba25..408062f722ce 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -5,6 +5,7 @@ #include #include #include +#include struct codetag_type { struct list_head link; @@ -206,12 +207,13 @@ void codetag_load_module(struct module *mod) mutex_unlock(&codetag_lock); } -void codetag_unload_module(struct module *mod) +bool codetag_unload_module(struct module *mod) { struct codetag_type *cttype; + bool unload_ok = true; if (!mod) - return; + return true; mutex_lock(&codetag_lock); list_for_each_entry(cttype, &codetag_types, link) { @@ -228,7 +230,8 @@ void codetag_unload_module(struct module *mod) } if (found) { if (cttype->desc.module_unload) - cttype->desc.module_unload(cttype, cmod); + if (!cttype->desc.module_unload(cttype, cmod)) + unload_ok = false; cttype->count -= range_size(cttype, &cmod->range); idr_remove(&cttype->mod_idr, mod_id); @@ -237,6 +240,8 @@ void codetag_unload_module(struct module *mod) up_write(&cttype->mod_lock); } mutex_unlock(&codetag_lock); + + return unload_ok; } #else /* CONFIG_MODULES */ From patchwork Thu Mar 21 16:36:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599170 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F9E9C6FD1F for ; Thu, 21 Mar 2024 16:37:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CED2F6B00A3; Thu, 21 Mar 2024 12:37:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C75F86B00A4; Thu, 21 Mar 2024 12:37:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA04F6B00A6; Thu, 21 Mar 2024 12:37:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 88EFD6B00A3 for ; Thu, 21 Mar 2024 12:37:41 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 508A5160516 for ; Thu, 21 Mar 2024 16:37:41 +0000 (UTC) X-FDA: 81921602322.06.3F26063 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf21.hostedemail.com (Postfix) with ESMTP id 22E511C001B for ; Thu, 21 Mar 2024 16:37:38 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Yap5CLwy; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of 3UmL8ZQYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3UmL8ZQYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039059; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kphxAsbZBCd3nBfcxo1fm19K6BicOHdsdiKRJJ1Q0RA=; b=I6JwcEyUEHjJaL4HvFSDkemmxMtGIFmhRc2f7LlaU7uXuLo/CGinSrcxEj8F6Jr37jXssT rgvw9QRcs6M2O8OE4ERP98h2UsmmDGpY0yM5NIyylbhmLjPHWjxeKG0NB7QTOyWtZdV6J3 DpfJM5/tMPzf541L+uqVQhfaR23GJK4= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Yap5CLwy; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of 3UmL8ZQYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3UmL8ZQYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039059; a=rsa-sha256; cv=none; b=MDeyGw2TBRn8AwdQIdw/nnuudOwurOQD5JgIqemtaxcPBj1IIH0IGOJHwgQDorHeivngwo uBCB8WkxRNL4aJ80Gjun1mWpNr7pQNqS614eX6YdjF2hM72iNZGO3+4YADHOakTnh5DhkY FDZ5jKvibhHiZM94QHteWYowZCjyG2U= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60a0a5bf550so21839567b3.3 for ; Thu, 21 Mar 2024 09:37:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039058; x=1711643858; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kphxAsbZBCd3nBfcxo1fm19K6BicOHdsdiKRJJ1Q0RA=; b=Yap5CLwyvRCiqBkpR6Keoujd2mnTYD5sHuecXaSteVLR+r0EgjdSGX+8jklUlQKndr VbVJHkgspu5rCQx9lYLaqobpkpQgt6/ZApJupZAPYhe20Ee5bZiCc64IlImW+OUUpxjp nIuze4ChaoY3A1SFhzcDpVQaOkYMKr/Fg0NTpSjHlhF+2e9RU6/ZRrw08z2iI4H8evXm ib4nc7UY9XxUsziv8T96T/muk/yt0RH07sZS2CTlBiR7rC0U2o35L1x5ChJp6QkteBnh ogjXj6zkP7hiL5IGUDfvdhmLzrzLgrSlZj6+08P5b3cXHTaDzBDm8i9qEOezhov5B43Q tLzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039058; x=1711643858; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kphxAsbZBCd3nBfcxo1fm19K6BicOHdsdiKRJJ1Q0RA=; b=TITejCvLLT69JB4yarYMnfssNDj10d0JXOJdJ4lZEXagS9aJK62dvtd5IHgcbT5Sk6 d9VQYxNfQJp9ly2W82G4z2OENbZTzbvWfMYy0WZF6yaRdASJOP5B/sXTOibDxYjlehO9 VIK63gqFGiy1UKe9VfmINdOmdavvvF4X+l0O8nQY7cwlblJqyzyH5UQCmGsFZDa/bidF 73risRCA2gfwjZpkqydl0MQRT3ctdUsGTCvolSOXQOLkygeMC8Drt9/Vah4oYboXeelW AyLdTMOBlzGmZObQwguU83Ce802opxccL62gMILfrCfN40BLLkFXzepkFBFgwvUt2+l/ gCMg== X-Forwarded-Encrypted: i=1; AJvYcCUp2qI/H7vaQsFhIJirY8xf1U0LjmO0AIR6CYQZz+6lizwRpo7b+txNZooFS8CwjTSgr73ahTm95n8MW9d9dkHl1fE= X-Gm-Message-State: AOJu0Yyx6r1eAo4kOOkxTVeExzpSIC9L2fZIwikfV2vcpDM1aAh+Furc 6250MpAaMXfZuZKXRvh9y1fTXE0UFY4Zs/7QnOqt30hfZ1C9wTwTtPnDLmW/nvlZLkEbKsalh1D ehQ== X-Google-Smtp-Source: AGHT+IG9K8rXgi/OQCC1iR8aLsSQm3nmRkS2m5A4/0GsXJ3PvPaiZRw0JqeQ/5KmjQA2BTVKONa8wniSERU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a0d:cc83:0:b0:611:19e0:dfcf with SMTP id o125-20020a0dcc83000000b0061119e0dfcfmr32561ywd.10.1711039058133; Thu, 21 Mar 2024 09:37:38 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:35 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-14-surenb@google.com> Subject: [PATCH v6 13/37] lib: add allocation tagging support for memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 22E511C001B X-Stat-Signature: h76tkogk98ty9xgtnttqs5qwp3aq3swx X-HE-Tag: 1711039058-983705 X-HE-Meta: U2FsdGVkX1+GOFJs/m0Sl01+pEHe0SY20PgNDHfJsCaE3FKmEAYjuqpknNAxJMMnSpjOYG0DkuqHuwZDYShIYRjU5OKdA1WQzdag5stFCTjmFybsiJyeBmL9FFwdUs/FxUWa4kMO3/BKRNWudYw4iMjiJtzhhLevAF7NslCMJLgS+8mjHtUCIV23ED/tjeYUUUxYxYer4p/XFWa+fO4ikaRdw0XuOgtTgF96SGLumzCK/6RnkV6DrBtA6Odbx8+1Vp7k2Jw+1lH6s6DDB9L5tuQqpxgyM16AyiccUXKkE7OpbJ1kbIFOXpDFk/roibEs/mp3Kdy40tZS1genuECP1eAiCDGUk/Oh4j5PF1XWBsJAHVjxt2mrcHJXhE8r0kc5yEoKRSMrsMa5es+9X5owoyMLYTi7Xx4XRoVwi78efSi3sALHWVZ3IxDotrx5oqNuYW605mImAo/wvOBNguou5tR8E/l+UIEu7tRG71cV+uj60FjKjrwIxNA06gSBPZOL9iaeZQHK2yLjqa9oxsE9lR8NXRdYpqA9JRSzP4fMidq6YpsAq30ojoMD24Ad/AVSO8OXyrrmONezproVYHm4tL0O9RnWevHqp7MTjAB3AMmxtXyzBIgkFPKlI7W4yjQCUf4uLnwQhNGZuBxzAb4rCoFUjpg50ET4h8Y9B8GT4bJVK+URzfFt6HaLZ69OKg6eiJULCqBqsQZ02TLkGfZXnS+8UrajUp6WYz/mCS2znmXhfy9tBxrmqnfsIlkQgkW39yXZfjEi1d0Yql7meaGJ3n0RAXvFH34/MH7mapatGmEDK8Kp/4ijyJFJbi75ZFqeL4HVzaoWxGJ+2D0ueBxrYLa4aeZuDpcdZB0cMQAf0/UHUeKAgaVZ0F7Sq0QplN0RONhJbOl7UKDKmduEgYkN6p2Sk88SO5/6LKW2UcN79sXQpPBY534gj1tEXNzUbMUq8At2NDAlpsYDLxKdlUI krC37wQv oHqeGa+gWLedRpVU5KuBGW3Ywbi0o9sWGpOYnxbnmmG5+hOkicw4o5krcII6REbezkbF2R2B6MTj1oddKLdeFqTFI7V2PFm+i8+88pYeKpwUsqPvsmDli3oUa/xKBkxuBMxUsC4w/DoCXAwnbLmXCS1mZmQRxbFg0aLJDzOZKIUC4ZBdDBdFxm7W7Odszsw99S8trpjBiRh9cmRWu+73XUCS8SrCuVQ4RdZHy/npBSrWTDLYZtax1V9Mwz8i79O/52gVuHOoBZEh8eDA8HXiJtT9zjwbtV6NbKeccRNV193r9hFEaw516CaQ0Snj8ZSrbYsfUQmX/kpeUXyoQARNVVKWwWVRjwzNHLplm3LyIGc+L5YS82R9dV36+JsFtAigA6AZBD+2JMS7VGJR+gOyqfmgMD7T7eQAFffOOzr9pnDQN39mMYJQ3ZRArK//D44LwFHZwsOf30nIehcsa41Z2Mrpw+M9d8Sjff95F5+WQuJWgClg69fn7ZgqxUqmVFotcSq3OppRyMFDSvXntKEmyCFHHTIVUwDc+Vx3J/iBJub7pCxlEVlMbNKt88yHqo0z+AuSIrN8saIw7kJ/gkO68H57QKxA8crUYqhiO7wAWbkvzh7il9QDBcVZa+zz9SrHZQzJeISNsW0Qyxc3E2xI1wvmYGbIijzgky+9tb9sjTedwD2RkFp8tTn1/zw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce CONFIG_MEM_ALLOC_PROFILING which provides definitions to easily instrument memory allocators. It registers an "alloc_tags" codetag type with /proc/allocinfo interface to output allocation tag information when the feature is enabled. CONFIG_MEM_ALLOC_PROFILING_DEBUG is provided for debugging the memory allocation profiling instrumentation. Memory allocation profiling can be enabled or disabled at runtime using /proc/sys/vm/mem_profiling sysctl when CONFIG_MEM_ALLOC_PROFILING_DEBUG=n. CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT enables memory allocation profiling by default. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- Documentation/admin-guide/sysctl/vm.rst | 16 +++ Documentation/filesystems/proc.rst | 29 +++++ include/asm-generic/codetag.lds.h | 14 +++ include/asm-generic/vmlinux.lds.h | 3 + include/linux/alloc_tag.h | 145 +++++++++++++++++++++++ include/linux/sched.h | 24 ++++ lib/Kconfig.debug | 25 ++++ lib/Makefile | 2 + lib/alloc_tag.c | 149 ++++++++++++++++++++++++ scripts/module.lds.S | 7 ++ 10 files changed, 414 insertions(+) create mode 100644 include/asm-generic/codetag.lds.h create mode 100644 include/linux/alloc_tag.h create mode 100644 lib/alloc_tag.c diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst index c59889de122b..e86c968a7a0e 100644 --- a/Documentation/admin-guide/sysctl/vm.rst +++ b/Documentation/admin-guide/sysctl/vm.rst @@ -43,6 +43,7 @@ Currently, these files are in /proc/sys/vm: - legacy_va_layout - lowmem_reserve_ratio - max_map_count +- mem_profiling (only if CONFIG_MEM_ALLOC_PROFILING=y) - memory_failure_early_kill - memory_failure_recovery - min_free_kbytes @@ -425,6 +426,21 @@ e.g., up to one or two maps per allocation. The default value is 65530. +mem_profiling +============== + +Enable memory profiling (when CONFIG_MEM_ALLOC_PROFILING=y) + +1: Enable memory profiling. + +0: Disable memory profiling. + +Enabling memory profiling introduces a small performance overhead for all +memory allocations. + +The default value depends on CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT. + + memory_failure_early_kill: ========================== diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst index c6a6b9df2104..5d2fc58b5b1f 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -688,6 +688,7 @@ files are there, and which are missing. ============ =============================================================== File Content ============ =============================================================== + allocinfo Memory allocations profiling information apm Advanced power management info bootconfig Kernel command line obtained from boot config, and, if there were kernel parameters from the @@ -953,6 +954,34 @@ also be allocatable although a lot of filesystem metadata may have to be reclaimed to achieve this. +allocinfo +~~~~~~~ + +Provides information about memory allocations at all locations in the code +base. Each allocation in the code is identified by its source file, line +number, module (if originates from a loadable module) and the function calling +the allocation. The number of bytes allocated and number of calls at each +location are reported. + +Example output. + +:: + + > sort -rn /proc/allocinfo + 127664128 31168 mm/page_ext.c:270 func:alloc_page_ext + 56373248 4737 mm/slub.c:2259 func:alloc_slab_page + 14880768 3633 mm/readahead.c:247 func:page_cache_ra_unbounded + 14417920 3520 mm/mm_init.c:2530 func:alloc_large_system_hash + 13377536 234 block/blk-mq.c:3421 func:blk_mq_alloc_rqs + 11718656 2861 mm/filemap.c:1919 func:__filemap_get_folio + 9192960 2800 kernel/fork.c:307 func:alloc_thread_stack_node + 4206592 4 net/netfilter/nf_conntrack_core.c:2567 func:nf_ct_alloc_hashtable + 4136960 1010 drivers/staging/ctagmod/ctagmod.c:20 [ctagmod] func:ctagmod_start + 3940352 962 mm/memory.c:4214 func:alloc_anon_folio + 2894464 22613 fs/kernfs/dir.c:615 func:__kernfs_new_node + ... + + meminfo ~~~~~~~ diff --git a/include/asm-generic/codetag.lds.h b/include/asm-generic/codetag.lds.h new file mode 100644 index 000000000000..64f536b80380 --- /dev/null +++ b/include/asm-generic/codetag.lds.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __ASM_GENERIC_CODETAG_LDS_H +#define __ASM_GENERIC_CODETAG_LDS_H + +#define SECTION_WITH_BOUNDARIES(_name) \ + . = ALIGN(8); \ + __start_##_name = .; \ + KEEP(*(_name)) \ + __stop_##_name = .; + +#define CODETAG_SECTIONS() \ + SECTION_WITH_BOUNDARIES(alloc_tags) + +#endif /* __ASM_GENERIC_CODETAG_LDS_H */ diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index f7749d0f2562..3e4497b5135a 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -50,6 +50,8 @@ * [__nosave_begin, __nosave_end] for the nosave data */ +#include + #ifndef LOAD_OFFSET #define LOAD_OFFSET 0 #endif @@ -366,6 +368,7 @@ . = ALIGN(8); \ BOUNDED_SECTION_BY(__dyndbg_classes, ___dyndbg_classes) \ BOUNDED_SECTION_BY(__dyndbg, ___dyndbg) \ + CODETAG_SECTIONS() \ LIKELY_PROFILE() \ BRANCH_PROFILE() \ TRACE_PRINTKS() \ diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h new file mode 100644 index 000000000000..b970ff1c80dc --- /dev/null +++ b/include/linux/alloc_tag.h @@ -0,0 +1,145 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * allocation tagging + */ +#ifndef _LINUX_ALLOC_TAG_H +#define _LINUX_ALLOC_TAG_H + +#include +#include +#include +#include +#include +#include +#include + +struct alloc_tag_counters { + u64 bytes; + u64 calls; +}; + +/* + * An instance of this structure is created in a special ELF section at every + * allocation callsite. At runtime, the special section is treated as + * an array of these. Embedded codetag utilizes codetag framework. + */ +struct alloc_tag { + struct codetag ct; + struct alloc_tag_counters __percpu *counters; +} __aligned(8); + +#ifdef CONFIG_MEM_ALLOC_PROFILING + +static inline struct alloc_tag *ct_to_alloc_tag(struct codetag *ct) +{ + return container_of(ct, struct alloc_tag, ct); +} + +#ifdef ARCH_NEEDS_WEAK_PER_CPU +/* + * When percpu variables are required to be defined as weak, static percpu + * variables can't be used inside a function (see comments for DECLARE_PER_CPU_SECTION). + */ +#error "Memory allocation profiling is incompatible with ARCH_NEEDS_WEAK_PER_CPU" +#endif + +#define DEFINE_ALLOC_TAG(_alloc_tag) \ + static DEFINE_PER_CPU(struct alloc_tag_counters, _alloc_tag_cntr); \ + static struct alloc_tag _alloc_tag __used __aligned(8) \ + __section("alloc_tags") = { \ + .ct = CODE_TAG_INIT, \ + .counters = &_alloc_tag_cntr }; + +DECLARE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + mem_alloc_profiling_key); + +static inline bool mem_alloc_profiling_enabled(void) +{ + return static_branch_maybe(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + &mem_alloc_profiling_key); +} + +static inline struct alloc_tag_counters alloc_tag_read(struct alloc_tag *tag) +{ + struct alloc_tag_counters v = { 0, 0 }; + struct alloc_tag_counters *counter; + int cpu; + + for_each_possible_cpu(cpu) { + counter = per_cpu_ptr(tag->counters, cpu); + v.bytes += counter->bytes; + v.calls += counter->calls; + } + + return v; +} + +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG +static inline void alloc_tag_add_check(union codetag_ref *ref, struct alloc_tag *tag) +{ + WARN_ONCE(ref && ref->ct, + "alloc_tag was not cleared (got tag for %s:%u)\n", + ref->ct->filename, ref->ct->lineno); + + WARN_ONCE(!tag, "current->alloc_tag not set"); +} + +static inline void alloc_tag_sub_check(union codetag_ref *ref) +{ + WARN_ONCE(ref && !ref->ct, "alloc_tag was not set\n"); +} +#else +static inline void alloc_tag_add_check(union codetag_ref *ref, struct alloc_tag *tag) {} +static inline void alloc_tag_sub_check(union codetag_ref *ref) {} +#endif + +/* Caller should verify both ref and tag to be valid */ +static inline void __alloc_tag_ref_set(union codetag_ref *ref, struct alloc_tag *tag) +{ + ref->ct = &tag->ct; + /* + * We need in increment the call counter every time we have a new + * allocation or when we split a large allocation into smaller ones. + * Each new reference for every sub-allocation needs to increment call + * counter because when we free each part the counter will be decremented. + */ + this_cpu_inc(tag->counters->calls); +} + +static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, size_t bytes) +{ + alloc_tag_add_check(ref, tag); + if (!ref || !tag) + return; + + __alloc_tag_ref_set(ref, tag); + this_cpu_add(tag->counters->bytes, bytes); +} + +static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) +{ + struct alloc_tag *tag; + + alloc_tag_sub_check(ref); + if (!ref || !ref->ct) + return; + + tag = ct_to_alloc_tag(ref->ct); + + this_cpu_sub(tag->counters->bytes, bytes); + this_cpu_dec(tag->counters->calls); + + ref->ct = NULL; +} + +#else /* CONFIG_MEM_ALLOC_PROFILING */ + +#define DEFINE_ALLOC_TAG(_alloc_tag) +static inline bool mem_alloc_profiling_enabled(void) { return false; } +static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, + size_t bytes) {} +static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING */ + +#endif /* _LINUX_ALLOC_TAG_H */ diff --git a/include/linux/sched.h b/include/linux/sched.h index 3c2abbc587b4..4118b3f959c3 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -770,6 +770,10 @@ struct task_struct { unsigned int flags; unsigned int ptrace; +#ifdef CONFIG_MEM_ALLOC_PROFILING + struct alloc_tag *alloc_tag; +#endif + #ifdef CONFIG_SMP int on_cpu; struct __call_single_node wake_entry; @@ -810,6 +814,7 @@ struct task_struct { struct task_group *sched_task_group; #endif + #ifdef CONFIG_UCLAMP_TASK /* * Clamp values requested for a scheduling entity. @@ -2187,4 +2192,23 @@ static inline int sched_core_idle_cpu(int cpu) { return idle_cpu(cpu); } extern void sched_set_stop_task(int cpu, struct task_struct *stop); +#ifdef CONFIG_MEM_ALLOC_PROFILING +static inline struct alloc_tag *alloc_tag_save(struct alloc_tag *tag) +{ + swap(current->alloc_tag, tag); + return tag; +} + +static inline void alloc_tag_restore(struct alloc_tag *tag, struct alloc_tag *old) +{ +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + WARN(current->alloc_tag != tag, "current->alloc_tag was changed:\n"); +#endif + current->alloc_tag = old; +} +#else +#define alloc_tag_save(_tag) NULL +#define alloc_tag_restore(_tag, _old) do {} while (0) +#endif + #endif diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index d2dbdd45fd9a..d9a6477afdb1 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -972,6 +972,31 @@ config CODE_TAGGING bool select KALLSYMS +config MEM_ALLOC_PROFILING + bool "Enable memory allocation profiling" + default n + depends on PROC_FS + depends on !DEBUG_FORCE_WEAK_PER_CPU + select CODE_TAGGING + help + Track allocation source code and record total allocation size + initiated at that code location. The mechanism can be used to track + memory leaks with a low performance and memory impact. + +config MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT + bool "Enable memory allocation profiling by default" + default y + depends on MEM_ALLOC_PROFILING + +config MEM_ALLOC_PROFILING_DEBUG + bool "Memory allocation profiler debugging" + default n + depends on MEM_ALLOC_PROFILING + select MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT + help + Adds warnings with helpful error messages for memory allocation + profiling. + source "lib/Kconfig.kasan" source "lib/Kconfig.kfence" source "lib/Kconfig.kmsan" diff --git a/lib/Makefile b/lib/Makefile index 910335da8f13..2f4e17bfb299 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -234,6 +234,8 @@ obj-$(CONFIG_OF_RECONFIG_NOTIFIER_ERROR_INJECT) += \ obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o obj-$(CONFIG_CODE_TAGGING) += codetag.o +obj-$(CONFIG_MEM_ALLOC_PROFILING) += alloc_tag.o + lib-$(CONFIG_GENERIC_BUG) += bug.o obj-$(CONFIG_HAVE_ARCH_TRACEHOOK) += syscall.o diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c new file mode 100644 index 000000000000..f09c8a422bc2 --- /dev/null +++ b/lib/alloc_tag.c @@ -0,0 +1,149 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include +#include +#include +#include +#include + +static struct codetag_type *alloc_tag_cttype; + +DEFINE_STATIC_KEY_MAYBE(CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT, + mem_alloc_profiling_key); + +static void *allocinfo_start(struct seq_file *m, loff_t *pos) +{ + struct codetag_iterator *iter; + struct codetag *ct; + loff_t node = *pos; + + iter = kzalloc(sizeof(*iter), GFP_KERNEL); + m->private = iter; + if (!iter) + return NULL; + + codetag_lock_module_list(alloc_tag_cttype, true); + *iter = codetag_get_ct_iter(alloc_tag_cttype); + while ((ct = codetag_next_ct(iter)) != NULL && node) + node--; + + return ct ? iter : NULL; +} + +static void *allocinfo_next(struct seq_file *m, void *arg, loff_t *pos) +{ + struct codetag_iterator *iter = (struct codetag_iterator *)arg; + struct codetag *ct = codetag_next_ct(iter); + + (*pos)++; + if (!ct) + return NULL; + + return iter; +} + +static void allocinfo_stop(struct seq_file *m, void *arg) +{ + struct codetag_iterator *iter = (struct codetag_iterator *)m->private; + + if (iter) { + codetag_lock_module_list(alloc_tag_cttype, false); + kfree(iter); + } +} + +static void alloc_tag_to_text(struct seq_buf *out, struct codetag *ct) +{ + struct alloc_tag *tag = ct_to_alloc_tag(ct); + struct alloc_tag_counters counter = alloc_tag_read(tag); + s64 bytes = counter.bytes; + + seq_buf_printf(out, "%12lli %8llu ", bytes, counter.calls); + codetag_to_text(out, ct); + seq_buf_putc(out, ' '); + seq_buf_putc(out, '\n'); +} + +static int allocinfo_show(struct seq_file *m, void *arg) +{ + struct codetag_iterator *iter = (struct codetag_iterator *)arg; + char *bufp; + size_t n = seq_get_buf(m, &bufp); + struct seq_buf buf; + + seq_buf_init(&buf, bufp, n); + alloc_tag_to_text(&buf, iter->ct); + seq_commit(m, seq_buf_used(&buf)); + return 0; +} + +static const struct seq_operations allocinfo_seq_op = { + .start = allocinfo_start, + .next = allocinfo_next, + .stop = allocinfo_stop, + .show = allocinfo_show, +}; + +static void __init procfs_init(void) +{ + proc_create_seq("allocinfo", 0444, NULL, &allocinfo_seq_op); +} + +static bool alloc_tag_module_unload(struct codetag_type *cttype, + struct codetag_module *cmod) +{ + struct codetag_iterator iter = codetag_get_ct_iter(cttype); + struct alloc_tag_counters counter; + bool module_unused = true; + struct alloc_tag *tag; + struct codetag *ct; + + for (ct = codetag_next_ct(&iter); ct; ct = codetag_next_ct(&iter)) { + if (iter.cmod != cmod) + continue; + + tag = ct_to_alloc_tag(ct); + counter = alloc_tag_read(tag); + + if (WARN(counter.bytes, + "%s:%u module %s func:%s has %llu allocated at module unload", + ct->filename, ct->lineno, ct->modname, ct->function, counter.bytes)) + module_unused = false; + } + + return module_unused; +} + +static struct ctl_table memory_allocation_profiling_sysctls[] = { + { + .procname = "mem_profiling", + .data = &mem_alloc_profiling_key, +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + .mode = 0444, +#else + .mode = 0644, +#endif + .proc_handler = proc_do_static_key, + }, + { } +}; + +static int __init alloc_tag_init(void) +{ + const struct codetag_type_desc desc = { + .section = "alloc_tags", + .tag_size = sizeof(struct alloc_tag), + .module_unload = alloc_tag_module_unload, + }; + + alloc_tag_cttype = codetag_register_type(&desc); + if (IS_ERR_OR_NULL(alloc_tag_cttype)) + return PTR_ERR(alloc_tag_cttype); + + register_sysctl_init("vm", memory_allocation_profiling_sysctls); + procfs_init(); + + return 0; +} +module_init(alloc_tag_init); diff --git a/scripts/module.lds.S b/scripts/module.lds.S index bf5bcf2836d8..45c67a0994f3 100644 --- a/scripts/module.lds.S +++ b/scripts/module.lds.S @@ -9,6 +9,8 @@ #define DISCARD_EH_FRAME *(.eh_frame) #endif +#include + SECTIONS { /DISCARD/ : { *(.discard) @@ -47,12 +49,17 @@ SECTIONS { .data : { *(.data .data.[0-9a-zA-Z_]*) *(.data..L*) + CODETAG_SECTIONS() } .rodata : { *(.rodata .rodata.[0-9a-zA-Z_]*) *(.rodata..L*) } +#else + .data : { + CODETAG_SECTIONS() + } #endif } From patchwork Thu Mar 21 16:36:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599171 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8915EC54E58 for ; Thu, 21 Mar 2024 16:37:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2649B6B00A6; Thu, 21 Mar 2024 12:37:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F0076B00A7; Thu, 21 Mar 2024 12:37:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0197C6B00A8; Thu, 21 Mar 2024 12:37:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D68936B00A6 for ; Thu, 21 Mar 2024 12:37:43 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A7D8A1601DA for ; Thu, 21 Mar 2024 16:37:43 +0000 (UTC) X-FDA: 81921602406.10.83C22CF Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf24.hostedemail.com (Postfix) with ESMTP id AEE1A18002D for ; Thu, 21 Mar 2024 16:37:41 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=IIJiKUky; spf=pass (imf24.hostedemail.com: domain of 3VGL8ZQYKCEEvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3VGL8ZQYKCEEvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039061; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VEuENejKrKvwi5jd3fEQOIFk9pMWLkQIYT8gJCjoD3E=; b=UR+lWZR0a6QywjMrM/qp9FMX+y7QMKGuhOXTlTqs/6HH151FnSbHeffqhhsJDJyImuOJVI Pk73gUZLn0S6ijMYoDL5Y2kQruaqPiy5hGjrexq5lXeDvtYoH88bDjT0k0Z0HY8K1HaKYe aCeRsRWktk61izV62IMkjBYVJmY8rq4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039061; a=rsa-sha256; cv=none; b=z5t30tu6aZ+MM2gshrUAA+MpOTk2K3Mx3u6Z17lI4NvEvhBomrmNmuUtdpk9+IE2YlV8aa ObV+N4zX3y//sk4tKQgk4THfF3u38blF2oF1Gic8PMPjRGyvX++gKplUZ2hMwATwR+KjPf Qzd2Flp5HIE1cacvi8F9/TmB0HqIwVk= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=IIJiKUky; spf=pass (imf24.hostedemail.com: domain of 3VGL8ZQYKCEEvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3VGL8ZQYKCEEvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc6ceade361so2069962276.0 for ; Thu, 21 Mar 2024 09:37:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039061; x=1711643861; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VEuENejKrKvwi5jd3fEQOIFk9pMWLkQIYT8gJCjoD3E=; b=IIJiKUkyktaBvN/XI+YvF583THPOZj/FCszOCDXnYI6l6POIlA3NM9I74EzYk7PS7Z NJ6FXcBMl7jUgl05Gm6T2AeIkAPEIp/zbyVttWZLGxdyKQVkld0o1IE/rcfZhjUy7HUT LxKd0guJteRnVFp3abZ6Dd9zdnq31MSNhndz33jVg0G3BIilhhVM6BWDmLrYM772nNRO EE5qVukId8AZR3Gvg9T4F3SsAuBjW73YbBROTOHqJ4t42RTnq0MVnpZuSIVrsPd6CqlA 2ZBRESqE7BznKI2GjqyrB14P+36YtIN4W7fw5KEPN0U6IcA9KSo0jvNiK8zi07x5fJPu stCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039061; x=1711643861; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VEuENejKrKvwi5jd3fEQOIFk9pMWLkQIYT8gJCjoD3E=; b=uh4RS/15PDgtjjf0sLAMMnUf0b61tbUfMuJ7wSx/Yb3WTcqFAbpOObBAwTzCdKIOIN Jh8Csm84AYmT385swN9WL884i52HljH4px7Ylcg0oB//QpPxe+SQ/5p8UUSlK5IwS9JR W6V0lLxOoaxS+vaP+nGt9022Tz48DsYxIvc733fXplgcqEOFaGZSHHbv8NiBkfdGusAQ ce32RU4iDDbQ3FF4NxMlmXTE/n3qCyYt54uzj6TpAtCbyeunWtOeN+KlxTG7pBAuuzcN meGeWAgsYmmEVvZyqG7qhDS/0ptN3fOMkHzEFchm5aBGw9pDYKho78L9QSThQXmtmFP5 i8pg== X-Forwarded-Encrypted: i=1; AJvYcCVBctOwCf1S0c40byCUmCkX7BbrixottiIItuc4h31Brpz0tulXecFh55Y1veCliQqLB8kgPbTCxD/SMwFKWooEjic= X-Gm-Message-State: AOJu0YytfwP4RQyrSi+3jiFV+ZNrqep+bNDDm6GAFSUiTnA7AllSthfA pzgb6Y71TNgqoJBsyEwm1XGG53MwULxFJ1wBRqupsvhV7zp3lc4PDPJBgSGqmHQEW1qZpZnca/o jdQ== X-Google-Smtp-Source: AGHT+IFNQMTWyzDytJBsiwO6OlHwO/XgSIlxcnK6gjsE8Q4BevLiMexdP0sp4DeK7t/kONH0yJROFT8KJZw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:1507:b0:dcd:ad52:6932 with SMTP id q7-20020a056902150700b00dcdad526932mr5791743ybu.5.1711039060482; Thu, 21 Mar 2024 09:37:40 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:36 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-15-surenb@google.com> Subject: [PATCH v6 14/37] lib: introduce support for page allocation tagging From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: AEE1A18002D X-Rspam-User: X-Stat-Signature: h5eodtujs3d3n9y9rstx89e9389744kp X-Rspamd-Server: rspam03 X-HE-Tag: 1711039061-678563 X-HE-Meta: U2FsdGVkX1+rcqvzpQz1tPJQnW7aLEJU0Ck8hTp4qrm8RJBLvqVVGEmqRoFs9995tBMBUAFF3vFdEhnDj+5HEZUwhs4IhEbHQ2LHYLnMnzCRGgoQDj/6DMyoDMkibWof6j8gLhWBydMR+/lYJJVxKkrCGYoGxNXtBdpDm5pSWbxq7KvdC9JUdnRguxOo1j8qZ+yb8xEg/J6twXRLUCkuDJv9rl/EuplnuhlE2DFDmbkzfT9JjMNH9xsFyZq5SXd/LnkWAUfGmJzjjajucyexnWTj6GBwQDzzeLrTpkHHSgRf6cUNsm7m8n2R5/wW7vnIR2038GSd0vIwkPqTMhYnPne8E5+rAdrp7AHceG2WynDM5dF4nAtexlgTzmJsEwPd2m7XLAse9aJUhuNaJ48KGIBbUqMetCxROU7da40BXZuTdY23MBQDAD73ZvhkBkyWWMWUzXV6HpGtgaSH5T5bBZKPTBdFoiH5R0tBGwPzmFgcSlWAl6QqdbyfsUfqa7NuokUr0ORKj18V/+YjjnrQjyt7GPJU3XHrpqveAZnE3xrdsWRZHyBfRmQWKd1pI6UcOcz9E8HsG7X5MOVeRHSziunvwa+vsNU3QrFxcr12S5XnhesTaZLI8GphURcLdzWUkRYYtkD5ZcZ1vjS/Hr17FNOyiOvyzlWPVchVwx5Vo193DEVE6/mWQymrZ9UI57rH1FZcNkDskkTUC/i4dnLHDKjBlq2XMFk+yfr4baROouoQB4t0JcbFSKyvICQ/TiNH3D+QEfLt16ZeBPg+3kFKISRuSr/evU76iQbdQOWjfIGkX/VRG38W54Pi7xf8iqx091LwXWEKw0v439v4yIy4kde9Xha4lQuZj7s3GKEaZgqiCv1OWIchGzAeccNqsHjdZFEdfxtuhF+Selm3M5v9thCKWtm66rPVfl58UPK6wSH524B3vn8jvFoa4fS94Sr77oWBci5xxCnPUNSCA2J +omPUmej s8HAQd2OJGubLDW6QGR7LVfM2H12NZkopz6YP+2TaaS6U9k07Iqx/Q+IqsounnqwZOpKKc51V8SnrGpi5W4DT5/s7DT5LoZPHtIZK/gG8U4MbHUkzdLXg3ZupPd6TddyymnvjMLYc+TzuVoDg2bABMtfUG0rsCRM3Vvc+FINe6OzAtU/tgWsjzOyaWXPQ5Hx1mwtxLRCOnD8xwy/VMOlgByMpx9IfJ0icEn35xCsrTSjJHZUgBlw8sUO6Bt6fqGpyezUPsuFTNn6alPVRt4pYvS5V+Nui33Wx1XAusXFloc9meZzKqPJ1QBML90Rz7iKZNY5B4dvsjIQTnukRb9Ljxd4FVu8wXEicoBwE+jLvy4j3cpPhLttVDLF/7y1KSIP7L/dRq70mHt9vF00vJARSbS7oObVPqCtlU0VZ1rRgzEZvb+1SJ7RJyV142Q4bHblpTnohIetC9GFQrU77fT7n2YhBv8jAKrVf+fLoU+i/qOJ6gmWW04uxsfAY4cJG8ielkfZNDphUOoBwwjMa2/rS10S0Qz8SO0rjp8o+BgjEd9dSn+aNM7YNss/koZDghaAuqsJWDzmgB3YnIzgz4gjnPxhBi4vAgfJ/NttaYIdFwSKdwR0dVqVEbLddsgH/5FXQdHeA2CsaaJwCPw+kA0ryDMWIRQbUreOJjjTwg1QvIJjhPA7kXSBpdF/Wew== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce helper functions to easily instrument page allocators by storing a pointer to the allocation tag associated with the code that allocated the page in a page_ext field. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Reviewed-by: Vlastimil Babka --- include/linux/page_ext.h | 1 - include/linux/pgalloc_tag.h | 78 +++++++++++++++++++++++++++++++++++++ lib/Kconfig.debug | 1 + lib/alloc_tag.c | 17 ++++++++ mm/mm_init.c | 1 + mm/page_alloc.c | 4 ++ mm/page_ext.c | 4 ++ 7 files changed, 105 insertions(+), 1 deletion(-) create mode 100644 include/linux/pgalloc_tag.h diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h index be98564191e6..07e0656898f9 100644 --- a/include/linux/page_ext.h +++ b/include/linux/page_ext.h @@ -4,7 +4,6 @@ #include #include -#include struct pglist_data; diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h new file mode 100644 index 000000000000..66bd021eb46e --- /dev/null +++ b/include/linux/pgalloc_tag.h @@ -0,0 +1,78 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * page allocation tagging + */ +#ifndef _LINUX_PGALLOC_TAG_H +#define _LINUX_PGALLOC_TAG_H + +#include + +#ifdef CONFIG_MEM_ALLOC_PROFILING + +#include + +extern struct page_ext_operations page_alloc_tagging_ops; +extern struct page_ext *page_ext_get(struct page *page); +extern void page_ext_put(struct page_ext *page_ext); + +static inline union codetag_ref *codetag_ref_from_page_ext(struct page_ext *page_ext) +{ + return (void *)page_ext + page_alloc_tagging_ops.offset; +} + +static inline struct page_ext *page_ext_from_codetag_ref(union codetag_ref *ref) +{ + return (void *)ref - page_alloc_tagging_ops.offset; +} + +/* Should be called only if mem_alloc_profiling_enabled() */ +static inline union codetag_ref *get_page_tag_ref(struct page *page) +{ + if (page) { + struct page_ext *page_ext = page_ext_get(page); + + if (page_ext) + return codetag_ref_from_page_ext(page_ext); + } + return NULL; +} + +static inline void put_page_tag_ref(union codetag_ref *ref) +{ + page_ext_put(page_ext_from_codetag_ref(ref)); +} + +static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, + unsigned int nr) +{ + if (mem_alloc_profiling_enabled()) { + union codetag_ref *ref = get_page_tag_ref(page); + + if (ref) { + alloc_tag_add(ref, task->alloc_tag, PAGE_SIZE * nr); + put_page_tag_ref(ref); + } + } +} + +static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) +{ + if (mem_alloc_profiling_enabled()) { + union codetag_ref *ref = get_page_tag_ref(page); + + if (ref) { + alloc_tag_sub(ref, PAGE_SIZE * nr); + put_page_tag_ref(ref); + } + } +} + +#else /* CONFIG_MEM_ALLOC_PROFILING */ + +static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, + unsigned int nr) {} +static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING */ + +#endif /* _LINUX_PGALLOC_TAG_H */ diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index d9a6477afdb1..ca2c466056d5 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -978,6 +978,7 @@ config MEM_ALLOC_PROFILING depends on PROC_FS depends on !DEBUG_FORCE_WEAK_PER_CPU select CODE_TAGGING + select PAGE_EXTENSION help Track allocation source code and record total allocation size initiated at that code location. The mechanism can be used to track diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index f09c8a422bc2..cb5adec4b2e2 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -115,6 +116,22 @@ static bool alloc_tag_module_unload(struct codetag_type *cttype, return module_unused; } +static __init bool need_page_alloc_tagging(void) +{ + return true; +} + +static __init void init_page_alloc_tagging(void) +{ +} + +struct page_ext_operations page_alloc_tagging_ops = { + .size = sizeof(union codetag_ref), + .need = need_page_alloc_tagging, + .init = init_page_alloc_tagging, +}; +EXPORT_SYMBOL(page_alloc_tagging_ops); + static struct ctl_table memory_allocation_profiling_sysctls[] = { { .procname = "mem_profiling", diff --git a/mm/mm_init.c b/mm/mm_init.c index 370a057dae97..3e48afcd0faa 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 4491d0240bc6..48cdd25261ea 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -54,6 +54,7 @@ #include #include #include +#include #include #include "internal.h" #include "shuffle.h" @@ -1101,6 +1102,7 @@ __always_inline bool free_pages_prepare(struct page *page, /* Do not let hwpoison pages hit pcplists/buddy */ reset_page_owner(page, order); page_table_check_free(page, order); + pgalloc_tag_sub(page, 1 << order); return false; } @@ -1140,6 +1142,7 @@ __always_inline bool free_pages_prepare(struct page *page, page->flags &= ~PAGE_FLAGS_CHECK_AT_PREP; reset_page_owner(page, order); page_table_check_free(page, order); + pgalloc_tag_sub(page, 1 << order); if (!PageHighMem(page)) { debug_check_no_locks_freed(page_address(page), @@ -1533,6 +1536,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order, set_page_owner(page, order, gfp_flags); page_table_check_alloc(page, order); + pgalloc_tag_add(page, current, 1 << order); } static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags, diff --git a/mm/page_ext.c b/mm/page_ext.c index 4548fcc66d74..3c58fe8a24df 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -10,6 +10,7 @@ #include #include #include +#include /* * struct page extension @@ -82,6 +83,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = { #if defined(CONFIG_PAGE_IDLE_FLAG) && !defined(CONFIG_64BIT) &page_idle_ops, #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + &page_alloc_tagging_ops, +#endif #ifdef CONFIG_PAGE_TABLE_CHECK &page_table_check_ops, #endif From patchwork Thu Mar 21 16:36:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599172 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD99AC54E58 for ; Thu, 21 Mar 2024 16:37:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2673D6B00A7; Thu, 21 Mar 2024 12:37:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1ED3B6B00A9; Thu, 21 Mar 2024 12:37:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F35456B00AA; Thu, 21 Mar 2024 12:37:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D59C36B00A7 for ; Thu, 21 Mar 2024 12:37:45 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B1A2D1C0466 for ; Thu, 21 Mar 2024 16:37:45 +0000 (UTC) X-FDA: 81921602490.13.663AC14 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf22.hostedemail.com (Postfix) with ESMTP id DBEF7C0006 for ; Thu, 21 Mar 2024 16:37:43 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=r1hRGjvb; spf=pass (imf22.hostedemail.com: domain of 3VmL8ZQYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3VmL8ZQYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039063; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BfPuXrQNw6sP43MOvy7juBQZEF5XCZI3rVOwMSUBcyU=; b=JfAFl3PQ+kSR9rAERiTRl3Xp/luy6JkmxcVdpmP7NIYlwjg/LcOVGbPEzuncyLvIu/cpCq 0flyqowmOPwxGMPRFlhW5OLsEbpHdu6gGpL4xUa9Gm0DdNzTTBGUw67ZSi6aPLkrcZMrFf tpPDljV6TQAkQlDe042xOdxaHxtxCk4= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=r1hRGjvb; spf=pass (imf22.hostedemail.com: domain of 3VmL8ZQYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3VmL8ZQYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039063; a=rsa-sha256; cv=none; b=yN35pch657ldKZioV4Ywz1gJ1ARPfCQ+9VuzS+vM1FFf3WpAK5MD15DrGPzhQU+3NKGk63 Y6jduU3gd3ZzS7qNyiwlwCDe7iDLSlE7+02YTxYqTKBVpILWzmfvXxA/o9kxS5Z1p2EfJ7 bNvCwQjlmFGMuK4mL/pivoTtu7qw458= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60a0a5bf550so21840857b3.3 for ; Thu, 21 Mar 2024 09:37:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039063; x=1711643863; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BfPuXrQNw6sP43MOvy7juBQZEF5XCZI3rVOwMSUBcyU=; b=r1hRGjvb71DQPdhxjE51t8lb483KqLPiH17qSU5di5QtoFwXiIw0OrpbqTqkFnujG4 9IY07H4zZlhj6M5kFpRxAH7pR9QG21+Th1NtvdKZf+fXHXg0tBZpTcL2VauuOUlam7di 7htj+dkAGpKLyD2eZ/CUcTcsEEX2h5mKD/H6VJmrtG3wvkNsxEPkvHgpnyoBRTgTBtZh Sf+ltVtCKLwwBHF8YwXy8dGgPup4Lpc4jLNxW35m28S5nTnIR+qagzaqL/hMCDelc3rs hWQXX+/KcvH95If7k7qoWxt3M1LUECp2CWXbRFVXwEAUY9MrFkCJp47dKNPBtMHQZEZL MPbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039063; x=1711643863; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BfPuXrQNw6sP43MOvy7juBQZEF5XCZI3rVOwMSUBcyU=; b=XOoett28MCPw8ZZmtPJOFrY3DWIvJSkPSgooU+GQwXPHNoaL70GmVK6PvjbvmkMN+W 7Y01DDjyoY69Fh9fRjbrWuJ3s7SoH4S6oIBvOkpFyqHFP8JruJ8ONzyyV3vNp9TV/JBv 4s0FxQyKBdZ/qteyxOw5vHVPmGCuH+Y4nSjX446rc+Yyev/nkPcYtSwiPeGCilNRaiyY NneMvrfYvrtw2Qm1TEAItw4wDs+ogiJp2RWP4WJryIlkE/UedJ6TUwGbByglEHtyc+XR Wb5eIMvTOMXL81XgGndO2r3fP4cH+dj1JntXwfHeUl2ZA/DllTXq7KIg27QWaB+54KF1 4l8A== X-Forwarded-Encrypted: i=1; AJvYcCX3Fqw/PAZCrXtBE/oyp3SygJgX4EBNB8TQXXI+nBaqc+qA8HsYI/TtrRyHQw4OqQf5OPWsnk5CO2UgjnwIuRohYTY= X-Gm-Message-State: AOJu0YxOeHgaPLXZpYzTNikDNMRxcu7r1B6vkYmVi48AtOh9tAYYE/d4 3tM85NCg7dtnK4u86SDyIpQOWQFAZfNZgJpKoTFiUZeZNDitZwj6OXtL1H3hBITeS/WrFTTsOlW vHw== X-Google-Smtp-Source: AGHT+IF0jEtLINTH4On1VBGvIr2Jc0SGzS38YLdjNwaqc8VhKxt4NeXt5PC/R2alQ3iBswciTecJTdFY/QM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:690c:39b:b0:60c:d162:7abc with SMTP id bh27-20020a05690c039b00b0060cd1627abcmr2322702ywb.1.1711039062719; Thu, 21 Mar 2024 09:37:42 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:37 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-16-surenb@google.com> Subject: [PATCH v6 15/37] lib: introduce early boot parameter to avoid page_ext memory overhead From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: DBEF7C0006 X-Rspam-User: X-Stat-Signature: j6dyw9d1deseds5j8aaw6yje7ac7t1be X-Rspamd-Server: rspam01 X-HE-Tag: 1711039063-380211 X-HE-Meta: U2FsdGVkX1/MV13HP0qOYMnhpAmev+diMWkikGqW00ojlqu+q7x7Uz5IIlBqXR9AMcK8WY1RYEY7pnPlN1U+rLaXTL/5stWyfIRMadOnpmJzD+1UcwVXKASDF9XT+WwrOwZNhvDEpc5+FQzMltMm5ECxpy8+9JTqBgtxAAQrF+eGNmecYc1ZdfyFG88iGCPsSQpM/1yvM/a0sOgzCjuQ7ba/KCKVkPHWHyDI0V1vSyrq5cD7FALAbiWWHZPWynTdHm83tKqbuaEXSzfABQEsqvW7Rpi8rVR8Be623QU7qLYhNlq1v/DFALOpjkPsVc5PGk6ol/w6XKrVfxPrDVmvahpZazA25e3vFNVTd1e8GE+v5MYbeSzRp7d6pVGBEsOq/OjYx8a1t2JbmLO/D+7LuqBExxpXlpxCzZvAFfdHyAX0Qeuf4Tc+s/0u+/V/afHdhhdaxBDv5l0KWMrhLU3BiC4J0hBW/qu0zbUITWv+4V3HkqdKomPvzpgRy+BDHrfSIrsVFJ2PBGZuBCavYH/7onZNHCMJFWVLTy0paiSRSRC9qTwuqNMQkF3br96tb3f0quQP91YkVrI7+THXNC45MeSso9zC6LoocwP0QX5KCPV+66SJl1UpKYEBdepny7gxwqZGzJCX9jNwITb2aoG9dPRSnSKZGzvIFMSLaWnTEQNrduzwYkajnN0d5RYSfCYDous46whrjj6S1V/aGGjNkuQE5FmgQe1Hn2jKea94IRnTRNhGX4I4qSpQOE3aqMBowNYqiqYV7ems9eIS7nUdRomEn+wC2IPKMAOloE3MyVnxFH9EufJ468EgRkMhNxuN+TdXZdoZopFpW+gPfjTEzPgmMkvbEXkGdhdS+Q8lp80Atl+xRLlWf7qt6tKwZldfPGV2Pi7G8zk+i3r/qMwNEFlZoZJY2zhDfww+0PVGPT9omk8mNAs6X20Epa6k4uFJPxazdmcgyktIRgxx1s7 Rys6TRCb VXANgUWeD+XIg0yRuY7YOunZ1L0iGET1HAaOgD7aSuAvFjVlpbRv2EWeKOHrbTxMB7OzXItY0pmcVhN2vZD5ANAHekwUvUkp9RMnOBYZsn5KwREPLLxG0WmIyedvPNPArhdXHvsM2dTZOcnO+BzmqvdFLYCKZjg/Ongkd7hq1Zbz9Y6CQW7nz24ovjxsWyvbKwjbE4DEtgOHK3ACUTf+1RS8oTrD8krzX0ydFktLNvhavB1/tgXZ8V8KP3XKrhH67VWsTUVUBEDO7hWFg7MJ5Liui9q4GC0am9egC3gBi41SWhFcNmia40VBa91wbSSydkm1qenEMKUCJu4lJH6PSE7M+c1WARYznaz5WAuiYHQ4iKN8DAxmO8FP7nP9c4/rrvhf3IBTQf4k7xX0yhdCm5zXHyAcHk6RduzJDU+NZwRrau9zGqYXyTVc3KViSM5BTual0UJa1cjMbB/IVmHIlU4QJWfDn68hDVl7CZzgN0WUijEN7LNeqpZmNkuNoZJFCbe2jtVuldbEJZgqi6/m/LYteTTlwTXSMUVKhlfhlqCH2/orkzTAx+mhFyWsAEQw+QPQtWWhII7WXfHi7ODbgcJncgM/i2sDT6DeMr9BvoIqXSbqIs6Pfydc/4TWzxc6yYAnKhNMOYvZjXkRmJI5p8IakNhpdrWY8hHYGa1aLW16vE9+4Pyc2v1/QF9MzkT4i2tSWjwx9yP72jNyI5E8vOZyh6BZegTyH2AcN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The highest memory overhead from memory allocation profiling comes from page_ext objects. This overhead exists even if the feature is disabled but compiled-in. To avoid it, introduce an early boot parameter that prevents page_ext object creation. The new boot parameter is a tri-state with possible values of 0|1|never. When it is set to "never" the memory allocation profiling support is disabled, and overhead is minimized (currently no page_ext objects are allocated, in the future more overhead might be eliminated). As a result we also lose ability to enable memory allocation profiling at runtime (because there is no space to store alloctag references). Runtime sysctrl becomes read-only if the early boot parameter was set to "never". Note that the default value of this boot parameter depends on the CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT configuration. When CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n the boot parameter is set to "never", therefore eliminating any overhead. CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y results in boot parameter being set to 1 (enabled). This allows distributions to avoid any overhead by setting CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n config and with no changes to the kernel command line. We reuse sysctl.vm.mem_profiling boot parameter name in order to avoid introducing yet another control. This change turns it into a tri-state early boot parameter. Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka --- lib/alloc_tag.c | 41 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index cb5adec4b2e2..617c2fbb6673 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -116,9 +116,46 @@ static bool alloc_tag_module_unload(struct codetag_type *cttype, return module_unused; } +#ifdef CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT +static bool mem_profiling_support __meminitdata = true; +#else +static bool mem_profiling_support __meminitdata; +#endif + +static int __init setup_early_mem_profiling(char *str) +{ + bool enable; + + if (!str || !str[0]) + return -EINVAL; + + if (!strncmp(str, "never", 5)) { + enable = false; + mem_profiling_support = false; + } else { + int res; + + res = kstrtobool(str, &enable); + if (res) + return res; + + mem_profiling_support = true; + } + + if (enable != static_key_enabled(&mem_alloc_profiling_key)) { + if (enable) + static_branch_enable(&mem_alloc_profiling_key); + else + static_branch_disable(&mem_alloc_profiling_key); + } + + return 0; +} +early_param("sysctl.vm.mem_profiling", setup_early_mem_profiling); + static __init bool need_page_alloc_tagging(void) { - return true; + return mem_profiling_support; } static __init void init_page_alloc_tagging(void) @@ -158,6 +195,8 @@ static int __init alloc_tag_init(void) if (IS_ERR_OR_NULL(alloc_tag_cttype)) return PTR_ERR(alloc_tag_cttype); + if (!mem_profiling_support) + memory_allocation_profiling_sysctls[0].mode = 0444; register_sysctl_init("vm", memory_allocation_profiling_sysctls); procfs_init(); From patchwork Thu Mar 21 16:36:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE7B1C54E58 for ; Thu, 21 Mar 2024 16:38:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E4C0F6B00AA; Thu, 21 Mar 2024 12:37:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DFC4B6B00AB; Thu, 21 Mar 2024 12:37:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C4EA26B00AC; Thu, 21 Mar 2024 12:37:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id AB4436B00AA for ; Thu, 21 Mar 2024 12:37:47 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 84652120499 for ; Thu, 21 Mar 2024 16:37:47 +0000 (UTC) X-FDA: 81921602574.20.437ADD4 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf02.hostedemail.com (Postfix) with ESMTP id BEE6A80025 for ; Thu, 21 Mar 2024 16:37:45 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=F79jPFKq; spf=pass (imf02.hostedemail.com: domain of 3WGL8ZQYKCEUz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3WGL8ZQYKCEUz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039065; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=adETXft81L3mg8VeiLZhJ9Sn5YTIW37aD8rIZRQJCFY=; b=uIWh1moq+USGeczzWWPwRNY0bMa/WrAGwdMapcDC5rbcEgF9AaqY1C35DdCTWSPVl/DUBC d6y08QGF/VMA0RqAlnoytVJYwQFA4I+Gw+r6ohNKZWaSVElwIg4UWGXeUovSDq5I8cVhN5 teMMwiPrDPHEJXrcPfXDZ+XaKWps9HE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039065; a=rsa-sha256; cv=none; b=oNXHAld/M3cdg1WM1pJL5tZMSnJu5NMmctbhI0AJJ90wHuu0fjvQb9P3HdK3Nbi2pfIYYK R7BGzEpZjuLApCrT0b3tnHIVunJN52T87FA1agxl8SFvYK+9HvPSWUpeG9TG4hJL5X8+bD aAhN9SoaRJqt8Ga102k9w+950S/YHpI= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=F79jPFKq; spf=pass (imf02.hostedemail.com: domain of 3WGL8ZQYKCEUz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3WGL8ZQYKCEUz1yluinvvnsl.jvtspu14-ttr2hjr.vyn@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-607838c0800so24231297b3.1 for ; Thu, 21 Mar 2024 09:37:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039065; x=1711643865; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=adETXft81L3mg8VeiLZhJ9Sn5YTIW37aD8rIZRQJCFY=; b=F79jPFKqHHq+X32OoJE34fj2cytFTRo8yb7n1NinyHkSXdIS4hBbv55wpP/bAu7WYC eAo/BGuAJw+3eppwUcjr1Qd2mdU65NyNm9OPnOugEv+mnRvVrqOC3m94sPFr8WTKSq0O 9MCtHKAKdGgUq9wm5UKU4TLkI5qfhIgMmP8gLdTsDwlX495PAMnaQEDQQcz7JDDiWIoZ y2L8mli1h4Ryndz9QDd4W0yiqmogmOSgu12m45Un1ODM2YjyP76lYzMMCne1w1uH76U5 8q+QMD3Iz+MAYOb4Izq/daPafruM1nZTHpOMzI3qwJywIsys/iM9bgfOWyoNu8lpmjFX XrhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039065; x=1711643865; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=adETXft81L3mg8VeiLZhJ9Sn5YTIW37aD8rIZRQJCFY=; b=Bem8P2b66F/pmH2jU/a6Y6hfec2Xa2o/mHTTRwzKW/2CZo0Z9D33eS6W7Flb5VBVeP dStBaxpVs2TgCnrTvxvjxOx7mu3Xa5GyNlBrYKeN4LmGLC11pV9ItmIJUFOoQOUf7PyR 364gYgVrwZ/Ntnn4GnDrUU0RX/m5GdCUbHtQVZCw4iwgzdGR8szMjl3lWrfci7k8kn6x RxDXzJ5m4giIiVgNPe98iQNlLspXVE6zJ4D1ARmov8L2hmXSSm/GbsDvAqHhoVl5ayfn Yzf7i8B6x4DJiLIoK6F9c8OUQBtKvRsxrLzR+NeTiPsrhT22sRtX4FQDLGwGw0aiGEC9 NB8Q== X-Forwarded-Encrypted: i=1; AJvYcCVow8R9/Wj7ptJaTDMCW0SLpM+9LltsADrdibgtC7dvEUz+PHG4qrwejfbeHJ7/s0LbWryENav+kB9ogHZCPO0BdTM= X-Gm-Message-State: AOJu0YxgZk7CoFOJsFlNGceKUHk47thM/ehrEVdnQFYXzVsALSo2BY1A mPZ5PW/8WJcsuKqVGuCbiFYuPzaGkq6jHFEKtwb/yzGSrYzYZLRUAP0GZCBsv7HZukqnFXU9alD agA== X-Google-Smtp-Source: AGHT+IEpbeHGmaH4ZP6HrzXLjRbRGOQFlxFrFvHpCJT+qt/SUASA+RVr+wlFFHpwWsqOIXtYDNaLSozJTUQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:690c:b06:b0:60c:cf91:53e0 with SMTP id cj6-20020a05690c0b0600b0060ccf9153e0mr3628ywb.1.1711039064823; Thu, 21 Mar 2024 09:37:44 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:38 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-17-surenb@google.com> Subject: [PATCH v6 16/37] mm: percpu: increase PERCPU_MODULE_RESERVE to accommodate allocation tags From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: i8rmagtzikb533qif8fszyqihjuu9pts X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: BEE6A80025 X-Rspam-User: X-HE-Tag: 1711039065-398121 X-HE-Meta: U2FsdGVkX1+3GjH9PMTDoXy0cBbpiwNnD0snn/7mBMGIRBX0I3/1HpuqM9DI615kTqWeoWT5017dTRCZRjscnF06aDiW37o/9nNXRPdMP6AIqwAak/r5u9qYdBAchM0iBT4DRyZU4pAP4XEXAgLRRgsGDyohACNIxuhcqhaq77mDZVpHreNgb3hn/rbFP9fP938X42LoejRXAevk8Ez9NYWTaRNZTsqwhJiIPgB7bpTyU83O+8kwyvnxfDZQCh62Sbnnq5utjVSnh4h5QP9VAbv3f+vO/mAB8tWkvX2wMo+hUR2ybs094TqmBurETkHIJdbi6IY8Sx51VTqCDBmlJsL9eehtVFMHic79cWShRB29ZM2Wpx9e4eu7EAL2Y3pDSkQowlnfxmf3l4cyPAzz6+FvGCXZK5vc9cwad5BIShgE9Fk/D6l2cCbNTch6O/pGm52o5/gYdOAJ/4ObNL8GJ/wf7CO1ZofWiXy3ccR6CzGEa4aX2C4ZVWCnyspSRkAXFIKGiCV6HVih4pqqqo63z+JCjKEX7FpBGx0NnOm7hvg7SyGFe3pUzZtzyJo9TxGKjbtwqfaw9OulibGW8lJdScBqoIjPWSEUi3ZWKu66Sqrlh9VvLXDYR9Exm96MqSnMgkuSDsL1arMVyZP+CtNqe8RhaPX7DSVXESfU5vVAGYzCByYM7H6KcC4jHowpEsOtELGKSwS4OPBUdZBGSguSSPsOWT05QqFy3swviF7jnjzBSVx3AbR34hQ046CKKpmhWnClv+Xmiw8FIHblAJkHDng346fJAGBcXm+y9mRCjg4ZroyqIBASfhSQE6ocmhPolMnnwe8RceUeUwoksg4CaMM6tXkACC3ARlzotsy6LwkaoYC9Hd0Lw8OLrNJJUoeMZo/POCfLtnHTmceFON09diSKgzrOvW/8u5kdkkMxXBijAVTRNrQ+uuG7ZaGCCmRaeagB6Y8uMk7v/S6cmD7 hT4pUmti NiPVOe3L7QYA9b0hGedFp/79dPilX7giiaF67KZCXgBzmpwFPJY84u/dG7q9Xnnuy+GyGM+Snl6GjCKr74Ynon7yoKNmV5FReGqZLkbTf9yGlNTvFHZfGwRyBQCNP+gZqbyuwciPmEJDYBaLdqsZ5RqFtnAoS1IxIR5PV28LeBOj0nN9ggiaWuEFPy4FlBQIOBOEQdqgTt0UB7FGbcg5eDJF5rOCIRV1BcOOLaPn3pfiPiLQ2bM28Hjdxovu/vN0O7KeFoK3guo6tKJ7rIqKww1WCYfnbPuEftukyPE0n3WcauRWkNWvt/KexiXJ/X2mPJUopQlXsClTq1qbMqQ4Jbfkz/z07sMY4h50eDYfqKIE1ggHzYqueEd+LQfileDD5bAGS/iV+4LznK7k6DvzTU98SLr6wJG8mTQeDzwPWXDj9H19SnII6BbOu6NMjhznvGiVW43ra4bktdsvxTqPrpQ9cQfS0lmBp4oJL2KZeddB6WZHp8+Rq0bspr9PLGHxoAX+kaQdvE3OLGE7jUJAdN8YSovT39UeZzcyJkaOCO6dBgQVlN8pz3TvCNUresG5LrX4aUIsJwZz3XzYCUtNQ1HqS4zxCl8ey+6R038f86T0K4jQtyuKbuzw7RlAsHk6uPT1818AhkBH9EmHTQNMsJcOP5DLcx9N7fbF5kO3pGLPtyxsM9HJ58uwr5Z9A8e5dQeKQkPLPAfcToKpJcRFIB90Ui0RMSI6dk4YU4t6+MvuGTRVxnBvMoxZLbHYPPElz/s33yIR/Q6NG6oET7fjSRjLFwCi4jySUJhW40C3LeoM3yUFnANnX8+ZwlKlZzNbKlB9G5vHtlxrxQE0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: As each allocation tag generates a per-cpu variable, more space is required to store them. Increase PERCPU_MODULE_RESERVE to provide enough area. A better long-term solution would be to allocate this memory dynamically. Signed-off-by: Suren Baghdasaryan Signed-off-by: Kent Overstreet Cc: Peter Zijlstra Cc: Tejun Heo --- include/linux/percpu.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 8c677f185901..62b5eb45bd89 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -14,7 +14,11 @@ /* enough to cover all DEFINE_PER_CPUs in modules */ #ifdef CONFIG_MODULES +#ifdef CONFIG_MEM_ALLOC_PROFILING +#define PERCPU_MODULE_RESERVE (8 << 12) +#else #define PERCPU_MODULE_RESERVE (8 << 10) +#endif #else #define PERCPU_MODULE_RESERVE 0 #endif From patchwork Thu Mar 21 16:36:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599174 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5DA58C6FD1F for ; Thu, 21 Mar 2024 16:38:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6D1776B00AB; Thu, 21 Mar 2024 12:37:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 659B66B00AD; Thu, 21 Mar 2024 12:37:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 485D46B00AE; Thu, 21 Mar 2024 12:37:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2D2AE6B00AB for ; Thu, 21 Mar 2024 12:37:50 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id F163980B19 for ; Thu, 21 Mar 2024 16:37:49 +0000 (UTC) X-FDA: 81921602658.27.7D590E9 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf25.hostedemail.com (Postfix) with ESMTP id 1B380A001F for ; Thu, 21 Mar 2024 16:37:47 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=UmrIvuzb; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3W2L8ZQYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3W2L8ZQYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039068; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jp7kzC4k0D+sc/q1IPxvwe52UJDwCcNnw+Zbq+7qIDQ=; b=CMAY0maTfs5Gu2J70uMEbGDG4ydnls2G/BnnG0swLi4iePIPHZCsWx7vgFtM0G2OYOoJuG aQtKLKoA5JF4tstQzTFP4SQUzjqBVsfcPE2VPfzHL40xOw+yWj53vNkowjOXKeb7PZHnIN dNanXY4uvHaW2umIsHmA3qP0tqCpPao= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=UmrIvuzb; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3W2L8ZQYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3W2L8ZQYKCEg241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039068; a=rsa-sha256; cv=none; b=6LwQi7ZgX7O8Wl1ycORPMp2tytv4f5v+qgi+SYzo+0KWdt6fya/DdOfHYwKV8rkBBMMU2q 04tKIvdo495f+heXUqcE6uzkhLBl7jAI88Zu5DlYGc89Cwy6V4AuLaNCPH1sgz3W9GznkR WOKLnteGMcNk2nCJ5r9nrdQsAJgDMos= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60a605154d0so14620177b3.0 for ; Thu, 21 Mar 2024 09:37:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039067; x=1711643867; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jp7kzC4k0D+sc/q1IPxvwe52UJDwCcNnw+Zbq+7qIDQ=; b=UmrIvuzbSkDAmQqKB9LOhD/hnghQoYP9xBauvt/jvW3Rn/9RSzW3zlTh2mwrSRlwy5 +PXoEBEK+RayKojkYidhvN9DT0bEsT6/SJuazocq7LH00M95UQNyV7EWPPhFYr864uTY klHFJv7kVCqb1R8ny2uHCcUbUmORdrS+RLRks+XSNrCcaqAfsFPLkvxYFN8ghCpTCEwI 91gtqEd0A3Mizf5e3zcbdTgk8d9EoCw6zeXeL/nyrU6NOqw08V8J73pQqzk5Jh+58fsi QtaShWoNIhgw/v5AjJ2Rr+t2pso4mkrEWCCaPh8W7z/r3G3IyCoaOSFSm+VdD/zIX+tL Ba+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039067; x=1711643867; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jp7kzC4k0D+sc/q1IPxvwe52UJDwCcNnw+Zbq+7qIDQ=; b=lGTZ/8vHM10CblqUqcCtXJmDLMqA55yMO0u4/1voLqvAtz/d5UT4eSZxhhvaN2pPte OJ+sS70NuZSn9y/QyeWqm1X22OjSUb0bhiTyg0rcT/W6lAUWFManwzOBohmp7uUmDfG+ 7J9ah5DrzsNvLMXYNV33kr3fir0TWHqtpAaCjBaMhtEcuU/juFQDZklszHf/xicE6bL4 ds14VQJESPjO5W8C4F5+xs/cm/cpVZV8UvtXfkfQUTwjIcrxO8WPEOn6XDuE/GtjKAjn CRP6vtC7Ulq/RmLGAz9KJCs7kwreozsX2vOT79nfXhSWUppG6YqXRXE8CCbazmEk6Gd6 xX0A== X-Forwarded-Encrypted: i=1; AJvYcCVqDKusFSj4HDAy7BQA64cZNgFrSwSoHuBFGQ+cT5HdAbZZUyix9UxCLdzAd7cHyaMDyD9SIm7z0J8/V/7047E2kvI= X-Gm-Message-State: AOJu0Yxl/UWbnF1WkUj1vTdxZajV9ml3Q7eU8Gw0x04IPsdmaC5ZapDB qIIR1M49ZPWqv8LLgIDhbmJoB29Wib1fdxQ3YzzSfftaqrrQBll8tpuPkRoAqHB/FxSxZWXQDBp Hpw== X-Google-Smtp-Source: AGHT+IFI5aLEMhGkJPVMYqKYBpHCCYs3laPPvxTJvnz1VdKYfIbwJ16TvQd2SO6GjZkf9rmaF0ZSWy/DcA4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:690c:6f91:b0:60a:1844:74ef with SMTP id je17-20020a05690c6f9100b0060a184474efmr881713ywb.1.1711039067006; Thu, 21 Mar 2024 09:37:47 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:39 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-18-surenb@google.com> Subject: [PATCH v6 17/37] change alloc_pages name in dma_map_ops to avoid name conflicts From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 1B380A001F X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: b3oztrsgbze93a4mcnt1fn4wquumnxae X-HE-Tag: 1711039067-318360 X-HE-Meta: U2FsdGVkX19vyDj2LQ2ziSgE4w6yIUJUvtwAQCPEeEneLCMe0Hyh3fGrGio5fSi0cBPqcEI7LOCKu2MvIFzoO0Ko/Jswk3mRg4C7ahq6MSpGn9Dm3nvKg6wVcmYjjpdy0b9+XO4QGSaUbNHRxHboAI0nHKrTklqBlS9TSo+jC43UWCpKjwPt+zQZxHWjqou2LymmcRkWavvB9su4O/8PKewMu+WhzTv3G8yhVVwEMnNCK3FHc6PEjFPeI9YpsoBO4XPbG3FoHStSw3x1oISDr/eTjtEkTUQgHGZdd8TNOy48hciyphmEAJP67ONExAUo+zD5Y2IguBheywbGy2veiMy7i4V/Bkp06K4sxIiJolnpykIy9Y3oVDoRBwnbvifgLi0p3ssjbywtyvMEFDjgMAPgMhXKKDR7uoACgB3CsRkZ/9hziozHloQSDZyHgulpBFIvzaKgk5FNozEcMxYEpZ10A5AvD1FgWoxUuEcQP2S69EgrQL1DOgod4uMSZ1aLqg0wDh8IOBiiuZwQA0Nar+spGfMwWGpOipk/C10HPjJp27d3X7pdenn/XAxQJbLPv9kdw6kpJrdPPVTsKS0HSh/lhR+qsewJHk+0U554wi66qFp4YKwsLPFHjkUONIXc+GwQEz0PhSd3sXPqBKMZfLqNoLn82GuvlwgFW9SHiczD++yMLd0CrlDsCP82tcRpayeJJjanlX6rLVhlAiSAfns7NhQakldVtzjN2Kg8UycJ6WUIxyBulptx40nhwDr3c9tg4pBdUkB0UwcAqOxD6Ze+sGIC+M/cVc8qF0dOrA3Xri5tItJdtq2Pme/DwADdJD6i3tIJNxO0Rh7r0Uli4lmlBXs0cGfpkMH9kuPo2eqWIOoJlLKx5IujIrpy42b4ktteQqG48K6UbWAfb4Jggjn0vKN6Xua3AdkTlygvOVBCSdyg/OQg9nbocHzp6aI+t5QxehJaWQ30t666haL 70QdLjz6 abHru8yUesHG6J9rLRAAuWcqZAHkd2yOmPEv4yTBvDGTQBd9sraYgqP+K+iZfonY5hQrwIF0TyTP3XAoQ3xMY1NRFzNtcaow/guXZCV/8A7YHT+VPSvyookKuKmZGOVztzlgR4SA1IfwdaCx+WONaIOm5q6qA/TjHoeG3dTBTVwSxElQi1M600CW/ISAV6kKVL1udzp5LhVOSS4/gxNQ2XWQV82EI13YNzP0153Fgz8k55nPkql4S4IMQ3XR1OBc6yeLdHkFk+uWts/nvFzi6/EfER4WON4CPBdaH/Ye24tD0Ps+MSgM2U6zpUvdgYcxehCxDqYtM06UHvl0R7z6UtHGpek8p6+RdwFmdPvpGAb87hVtJfFugYM9K56qyP9XaQ4blRYpm0Ycn7nwl3rxNLAkC/dedNTdlQAAU7ghyum93nzFu27Gci1mffwhne7rXfgp6kuaUzpuewZcRnPFBm5/cwj5BrHxXgedw+rdoK6geOgmBOiCWPLlkuzMzL9tSeeowj5i18UNuzQ8I5kfa3FatfEw9PVRpRyv567RKijzD/ri7sqODZGbgmw+LuGqXKA6//Y0LPMqI7rR0TPwPC2h1VykCnKv9cNSVqvnLuBMRv4T/vuN66M6BeuhsfXFvDI3G+V3mFNolL1u2+yEsul/aX1d78fkTanr94mccdA4hI8OuONCdxkXF/mDruQYFSI6WMlhOSD3ayhOOnDwgH5dlQHzmS6bB+DWh X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: After redefining alloc_pages, all uses of that name are being replaced. Change the conflicting names to prevent preprocessor from replacing them when it's not intended. Signed-off-by: Suren Baghdasaryan --- arch/alpha/kernel/pci_iommu.c | 2 +- arch/mips/jazz/jazzdma.c | 2 +- arch/powerpc/kernel/dma-iommu.c | 2 +- arch/powerpc/platforms/ps3/system-bus.c | 4 ++-- arch/powerpc/platforms/pseries/vio.c | 2 +- arch/x86/kernel/amd_gart_64.c | 2 +- drivers/iommu/dma-iommu.c | 2 +- drivers/parisc/ccio-dma.c | 2 +- drivers/parisc/sba_iommu.c | 2 +- drivers/xen/grant-dma-ops.c | 2 +- drivers/xen/swiotlb-xen.c | 2 +- include/linux/dma-map-ops.h | 2 +- kernel/dma/mapping.c | 4 ++-- 13 files changed, 15 insertions(+), 15 deletions(-) diff --git a/arch/alpha/kernel/pci_iommu.c b/arch/alpha/kernel/pci_iommu.c index c81183935e97..7fcf3e9b7103 100644 --- a/arch/alpha/kernel/pci_iommu.c +++ b/arch/alpha/kernel/pci_iommu.c @@ -929,7 +929,7 @@ const struct dma_map_ops alpha_pci_ops = { .dma_supported = alpha_pci_supported, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; EXPORT_SYMBOL(alpha_pci_ops); diff --git a/arch/mips/jazz/jazzdma.c b/arch/mips/jazz/jazzdma.c index eabddb89d221..c97b089b9902 100644 --- a/arch/mips/jazz/jazzdma.c +++ b/arch/mips/jazz/jazzdma.c @@ -617,7 +617,7 @@ const struct dma_map_ops jazz_dma_ops = { .sync_sg_for_device = jazz_dma_sync_sg_for_device, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; EXPORT_SYMBOL(jazz_dma_ops); diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c index 8920862ffd79..f0ae39e77e37 100644 --- a/arch/powerpc/kernel/dma-iommu.c +++ b/arch/powerpc/kernel/dma-iommu.c @@ -216,6 +216,6 @@ const struct dma_map_ops dma_iommu_ops = { .get_required_mask = dma_iommu_get_required_mask, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/arch/powerpc/platforms/ps3/system-bus.c b/arch/powerpc/platforms/ps3/system-bus.c index d6b5f5ecd515..56dc6b29a3e7 100644 --- a/arch/powerpc/platforms/ps3/system-bus.c +++ b/arch/powerpc/platforms/ps3/system-bus.c @@ -695,7 +695,7 @@ static const struct dma_map_ops ps3_sb_dma_ops = { .unmap_page = ps3_unmap_page, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; @@ -709,7 +709,7 @@ static const struct dma_map_ops ps3_ioc0_dma_ops = { .unmap_page = ps3_unmap_page, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/arch/powerpc/platforms/pseries/vio.c b/arch/powerpc/platforms/pseries/vio.c index 90ff85c879bf..477c1d5e1737 100644 --- a/arch/powerpc/platforms/pseries/vio.c +++ b/arch/powerpc/platforms/pseries/vio.c @@ -611,7 +611,7 @@ static const struct dma_map_ops vio_dma_mapping_ops = { .get_required_mask = dma_iommu_get_required_mask, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/arch/x86/kernel/amd_gart_64.c b/arch/x86/kernel/amd_gart_64.c index 2ae98f754e59..c884deca839b 100644 --- a/arch/x86/kernel/amd_gart_64.c +++ b/arch/x86/kernel/amd_gart_64.c @@ -676,7 +676,7 @@ static const struct dma_map_ops gart_dma_ops = { .get_sgtable = dma_common_get_sgtable, .dma_supported = dma_direct_supported, .get_required_mask = dma_direct_get_required_mask, - .alloc_pages = dma_direct_alloc_pages, + .alloc_pages_op = dma_direct_alloc_pages, .free_pages = dma_direct_free_pages, }; diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index b58f5a3311c3..6ba52481a1d7 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1715,7 +1715,7 @@ static const struct dma_map_ops iommu_dma_ops = { .flags = DMA_F_PCI_P2PDMA_SUPPORTED, .alloc = iommu_dma_alloc, .free = iommu_dma_free, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, .alloc_noncontiguous = iommu_dma_alloc_noncontiguous, .free_noncontiguous = iommu_dma_free_noncontiguous, diff --git a/drivers/parisc/ccio-dma.c b/drivers/parisc/ccio-dma.c index 9ce0d20a6c58..feef537257d0 100644 --- a/drivers/parisc/ccio-dma.c +++ b/drivers/parisc/ccio-dma.c @@ -1022,7 +1022,7 @@ static const struct dma_map_ops ccio_ops = { .map_sg = ccio_map_sg, .unmap_sg = ccio_unmap_sg, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/drivers/parisc/sba_iommu.c b/drivers/parisc/sba_iommu.c index 784037837f65..fc3863c09f83 100644 --- a/drivers/parisc/sba_iommu.c +++ b/drivers/parisc/sba_iommu.c @@ -1090,7 +1090,7 @@ static const struct dma_map_ops sba_ops = { .map_sg = sba_map_sg, .unmap_sg = sba_unmap_sg, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, }; diff --git a/drivers/xen/grant-dma-ops.c b/drivers/xen/grant-dma-ops.c index 76f6f26265a3..29257d2639db 100644 --- a/drivers/xen/grant-dma-ops.c +++ b/drivers/xen/grant-dma-ops.c @@ -282,7 +282,7 @@ static int xen_grant_dma_supported(struct device *dev, u64 mask) static const struct dma_map_ops xen_grant_dma_ops = { .alloc = xen_grant_dma_alloc, .free = xen_grant_dma_free, - .alloc_pages = xen_grant_dma_alloc_pages, + .alloc_pages_op = xen_grant_dma_alloc_pages, .free_pages = xen_grant_dma_free_pages, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c index 0e6c6c25d154..1c4ef5111651 100644 --- a/drivers/xen/swiotlb-xen.c +++ b/drivers/xen/swiotlb-xen.c @@ -403,7 +403,7 @@ const struct dma_map_ops xen_swiotlb_dma_ops = { .dma_supported = xen_swiotlb_dma_supported, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, - .alloc_pages = dma_common_alloc_pages, + .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, .max_mapping_size = swiotlb_max_mapping_size, }; diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index 4abc60f04209..9ee319851b5f 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -29,7 +29,7 @@ struct dma_map_ops { unsigned long attrs); void (*free)(struct device *dev, size_t size, void *vaddr, dma_addr_t dma_handle, unsigned long attrs); - struct page *(*alloc_pages)(struct device *dev, size_t size, + struct page *(*alloc_pages_op)(struct device *dev, size_t size, dma_addr_t *dma_handle, enum dma_data_direction dir, gfp_t gfp); void (*free_pages)(struct device *dev, size_t size, struct page *vaddr, diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 58db8fd70471..5e2d51e1cdf6 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -570,9 +570,9 @@ static struct page *__dma_alloc_pages(struct device *dev, size_t size, size = PAGE_ALIGN(size); if (dma_alloc_direct(dev, ops)) return dma_direct_alloc_pages(dev, size, dma_handle, dir, gfp); - if (!ops->alloc_pages) + if (!ops->alloc_pages_op) return NULL; - return ops->alloc_pages(dev, size, dma_handle, dir, gfp); + return ops->alloc_pages_op(dev, size, dma_handle, dir, gfp); } struct page *dma_alloc_pages(struct device *dev, size_t size, From patchwork Thu Mar 21 16:36:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD136C54E58 for ; Thu, 21 Mar 2024 16:38:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD6E66B00AE; Thu, 21 Mar 2024 12:37:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D877A6B00AF; Thu, 21 Mar 2024 12:37:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BDA396B00B0; Thu, 21 Mar 2024 12:37:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A1CFE6B00AE for ; Thu, 21 Mar 2024 12:37:52 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 7A1A214157B for ; Thu, 21 Mar 2024 16:37:52 +0000 (UTC) X-FDA: 81921602784.02.3502E18 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf24.hostedemail.com (Postfix) with ESMTP id 8363E18001F for ; Thu, 21 Mar 2024 16:37:50 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nWWRx6rn; spf=pass (imf24.hostedemail.com: domain of 3XWL8ZQYKCEo463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3XWL8ZQYKCEo463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039070; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7FdsBQ7dxKBT3Q0hroP+RtUALgw5JCKh9olno1yd0hA=; b=aL9gZ/gitEzwnojCEATuyoh0XqZL+wALZcOTAAP8/J7hndRRAXqJsyB1eeA9uo5PVIABUG ADzptN8onSAfyqRVOcfa958axZ+w97fALdnbvxORvP/Lf5PdIxFpy/W78I+/qtJkSd5Vgt LakGB782PvrCy8je1Wb/dvEuRI5etB8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039070; a=rsa-sha256; cv=none; b=IZI5AlG+E7DRoUrWU4CBICBDa+nvtQhbR+dCXmT+jb7dqivDXIs6oFmEAvosus2Pc18o+3 3JtvvsNWq/fkpJdslkuIfM8gs20YFkOSbfZmtyw2ummN0XgLLDyRjzOW+K4ppptjHSqKEg q5JxxcJB5C6tdB+11fEM42QsE169wOc= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nWWRx6rn; spf=pass (imf24.hostedemail.com: domain of 3XWL8ZQYKCEo463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3XWL8ZQYKCEo463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-610fe0406e9so18742417b3.2 for ; Thu, 21 Mar 2024 09:37:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039069; x=1711643869; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7FdsBQ7dxKBT3Q0hroP+RtUALgw5JCKh9olno1yd0hA=; b=nWWRx6rntOxS5tiwZF7Bz0odOU9ZM+a8SlFzyrIAtMBb7ZLXL7kpCeG16YsXH5FhLO 1e+GZMiCg1OwQU5DjjYD6JIqVwCKA7hL02Mp5BjxihcT2axv8vXPUpLvAuigxF48gtA6 Mv6ZsDajcEj0EplbJjoteOgFzA0l+AsG9+xpIySLbX2ocfpwcEVY9ULd2mwdWhfsuwyK rWCHdVbpRzQuBwHJBdlVUjJTXYu0QXta36wbWf30nyvbTP9FUOIOwVx5m5AWSlMIphvW Pfd7yPnoB31yoauTHRjZ7KYe1mh2OEP2dspVhMvaQtxl4ue6ouLdvoA2xVRqF2Jq7fID znlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039069; x=1711643869; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7FdsBQ7dxKBT3Q0hroP+RtUALgw5JCKh9olno1yd0hA=; b=mvja4MZPoJfzVA78pgF3FeDeh4D8LQych9wc3Dc4wxI4KnUxaLQs2T8CvVVFPMLCz9 sYhuggeytfav2ecF/riZOm0lFqjKDMjxSZsBi3uDwI6+F5M4JW/qDIiyY4pHeJ3dX1Pm +g6Uauvn1yt60knIddMNn+JVzshjiyd/0DiovTbb5zy9+V0XMCufjeIegHGRmJ4tY7dl uSe0/BnYJQVRWhvEpGTxxa68/pzto/IGMgwveeuJN6Nuig4A7qAiUDrrGGWjqEdt3VjV UUrTpkAqHuN9/TwS4omyQM8v5rqe9ApOYITb7k1S+jkxGadmrePE1HViV1GyXMyl5tG2 qKIA== X-Forwarded-Encrypted: i=1; AJvYcCVjaEMIFO00NMRG4FMaN1jYvqvAtwIMmdrAS2ruaXBTEQDQdtrslyI4PJ3Gydfi6dsu8cdLjkfECsnEL1B6ruCBCyA= X-Gm-Message-State: AOJu0YyAAaKP/xjPKzuKbWTG7+a0xkmHP8z1Pd40oenLv7WbGM7AEbx7 i7z+MmkZdfalqmgi2q7+d7EgMFdpfSXNLNR2BM9GIlhE7MvTL+fiiniRJFsd6FzX1BCFGz7EZx9 3Aw== X-Google-Smtp-Source: AGHT+IG6Y/ms7n//UQpkW32UEhUN38Qylpu3mQd5KGWXP2OQ9MTMou19lVeqkh7+swYKkGK0IHqdnETLoRE= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a0d:eb41:0:b0:60a:6a7b:3c61 with SMTP id u62-20020a0deb41000000b0060a6a7b3c61mr3979557ywe.5.1711039069400; Thu, 21 Mar 2024 09:37:49 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:40 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-19-surenb@google.com> Subject: [PATCH v6 18/37] mm: enable page allocation tagging From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 8363E18001F X-Rspam-User: X-Stat-Signature: skoi8pti6xquzui1c1mbdch9aogafztp X-Rspamd-Server: rspam03 X-HE-Tag: 1711039070-794321 X-HE-Meta: U2FsdGVkX18van9ivZ3CsqfE8GB7GikKVWG6PnVqXhlU5nMDOM9AqKnieDP9MytbyEmx5GDq425hWVjIMxmaMah9rVblNW1KnlyV/iiFae3Px9Q546hQCjxkzvrN8X0oxjl2VsVr9cpqWJ3YR5j1r0+SW7AQ+yAvBlr/01Ce7ET5+yWjPyBakeCEUTrY2CK0Pz4qOzEas4bRqwU1rFlAe3dFQfWo6Mj4cDKjW/VshJ0m0N4x3KzauhatIy+XmW4XMoog09s3D5UVl2F4kuj6zxT2rdWiidLcnjWmBASutf6fUAHbbLXSGtvtyF3NwQKOzGaa9gk894JjfelLm2ulCG2yliwuFIK8f7InePhk9cxTA6I64LHzpwYjGo6DCoXSLRmUEO1hd3NYCfmzIU/TZi6VpJi4OoLx+1Fd9oLJQi6m3wgdu0fNr90w0aDLDE+qgxgG866jILKDocXSJW0LhHjPY6w9lG959yasqJwH8j0LXCWlQKA7flTkEFwrK5UYe+9OJl23zDkkuz0NExIsV2Bnpo0CS8EtDx2ZJnhEw8rurM0BeKc0zQUfhycWmmwLnzf7wHaDcd2mi1k5WTTizX3gcqKzXnnJ1LTssseghKUw+LlcDlxr6XFE724ZSi8CQPvv291Ww3whOlrxfJ6uNLrb7wWggtjZplaiivP7/1wC2+mJefLPH7qJz/gmZORAuUaIGDLOprygkzOYXGhbJ9fdNOZr4CFUnfCd9BquEEPuPe1VkM4ULI81mQD2dpZok+Viapbi49KpRtd/2AQYC3ffsX5Z8biaDw0ZTkufnUNIGLA9MuyXKNasoVLwWwScqu+IrFM2qVNDQqEme+CQ6d4I1BKfQWSCPQhnC6l+nE187nlgKJSfVs6NGLwAJ09UluzycOcEHxnQNZ6a0jw1UUmi1TeEyGFKe2fIinDbI85lfxqbH3PgGu0JsCVrrZu2ocCo0qzHpeF7CevJTDy C/yt6XAE BRyQcqXXwtzT1AGHTM3glb9aahYaUxfPKA2csi4OQI1CEflyM5FDbaOdAooGm3f1r3BQU+3JxSbpcdJen1lhy+/NvGn1c3xcFZ+iSy81H5dYbt8QRLVGdcjoOAcI+7J4RJV4j7bKKly4M+6TLo4R/4fNCkP39z2CjpZyR3HcHe/PHnbl9IGPejSsSrt+Nujntf+oGg/rv6DXe47xsmVlG8h6CiIBRzZRT33vrytH9nxOZdqTeaX5URWb1Q2J1iyZbtvTLg9yuserlA73zcSvdU3aQhuZrbyz5ggk+kbje4MDTvm0GVPfPF0dnOH+QmiaSOEExlhXcuBv1KI5gkhZOTTWsgmLxkaa/nFhk358iwq7FwD6xtTrNtJlfibjO2mHmJO1C72NSJtQlU1fCNxuUFrQ2k/+hh/t8zuD+MLI0aKInTVn0EmmX18nE74g3ACxK3YmvjGmgAWeUC4yc6O7gThkVsbMAs3hO7QvxZ25fCCA57rEfVcrgt37oPqNiSBuUB8C0hzdkN80KdqdpkfdkJEgzlnWfohbS/bpSjxBCYLAwEY2c4pH3yZagw2kyt6WWGy0xbG+Omjx6UUlSNkV/XK+j4vbj1VPMse5l6ftW6J7T9g3JPttiMlucbkzOuX3WsQn8sJsb17bLWvcVhUuieSZ/puw1JYIPIPta7A0SYQ3NezuyFnviELHwDdMH5Zb76tTnBBX21wBSP8jkQ/DIV+ZBtJFfM2OBvSweTI1zsVW0iV4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Redefine page allocators to record allocation tags upon their invocation. Instrument post_alloc_hook and free_pages_prepare to modify current allocation tag. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Reviewed-by: Kees Cook --- include/linux/alloc_tag.h | 14 +++++ include/linux/gfp.h | 126 ++++++++++++++++++++++++-------------- include/linux/pagemap.h | 9 ++- mm/compaction.c | 7 ++- mm/filemap.c | 6 +- mm/mempolicy.c | 52 ++++++++-------- mm/page_alloc.c | 60 +++++++++--------- 7 files changed, 164 insertions(+), 110 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index b970ff1c80dc..28c0005edae1 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -142,4 +142,18 @@ static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) {} #endif /* CONFIG_MEM_ALLOC_PROFILING */ +#define alloc_hooks_tag(_tag, _do_alloc) \ +({ \ + struct alloc_tag * __maybe_unused _old = alloc_tag_save(_tag); \ + typeof(_do_alloc) _res = _do_alloc; \ + alloc_tag_restore(_tag, _old); \ + _res; \ +}) + +#define alloc_hooks(_do_alloc) \ +({ \ + DEFINE_ALLOC_TAG(_alloc_tag); \ + alloc_hooks_tag(&_alloc_tag, _do_alloc); \ +}) + #endif /* _LINUX_ALLOC_TAG_H */ diff --git a/include/linux/gfp.h b/include/linux/gfp.h index c775ea3c6015..450c2cbcf04b 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -6,6 +6,8 @@ #include #include +#include +#include struct vm_area_struct; struct mempolicy; @@ -175,42 +177,46 @@ static inline void arch_free_page(struct page *page, int order) { } static inline void arch_alloc_page(struct page *page, int order) { } #endif -struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, +struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask); -struct folio *__folio_alloc(gfp_t gfp, unsigned int order, int preferred_nid, +#define __alloc_pages(...) alloc_hooks(__alloc_pages_noprof(__VA_ARGS__)) + +struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask); +#define __folio_alloc(...) alloc_hooks(__folio_alloc_noprof(__VA_ARGS__)) -unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, +unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, nodemask_t *nodemask, int nr_pages, struct list_head *page_list, struct page **page_array); +#define __alloc_pages_bulk(...) alloc_hooks(alloc_pages_bulk_noprof(__VA_ARGS__)) -unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, +unsigned long alloc_pages_bulk_array_mempolicy_noprof(gfp_t gfp, unsigned long nr_pages, struct page **page_array); +#define alloc_pages_bulk_array_mempolicy(...) \ + alloc_hooks(alloc_pages_bulk_array_mempolicy_noprof(__VA_ARGS__)) /* Bulk allocate order-0 pages */ -static inline unsigned long -alloc_pages_bulk_list(gfp_t gfp, unsigned long nr_pages, struct list_head *list) -{ - return __alloc_pages_bulk(gfp, numa_mem_id(), NULL, nr_pages, list, NULL); -} +#define alloc_pages_bulk_list(_gfp, _nr_pages, _list) \ + __alloc_pages_bulk(_gfp, numa_mem_id(), NULL, _nr_pages, _list, NULL) -static inline unsigned long -alloc_pages_bulk_array(gfp_t gfp, unsigned long nr_pages, struct page **page_array) -{ - return __alloc_pages_bulk(gfp, numa_mem_id(), NULL, nr_pages, NULL, page_array); -} +#define alloc_pages_bulk_array(_gfp, _nr_pages, _page_array) \ + __alloc_pages_bulk(_gfp, numa_mem_id(), NULL, _nr_pages, NULL, _page_array) static inline unsigned long -alloc_pages_bulk_array_node(gfp_t gfp, int nid, unsigned long nr_pages, struct page **page_array) +alloc_pages_bulk_array_node_noprof(gfp_t gfp, int nid, unsigned long nr_pages, + struct page **page_array) { if (nid == NUMA_NO_NODE) nid = numa_mem_id(); - return __alloc_pages_bulk(gfp, nid, NULL, nr_pages, NULL, page_array); + return alloc_pages_bulk_noprof(gfp, nid, NULL, nr_pages, NULL, page_array); } +#define alloc_pages_bulk_array_node(...) \ + alloc_hooks(alloc_pages_bulk_array_node_noprof(__VA_ARGS__)) + static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask) { gfp_t warn_gfp = gfp_mask & (__GFP_THISNODE|__GFP_NOWARN); @@ -230,82 +236,104 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask) * online. For more general interface, see alloc_pages_node(). */ static inline struct page * -__alloc_pages_node(int nid, gfp_t gfp_mask, unsigned int order) +__alloc_pages_node_noprof(int nid, gfp_t gfp_mask, unsigned int order) { VM_BUG_ON(nid < 0 || nid >= MAX_NUMNODES); warn_if_node_offline(nid, gfp_mask); - return __alloc_pages(gfp_mask, order, nid, NULL); + return __alloc_pages_noprof(gfp_mask, order, nid, NULL); } +#define __alloc_pages_node(...) alloc_hooks(__alloc_pages_node_noprof(__VA_ARGS__)) + static inline -struct folio *__folio_alloc_node(gfp_t gfp, unsigned int order, int nid) +struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid) { VM_BUG_ON(nid < 0 || nid >= MAX_NUMNODES); warn_if_node_offline(nid, gfp); - return __folio_alloc(gfp, order, nid, NULL); + return __folio_alloc_noprof(gfp, order, nid, NULL); } +#define __folio_alloc_node(...) alloc_hooks(__folio_alloc_node_noprof(__VA_ARGS__)) + /* * Allocate pages, preferring the node given as nid. When nid == NUMA_NO_NODE, * prefer the current CPU's closest node. Otherwise node must be valid and * online. */ -static inline struct page *alloc_pages_node(int nid, gfp_t gfp_mask, - unsigned int order) +static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask, + unsigned int order) { if (nid == NUMA_NO_NODE) nid = numa_mem_id(); - return __alloc_pages_node(nid, gfp_mask, order); + return __alloc_pages_node_noprof(nid, gfp_mask, order); } +#define alloc_pages_node(...) alloc_hooks(alloc_pages_node_noprof(__VA_ARGS__)) + #ifdef CONFIG_NUMA -struct page *alloc_pages(gfp_t gfp, unsigned int order); -struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, +struct page *alloc_pages_noprof(gfp_t gfp, unsigned int order); +struct page *alloc_pages_mpol_noprof(gfp_t gfp, unsigned int order, struct mempolicy *mpol, pgoff_t ilx, int nid); -struct folio *folio_alloc(gfp_t gfp, unsigned int order); -struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, +struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order); +struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order, struct vm_area_struct *vma, unsigned long addr, bool hugepage); #else -static inline struct page *alloc_pages(gfp_t gfp_mask, unsigned int order) +static inline struct page *alloc_pages_noprof(gfp_t gfp_mask, unsigned int order) { - return alloc_pages_node(numa_node_id(), gfp_mask, order); + return alloc_pages_node_noprof(numa_node_id(), gfp_mask, order); } -static inline struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, +static inline struct page *alloc_pages_mpol_noprof(gfp_t gfp, unsigned int order, struct mempolicy *mpol, pgoff_t ilx, int nid) { - return alloc_pages(gfp, order); + return alloc_pages_noprof(gfp, order); } -static inline struct folio *folio_alloc(gfp_t gfp, unsigned int order) +static inline struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order) { return __folio_alloc_node(gfp, order, numa_node_id()); } -#define vma_alloc_folio(gfp, order, vma, addr, hugepage) \ - folio_alloc(gfp, order) +#define vma_alloc_folio_noprof(gfp, order, vma, addr, hugepage) \ + folio_alloc_noprof(gfp, order) #endif + +#define alloc_pages(...) alloc_hooks(alloc_pages_noprof(__VA_ARGS__)) +#define alloc_pages_mpol(...) alloc_hooks(alloc_pages_mpol_noprof(__VA_ARGS__)) +#define folio_alloc(...) alloc_hooks(folio_alloc_noprof(__VA_ARGS__)) +#define vma_alloc_folio(...) alloc_hooks(vma_alloc_folio_noprof(__VA_ARGS__)) + #define alloc_page(gfp_mask) alloc_pages(gfp_mask, 0) -static inline struct page *alloc_page_vma(gfp_t gfp, + +static inline struct page *alloc_page_vma_noprof(gfp_t gfp, struct vm_area_struct *vma, unsigned long addr) { - struct folio *folio = vma_alloc_folio(gfp, 0, vma, addr, false); + struct folio *folio = vma_alloc_folio_noprof(gfp, 0, vma, addr, false); return &folio->page; } +#define alloc_page_vma(...) alloc_hooks(alloc_page_vma_noprof(__VA_ARGS__)) + +extern unsigned long get_free_pages_noprof(gfp_t gfp_mask, unsigned int order); +#define __get_free_pages(...) alloc_hooks(get_free_pages_noprof(__VA_ARGS__)) -extern unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order); -extern unsigned long get_zeroed_page(gfp_t gfp_mask); +extern unsigned long get_zeroed_page_noprof(gfp_t gfp_mask); +#define get_zeroed_page(...) alloc_hooks(get_zeroed_page_noprof(__VA_ARGS__)) + +void *alloc_pages_exact_noprof(size_t size, gfp_t gfp_mask) __alloc_size(1); +#define alloc_pages_exact(...) alloc_hooks(alloc_pages_exact_noprof(__VA_ARGS__)) -void *alloc_pages_exact(size_t size, gfp_t gfp_mask) __alloc_size(1); void free_pages_exact(void *virt, size_t size); -__meminit void *alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) __alloc_size(2); -#define __get_free_page(gfp_mask) \ - __get_free_pages((gfp_mask), 0) +__meminit void *alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_mask) __alloc_size(2); +#define alloc_pages_exact_nid(...) \ + alloc_hooks(alloc_pages_exact_nid_noprof(__VA_ARGS__)) + +#define __get_free_page(gfp_mask) \ + __get_free_pages((gfp_mask), 0) -#define __get_dma_pages(gfp_mask, order) \ - __get_free_pages((gfp_mask) | GFP_DMA, (order)) +#define __get_dma_pages(gfp_mask, order) \ + __get_free_pages((gfp_mask) | GFP_DMA, (order)) extern void __free_pages(struct page *page, unsigned int order); extern void free_pages(unsigned long addr, unsigned int order); @@ -374,10 +402,14 @@ extern gfp_t vma_thp_gfp_mask(struct vm_area_struct *vma); #ifdef CONFIG_CONTIG_ALLOC /* The below functions must be run on a range from a single zone. */ -extern int alloc_contig_range(unsigned long start, unsigned long end, +extern int alloc_contig_range_noprof(unsigned long start, unsigned long end, unsigned migratetype, gfp_t gfp_mask); -extern struct page *alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask, - int nid, nodemask_t *nodemask); +#define alloc_contig_range(...) alloc_hooks(alloc_contig_range_noprof(__VA_ARGS__)) + +extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask, + int nid, nodemask_t *nodemask); +#define alloc_contig_pages(...) alloc_hooks(alloc_contig_pages_noprof(__VA_ARGS__)) + #endif void free_contig_range(unsigned long pfn, unsigned long nr_pages); diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 2df35e65557d..35636e67e2e1 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -542,14 +542,17 @@ static inline void *detach_page_private(struct page *page) #endif #ifdef CONFIG_NUMA -struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order); +struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order); #else -static inline struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order) +static inline struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order) { - return folio_alloc(gfp, order); + return folio_alloc_noprof(gfp, order); } #endif +#define filemap_alloc_folio(...) \ + alloc_hooks(filemap_alloc_folio_noprof(__VA_ARGS__)) + static inline struct page *__page_cache_alloc(gfp_t gfp) { return &filemap_alloc_folio(gfp, 0)->page; diff --git a/mm/compaction.c b/mm/compaction.c index 807b58e6eb68..70b01190d2f3 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1851,7 +1851,7 @@ static void isolate_freepages(struct compact_control *cc) * This is a migrate-callback that "allocates" freepages by taking pages * from the isolated freelists in the block we are migrating to. */ -static struct folio *compaction_alloc(struct folio *src, unsigned long data) +static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long data) { struct compact_control *cc = (struct compact_control *)data; struct folio *dst; @@ -1898,6 +1898,11 @@ static struct folio *compaction_alloc(struct folio *src, unsigned long data) return page_rmappable_folio(&dst->page); } +static struct folio *compaction_alloc(struct folio *src, unsigned long data) +{ + return alloc_hooks(compaction_alloc_noprof(src, data)); +} + /* * This is a migrate-callback that "frees" freepages back to the isolated * freelist. All pages on the freelist are from the same zone, so there is no diff --git a/mm/filemap.c b/mm/filemap.c index a2ca78d13321..ba6d1ba60a2a 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -966,7 +966,7 @@ int filemap_add_folio(struct address_space *mapping, struct folio *folio, EXPORT_SYMBOL_GPL(filemap_add_folio); #ifdef CONFIG_NUMA -struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order) +struct folio *filemap_alloc_folio_noprof(gfp_t gfp, unsigned int order) { int n; struct folio *folio; @@ -981,9 +981,9 @@ struct folio *filemap_alloc_folio(gfp_t gfp, unsigned int order) return folio; } - return folio_alloc(gfp, order); + return folio_alloc_noprof(gfp, order); } -EXPORT_SYMBOL(filemap_alloc_folio); +EXPORT_SYMBOL(filemap_alloc_folio_noprof); #endif /* diff --git a/mm/mempolicy.c b/mm/mempolicy.c index c4d439efcadf..b270b8f8519a 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2201,15 +2201,15 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, */ preferred_gfp = gfp | __GFP_NOWARN; preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - page = __alloc_pages(preferred_gfp, order, nid, nodemask); + page = __alloc_pages_noprof(preferred_gfp, order, nid, nodemask); if (!page) - page = __alloc_pages(gfp, order, nid, NULL); + page = __alloc_pages_noprof(gfp, order, nid, NULL); return page; } /** - * alloc_pages_mpol - Allocate pages according to NUMA mempolicy. + * alloc_pages_mpol_noprof - Allocate pages according to NUMA mempolicy. * @gfp: GFP flags. * @order: Order of the page allocation. * @pol: Pointer to the NUMA mempolicy. @@ -2218,7 +2218,7 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, * * Return: The page on success or NULL if allocation fails. */ -struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, +struct page *alloc_pages_mpol_noprof(gfp_t gfp, unsigned int order, struct mempolicy *pol, pgoff_t ilx, int nid) { nodemask_t *nodemask; @@ -2249,7 +2249,7 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, * First, try to allocate THP only on local node, but * don't reclaim unnecessarily, just compact. */ - page = __alloc_pages_node(nid, + page = __alloc_pages_node_noprof(nid, gfp | __GFP_THISNODE | __GFP_NORETRY, order); if (page || !(gfp & __GFP_DIRECT_RECLAIM)) return page; @@ -2262,7 +2262,7 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, } } - page = __alloc_pages(gfp, order, nid, nodemask); + page = __alloc_pages_noprof(gfp, order, nid, nodemask); if (unlikely(pol->mode == MPOL_INTERLEAVE) && page) { /* skip NUMA_INTERLEAVE_HIT update if numa stats is disabled */ @@ -2278,7 +2278,7 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, } /** - * vma_alloc_folio - Allocate a folio for a VMA. + * vma_alloc_folio_noprof - Allocate a folio for a VMA. * @gfp: GFP flags. * @order: Order of the folio. * @vma: Pointer to VMA. @@ -2293,7 +2293,7 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, * * Return: The folio on success or NULL if allocation fails. */ -struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, +struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order, struct vm_area_struct *vma, unsigned long addr, bool hugepage) { struct mempolicy *pol; @@ -2301,15 +2301,15 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, struct page *page; pol = get_vma_policy(vma, addr, order, &ilx); - page = alloc_pages_mpol(gfp | __GFP_COMP, order, - pol, ilx, numa_node_id()); + page = alloc_pages_mpol_noprof(gfp | __GFP_COMP, order, + pol, ilx, numa_node_id()); mpol_cond_put(pol); return page_rmappable_folio(page); } -EXPORT_SYMBOL(vma_alloc_folio); +EXPORT_SYMBOL(vma_alloc_folio_noprof); /** - * alloc_pages - Allocate pages. + * alloc_pages_noprof - Allocate pages. * @gfp: GFP flags. * @order: Power of two of number of pages to allocate. * @@ -2322,7 +2322,7 @@ EXPORT_SYMBOL(vma_alloc_folio); * flags are used. * Return: The page on success or NULL if allocation fails. */ -struct page *alloc_pages(gfp_t gfp, unsigned int order) +struct page *alloc_pages_noprof(gfp_t gfp, unsigned int order) { struct mempolicy *pol = &default_policy; @@ -2333,16 +2333,16 @@ struct page *alloc_pages(gfp_t gfp, unsigned int order) if (!in_interrupt() && !(gfp & __GFP_THISNODE)) pol = get_task_policy(current); - return alloc_pages_mpol(gfp, order, - pol, NO_INTERLEAVE_INDEX, numa_node_id()); + return alloc_pages_mpol_noprof(gfp, order, pol, NO_INTERLEAVE_INDEX, + numa_node_id()); } -EXPORT_SYMBOL(alloc_pages); +EXPORT_SYMBOL(alloc_pages_noprof); -struct folio *folio_alloc(gfp_t gfp, unsigned int order) +struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order) { - return page_rmappable_folio(alloc_pages(gfp | __GFP_COMP, order)); + return page_rmappable_folio(alloc_pages_noprof(gfp | __GFP_COMP, order)); } -EXPORT_SYMBOL(folio_alloc); +EXPORT_SYMBOL(folio_alloc_noprof); static unsigned long alloc_pages_bulk_array_interleave(gfp_t gfp, struct mempolicy *pol, unsigned long nr_pages, @@ -2361,13 +2361,13 @@ static unsigned long alloc_pages_bulk_array_interleave(gfp_t gfp, for (i = 0; i < nodes; i++) { if (delta) { - nr_allocated = __alloc_pages_bulk(gfp, + nr_allocated = alloc_pages_bulk_noprof(gfp, interleave_nodes(pol), NULL, nr_pages_per_node + 1, NULL, page_array); delta--; } else { - nr_allocated = __alloc_pages_bulk(gfp, + nr_allocated = alloc_pages_bulk_noprof(gfp, interleave_nodes(pol), NULL, nr_pages_per_node, NULL, page_array); } @@ -2504,11 +2504,11 @@ static unsigned long alloc_pages_bulk_array_preferred_many(gfp_t gfp, int nid, preferred_gfp = gfp | __GFP_NOWARN; preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - nr_allocated = __alloc_pages_bulk(preferred_gfp, nid, &pol->nodes, + nr_allocated = alloc_pages_bulk_noprof(preferred_gfp, nid, &pol->nodes, nr_pages, NULL, page_array); if (nr_allocated < nr_pages) - nr_allocated += __alloc_pages_bulk(gfp, numa_node_id(), NULL, + nr_allocated += alloc_pages_bulk_noprof(gfp, numa_node_id(), NULL, nr_pages - nr_allocated, NULL, page_array + nr_allocated); return nr_allocated; @@ -2520,7 +2520,7 @@ static unsigned long alloc_pages_bulk_array_preferred_many(gfp_t gfp, int nid, * It can accelerate memory allocation especially interleaving * allocate memory. */ -unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, +unsigned long alloc_pages_bulk_array_mempolicy_noprof(gfp_t gfp, unsigned long nr_pages, struct page **page_array) { struct mempolicy *pol = &default_policy; @@ -2544,8 +2544,8 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, nid = numa_node_id(); nodemask = policy_nodemask(gfp, pol, NO_INTERLEAVE_INDEX, &nid); - return __alloc_pages_bulk(gfp, nid, nodemask, - nr_pages, NULL, page_array); + return alloc_pages_bulk_noprof(gfp, nid, nodemask, + nr_pages, NULL, page_array); } int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 48cdd25261ea..9c86ef2a0296 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4427,7 +4427,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, * * Returns the number of pages on the list or array. */ -unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, +unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, nodemask_t *nodemask, int nr_pages, struct list_head *page_list, struct page **page_array) @@ -4563,7 +4563,7 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, pcp_trylock_finish(UP_flags); failed: - page = __alloc_pages(gfp, 0, preferred_nid, nodemask); + page = __alloc_pages_noprof(gfp, 0, preferred_nid, nodemask); if (page) { if (page_list) list_add(&page->lru, page_list); @@ -4574,13 +4574,13 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, goto out; } -EXPORT_SYMBOL_GPL(__alloc_pages_bulk); +EXPORT_SYMBOL_GPL(alloc_pages_bulk_noprof); /* * This is the 'heart' of the zoned buddy allocator. */ -struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, - nodemask_t *nodemask) +struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, + int preferred_nid, nodemask_t *nodemask) { struct page *page; unsigned int alloc_flags = ALLOC_WMARK_LOW; @@ -4642,38 +4642,38 @@ struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, return page; } -EXPORT_SYMBOL(__alloc_pages); +EXPORT_SYMBOL(__alloc_pages_noprof); -struct folio *__folio_alloc(gfp_t gfp, unsigned int order, int preferred_nid, +struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask) { - struct page *page = __alloc_pages(gfp | __GFP_COMP, order, + struct page *page = __alloc_pages_noprof(gfp | __GFP_COMP, order, preferred_nid, nodemask); return page_rmappable_folio(page); } -EXPORT_SYMBOL(__folio_alloc); +EXPORT_SYMBOL(__folio_alloc_noprof); /* * Common helper functions. Never use with __GFP_HIGHMEM because the returned * address cannot represent highmem pages. Use alloc_pages and then kmap if * you need to access high mem. */ -unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order) +unsigned long get_free_pages_noprof(gfp_t gfp_mask, unsigned int order) { struct page *page; - page = alloc_pages(gfp_mask & ~__GFP_HIGHMEM, order); + page = alloc_pages_noprof(gfp_mask & ~__GFP_HIGHMEM, order); if (!page) return 0; return (unsigned long) page_address(page); } -EXPORT_SYMBOL(__get_free_pages); +EXPORT_SYMBOL(get_free_pages_noprof); -unsigned long get_zeroed_page(gfp_t gfp_mask) +unsigned long get_zeroed_page_noprof(gfp_t gfp_mask) { - return __get_free_page(gfp_mask | __GFP_ZERO); + return get_free_pages_noprof(gfp_mask | __GFP_ZERO, 0); } -EXPORT_SYMBOL(get_zeroed_page); +EXPORT_SYMBOL(get_zeroed_page_noprof); /** * __free_pages - Free pages allocated with alloc_pages(). @@ -4875,7 +4875,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, } /** - * alloc_pages_exact - allocate an exact number physically-contiguous pages. + * alloc_pages_exact_noprof - allocate an exact number physically-contiguous pages. * @size: the number of bytes to allocate * @gfp_mask: GFP flags for the allocation, must not contain __GFP_COMP * @@ -4889,7 +4889,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, * * Return: pointer to the allocated area or %NULL in case of error. */ -void *alloc_pages_exact(size_t size, gfp_t gfp_mask) +void *alloc_pages_exact_noprof(size_t size, gfp_t gfp_mask) { unsigned int order = get_order(size); unsigned long addr; @@ -4897,13 +4897,13 @@ void *alloc_pages_exact(size_t size, gfp_t gfp_mask) if (WARN_ON_ONCE(gfp_mask & (__GFP_COMP | __GFP_HIGHMEM))) gfp_mask &= ~(__GFP_COMP | __GFP_HIGHMEM); - addr = __get_free_pages(gfp_mask, order); + addr = get_free_pages_noprof(gfp_mask, order); return make_alloc_exact(addr, order, size); } -EXPORT_SYMBOL(alloc_pages_exact); +EXPORT_SYMBOL(alloc_pages_exact_noprof); /** - * alloc_pages_exact_nid - allocate an exact number of physically-contiguous + * alloc_pages_exact_nid_noprof - allocate an exact number of physically-contiguous * pages on a node. * @nid: the preferred node ID where memory should be allocated * @size: the number of bytes to allocate @@ -4914,7 +4914,7 @@ EXPORT_SYMBOL(alloc_pages_exact); * * Return: pointer to the allocated area or %NULL in case of error. */ -void * __meminit alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) +void * __meminit alloc_pages_exact_nid_noprof(int nid, size_t size, gfp_t gfp_mask) { unsigned int order = get_order(size); struct page *p; @@ -4922,7 +4922,7 @@ void * __meminit alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) if (WARN_ON_ONCE(gfp_mask & (__GFP_COMP | __GFP_HIGHMEM))) gfp_mask &= ~(__GFP_COMP | __GFP_HIGHMEM); - p = alloc_pages_node(nid, gfp_mask, order); + p = alloc_pages_node_noprof(nid, gfp_mask, order); if (!p) return NULL; return make_alloc_exact((unsigned long)page_address(p), order, size); @@ -6359,7 +6359,7 @@ int __alloc_contig_migrate_range(struct compact_control *cc, } /** - * alloc_contig_range() -- tries to allocate given range of pages + * alloc_contig_range_noprof() -- tries to allocate given range of pages * @start: start PFN to allocate * @end: one-past-the-last PFN to allocate * @migratetype: migratetype of the underlying pageblocks (either @@ -6379,7 +6379,7 @@ int __alloc_contig_migrate_range(struct compact_control *cc, * pages which PFN is in [start, end) are allocated for the caller and * need to be freed with free_contig_range(). */ -int alloc_contig_range(unsigned long start, unsigned long end, +int alloc_contig_range_noprof(unsigned long start, unsigned long end, unsigned migratetype, gfp_t gfp_mask) { unsigned long outer_start, outer_end; @@ -6503,15 +6503,15 @@ int alloc_contig_range(unsigned long start, unsigned long end, undo_isolate_page_range(start, end, migratetype); return ret; } -EXPORT_SYMBOL(alloc_contig_range); +EXPORT_SYMBOL(alloc_contig_range_noprof); static int __alloc_contig_pages(unsigned long start_pfn, unsigned long nr_pages, gfp_t gfp_mask) { unsigned long end_pfn = start_pfn + nr_pages; - return alloc_contig_range(start_pfn, end_pfn, MIGRATE_MOVABLE, - gfp_mask); + return alloc_contig_range_noprof(start_pfn, end_pfn, MIGRATE_MOVABLE, + gfp_mask); } static bool pfn_range_valid_contig(struct zone *z, unsigned long start_pfn, @@ -6546,7 +6546,7 @@ static bool zone_spans_last_pfn(const struct zone *zone, } /** - * alloc_contig_pages() -- tries to find and allocate contiguous range of pages + * alloc_contig_pages_noprof() -- tries to find and allocate contiguous range of pages * @nr_pages: Number of contiguous pages to allocate * @gfp_mask: GFP mask to limit search and used during compaction * @nid: Target node @@ -6566,8 +6566,8 @@ static bool zone_spans_last_pfn(const struct zone *zone, * * Return: pointer to contiguous pages on success, or NULL if not successful. */ -struct page *alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask, - int nid, nodemask_t *nodemask) +struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask, + int nid, nodemask_t *nodemask) { unsigned long ret, pfn, flags; struct zonelist *zonelist; From patchwork Thu Mar 21 16:36:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599176 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27939CD11C2 for ; Thu, 21 Mar 2024 16:38:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CA0696B00B0; Thu, 21 Mar 2024 12:37:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C4F616B00B1; Thu, 21 Mar 2024 12:37:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A51EB6B00B2; Thu, 21 Mar 2024 12:37:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 8A7636B00B0 for ; Thu, 21 Mar 2024 12:37:54 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 387BBC01ED for ; Thu, 21 Mar 2024 16:37:54 +0000 (UTC) X-FDA: 81921602868.17.4492CD5 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf16.hostedemail.com (Postfix) with ESMTP id 92A3518000F for ; Thu, 21 Mar 2024 16:37:52 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ErEUPVvl; spf=pass (imf16.hostedemail.com: domain of 3X2L8ZQYKCEw685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3X2L8ZQYKCEw685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039072; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VNGm+8y6Ss+mlVcFwBeCqCAgvw3+NOVbq/tJOyMa3ww=; b=4/15QAj2hK73Uu7SsNt+IHQ1eWYjJPcu6vQ1Z9T79db+C17bL2Vkrj0MvN6+IXr9lLae5p 4/ptjMd462wyxX5F1ha+xyDW+vWTU7T07Qnsc+HO7J/wbezI9/0cWMnywKpiULg3rDsrcp KnVB4fFnGqM54itqPmznu+4TlDX303M= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039072; a=rsa-sha256; cv=none; b=GFsgv1MOjKA1+kv9lh3m0Ue3AvbcO/aUKvfT0urgVjrAokNRzl1XPh6nkX/W7mXFovKFtx 9qGcxR70w+TO2wkELcXLfj7pvRma6zM/6aohmegl1Czn9Ls/sjevwRh3U3V8K9Lr/feMc1 lmwmuWgaCx1bLZD9X/KhFVAKt9NZxcg= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ErEUPVvl; spf=pass (imf16.hostedemail.com: domain of 3X2L8ZQYKCEw685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3X2L8ZQYKCEw685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dcbee93a3e1so1860765276.3 for ; Thu, 21 Mar 2024 09:37:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039071; x=1711643871; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VNGm+8y6Ss+mlVcFwBeCqCAgvw3+NOVbq/tJOyMa3ww=; b=ErEUPVvl3oLXhsZkpm8au07HJNeO9uBhEK6OuDKX66gMbfrGvO0OmDAgAdaLfbnK+x 0Ed1nP8xBAcmpuries1V2ePle7QN2DWZR6mQoaEbFTLX/8GjKaqhftspQ2pxbLVFMGZ9 1QFm35+QKP/8ShIBYhPEbXJICUULU2+BCvtkmmc1+jamtG3VP/GjSXiF4QHyU3DDIt1+ lNQBY4sKe5zi/YSRpZuV2xOBDaqfMji8ashAj9kDNkYxapYJOZ0qJdNx+EbE97j6z5cB UnGH+bS4EF4sUmN/RMpjNU7TdHvjgcQmyrfVq28GKbaXpYC3HL8GWndrgzAuBlxqZF7T g24g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039071; x=1711643871; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VNGm+8y6Ss+mlVcFwBeCqCAgvw3+NOVbq/tJOyMa3ww=; b=WCwYGGJfaojcmk8mINMQI+KOJ4StxSPMDHERmfh1AIA1PTP1pwao8oU6yL2WThMlwE +i9F4od+BvYiDrlQEHz3lZ5jwxD4iBFB4XJ7LF9Wotmwut5ydR51ACuumk3D7PIzX7BC iovWaO/XFFK2bgchtrVx7YSICS3jYb9q3s8vpmbg0BPs8ZlBHTd6svq/piskf2R8Wx5W +Fzn2h5/BH0r/9NhPR2hKU/EqEv/WKYbwfInwg87jgx4IiKRlAljlxrP5oyT0HXPGOFb KIDxNzhc/vfUTiUFjXYmhnEyTpko3c364Nz+2mp/3jH1JdC1/k0gQVOod8O966mrkSRY 0bpg== X-Forwarded-Encrypted: i=1; AJvYcCXkRxxgeDTdzztM1vrACjNIu4K/pH9BtAjgypFPolPb9hSih5iDTV/PdWuhqaBTId9kpi0hbd2lISYi9uPD4xV9EKg= X-Gm-Message-State: AOJu0YySUEDZp2gfmxcFgubw479R87T1OdyQjiVlpd2SZ4d3Da1on4jH 2Mt3HxFESfcJSZrIaMO2nEs1KwP+NGi+ZH5dO/9jvkC7PEOSmYktnps92G3OpQKfRvadt8afK/P zSA== X-Google-Smtp-Source: AGHT+IFHuXgNI5SVUJASc8Fg628bHOvpHsCOaQtXol/znxQ1foKXK9QPyvtw6ludSGd9C7o69Hlt2+8i3RA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:1004:b0:dc6:44d4:bee0 with SMTP id w4-20020a056902100400b00dc644d4bee0mr1149298ybt.7.1711039071427; Thu, 21 Mar 2024 09:37:51 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:41 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-20-surenb@google.com> Subject: [PATCH v6 19/37] mm: create new codetag references during page splitting From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 92A3518000F X-Rspam-User: X-Stat-Signature: fwnfweiojia5s7ygjj6rjnbxg93m7yqt X-Rspamd-Server: rspam03 X-HE-Tag: 1711039072-691852 X-HE-Meta: U2FsdGVkX18HhLita1f2z1FwE+QH/ittQGZpF5cnYVqkAQSR1v/Rw+myoJqKMCQtYO3UT7btr8ozEBazmdgOTOS/scadB1OUb4XoBt1iBOIdSaLkPApDF1jzIYNt8sScCs2O/5y5fmlGoREQhUla1lq0hdamvQqu31NpVwFEfBe7koFx6t4Zr3BSnVbYWpMgSxBKxK+TyZF80pmLIKE2VoTBiohSxci6jlRGXE+uK/PvecsfKourwIj/J0+bdLasKnjuhjCvqK7FIvdd5lWHpSdl+OGMykrTWV72/pSY6dtH+3SmrOPOyfCySuPRtqevEbmD7FO+dWE/j3s7OqzrRGUZD2o0NKeyniw1hDvr4Kn+dtjVD/dEnAykusZfHZFDYMPGQlU0ZDtX1cksSQ2G/YZGJXcD4DHVmH+7Qr0MwsW+0IfQdfugjjdzKhvPlBhDe4a/kxKs4RK8Yk0+oxwh76VJzb9fIFhUTllGc1lNC7fz7n4J30Hs2e3LhPhuTAfGitO9N/QMPwL2wJfRhRjfNeC9jR3TapC7xS9X1A3cPPKsmsrtdyBwfVi25+L2iW7BcFwyN2FZv9Oz5trF0UTob+UUdkCfwG9A+u3e53ZhuWiKEdqFX3Foz7xTdboNFUMmnrdBhxfx2m/WL0jqVlqfboyRrvW9uGNKKQdO4Khs4TP99DMUlgUiOT4tXiD7By2yJBYoMgUi+V2H7nvEmKALCR/6A1tQa0TMfn1OBoGzNQx0igxszURjFJKYxZk3FefXz9bbVQjsqIVL6aE0hyH+m+80tXA5bsWqQO7MlZxOu5UcfO3eV8ratLMkun1PBxzNvGQosICYOARgPBHKTInZ+67yXAm/+JIK0Md6pZpxuzzUoeGU4GHX4L2ShafojDFFw8gYJfZljJzzV5RP4YqTU8aDghqA0m+AhC/zDOv7J98TigQ2BnygMJ+4hYe9kUdEFotAlgEpBkBDyTCe1vX 7rjn2/C7 IdER9RG3g8ZTmudJ5j4fqpnnqKxyeggd57ef9w+DEQ/Zx0z9GKAzW91sXcPQ7wbnpYu3vaslCcBvtDUsY1AeYeDOdeYtYkbzC6PS3abtIjjVtWhVDA36ohGJtYKnTpdCmYYxOsBl5WhsRrgkHsRAau8+Gn6smLGudxFxYT+0XjfLQuJIy42ZwFJeCLYJzc6wkA/mNJwAy56/KHV0Pt7WR5K8yI5h7YIcu1uiqNIBqrAVqTprZIeJwPzBzXNalnpenIJl42DUvYogwHYyoOyEjT76Q+Zqb8CXIsqUfGYTTtbax8Ke5Qa70wNCw9A5nAzwQJDr37jPumN4J0dlKpufiXlwvzns3Z2UKXTyRU491ETmZnkRYexnXBO5v9JEzc1Gt5fUFZvQYZ+HhZ+mPl1s8ss9y30sMtusyWHfAS5hC9Ausl7A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a high-order page is split into smaller ones, each newly split page should get its codetag. After the split each split page will be referencing the original codetag. The codetag's "bytes" counter remains the same because the amount of allocated memory has not changed, however the "calls" counter gets increased to keep the counter correct when these individual pages get freed. Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka --- include/linux/alloc_tag.h | 9 +++++++++ include/linux/pgalloc_tag.h | 30 ++++++++++++++++++++++++++++++ mm/huge_memory.c | 2 ++ mm/page_alloc.c | 2 ++ 4 files changed, 43 insertions(+) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 28c0005edae1..bc9b1b99a55b 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -106,6 +106,15 @@ static inline void __alloc_tag_ref_set(union codetag_ref *ref, struct alloc_tag this_cpu_inc(tag->counters->calls); } +static inline void alloc_tag_ref_set(union codetag_ref *ref, struct alloc_tag *tag) +{ + alloc_tag_add_check(ref, tag); + if (!ref || !tag) + return; + + __alloc_tag_ref_set(ref, tag); +} + static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, size_t bytes) { alloc_tag_add_check(ref, tag); diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 66bd021eb46e..093edf98c3d7 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -67,11 +67,41 @@ static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) } } +static inline void pgalloc_tag_split(struct page *page, unsigned int nr) +{ + int i; + struct page_ext *page_ext; + union codetag_ref *ref; + struct alloc_tag *tag; + + if (!mem_alloc_profiling_enabled()) + return; + + page_ext = page_ext_get(page); + if (unlikely(!page_ext)) + return; + + ref = codetag_ref_from_page_ext(page_ext); + if (!ref->ct) + goto out; + + tag = ct_to_alloc_tag(ref->ct); + page_ext = page_ext_next(page_ext); + for (i = 1; i < nr; i++) { + /* Set new reference to point to the original tag */ + alloc_tag_ref_set(codetag_ref_from_page_ext(page_ext), tag); + page_ext = page_ext_next(page_ext); + } +out: + page_ext_put(page_ext); +} + #else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int nr) {} static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) {} +static inline void pgalloc_tag_split(struct page *page, unsigned int nr) {} #endif /* CONFIG_MEM_ALLOC_PROFILING */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c77cedf45f3a..b29f9ef0fcb2 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -38,6 +38,7 @@ #include #include #include +#include #include #include @@ -2924,6 +2925,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, /* Caller disabled irqs, so they are still disabled here */ split_page_owner(head, order, new_order); + pgalloc_tag_split(head, 1 << order); /* See comment in __split_huge_page_tail() */ if (folio_test_anon(folio)) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 9c86ef2a0296..fd1cc5b80a56 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2666,6 +2666,7 @@ void split_page(struct page *page, unsigned int order) for (i = 1; i < (1 << order); i++) set_page_refcounted(page + i); split_page_owner(page, order, 0); + pgalloc_tag_split(page, 1 << order); split_page_memcg(page, order, 0); } EXPORT_SYMBOL_GPL(split_page); @@ -4863,6 +4864,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, struct page *last = page + nr; split_page_owner(page, order, 0); + pgalloc_tag_split(page, 1 << order); split_page_memcg(page, order, 0); while (page < --last) set_page_refcounted(last); From patchwork Thu Mar 21 16:36:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40922C54E58 for ; Thu, 21 Mar 2024 16:38:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EEAC46B00B1; Thu, 21 Mar 2024 12:37:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E989B6B00B2; Thu, 21 Mar 2024 12:37:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC38A6B00B3; Thu, 21 Mar 2024 12:37:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AA33F6B00B1 for ; Thu, 21 Mar 2024 12:37:56 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 7EE2B14048C for ; Thu, 21 Mar 2024 16:37:56 +0000 (UTC) X-FDA: 81921602952.22.93EF7E4 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf15.hostedemail.com (Postfix) with ESMTP id 75784A0034 for ; Thu, 21 Mar 2024 16:37:54 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=wTEecQ2c; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of 3YWL8ZQYKCE48A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3YWL8ZQYKCE48A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039074; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/zyX0GCVbwAVqAn+thP4TQZbJ1DZ8xPrkSKdxK+srAs=; b=l/zCb4yVpttw8hqL62QxWhfDiN4/bmPmSQr5IDzXSjU/3DOOeO2U0KIIsdOiK2y41dHHKe F2FW6lmaVNwsAUVidtAbdh9wP8HGsABvjvheuyPXMJuvBlsy5r1hmPj/4uKwWGURqodfgT TQF9JHen0q5KOhPji0aVOPMCmsFFS50= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=wTEecQ2c; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of 3YWL8ZQYKCE48A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3YWL8ZQYKCE48A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039074; a=rsa-sha256; cv=none; b=qkEsbKCCrhW++ZPwB3FgtoICzPs6v4e+s3kMk+iyWvceap+PgOFHc2aKTgBEx4ej8MIKEF /XuDi/46ZC9Y0aWhV5L2en23T4mcPFdGrwwdHBUMIa50KVq+TVbJC/dRtETCBzsV7kMQJd IFqpryNIsGpwKPmJDcmTmrra+8jZMbk= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcc15b03287so1685963276.3 for ; Thu, 21 Mar 2024 09:37:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039073; x=1711643873; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/zyX0GCVbwAVqAn+thP4TQZbJ1DZ8xPrkSKdxK+srAs=; b=wTEecQ2cDrHQxDnAb+Tf7sKFoqd9o90gnp7F6AYfrfXFAD+vJat3N0KOR02/V3fpIg xeS+2wL5b+0aotpmK4QmHwJSsmYAoA1MMaD8DxAW5eAOwtrX/ppBzkLFnThEP0TJCaWF g60lV+zfdidOd8iayxqOGNaxrijW3vHIYCrO7Wl/FP6w4Z0x5C+dKnHKwHG69mCIZu3i /KhH17a3uSjTxHLjo1rNI/tMO8qDHBob6f5ozkqpfys0H5yZU4cqVjTM4+Kbu+fIUhFF 1dlKLHqNCGNkBuu/ZZSMMAas1ORxxaS8AkdsTHDKRYELcN1H117q8dxTO9ZsjqyIVHZw Wx2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039073; x=1711643873; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/zyX0GCVbwAVqAn+thP4TQZbJ1DZ8xPrkSKdxK+srAs=; b=AdEan5N1r/yZ3D6yBcoycIabRhQ8p9j4fHiC7vPZCoT3qHWEywV97RtrCnjM2877gT 3KIl08lqDx3RSjYFYwg7+OvrGZBAsq4m+iz5fGzPkD3TtIbhCYUrItND3hsDtUEf5/AT nnLQnR4DhduOtJasM5GQFAgtIhznVz8y8MTJOl92j/Izx6OE48Stg7w1BqAW33v83t7j tR6jV0MDScnrhr8zPXL5BYRVrCE02FMC3cIL3mhlawcJ8kZj8fRbZEKNBhCuBCeYeTev hxkSXEyXnGcW76GZTe4m6FXd8hLIoGqR4mPPXDWb4tK9dgnVKrj89HMqlbsf2QgiYlBy a4Ag== X-Forwarded-Encrypted: i=1; AJvYcCUCgZ7BOfKTZuSvKLDR7NPgyOPEtgF5t6aqKO7i2qf8nLewjOV1TGH1i6UNZAbWY8b962G73IT2G9xQR4sdFeMaXuE= X-Gm-Message-State: AOJu0Yzs8ICxyzdigJ5T7IF/tzRtH9I+kUTcsEpaX93VGMkciBl8/vyc 2ETuEbButjcIf9qdnYyIUHQZjbnaGLAMZk1od0qspbhw2LUFqgFVsZMTdFpoaU+Ty8lBAE3ivrz dhQ== X-Google-Smtp-Source: AGHT+IGl7/sqfjKOff0ZjNp0/9geWcByH3EbVIixPqUdOWI6lFEA8KUqGJXoRB+ynXILrSpbs71KJ/Vx4lo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:218c:b0:dcc:4785:b51e with SMTP id dl12-20020a056902218c00b00dcc4785b51emr980431ybb.12.1711039073381; Thu, 21 Mar 2024 09:37:53 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:42 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-21-surenb@google.com> Subject: [PATCH v6 20/37] mm: fix non-compound multi-order memory accounting in __free_pages From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 75784A0034 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 79yo6c3xsetqdgxy6erd7aeky1u3esg7 X-HE-Tag: 1711039074-364218 X-HE-Meta: U2FsdGVkX19xSHKrwuxbZ8L6rz4UXLebdqwtj5kTeiUSuACJe38TKQ0TPLCPom37DZNXw/bFKePurMYgTvAlrdeg9NteSmgi7n6Drilb344F7Mw0SYebfSbZSzlo6rN2k85SLysDijUS9X/YRtKAheMZLBUyPy8wTiAK6/LtVqIjMtWD7SImzVjCdj4mJ9AUA0cNChroTFlhmZktEzX6IT7+d0WHOUNWOda9g2Q/aPJLpiiq4EPAxsDhVCneZNz+LKKy+2Cyn1kTNSxqivzz80uE2wBIbkrBI0SNs3yZE04jEEDvL4ZH3dLXuXAmi5RV9MffQqlrZQvD7eFpW3rrP3rXG5W1bT/NlXWjXpRnYomQdvVCTZ8dGHyeiqOXC/TuqJQq7VIJPmH+zAoSgnCzj1aBXIWqnvpaN8jBsMsfKvgwTlj9WAA1yUMRSrLpsPekZwFcxabIbFfM0/uJcOhZJmK/ifED6fKqPzqLkwWIxWXmyWZLftrXHYilf8maBvSmb3NmxsjYVyGyNfhy81SMUW9+SZxtfF3Km0/UBmJdS/tYUUqNrb2A7PdMcGzO1dQLO9uXUB1A4lDMe34XjYBxVS3BXqgOhE6ycr1nS5Vr6z7zDKppfhK2syuAc3U4USU/Sudh3JGhMlaUUDDGfoYwRWzOPJhxGpyOzr/FpGQ2KlB8fXclKmBkYJhzSt3oF6uLrIY7fG59Wg6Nf5J/elpZDeiCuQQrcroyRBIglea05gZtV9M5QYRtoupLEX02IMDPImb+Vsc83OQICaV5x+JFk4GRMSCBTGgQDmOxTTEXMHlubyKM3jjbpNOCma0N07tvpXfBBOdetAlmrroZARf46d9F5tnRA1rQpIaHcwAfyErXZo747c2f7tyZl+j9KtC9W74LJnU/SAm0sQlrbPBr8xIHlnQ2RAgD0JCIZCYdYC5Or5CtP+Zd0/PxGDOjfnmjV7r8wuD2XmXdA730uHU j9LVhcSH R+yR7Dqhhqy/ya8mvXDRyFHWwT0F1IONqVPItZrbp8CzwXganoYyEQj/MhjmzgMoB35GRSdfFCzkeUwCI/HLPRo2ig/Gey1X9lIQSYlDXVYrEHKfgdcYaHpVHqPFJsW5i60vmar+MPAezVi5cixN8C4FaIpK9m3DLmoyDlozF5qSOd1gph8IjUd3vBKJNg3AOpQaCSkb2zvFFS6V7CN4vtEm1BoSuIDbNHLCzqKQLudTz0b7GJX+tWYOv94yB+7JZC2u+YQt3N3W+EAj5XmxO/B/iouTuLcJ0Osw5UO2dpTDN1U7nQCZ9+fbXJqfeFKMT6oYz6fyXTCSsUHh7/p9i0hWflljm1PVc1C8xQKS3VqaQAQiZil6E/SJpQ5xFRwVkOTHigzMVRJ8IU+y6A+LNfvQwLcYUyn+2bWjLko1/0vmvLWJ3mwXD7mmCTA23zPfveZ4riqiGOZ7TbuisKRW4YcsBOLE4WRo4gW+MgxxQlm4gnPZnAAuh8DkO/KpX09+hL0WDBnMYOpcYeVw2fP3bXQTw7+aZluqsTQ+BqwM5M7biDzdHTQZWThnBGAmhfOX2CZ9hwCe/iRL2QHHNCcdAObIX1fxGYQHDH//CQyARcVpuv0UkPCBcr12yXIzutoyO9GR7j3KEUWSE+civk427GpN0rOsv49s/FkQ/lS1xEH9LjF8iub/xD+0miLKDGuM9DmdNIPmjrwOJKks/trWuwI7NqQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a non-compound multi-order page is freed, it is possible that a speculative reference keeps the page pinned. In this case we free all pages except for the first page, which will be freed later by the last put_page(). However the page passed to put_page() is indistinguishable from an order-0 page, so it cannot do the accounting, just as it cannot free the subsequent pages. Do the accounting here, where we free the pages. Reported-by: Vlastimil Babka Signed-off-by: Suren Baghdasaryan --- include/linux/pgalloc_tag.h | 24 ++++++++++++++++++++++++ mm/page_alloc.c | 5 ++++- 2 files changed, 28 insertions(+), 1 deletion(-) diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 093edf98c3d7..50d212330bbb 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -96,12 +96,36 @@ static inline void pgalloc_tag_split(struct page *page, unsigned int nr) page_ext_put(page_ext); } +static inline struct alloc_tag *pgalloc_tag_get(struct page *page) +{ + struct alloc_tag *tag = NULL; + + if (mem_alloc_profiling_enabled()) { + union codetag_ref *ref = get_page_tag_ref(page); + + alloc_tag_sub_check(ref); + if (ref && ref->ct) + tag = ct_to_alloc_tag(ref->ct); + put_page_tag_ref(ref); + } + + return tag; +} + +static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) +{ + if (mem_alloc_profiling_enabled() && tag) + this_cpu_sub(tag->counters->bytes, PAGE_SIZE * nr); +} + #else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int nr) {} static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) {} static inline void pgalloc_tag_split(struct page *page, unsigned int nr) {} +static inline struct alloc_tag *pgalloc_tag_get(struct page *page) { return NULL; } +static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) {} #endif /* CONFIG_MEM_ALLOC_PROFILING */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index fd1cc5b80a56..00e0ae4cbf2d 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4700,12 +4700,15 @@ void __free_pages(struct page *page, unsigned int order) { /* get PageHead before we drop reference */ int head = PageHead(page); + struct alloc_tag *tag = pgalloc_tag_get(page); if (put_page_testzero(page)) free_the_page(page, order); - else if (!head) + else if (!head) { + pgalloc_tag_sub_pages(tag, (1 << order) - 1); while (order-- > 0) free_the_page(page + (1 << order), order); + } } EXPORT_SYMBOL(__free_pages); From patchwork Thu Mar 21 16:36:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599178 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62C16C6FD1F for ; Thu, 21 Mar 2024 16:38:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D61666B00B3; Thu, 21 Mar 2024 12:37:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CEC206B00B4; Thu, 21 Mar 2024 12:37:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AF8596B00B5; Thu, 21 Mar 2024 12:37:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 976116B00B3 for ; Thu, 21 Mar 2024 12:37:58 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 6D1F5120499 for ; Thu, 21 Mar 2024 16:37:58 +0000 (UTC) X-FDA: 81921603036.01.16B62AD Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf26.hostedemail.com (Postfix) with ESMTP id 8C9BD14001D for ; Thu, 21 Mar 2024 16:37:56 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=3EjzCEva; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3Y2L8ZQYKCFAAC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Y2L8ZQYKCFAAC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039076; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=l0NoLpdPXZUI9jv+GBaFWeUdE0u/LS8GwKny2YwhxYE=; b=a4CLVehOF2RsTEzFdzo3rKjnZXsTMWukhx+tepBFEB/TBBzb2SraGUMkkb7Y6I4+wSO15f i0tNpsGvrY3WlBA0FwKXuyZf1+Zz7dss3iPmSLm1u2CGw44yRMCXv69hesplLuyqsbfGj/ spjUh4fTglFCbP00vBMGcATBaeYxNvM= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=3EjzCEva; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3Y2L8ZQYKCFAAC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Y2L8ZQYKCFAAC9w5ty66y3w.u64305CF-442Dsu2.69y@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039076; a=rsa-sha256; cv=none; b=a83DlxaDJXWOCl/MtpIJ0iUs2W0pi/Qr+PFb8vju+F0H7CU8czFaIwRR6TO/c4hwYr5Pr4 wzfRe0V4ApQU/Dl3tbERNVgeTc3jbze7QXj8wALu3ccCAKke64B69wBxGT+WntdcIFTF07 SIX8eccCcWY+/Uou3ypUjii2IsNfmno= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-607838c0800so24232527b3.1 for ; Thu, 21 Mar 2024 09:37:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039075; x=1711643875; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=l0NoLpdPXZUI9jv+GBaFWeUdE0u/LS8GwKny2YwhxYE=; b=3EjzCEvacTZm5kEC1Me3Sw0wMSSjtj+KkTVUIeStaB+c7nNKelXTc8HfZTvd4qnUJr WSEUhcpq/chECrXgpoYrpoDYsjmScdxxK0Yf3MgAMfnOwXO1hOEyAzjT7AqERPASkU0R x+cTVPp5ZPuvt47Z4747Qh7m367718ToncDkgnJw0q2U+259DpC85qFH+WHDuNWv7LYX PL1dlflIEeYmJryrt0gC28F55ua6+Fh6PGLRLiRwpYizi8gRI5FedGuuwnLWZUK/M7nN DwEPFsPLnyUCR6g3kriGEDiYaZP6FAnTqkWvGyd7UpHB3PbFw3S+RzsHGKfVNXrNbDSF YrSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039075; x=1711643875; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=l0NoLpdPXZUI9jv+GBaFWeUdE0u/LS8GwKny2YwhxYE=; b=HUHWQGN6Mn4qIk5nWn4MdI1qPcGgj65gIVRppsY/fBM1kvvCIvPKTfsRSSGfJ/SwTZ CKHiBPzfF2WWEabO1xuND9mnPeWd41yhmLQKFq0FA3vH747r16VKjV6h2p0wNBOeqyTw XAmnSQtkBkgcKqiqzHQEoEiMIdmi1j/En3CKxgEFUge1IGQPqHjd6D+BWXNixCuzKJZ0 ho4nxAounGCQsxPcg0ltZ9pYzTGOr3XXQHYIjoYip2twdhuzB54Q8eVH22Z4V00+zuwo LbpS1KepmVOsKdm+kfRsdP9ULfVUSgzcp/X+G+qOcXc5sEbDgL/EAX54n+gY2a4sQsin JKfQ== X-Forwarded-Encrypted: i=1; AJvYcCUQBbwKxo850zkuuXp5PyH9jGzfwbrzNZdyO4ywSUDrMKeimLXimF5FOCv0dn2KBOxEjlEOz+yGeKIGsECWo2YHEa4= X-Gm-Message-State: AOJu0YzLeCGrXNDIHQau7KWysZwxK8zzoVyWZ/8NvIianoAJNw+TNjqS sbamJivvq8Pqmyb6zmnlY3upn+5AkLV/thJ4QoNGyhxB3qZHcRicQqUs1BeHlMjAjk+bN7YnN7Z yvw== X-Google-Smtp-Source: AGHT+IEretF+q2ZgfU/nK2U7IsM8qgxzJOcM6j0PHFTxTRvud7ewavgn/Hqbg7folhUnncWwNi0CsjWDS2Y= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a25:ce51:0:b0:dc7:4ca0:cbf0 with SMTP id x78-20020a25ce51000000b00dc74ca0cbf0mr607569ybe.3.1711039075564; Thu, 21 Mar 2024 09:37:55 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:43 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-22-surenb@google.com> Subject: [PATCH v6 21/37] mm/page_ext: enable early_page_ext when CONFIG_MEM_ALLOC_PROFILING_DEBUG=y From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 8C9BD14001D X-Stat-Signature: 65uzpfy8hwkcxqmcqywbpn5getmk8if3 X-HE-Tag: 1711039076-432134 X-HE-Meta: U2FsdGVkX19v01D4kBAkR+ZOy/Dm4gDYNf3mzbOl4OQ5KA2uGYcRUp2AwfjZdvM6mk9VFdMzrvbvWGcPbEp4p0mkJ5OljWrmm1ed7b86rhzJsG8X0SeO2Dit9A5KJjroHALhzY2D6wFct3kn2QKb9FCsMnNv6dAZCsvP76BnznJS24D29IETTWPNwICq4DRMllT7Yk6yXjY9xrxBNWVOfp3WXqBmYmR9czPLVymJSI1dJJY5WEO9TgDUb9J7Uo1gJ7jD47CjEab1keo404aRI/2IsfMNK2KCspVffEze1S4eKDI0PzfT5EpWUXpP2x9YyZ7AcbofXmTb8I2oY1Yd1YjIgE0Igs8E+UCOt+SGLaMri0KzXI+3jmK2IPoxC3erdZTfWsgW5xhxe1WSPJ82NYA0+k9NCCWE4au0+PnNcNGIG+ozFiZRy0R/BKNlhE2IJRnlhoaWPeSnKJkjUpVrHPwKSJDIGNfle0njT+nbLTIic2Yj6dnH/+OeNcgQm50TFHAwVdUH6GFUl+kdP6sFbvhWjiZ/AUpKqdESxXOWTSzP2uECoDO5uTVDu7PYeONdYzqCNgRkPuJbaKccAufVXEUm0kH8WrpPqS7uNgq59oTY69H6MmGWbTZ9bt3dPSrTt/iAaPXQNzg9YF59fbac4AKTHd6rGrMsoOtMty+DsRGGTtzbham+FLaRbCdGKFYpgJRr5zZSdJ/Au9pyRQMUsybw9hS4Xgz+TAqsjU/qyEW9iHWYyPlppus7PemxRQwLsYHyw0ItED/K3mpCm8E94AKfBgZCHxK8/LMAk/RXW6RtBZzRLekkA/urYvROi6WKH6YtHgVYmMpQ6FQF6jLKwsDSNC2pwKqeswITOY16yPCuoQ6+Zqkb+xJ9trkljxsV3ydwYgSAmlTX/6ppKljyOb0nbG5o+LC9aiEl3E2sgdn4mX1D+sgRn/y06ebtmfda4Eu8u77tLAVAS6YrJXV zQo6IfbH Ku6uIP6HAUXroK9MFLlVpGsihn0Ic+YX9CoatebFp4xhLJJeGdy8RyYyKomp64t5isp8mwbk/Lk1u2OShVNDgLyOsP42HwlrnRtCIO7IwCdPlelM7UIzOm4M1C+peilYX91ffceK2x2FSHFhgklFFjlfY5D8MejWQBIsZCqshkfppwp4oOSYTme+zcfNacResEvbcl2f1gT0nGifGY5atgb9jWb5VfGu0FtGAMkeaZGbQ8S+d7zDIWS2xs1pmFdp4VNB8gJv2mP7QCtbKIworu0oufZxuooOxSpbCBPRp/Qy9EJiNlSnkDHn395k7F2znZS5HCxYsM09F74kAGpsh97O8ec/DkOO/kvaFyAOcfJEUoeYz5baGBA2d/HYQK2GWjyRU5hVcxqtbFI42onr8/xnm9/gHPzWKyau7uSKF/7aySrcJSH0ZtE9rCyHDiNSnSuLYM7+tikPTLWIbhVZNYJ10R8Wi8utMGhPJbvTl0AqMAsg/kxlAYGZTDggAgFbSDTzsN9VQgltYU0HncQdpxvu1PqK8AXbeNN1sPnlsW2fVKXb7+H4vVMf88CjVj1K1NSSz6TcKf0kOUYe2u36nv7RlKdhhB/u8A4zXBgQ1MjiDwnmsQrw9CAEF/F2ExW7iqyXcWXOUp5IdPoj1lIfn8L6uXvB2JyhkZXcVgTGZHOLF/Nl0bodF6Hbbz34Wt6xby66NpYsVwUPpn6i+t2PXjcdEevJ8up72TU8f X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For all page allocations to be tagged, page_ext has to be initialized before the first page allocation. Early tasks allocate their stacks using page allocator before alloc_node_page_ext() initializes page_ext area, unless early_page_ext is enabled. Therefore these allocations will generate a warning when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled. Enable early_page_ext whenever CONFIG_MEM_ALLOC_PROFILING_DEBUG=y to ensure page_ext initialization prior to any page allocation. This will have all the negative effects associated with early_page_ext, such as possible longer boot time, therefore we enable it only when debugging with CONFIG_MEM_ALLOC_PROFILING_DEBUG enabled and not universally for CONFIG_MEM_ALLOC_PROFILING. Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka --- mm/page_ext.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/page_ext.c b/mm/page_ext.c index 3c58fe8a24df..e7d8f1a5589e 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -95,7 +95,16 @@ unsigned long page_ext_size; static unsigned long total_usage; +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG +/* + * To ensure correct allocation tagging for pages, page_ext should be available + * before the first page allocation. Otherwise early task stacks will be + * allocated before page_ext initialization and missing tags will be flagged. + */ +bool early_page_ext __meminitdata = true; +#else bool early_page_ext __meminitdata; +#endif static int __init setup_early_page_ext(char *str) { early_page_ext = true; From patchwork Thu Mar 21 16:36:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599179 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5500DC54E58 for ; Thu, 21 Mar 2024 16:38:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 12CFA6B00B4; Thu, 21 Mar 2024 12:38:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 091476B00B6; Thu, 21 Mar 2024 12:38:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D60376B00B7; Thu, 21 Mar 2024 12:38:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BE4B46B00B4 for ; Thu, 21 Mar 2024 12:38:00 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 9168EA0A7D for ; Thu, 21 Mar 2024 16:38:00 +0000 (UTC) X-FDA: 81921603120.03.2BDC95B Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf21.hostedemail.com (Postfix) with ESMTP id C0DDA1C001F for ; Thu, 21 Mar 2024 16:37:58 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="QFwO/4dg"; spf=pass (imf21.hostedemail.com: domain of 3ZWL8ZQYKCFICEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3ZWL8ZQYKCFICEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039078; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pOwNrbZCC43nJVxac3Uw9hAvg9K+bdh2+QxQ9H95ZLk=; b=B5MbQUVKBxIc0JttZzMLdg5mKXLE1Xwyn/5aQbQqNJ9BKA2CX9AC8P0mizycId6R1PPtIg aBcYTivbfsuC7jXwRFQgUON7q7BpsaJlDbF9ZLxQO3v5rXvSvgAGp3kruBO4jj9p9UbcGd hZnynI2YpuA6rxuAdlJJf62s+dO9HXM= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="QFwO/4dg"; spf=pass (imf21.hostedemail.com: domain of 3ZWL8ZQYKCFICEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3ZWL8ZQYKCFICEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039078; a=rsa-sha256; cv=none; b=bHbikyMrtfue+Nu6dGWe+BVAmaEXRBxydNP5BNAQXPW09/rH+l9n/0mSyZftMDIm/mFc+G 6q4AXt/lycq2B6t7frNFJv1XeZ7XpGyyRMe7O3Hz+IDkjsQzOHKt8YoFaNSZyfamX5qNB/ S9aaGzt7bHJogYGtytSbqjTdyGLILw8= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60cd041665bso21148677b3.0 for ; Thu, 21 Mar 2024 09:37:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039078; x=1711643878; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pOwNrbZCC43nJVxac3Uw9hAvg9K+bdh2+QxQ9H95ZLk=; b=QFwO/4dgq74wPf4IWAurAET6IuOUiYnSAs3sZ823RKYINpCFnWB0XzND/wqITjDQ6F fH3YLXPXBwILc6IZUh1kCyPEDoQB0ZDVsgPfgFYXwYhMgqJqOEE9mCgzMCHtQAP34dAQ WEIyXnaK0e4Izw0Ff9tdQoPUwpt9WXyE9LpfeeBH3B6Mm970kke8LThIgAffyUAu76nF meY2Ai/Bdp7Vj5hVOzaCOzM4eKOB934T6zgCjBvM1Rz3T/aXbAq1yEqtGip/WbIu59oa 0GWNaxhPFNROww4PF5bgzVfSqQkv5OsQL9FSlPUq+siGaqEdyvumyKNtIIBhKGlB89tF zxjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039078; x=1711643878; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pOwNrbZCC43nJVxac3Uw9hAvg9K+bdh2+QxQ9H95ZLk=; b=jpttifuc/ynXE3D+AZ0FCdgMGo1cyV+slrW9eQLVqxqjPOVNX9+knpvngOsy3b2GUC FDGhAGJchEvS8cQZV/61myfZR6jdBqJQKDky0A648PVIKWwu5+X0gLa460vlgO4faVdN ao/YCU3+rp1pmDx3Wmrx9kaIPFvG3WXPBuuBCxitV4zVPLqjrDf+K14yA46cX0m3pVbA aK20p5mMncWsx9rxTxsS6LPC0OBWlJpCQQq57tjt0pVbvKGjtnBMJVyYIE9zrQL+8p+h sqOiZmIdIWN1DlDQjR5qqp6yf3PsZs2ZqLc0mhLXySEszPdhuhvhdzFNH/lHxelTSCkU R+Ig== X-Forwarded-Encrypted: i=1; AJvYcCUFP4SVc2/94mXTKrwqSwEfavIhzkTjgtZlrxWYU+tvPeB4QPRYodBb9pBJM1Lj8WGhTx7mIOe2vKKYRvQF33ekrR8= X-Gm-Message-State: AOJu0YyAYGZRKHElsPD/veV2HH/b69tBwDJiRd+nl76Xvv0RhSdcVWKn u/CcV6L1hJXSwWD5l8t9HQlb76Oaj6c3Inctc/cUs562qnFzgiVJrLb5xtIZstOFiYmHZvH0CkL PdQ== X-Google-Smtp-Source: AGHT+IEW+wzJaTFBqhE05AsZZeHucxp+/bXHicD4m92IkMS+am09AVJUNTX8XqOhkKHJop6Lzvm1ThHm3tA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:1004:b0:dc7:5aad:8965 with SMTP id w4-20020a056902100400b00dc75aad8965mr5894352ybt.0.1711039077542; Thu, 21 Mar 2024 09:37:57 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:44 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-23-surenb@google.com> Subject: [PATCH v6 22/37] lib: add codetag reference into slabobj_ext From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: C0DDA1C001F X-Rspam-User: X-Stat-Signature: 4t87d5xybet7kzku79ghhignj7kdho1x X-Rspamd-Server: rspam01 X-HE-Tag: 1711039078-891810 X-HE-Meta: U2FsdGVkX18ibjKTcNQtJT/S2dmgJZ5uqzNrd4MbYYaf25OiyONoAMBP1wt5bD1qdbStgqKCcghVYFb1Wp1+i08fJhluaDAcAXgKgjXRV5lPuu1xsRQ5ZovXgzp/VGbQDwf8Ii1K7qqPx8/GjkLUtxQ4zT2jL9GiV43FgzVJlhKcRSM/Huq882z1B2fmt4b0JfsQBHirfG3rYzxNT+jX6x5/LaiIAcqT/UnVRe2HFhzmGq9iVu8belsWvenZgGrs0E5nBtePn33wpW7oBYIym5GYTbUap5NIGItZodf+Ac+UPkAsck15LHRrUSZvPrDg9SxFHAkCGYrj73Q0Q8i5ZgV1qSzVjQ862s0YRPmyRrCmBS+oqOnGEILJ6gfT4DHkCWODV41r5X0H5Vlim5zvK3Hpf2ST63DMl55YtZgf6x394cUVtfIxYuzy2+PxA50FhINUabr7wjEUCDZRzGxlmpnV1P+sXtTsWk1EGLWqZTwbpQSiyF7xkND92vFE2Usd/5wC2m9CqrPX6BA9YHk/RlGWqLmWEFOymXpQ+jCbEmuYx9XqcYZWN6nzkLSqxeavNIim/OeCTyDstlKlrIDms+KdN2+Yj0fD6dQBnqjKuWtXmzetT44z4suTg4toj5vbJugLm61WTXA0vYy1H61QpObi2Jd1WvAHGvdiIF+Lf6A20RbSVE3nXfdQsDv3xFUNktfm6W2stXfk/M49EQL7xAGM+hpB0dfcfbZGMMoOsH03JqoC0ce3qjh5tI/NafrQy3jvpnDHNtJtKnL/7j2OdDp+CGNROudhilysM9QoBk55SDRNlwYqiZww43F9MjQOjGLRAC7D77EyEutBMoqYoW61SbYvFS78yMzVc8nkp6tH9YeNLp1X9kJd66Pq9Pka957G4bL+ZVl+RzC1CdawQng4UQlW7/vDTynsejrN06qmdFuc9Q22EIU5g2Sgmz4OLUxYNXKSVbjYjtWIpDx 87ZV3xS3 CoOqREW0oQFQyZEF8J+c2VAbAnbqdFUd7XX8l9J6oR/PFHQnBb/JsvHWXdF/MD7bZ1l1t3TMoG/DjV9f9dGeUKJl1gNovqYKl/hRnWJhTp7pH7AmaBql8T0VdDShrYIk5wok6ckrcnK5qt9olV0BYal/DmPMnlGNTantOBxtoKw+ijE2ZNvVRI3A4YiElJXdLYIwO3saszVFx1iRsfGrmV6NJ0y7KnV3tpKTLbTah468Vg05Y3XJ5Mm05L3NB3+6wJeaMJW3OMEShwbVISkAdRsv7mceOQymAmKR1+s8PtydTj/NdcAppsfWWjSH/bWg1/0l5Wj1qDhElPAbwZwUNhTsqx5ejaA72pzID37pQdR4BujpcgozvFR2dDvd87pqnyC7nef2pEQnyb+LxSXpDmJqJnLFV0pkw1bb6p1POpqBYI8PXKjQ8jali9OV31pqJIjHCoN793WY8UgrBd+JrEIe3uZP6tOM2v8mboS9u2dLCuXAakxcMMTOMYKSTwaKkEaEOv/uSOlAS/GLYqwILryXpDWqqO9UUnEvb99x9UUkELjRrnQbLhGpcVVQXqreCLWJ3VWp5jv/TxuxJSnjR+0zRveCQH6m1IoP3YBMTl12xLfLsMl3Siiz0YhfD8VgFElwj6PDyPeYxV+4RmZ5uvshlGeoPgz+MRFxAjbrP5sftbZNxufI6WN0AD4WWWq1UomIJA8AyWIF9oQFJgBypRzmyW0Tik7BmvqxocVq7ASWTb/5KvGLIocFSzCpcOgjDSZKj X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To store code tag for every slab object, a codetag reference is embedded into slabobj_ext when CONFIG_MEM_ALLOC_PROFILING=y. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Reviewed-by: Vlastimil Babka --- include/linux/memcontrol.h | 5 +++++ lib/Kconfig.debug | 1 + 2 files changed, 6 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 12afc2647cf0..24a6df30be49 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1653,7 +1653,12 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, * if MEMCG_DATA_OBJEXTS is set. */ struct slabobj_ext { +#ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *objcg; +#endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + union codetag_ref ref; +#endif } __aligned(8); static inline void __inc_lruvec_kmem_state(void *p, enum node_stat_item idx) diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index ca2c466056d5..dd44118e7337 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -979,6 +979,7 @@ config MEM_ALLOC_PROFILING depends on !DEBUG_FORCE_WEAK_PER_CPU select CODE_TAGGING select PAGE_EXTENSION + select SLAB_OBJ_EXT help Track allocation source code and record total allocation size initiated at that code location. The mechanism can be used to track From patchwork Thu Mar 21 16:36:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599180 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 315FBC6FD1F for ; Thu, 21 Mar 2024 16:38:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3A5986B00B6; Thu, 21 Mar 2024 12:38:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B1E06B00B7; Thu, 21 Mar 2024 12:38:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 104B76B00B8; Thu, 21 Mar 2024 12:38:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E9FE06B00B6 for ; Thu, 21 Mar 2024 12:38:02 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B3934A1FC3 for ; Thu, 21 Mar 2024 16:38:02 +0000 (UTC) X-FDA: 81921603204.12.E4B3891 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf13.hostedemail.com (Postfix) with ESMTP id C6CAB2002D for ; Thu, 21 Mar 2024 16:38:00 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=trdO50Vm; spf=pass (imf13.hostedemail.com: domain of 3Z2L8ZQYKCFQEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Z2L8ZQYKCFQEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039080; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qDb8iz8I8QS6d+rOBOAu9Y4ek6cz/wlqKmX9e9H3T5g=; b=hdapQ/3J9U7Mj6Qj+hVvNwqXR0Y6oJCu7eG6hRWn6x1ojpHn56b36WcbCdke6c0dG/wqrw x8pXR3jML0of7FbkQCZNTk2mfHUvFdO+kv2tMSaPNbdl9sKPvS4/lPXNeB9W0LZnJuMC4+ AfKt1tTCkS4xWLmbLQengKqIaWUAJlY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039080; a=rsa-sha256; cv=none; b=EvY8ykVje+WBEGr9pY26QR0w6U3V6RzgFEX5b8VWF2cAg9zs4CXnbNw/Sof2GGQSPdZRvD vNP6Y4daXk9bPDepTlx+fz9SwrXYIRJejqRSP0FiTt9RrsSmHNFNeTlv0w/28CX/CLVVwX XTMMtMBaGG+HGidRGWsZz81UpBGIfgE= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=trdO50Vm; spf=pass (imf13.hostedemail.com: domain of 3Z2L8ZQYKCFQEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Z2L8ZQYKCFQEGD09x2AA270.yA8749GJ-886Hwy6.AD2@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dd0ae66422fso2413457276.0 for ; Thu, 21 Mar 2024 09:38:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039080; x=1711643880; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qDb8iz8I8QS6d+rOBOAu9Y4ek6cz/wlqKmX9e9H3T5g=; b=trdO50VmBO7OVgGtubmaDDwQ4RXKdAWS1J9dGjP6aNdzfowW5jgN0PpG0RKxENMYJc XOFTbpT/vM+lTBR8CsHgqj6Cj6czTo8gAlZxCAsliLt/uH/2R93zGG17XcftF94xQGPz Dau81PY4C2E/hI15E28bHgkoNYXSGGK5bH/6QfHbEXXHoAXF+G1pVGdfOzW6OmcHeLWA GpZGrkB1Aq5+32UPXMzycDfI0Hw/nNCB3IyGw+13TuX4DqpBoC9CYbhsgwAlf83RMibm 6n1QZ1LI561FZA2BWzWpKt4d5UaYDZJ/Xof1xlKHZtVSMIC+yPTrRj8LrORBSjlpvn/P RR5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039080; x=1711643880; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qDb8iz8I8QS6d+rOBOAu9Y4ek6cz/wlqKmX9e9H3T5g=; b=Qy/ZmiyrkVOlViLH7eL63enOs7ukLgygvDPOiuz9xkd9sq24JwP87bt3eO8PT9GTFO wBQYe3j0TM7QFB1cjbPSN3h1fE2sqMEBEi5t4/oGZpP+8pyJqglWlq7UiGo5JIKKZbKu +RVwZUIOc9TEoc0g/+QVACtAQTWlNRbz/Lugd1adMtFIAMozZKnhNaibNMLoUBGyjT81 huLtO56yJZlJ/VRetLsmxHQlY5uWmhH4YT6OfJHH+hdjBM+AzEP275BHyiKby8sA8ez1 c6R4Nfd4+eyCEiZUzanpXE9sfBoohsQcjrel+DcpdDVv+byiQ/7XBggzh3h+Wmec+sIK tJDA== X-Forwarded-Encrypted: i=1; AJvYcCVPhqBDds8GXUydvq4nkTGrxiT3sbj+TPhgrkUmTRNkAmKGzxh8R/W7PmWOtm100DL1y2E24prZSIJvWZBDP+RbVCY= X-Gm-Message-State: AOJu0YzQlo+ZY+OtZomRe2lvS5h1ho2AuqzH5stblg15+kZ3L9UBX+Vz y+VNJEG6beWev9Xmc9GuDbxx5UPW9cWNM0QeuXbUettBeIpCkQRxDcbc1FA9hBWp/KZW3kRbMYW HrA== X-Google-Smtp-Source: AGHT+IHDeZBFlJ0NtCqkUUNT0gfkHkrQ10cEoA8CJOTWwHwDSwKuqC04Vl0uLsTIeS1kWMwDWt3A7ZdyM4A= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:1604:b0:dd9:2782:d1c6 with SMTP id bw4-20020a056902160400b00dd92782d1c6mr9210ybb.1.1711039079759; Thu, 21 Mar 2024 09:37:59 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:45 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-24-surenb@google.com> Subject: [PATCH v6 23/37] mm/slab: add allocation accounting into slab allocation and free paths From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: C6CAB2002D X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 3in6sjf9t4w8ppoyryiy3abpqcfn7mzq X-HE-Tag: 1711039080-670577 X-HE-Meta: U2FsdGVkX18JkfMoasJUQ1oYet2kvBh3ylYUmVOUpOHMfG3MbyOnKu95sPfjtmoDvqC/+Zoc23ErYMnk9aTCdVxsTwSeKrzkH8bMNmj1DTrUfZ4PkGtiymWOgfXOafekWgaOMiUfAkcXYz1SopLgXOb49DMPwsUnPHixj7Gxo0j93rftr86zlBtYQKySlZek52Amxw8wKawKc1O94jG19J4qF7ApDPSprzUg6RtLh+E7qnop02+ndNVIUk333Id+xVIoC7ua0N8f7XsWu4kk6bjkDRYMMw2Qu5cPU2+azHr9T26t5bcqQohd5zbzm2SG4sGbMO2Y/tSUotPPbC8xm5Ox+U4ufhqPmJZeg8azsEebdRFAwYHLfdYAfi707ta7ymRdXZYuJN+mDctDrvrfD74DJzShJb7koeQx9MPJomyhEizRFv/KvIKJ56nfj1/JzXAkQzxNdAuRPXLES9ZCv1Pr9+YuiQNTBhlhicesY2Lzup8v4YzWm9JUWMVpqyCJTwcPmFTDN9iw9rz5p4YfavpB5VgMTopjs4f6Z2PZm59Y269FE7lVnNyk0vJNRtTcAj1LebDuSnOISmUMaNgxYabVpIVuEjSxoCWl2gzae9xuOJ/dn/NCpg73VISzsrMyAEqqhJJViqOIUnob4jpCMZtGzlkHXR/HpY+Ytu6pO3R8rrHrgFKrWqwFnw3wugmPFyRu0WVCtLfYsql7A5km1KCfjO/sDYZZK95RAUREp0NwyzsB4h85RVjISeN6psz5637jGErfThYjAuwFppNlu5m+StvpJX3CaeH274a7dchejX3eWbnaUTg5xPFOGLvBtd1/ax44HGZGjX1oLKkN7nBXklGCC4DlWziElYUPwZi2n5BMQP/qSP43FEkPaFoK/brrSClYmVnrhWJGnT1jiN2BbLytsx9UB8vobz6iQqyfcOCEM8T0SkRY/SLp/vWn3ck7EeH7XblIV0/l+Z1 mcWe5t1x t4K24zXIOk/leuW5ep5sM68CdhD1F4+ODodt1YZUiOgqTRokx8+oQDldunZSK9ha4CZ02boPBiy196XxbIOwEB9P6nSw4i2upsXSS6kL1tmlIyo+Xw8hWTRNw2vOTmRqI2tasYe1D50xY5Hk1QoM5Lps/pM1rzkNQZqKExPQPvud5XGC1HSCQShYCVyrtpwhmYlgddfI3T2VLDCKlEFcGPUp1FfVGZBv8lg3r9paS5nS+JVtyn4TBAxBmAm4tw6KyTUyeM+6Us3O6zOHGQgoFehYTe1KPHNc0UPZXL7Z7hWFtoYSucY0zFSIv0JlH86eJzDWz2nGgNzv3GJkCoOFaBOhG3crgomTvE88gvwjRneYQzrj5g6Zsy8r2Ooh1HRh6YyNnMBT+hBHoU3I4zkyPZYXUGniz/PoJCHur X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Account slab allocations using codetag reference embedded into slabobj_ext. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Reviewed-by: Kees Cook Reviewed-by: Vlastimil Babka --- mm/slub.c | 96 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 95 insertions(+), 1 deletion(-) diff --git a/mm/slub.c b/mm/slub.c index 666dcc3b8a26..5840ab963319 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1932,7 +1932,68 @@ static inline void free_slab_obj_exts(struct slab *slab) kfree(obj_exts); slab->obj_exts = 0; } + +static inline bool need_slab_obj_ext(void) +{ + if (mem_alloc_profiling_enabled()) + return true; + + /* + * CONFIG_MEMCG_KMEM creates vector of obj_cgroup objects conditionally + * inside memcg_slab_post_alloc_hook. No other users for now. + */ + return false; +} + +static inline struct slabobj_ext * +prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) +{ + struct slab *slab; + + if (!p) + return NULL; + + if (s->flags & SLAB_NO_OBJ_EXT) + return NULL; + + if (flags & __GFP_NO_OBJ_EXT) + return NULL; + + slab = virt_to_slab(p); + if (!slab_obj_exts(slab) && + WARN(alloc_slab_obj_exts(slab, s, flags, false), + "%s, %s: Failed to create slab extension vector!\n", + __func__, s->name)) + return NULL; + + return slab_obj_exts(slab) + obj_to_index(s, slab, p); +} + +static inline void +alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, + int objects) +{ +#ifdef CONFIG_MEM_ALLOC_PROFILING + struct slabobj_ext *obj_exts; + int i; + + if (!mem_alloc_profiling_enabled()) + return; + + obj_exts = slab_obj_exts(slab); + if (!obj_exts) + return; + + for (i = 0; i < objects; i++) { + unsigned int off = obj_to_index(s, slab, p[i]); + + alloc_tag_sub(&obj_exts[off].ref, s->size); + } +#endif +} + #else /* CONFIG_SLAB_OBJ_EXT */ + static int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, gfp_t gfp, bool new_slab) { @@ -1942,6 +2003,24 @@ static int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, static inline void free_slab_obj_exts(struct slab *slab) { } + +static inline bool need_slab_obj_ext(void) +{ + return false; +} + +static inline struct slabobj_ext * +prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) +{ + return NULL; +} + +static inline void +alloc_tagging_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, + int objects) +{ +} + #endif /* CONFIG_SLAB_OBJ_EXT */ #ifdef CONFIG_MEMCG_KMEM @@ -2370,7 +2449,7 @@ static __always_inline void account_slab(struct slab *slab, int order, static __always_inline void unaccount_slab(struct slab *slab, int order, struct kmem_cache *s) { - if (memcg_kmem_online()) + if (memcg_kmem_online() || need_slab_obj_ext()) free_slab_obj_exts(slab); mod_node_page_state(slab_pgdat(slab), cache_vmstat_idx(s), @@ -3823,6 +3902,7 @@ void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, unsigned int orig_size) { unsigned int zero_size = s->object_size; + struct slabobj_ext *obj_exts; bool kasan_init = init; size_t i; gfp_t init_flags = flags & gfp_allowed_mask; @@ -3865,6 +3945,18 @@ void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, kmemleak_alloc_recursive(p[i], s->object_size, 1, s->flags, init_flags); kmsan_slab_alloc(s, p[i], init_flags); + if (need_slab_obj_ext()) { + obj_exts = prepare_slab_obj_exts_hook(s, flags, p[i]); +#ifdef CONFIG_MEM_ALLOC_PROFILING + /* + * Currently obj_exts is used only for allocation profiling. + * If other users appear then mem_alloc_profiling_enabled() + * check should be added before alloc_tag_add(). + */ + if (likely(obj_exts)) + alloc_tag_add(&obj_exts->ref, current->alloc_tag, s->size); +#endif + } } memcg_slab_post_alloc_hook(s, objcg, flags, size, p); @@ -4339,6 +4431,7 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object, unsigned long addr) { memcg_slab_free_hook(s, slab, &object, 1); + alloc_tagging_slab_free_hook(s, slab, &object, 1); if (likely(slab_free_hook(s, object, slab_want_init_on_free(s)))) do_slab_free(s, slab, object, object, 1, addr); @@ -4349,6 +4442,7 @@ void slab_free_bulk(struct kmem_cache *s, struct slab *slab, void *head, void *tail, void **p, int cnt, unsigned long addr) { memcg_slab_free_hook(s, slab, p, cnt); + alloc_tagging_slab_free_hook(s, slab, p, cnt); /* * With KASAN enabled slab_free_freelist_hook modifies the freelist * to remove objects, whose reuse must be delayed. From patchwork Thu Mar 21 16:36:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599181 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30FB4C54E58 for ; Thu, 21 Mar 2024 16:38:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD6A66B00B8; Thu, 21 Mar 2024 12:38:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D5F556B00B9; Thu, 21 Mar 2024 12:38:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B8D796B00BA; Thu, 21 Mar 2024 12:38:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9D5B16B00B8 for ; Thu, 21 Mar 2024 12:38:04 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7DA3712104E for ; Thu, 21 Mar 2024 16:38:04 +0000 (UTC) X-FDA: 81921603288.26.A13C279 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf19.hostedemail.com (Postfix) with ESMTP id D0EEE1A0003 for ; Thu, 21 Mar 2024 16:38:02 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=KlioJpZB; spf=pass (imf19.hostedemail.com: domain of 3aWL8ZQYKCFYGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3aWL8ZQYKCFYGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039082; a=rsa-sha256; cv=none; b=cuF8W3FNhhdppyhnR9ag7rooUywf+ZgQR+3Glinm67S0uFDgjgA79NvtLztSPApjAqwCvl Nr24Hocd6kSwdf/2xqHAkF6EKTLVEsKuBId4lsQTyZTgjMTAvv1c8hx20L2lgoqGbkEq+G O91tT/v93LRjvpwwT3O8foBu3P09XyM= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=KlioJpZB; spf=pass (imf19.hostedemail.com: domain of 3aWL8ZQYKCFYGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3aWL8ZQYKCFYGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039082; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=s40WFNE8AOKkW3fMlFNAMtyrdkqCS9mljSRdY//hl9w=; b=MQd/js7AODVXhXe8eDV/aeuI5/87Rx3SPi+HfnJD485Bhaot/jX1BLHnOIQfpxcUK3+l4q t0DBQxTjCpx0PSmCLJux0TmeI+HO75k4GQqUzrfYuN3N0Whg9c3xmGwtM4DzZa6z6H8t/U 520mbp73UYK6DdoRq5LYGJxKtvo4rhI= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60a54004e9fso21458087b3.3 for ; Thu, 21 Mar 2024 09:38:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039082; x=1711643882; darn=kvack.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=s40WFNE8AOKkW3fMlFNAMtyrdkqCS9mljSRdY//hl9w=; b=KlioJpZBvXl0OOuJiIG4vgwvWwnja4FOJxl9t6cXp2M9XhWtf6aRJ2h9Dqpnza/wK0 5WchBX+VD8oq0skCVcj20KL4/jdLbYITYN+ri659hV0QDLiIxeItJ4pcoxixtBwpNRz9 m9hqj2+1yY5L1RpNSztUp8N5gb/3Pig9/s4jLFJlWHf17w1p5dbvyE63BZ/psNXa8zYS Py6AMf+6m8nFZ/o3TsIMiIPs22PiyYmY+p6O7la3k2EVRzMgIUdhoXLEqnrsIHn6cEGD ImJqrw2baw4+Y4CSReB55z8AZl5W2xHdbRSZ1/0nOy0QW+K3m/mZbJnthQl0udbVDTz4 Z/+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039082; x=1711643882; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=s40WFNE8AOKkW3fMlFNAMtyrdkqCS9mljSRdY//hl9w=; b=T8iC5LxNP9dE+3xuIYEeetYrGWzeSbtNRdLDiyxKTtibwqEssSch2YuOfn4GuUA4w5 59zQCE91zfK1yGSJds0hr4nPvCGWmMxYVYVUHW+uxODSweAxyYrRV/n5tItHWwx/9nGF TasQCamfic9fAJAJh1VA7O/+U9NKohIo2WAcylsYylN9U4KAASYHnrnLQqNpfYnGCGB3 mnJAC7yyvENn/TnIWJPjej9JxGIWhT6Nc0ac13itcITx56M6lpy2GUJPvXxiYBwyu1Fm DnyvGOpQC8hhrMU1ETRpEy+hxninILx5cD9evavlQQB4zHJfZ53gbzUX4XgNDqzLqRbe rEQQ== X-Forwarded-Encrypted: i=1; AJvYcCWX+r2j7yRY6J7FYydfL7B+3yP38aSxS/hMHcwT3owrf5ccKdXXr8Lx3p916aSyYvFxZ43G+bJfBGaOgVOoczgjPLU= X-Gm-Message-State: AOJu0Yx0qi9x5oT6CW1hrfo9pcnYr0e+I+hCg4eKF67z7Tn3GQHMaV5J GUmp960Ykz+p61dZKf1UY7Lcv2x4wwqvuPKt3MvTj8UKWM2jK/awqAAANGkPykLwurP9OvmO4Jz 1Fw== X-Google-Smtp-Source: AGHT+IFs+Nf82Ul1t74I4vPuVX4weNJROFN7yWXEOAVfX30LUtXPCkGRPdyvAgyHRn4LwSGt1RsLpaikF8g= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a0d:ca91:0:b0:610:fc58:5b83 with SMTP id m139-20020a0dca91000000b00610fc585b83mr1061193ywd.8.1711039081806; Thu, 21 Mar 2024 09:38:01 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:46 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-25-surenb@google.com> Subject: [PATCH v6 24/37] rust: Add a rust helper for krealloc() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org, Miguel Ojeda , Alex Gaynor , Wedson Almeida Filho , Boqun Feng , Gary Guo , " =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= " , Benno Lossin , Andreas Hindborg , rust-for-linux@vger.kernel.org X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: D0EEE1A0003 X-Stat-Signature: wb4mfyky9qrbhrojwt6dknhdjmpti53z X-Rspam-User: X-HE-Tag: 1711039082-430054 X-HE-Meta: U2FsdGVkX1/BjQqeFUh1pBfKYUr0XUb2aQmOO7nvozZf6a+NRnSm12Pz1XLDQBbIDFiizvAULQm87Y2Ka0hJAOgTpuV4dpVTN2jd53lJRnHkza7uoWdjMbI3erfCkU94UZVdGyDqr/awHWIlQiXzHCtNQI+5dBaIZe/q/LOtjp5v7k7QXCq7TMGx8I+l3lGZ5tp6Fqw9jVO9bsS/s96tC8Jp1Xn4IZR5gjgRQdd6O8ZGTL9zUeyM0R50F21H978DxvSqTQWsidurBL2wQ+Ey/uwiOWNaX6zXmGAIQ66HuFjKyDR4R5aujK44S92r1bWHX0epGQ1Jn96y60yzj3+qyNeg7YdTZYC4rduouTDqqgwgJVftemJgIRQZ2kctXiXmJiQZC73mf5oQ1ctGGUP2zKy+gYA1Zo3gzVfWA9gxKlQILdtaNWM+ydE83RYO6S3HJToSGe6OB6kTLbrytRqymHbC1it/dapz/hp0zzfefOdR4abY+0QWUBFJh5ww7qljQys/cNAQSn3qG3O2bTZtTl8TF2YNmjWNSnyP2/b0a4boQS68ZoYm+xL/yIdBJ0PUvI45W7qXWmYstl9/gZ/I7JJCYhSiLBJgTCvH5IrkYxKSMlbCgz72qKm0yNBJ25RXMd8NHjsZA+Yg1vKQddwNysG5t/WBLrMK8rqNc/YPOJInEz7TiAUkd7IKt68bS8o5WT95ZxPNM7TbdLRK4gpJH+ts0eemUtha5kyLq8EJTy61QZYdPlucxASigv0yEXzms4ClSqbB65iPtKJmdgPqmvq2dDs0gt7AoBpcVLeh0RqEiwnkMDKCBO5rXBIACfRg5le/lNNn9cC3GoxizEJBIbbjsscXKh/uH8lJPBRsmYrdGOUkjQbrtCGC88t/DxKxi73Bzu4CBaC0yl4pRYHWCQx/dvttBI/PJbL3d4z3jm2gI6bgsbuKO+6JSrHigfnWAmvEby8YIAFazark9PO wiKVDjLF 9YZvmYoOj78ZvlK80ivutWNSDdg27wC/OlL5nPodTw61RN8SLd0nLhxmdG+LLwSwU+cTTDLH1W15NCoJ9vqwnjCAsqMqm65c0gYwegmPd7TxWrnblQtNdgSq925pc1bVWs77OhiQk8ggEPctLIa/qU5Q3PshDiHWSbuOzXP5wBdTVRp0HPLBHCc9iDTQzANegdDf2zSGgzsRKJsWz8+dpysmg+sWzK8rtPM5biH2qm/w6i6WL87Lw/GbeNERwL1P4aZd2Plmy+KjNqs81VkZ+k56soGRT+WVWitoM+mDgXNfDhOmDaa9xTFnXJKCC+3tDTkZRgLpQn2Ezpvpv7ZvyHoO06/zCHNp+ggyy8F4Xls5zrOtklcI906QEX9ArD6TjZsSBtwPZXxhSDhLpdgRe7i6n73zsfwHoBHtv X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet Memory allocation profiling is turning krealloc() into a nontrivial macro - so for now, we need a helper for it. Until we have proper support on the rust side for memory allocation profiling this does mean that all Rust allocations will be accounted to the helper. Signed-off-by: Kent Overstreet Cc: Miguel Ojeda Cc: Alex Gaynor Cc: Wedson Almeida Filho Cc: Boqun Feng Cc: Gary Guo Cc: "Björn Roy Baron" Cc: Benno Lossin Cc: Andreas Hindborg Cc: Alice Ryhl Cc: rust-for-linux@vger.kernel.org Signed-off-by: Suren Baghdasaryan Reviewed-by: Alice Ryhl Acked-by: Miguel Ojeda --- rust/helpers.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/rust/helpers.c b/rust/helpers.c index 70e59efd92bc..858d802abd11 100644 --- a/rust/helpers.c +++ b/rust/helpers.c @@ -28,6 +28,7 @@ #include #include #include +#include #include #include #include @@ -157,6 +158,13 @@ void rust_helper_init_work_with_key(struct work_struct *work, work_func_t func, } EXPORT_SYMBOL_GPL(rust_helper_init_work_with_key); +void * __must_check __realloc_size(2) +rust_helper_krealloc(const void *objp, size_t new_size, gfp_t flags) +{ + return krealloc(objp, new_size, flags); +} +EXPORT_SYMBOL_GPL(rust_helper_krealloc); + /* * `bindgen` binds the C `size_t` type as the Rust `usize` type, so we can * use it in contexts where Rust expects a `usize` like slice (array) indices. From patchwork Thu Mar 21 16:36:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599182 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62481C6FD1F for ; Thu, 21 Mar 2024 16:38:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 87FE26B00BA; Thu, 21 Mar 2024 12:38:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 82EE56B00BB; Thu, 21 Mar 2024 12:38:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 634636B00BC; Thu, 21 Mar 2024 12:38:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4444E6B00BA for ; Thu, 21 Mar 2024 12:38:07 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 246B91A02FE for ; Thu, 21 Mar 2024 16:38:07 +0000 (UTC) X-FDA: 81921603414.27.9D52652 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf27.hostedemail.com (Postfix) with ESMTP id 69D6440009 for ; Thu, 21 Mar 2024 16:38:05 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=f2Di49Rh; spf=pass (imf27.hostedemail.com: domain of 3bGL8ZQYKCFkJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3bGL8ZQYKCFkJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039085; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v/NhR2ru/rhbwEka1DUrUYBdPl/5//ujLvCmXSkfW7o=; b=bjZJFpEopQk2iB2damr1z4K2y2sPYZ62ls9GyyRZfgKct4kPDj0krZIgKxhWMrZVrg89sL Qn8NhHZqfJM4MdbyOn/JoN66TPkWyySU5p9NB243LfVfmKadri6eoaXUgAlIbPoCCEAUZS VS1PgleY6m2eizHNdALuOCAPT0YjyQs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039085; a=rsa-sha256; cv=none; b=DJMT35M8cfsIB03XsX+xkdw1T9UYc5INEYsrYgx/pxt8ss0wmsWAO4/OUnP3Pr9/3h5f+D rVfkBQ5PtBNj5Em6RndBha9vK4AuXslqUlNdaUYVXqP/02goGIEIZHvXXddHK6n9IrJYdr +8KQ7S9dy7VwTDszD9GKjG6EVH/RPz0= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=f2Di49Rh; spf=pass (imf27.hostedemail.com: domain of 3bGL8ZQYKCFkJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3bGL8ZQYKCFkJLI5E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-61100c749afso20635277b3.2 for ; Thu, 21 Mar 2024 09:38:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039084; x=1711643884; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=v/NhR2ru/rhbwEka1DUrUYBdPl/5//ujLvCmXSkfW7o=; b=f2Di49Rh1Vs3TGY7gCvTpOH/KKToIWwWPSXt72bz6tdvvNKThIFF9myK4uP6Cwcuj2 H5npoLm7tz3WJD5P30R+3CHTHk2hpyLIw/YYw2yidjZW0KOm18TYzW6LtLf6bS4arnvG vz5qCeDachebqnKqav49SyuCcfTGdtRFEI2/uEOFKTP31rNM90q3mJtt5tVZGXbhE4JW k7zy8GY7eh151HjEyV36+eu2tJELtLrGf/51dSpl4WzFkDZVUXKZCZU808xbGftfT/G5 qsQfdMGs2zG6nA/F9hBERWK6FE2/VyU/0x4FWmKgdZZOjSw4xlrRKGcl1JCJe5STfaDQ I5vQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039084; x=1711643884; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=v/NhR2ru/rhbwEka1DUrUYBdPl/5//ujLvCmXSkfW7o=; b=vCQ17i8EgtWRPhQV2s3YOoUl6Nauj3URdp6n018fE25gpc6bLLa1Q2HQKdM7jvtzer YedfkWef/ZvIEkOJAoV8ZEAOZ3iuR1xp19/WtaQsJJck6pbytJKf+Y3dtry6jXelKi0X s3GdD2Rwb1tJPSjnpvl8ufaUMzv00fNS3+fhBukpY6C7PN/SxTUqcToz5plMoMC55h7f PNLp0YhdlOZivW9LA1CvGRwmqwXx93aYmnbAQturBu5srcBhuEgS5qT9mKI+BfTQvOz/ x1US02iFBqkduHdL8XLB7zJd+STAH3uwNLoHFZP2Cdxm38cgSoaClz5ZvFwzQiaSiW+F GkYg== X-Forwarded-Encrypted: i=1; AJvYcCXPCBhL3RzRD4bMVbxG0N1KWfNk7pypElY5jJcS9vtbtnRno/Csn7vqJ6y5tl50dl974X6Te2Mqk7oj8JoeUwGr8aY= X-Gm-Message-State: AOJu0Yyoks7ods48efNI7crL35MNcVu5qGmhTokMMtOkYHh+ESGNTwQI QVU3pCaywySujL986nJXDSwcCQvoXK0D21GF84z91rcdZ1lEKLZeVsEUvtl1tLfr6sXCMXopu9X Sjw== X-Google-Smtp-Source: AGHT+IHKi6ePxpLIm6Oqje/c12zdGr4Jgt9SCpXPqMsrhX3ojIaLFpQRURhZlXL5evQVhl06fPm8kIpNoTU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:690c:6f03:b0:60c:d637:afff with SMTP id jd3-20020a05690c6f0300b0060cd637afffmr4524896ywb.4.1711039084264; Thu, 21 Mar 2024 09:38:04 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:47 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-26-surenb@google.com> Subject: [PATCH v6 25/37] mm/slab: enable slab allocation tagging for kmalloc and friends From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: xnp56zxhrz54h7u4hmhmam9jnyd9jukg X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 69D6440009 X-Rspam-User: X-HE-Tag: 1711039085-639583 X-HE-Meta: U2FsdGVkX18raWYnPYdTN8ZUNDUz/S3y+J9dh5wm0qhgwbcBT0CdZSH7v7N/PXq8spqd+G0+JBPj9KetzxQCT93jZrgHU7mK2Ko98t8jgJwDvRy0A9RN5Ezs6BNyksdl1GeIQJtjvFDYoK1igt5yCF5TcbIXwwafw7XI57FaAJSLfU/8G40d4omOCOYYW/l6vhaIlSjYYe/02vaHkebqz4Brrlob/wcmD+783Wu8sFibj3HBtJpBt6zy74ttENQSPcJgiX9i+WcyHPr3rTiK4tjtkF7wKuOXR6jDsd6XBKwu3pZN9oMv9KmPFSpix8sXggS/4oVtH9j8e2uQp/6PV8/P2MnIkgepi/aF8q6lbjxvU7W2F7hHXHq9e8LZkXLvmUK3TzVT6DYp6L/GMpuhyVTuGKmcsNbxNEX/HeEVB00LbeM873C9VDffSPo29VEiq8DiRUWLp/kgcsP3XiBlGdpkBzg5uaph7TOwwxQgsCMTMY+zaHXWQrF5Qd6CNhJZ516X/mof5U42Jt40+W7Nr5Gb9JR+5V9gLNX5yCwmzcR9xc/TEuJEGg0zWNFGrhpAQeKtvaG67a5QlF1849H7aWc7joA0C2BpFg3joSSo9/4mtRUGVHSHxhdkLL1VGeOzjUxt+i418zQhF9StAmwT78mGno7s8aMnqQlfDKfXEnMDTP+3V+SE4gDz6mG54/g7X/Z68VodAYjnNIf+w7ES29isyz9eg6a3+aBje/8QFuY89prb7B9bH/yAwxV2tYMFj6N+FLgZWcLgH2r/grBwRKHFxKCnlu/6jbDKn7JYGsBEhLybWVSQ/BIc0lVl9ZXcQvoAbrG2NVbaAWvNz5jy+P4nyhmEq/RYAaV3QYdqwTlcoUDH0JWpUsZ7nf/NIOuuGLppN2/2JdCeq10Xbz3snvrkan76eUFQHJ4A0i77z4OXAkr59tYtYKvhLraZIFM/HKeX38AwV+GjyMzTlC7 /Ba+fu38 kC08NPZsJ8j+JjtYqBYx25Quz6kxhO0eTcHeE8bKDWEsUGXwwSRQOlEGjaf+bQk/zrm77hsvBZHE9RPn2INRx39Aqv/8xvXJZK/wLiTGGB9A+tIPEqRv0QMBOyWu5Xh8/vtk6qJkUDrhjrqYgBjpDc0Ui53+tiiOAFIMR/s11kvuyn/TJCPOIBqaUFQLXhOeLr3jO3ORfRed95v8IIHKP56m1yonwj5aZtixKFJp1F7sHFy7aSupsGVky8CVF+KsH3VJzcZhGTS1P5qB0oCR1i/NGMynqTA1Be8VsjHAHLwtFOut/gWKJGMLhXmHUgLoF+DYLx0y2is29taGJ4q4h+cVLIGP37eoEGiC6Fd1ERBHCFFHYCtOZUhYOg8MjflyMvf0nBDQsXQCavBSA6x6H/I9EreYxGPzX81Fl X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Redefine kmalloc, krealloc, kzalloc, kcalloc, etc. to record allocations and deallocations done by these functions. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet Reviewed-by: Kees Cook --- include/linux/fortify-string.h | 5 +- include/linux/slab.h | 169 +++++++++++++++++---------------- include/linux/string.h | 4 +- mm/slab_common.c | 6 +- mm/slub.c | 52 +++++----- mm/util.c | 20 ++-- 6 files changed, 130 insertions(+), 126 deletions(-) diff --git a/include/linux/fortify-string.h b/include/linux/fortify-string.h index 6aeebe0a6777..b24d62bad0b3 100644 --- a/include/linux/fortify-string.h +++ b/include/linux/fortify-string.h @@ -725,9 +725,9 @@ __FORTIFY_INLINE void *memchr_inv(const void * const POS0 p, int c, size_t size) return __real_memchr_inv(p, c, size); } -extern void *__real_kmemdup(const void *src, size_t len, gfp_t gfp) __RENAME(kmemdup) +extern void *__real_kmemdup(const void *src, size_t len, gfp_t gfp) __RENAME(kmemdup_noprof) __realloc_size(2); -__FORTIFY_INLINE void *kmemdup(const void * const POS0 p, size_t size, gfp_t gfp) +__FORTIFY_INLINE void *kmemdup_noprof(const void * const POS0 p, size_t size, gfp_t gfp) { const size_t p_size = __struct_size(p); @@ -737,6 +737,7 @@ __FORTIFY_INLINE void *kmemdup(const void * const POS0 p, size_t size, gfp_t gfp fortify_panic(FORTIFY_FUNC_kmemdup, FORTIFY_READ, p_size, size, NULL); return __real_kmemdup(p, size, gfp); } +#define kmemdup(...) alloc_hooks(kmemdup_noprof(__VA_ARGS__)) /** * strcpy - Copy a string into another string buffer diff --git a/include/linux/slab.h b/include/linux/slab.h index 68ff754b85a4..373862f17694 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -271,7 +271,10 @@ int kmem_cache_shrink(struct kmem_cache *s); /* * Common kmalloc functions provided by all allocators */ -void * __must_check krealloc(const void *objp, size_t new_size, gfp_t flags) __realloc_size(2); +void * __must_check krealloc_noprof(const void *objp, size_t new_size, + gfp_t flags) __realloc_size(2); +#define krealloc(...) alloc_hooks(krealloc_noprof(__VA_ARGS__)) + void kfree(const void *objp); void kfree_sensitive(const void *objp); size_t __ksize(const void *objp); @@ -523,7 +526,10 @@ static __always_inline unsigned int __kmalloc_index(size_t size, static_assert(PAGE_SHIFT <= 20); #define kmalloc_index(s) __kmalloc_index(s, true) -void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_size(1); +#include + +void *__kmalloc_noprof(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_size(1); +#define __kmalloc(...) alloc_hooks(__kmalloc_noprof(__VA_ARGS__)) /** * kmem_cache_alloc - Allocate an object @@ -535,9 +541,14 @@ void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_siz * * Return: pointer to the new object or %NULL in case of error */ -void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags) __assume_slab_alignment __malloc; -void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, - gfp_t gfpflags) __assume_slab_alignment __malloc; +void *kmem_cache_alloc_noprof(struct kmem_cache *cachep, + gfp_t flags) __assume_slab_alignment __malloc; +#define kmem_cache_alloc(...) alloc_hooks(kmem_cache_alloc_noprof(__VA_ARGS__)) + +void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru, + gfp_t gfpflags) __assume_slab_alignment __malloc; +#define kmem_cache_alloc_lru(...) alloc_hooks(kmem_cache_alloc_lru_noprof(__VA_ARGS__)) + void kmem_cache_free(struct kmem_cache *s, void *objp); /* @@ -548,29 +559,40 @@ void kmem_cache_free(struct kmem_cache *s, void *objp); * Note that interrupts must be enabled when calling these functions. */ void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p); -int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p); + +int kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags, size_t size, void **p); +#define kmem_cache_alloc_bulk(...) alloc_hooks(kmem_cache_alloc_bulk_noprof(__VA_ARGS__)) static __always_inline void kfree_bulk(size_t size, void **p) { kmem_cache_free_bulk(NULL, size, p); } -void *__kmalloc_node(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment +void *__kmalloc_node_noprof(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment __alloc_size(1); -void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t flags, int node) __assume_slab_alignment - __malloc; +#define __kmalloc_node(...) alloc_hooks(__kmalloc_node_noprof(__VA_ARGS__)) + +void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t flags, + int node) __assume_slab_alignment __malloc; +#define kmem_cache_alloc_node(...) alloc_hooks(kmem_cache_alloc_node_noprof(__VA_ARGS__)) -void *kmalloc_trace(struct kmem_cache *s, gfp_t flags, size_t size) +void *kmalloc_trace_noprof(struct kmem_cache *s, gfp_t flags, size_t size) __assume_kmalloc_alignment __alloc_size(3); -void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, - int node, size_t size) __assume_kmalloc_alignment +void *kmalloc_node_trace_noprof(struct kmem_cache *s, gfp_t gfpflags, + int node, size_t size) __assume_kmalloc_alignment __alloc_size(4); -void *kmalloc_large(size_t size, gfp_t flags) __assume_page_alignment +#define kmalloc_trace(...) alloc_hooks(kmalloc_trace_noprof(__VA_ARGS__)) + +#define kmalloc_node_trace(...) alloc_hooks(kmalloc_node_trace_noprof(__VA_ARGS__)) + +void *kmalloc_large_noprof(size_t size, gfp_t flags) __assume_page_alignment __alloc_size(1); +#define kmalloc_large(...) alloc_hooks(kmalloc_large_noprof(__VA_ARGS__)) -void *kmalloc_large_node(size_t size, gfp_t flags, int node) __assume_page_alignment +void *kmalloc_large_node_noprof(size_t size, gfp_t flags, int node) __assume_page_alignment __alloc_size(1); +#define kmalloc_large_node(...) alloc_hooks(kmalloc_large_node_noprof(__VA_ARGS__)) /** * kmalloc - allocate kernel memory @@ -626,37 +648,39 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node) __assume_page_align * Try really hard to succeed the allocation but fail * eventually. */ -static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) +static __always_inline __alloc_size(1) void *kmalloc_noprof(size_t size, gfp_t flags) { if (__builtin_constant_p(size) && size) { unsigned int index; if (size > KMALLOC_MAX_CACHE_SIZE) - return kmalloc_large(size, flags); + return kmalloc_large_noprof(size, flags); index = kmalloc_index(size); - return kmalloc_trace( + return kmalloc_trace_noprof( kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], flags, size); } - return __kmalloc(size, flags); + return __kmalloc_noprof(size, flags); } +#define kmalloc(...) alloc_hooks(kmalloc_noprof(__VA_ARGS__)) -static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node) +static __always_inline __alloc_size(1) void *kmalloc_node_noprof(size_t size, gfp_t flags, int node) { if (__builtin_constant_p(size) && size) { unsigned int index; if (size > KMALLOC_MAX_CACHE_SIZE) - return kmalloc_large_node(size, flags, node); + return kmalloc_large_node_noprof(size, flags, node); index = kmalloc_index(size); - return kmalloc_node_trace( + return kmalloc_node_trace_noprof( kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], flags, node, size); } - return __kmalloc_node(size, flags, node); + return __kmalloc_node_noprof(size, flags, node); } +#define kmalloc_node(...) alloc_hooks(kmalloc_node_noprof(__VA_ARGS__)) /** * kmalloc_array - allocate memory for an array. @@ -664,16 +688,17 @@ static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t fla * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -static inline __alloc_size(1, 2) void *kmalloc_array(size_t n, size_t size, gfp_t flags) +static inline __alloc_size(1, 2) void *kmalloc_array_noprof(size_t n, size_t size, gfp_t flags) { size_t bytes; if (unlikely(check_mul_overflow(n, size, &bytes))) return NULL; if (__builtin_constant_p(n) && __builtin_constant_p(size)) - return kmalloc(bytes, flags); - return __kmalloc(bytes, flags); + return kmalloc_noprof(bytes, flags); + return kmalloc_noprof(bytes, flags); } +#define kmalloc_array(...) alloc_hooks(kmalloc_array_noprof(__VA_ARGS__)) /** * krealloc_array - reallocate memory for an array. @@ -682,18 +707,19 @@ static inline __alloc_size(1, 2) void *kmalloc_array(size_t n, size_t size, gfp_ * @new_size: new size of a single member of the array * @flags: the type of memory to allocate (see kmalloc) */ -static inline __realloc_size(2, 3) void * __must_check krealloc_array(void *p, - size_t new_n, - size_t new_size, - gfp_t flags) +static inline __realloc_size(2, 3) void * __must_check krealloc_array_noprof(void *p, + size_t new_n, + size_t new_size, + gfp_t flags) { size_t bytes; if (unlikely(check_mul_overflow(new_n, new_size, &bytes))) return NULL; - return krealloc(p, bytes, flags); + return krealloc_noprof(p, bytes, flags); } +#define krealloc_array(...) alloc_hooks(krealloc_array_noprof(__VA_ARGS__)) /** * kcalloc - allocate memory for an array. The memory is set to zero. @@ -701,16 +727,12 @@ static inline __realloc_size(2, 3) void * __must_check krealloc_array(void *p, * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -static inline __alloc_size(1, 2) void *kcalloc(size_t n, size_t size, gfp_t flags) -{ - return kmalloc_array(n, size, flags | __GFP_ZERO); -} +#define kcalloc(_n, _size, _flags) kmalloc_array(_n, _size, (_flags) | __GFP_ZERO) -void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, +void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, int node, unsigned long caller) __alloc_size(1); -#define kmalloc_node_track_caller(size, flags, node) \ - __kmalloc_node_track_caller(size, flags, node, \ - _RET_IP_) +#define kmalloc_node_track_caller(...) \ + alloc_hooks(kmalloc_node_track_caller_noprof(__VA_ARGS__, _RET_IP_)) /* * kmalloc_track_caller is a special version of kmalloc that records the @@ -720,11 +742,9 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, * allocator where we care about the real place the memory allocation * request comes from. */ -#define kmalloc_track_caller(size, flags) \ - __kmalloc_node_track_caller(size, flags, \ - NUMA_NO_NODE, _RET_IP_) +#define kmalloc_track_caller(...) kmalloc_node_track_caller(__VA_ARGS__, NUMA_NO_NODE) -static inline __alloc_size(1, 2) void *kmalloc_array_node(size_t n, size_t size, gfp_t flags, +static inline __alloc_size(1, 2) void *kmalloc_array_node_noprof(size_t n, size_t size, gfp_t flags, int node) { size_t bytes; @@ -732,75 +752,56 @@ static inline __alloc_size(1, 2) void *kmalloc_array_node(size_t n, size_t size, if (unlikely(check_mul_overflow(n, size, &bytes))) return NULL; if (__builtin_constant_p(n) && __builtin_constant_p(size)) - return kmalloc_node(bytes, flags, node); - return __kmalloc_node(bytes, flags, node); + return kmalloc_node_noprof(bytes, flags, node); + return __kmalloc_node_noprof(bytes, flags, node); } +#define kmalloc_array_node(...) alloc_hooks(kmalloc_array_node_noprof(__VA_ARGS__)) -static inline __alloc_size(1, 2) void *kcalloc_node(size_t n, size_t size, gfp_t flags, int node) -{ - return kmalloc_array_node(n, size, flags | __GFP_ZERO, node); -} +#define kcalloc_node(_n, _size, _flags, _node) \ + kmalloc_array_node(_n, _size, (_flags) | __GFP_ZERO, _node) /* * Shortcuts */ -static inline void *kmem_cache_zalloc(struct kmem_cache *k, gfp_t flags) -{ - return kmem_cache_alloc(k, flags | __GFP_ZERO); -} +#define kmem_cache_zalloc(_k, _flags) kmem_cache_alloc(_k, (_flags)|__GFP_ZERO) /** * kzalloc - allocate memory. The memory is set to zero. * @size: how many bytes of memory are required. * @flags: the type of memory to allocate (see kmalloc). */ -static inline __alloc_size(1) void *kzalloc(size_t size, gfp_t flags) +static inline __alloc_size(1) void *kzalloc_noprof(size_t size, gfp_t flags) { - return kmalloc(size, flags | __GFP_ZERO); + return kmalloc_noprof(size, flags | __GFP_ZERO); } +#define kzalloc(...) alloc_hooks(kzalloc_noprof(__VA_ARGS__)) +#define kzalloc_node(_size, _flags, _node) kmalloc_node(_size, (_flags)|__GFP_ZERO, _node) -/** - * kzalloc_node - allocate zeroed memory from a particular memory node. - * @size: how many bytes of memory are required. - * @flags: the type of memory to allocate (see kmalloc). - * @node: memory node from which to allocate - */ -static inline __alloc_size(1) void *kzalloc_node(size_t size, gfp_t flags, int node) -{ - return kmalloc_node(size, flags | __GFP_ZERO, node); -} +extern void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) __alloc_size(1); +#define kvmalloc_node(...) alloc_hooks(kvmalloc_node_noprof(__VA_ARGS__)) -extern void *kvmalloc_node(size_t size, gfp_t flags, int node) __alloc_size(1); -static inline __alloc_size(1) void *kvmalloc(size_t size, gfp_t flags) -{ - return kvmalloc_node(size, flags, NUMA_NO_NODE); -} -static inline __alloc_size(1) void *kvzalloc_node(size_t size, gfp_t flags, int node) -{ - return kvmalloc_node(size, flags | __GFP_ZERO, node); -} -static inline __alloc_size(1) void *kvzalloc(size_t size, gfp_t flags) -{ - return kvmalloc(size, flags | __GFP_ZERO); -} +#define kvmalloc(_size, _flags) kvmalloc_node(_size, _flags, NUMA_NO_NODE) +#define kvzalloc(_size, _flags) kvmalloc(_size, _flags|__GFP_ZERO) + +#define kvzalloc_node(_size, _flags, _node) kvmalloc_node(_size, _flags|__GFP_ZERO, _node) -static inline __alloc_size(1, 2) void *kvmalloc_array(size_t n, size_t size, gfp_t flags) +static inline __alloc_size(1, 2) void *kvmalloc_array_noprof(size_t n, size_t size, gfp_t flags) { size_t bytes; if (unlikely(check_mul_overflow(n, size, &bytes))) return NULL; - return kvmalloc(bytes, flags); + return kvmalloc_node_noprof(bytes, flags, NUMA_NO_NODE); } -static inline __alloc_size(1, 2) void *kvcalloc(size_t n, size_t size, gfp_t flags) -{ - return kvmalloc_array(n, size, flags | __GFP_ZERO); -} +#define kvmalloc_array(...) alloc_hooks(kvmalloc_array_noprof(__VA_ARGS__)) +#define kvcalloc(_n, _size, _flags) kvmalloc_array(_n, _size, _flags|__GFP_ZERO) -extern void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) +extern void *kvrealloc_noprof(const void *p, size_t oldsize, size_t newsize, gfp_t flags) __realloc_size(3); +#define kvrealloc(...) alloc_hooks(kvrealloc_noprof(__VA_ARGS__)) + extern void kvfree(const void *addr); DEFINE_FREE(kvfree, void *, if (_T) kvfree(_T)) diff --git a/include/linux/string.h b/include/linux/string.h index 9ba8b4597009..793c27ad7c0d 100644 --- a/include/linux/string.h +++ b/include/linux/string.h @@ -282,7 +282,9 @@ extern void kfree_const(const void *x); extern char *kstrdup(const char *s, gfp_t gfp) __malloc; extern const char *kstrdup_const(const char *s, gfp_t gfp); extern char *kstrndup(const char *s, size_t len, gfp_t gfp); -extern void *kmemdup(const void *src, size_t len, gfp_t gfp) __realloc_size(2); +extern void *kmemdup_noprof(const void *src, size_t len, gfp_t gfp) __realloc_size(2); +#define kmemdup(...) alloc_hooks(kmemdup_noprof(__VA_ARGS__)) + extern void *kvmemdup(const void *src, size_t len, gfp_t gfp) __realloc_size(2); extern char *kmemdup_nul(const char *s, size_t len, gfp_t gfp); extern void *kmemdup_array(const void *src, size_t element_size, size_t count, gfp_t gfp); diff --git a/mm/slab_common.c b/mm/slab_common.c index f5234672f03c..3179a6aeffc5 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1189,7 +1189,7 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags) return (void *)p; } - ret = kmalloc_track_caller(new_size, flags); + ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_); if (ret && p) { /* Disable KASAN checks as the object's redzone is accessed. */ kasan_disable_current(); @@ -1213,7 +1213,7 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags) * * Return: pointer to the allocated memory or %NULL in case of error */ -void *krealloc(const void *p, size_t new_size, gfp_t flags) +void *krealloc_noprof(const void *p, size_t new_size, gfp_t flags) { void *ret; @@ -1228,7 +1228,7 @@ void *krealloc(const void *p, size_t new_size, gfp_t flags) return ret; } -EXPORT_SYMBOL(krealloc); +EXPORT_SYMBOL(krealloc_noprof); /** * kfree_sensitive - Clear sensitive information in memory before freeing diff --git a/mm/slub.c b/mm/slub.c index 5840ab963319..a05d4daf1efd 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4002,7 +4002,7 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list return object; } -void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags) +void *kmem_cache_alloc_noprof(struct kmem_cache *s, gfp_t gfpflags) { void *ret = slab_alloc_node(s, NULL, gfpflags, NUMA_NO_NODE, _RET_IP_, s->object_size); @@ -4011,9 +4011,9 @@ void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags) return ret; } -EXPORT_SYMBOL(kmem_cache_alloc); +EXPORT_SYMBOL(kmem_cache_alloc_noprof); -void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, +void *kmem_cache_alloc_lru_noprof(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags) { void *ret = slab_alloc_node(s, lru, gfpflags, NUMA_NO_NODE, _RET_IP_, @@ -4023,10 +4023,10 @@ void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, return ret; } -EXPORT_SYMBOL(kmem_cache_alloc_lru); +EXPORT_SYMBOL(kmem_cache_alloc_lru_noprof); /** - * kmem_cache_alloc_node - Allocate an object on the specified node + * kmem_cache_alloc_node_noprof - Allocate an object on the specified node * @s: The cache to allocate from. * @gfpflags: See kmalloc(). * @node: node number of the target node. @@ -4038,7 +4038,7 @@ EXPORT_SYMBOL(kmem_cache_alloc_lru); * * Return: pointer to the new object or %NULL in case of error */ -void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node) +void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t gfpflags, int node) { void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, s->object_size); @@ -4046,7 +4046,7 @@ void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node) return ret; } -EXPORT_SYMBOL(kmem_cache_alloc_node); +EXPORT_SYMBOL(kmem_cache_alloc_node_noprof); /* * To avoid unnecessary overhead, we pass through large allocation requests @@ -4063,7 +4063,7 @@ static void *__kmalloc_large_node(size_t size, gfp_t flags, int node) flags = kmalloc_fix_flags(flags); flags |= __GFP_COMP; - folio = (struct folio *)alloc_pages_node(node, flags, order); + folio = (struct folio *)alloc_pages_node_noprof(node, flags, order); if (folio) { ptr = folio_address(folio); lruvec_stat_mod_folio(folio, NR_SLAB_UNRECLAIMABLE_B, @@ -4078,7 +4078,7 @@ static void *__kmalloc_large_node(size_t size, gfp_t flags, int node) return ptr; } -void *kmalloc_large(size_t size, gfp_t flags) +void *kmalloc_large_noprof(size_t size, gfp_t flags) { void *ret = __kmalloc_large_node(size, flags, NUMA_NO_NODE); @@ -4086,9 +4086,9 @@ void *kmalloc_large(size_t size, gfp_t flags) flags, NUMA_NO_NODE); return ret; } -EXPORT_SYMBOL(kmalloc_large); +EXPORT_SYMBOL(kmalloc_large_noprof); -void *kmalloc_large_node(size_t size, gfp_t flags, int node) +void *kmalloc_large_node_noprof(size_t size, gfp_t flags, int node) { void *ret = __kmalloc_large_node(size, flags, node); @@ -4096,7 +4096,7 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node) flags, node); return ret; } -EXPORT_SYMBOL(kmalloc_large_node); +EXPORT_SYMBOL(kmalloc_large_node_noprof); static __always_inline void *__do_kmalloc_node(size_t size, gfp_t flags, int node, @@ -4123,26 +4123,26 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, return ret; } -void *__kmalloc_node(size_t size, gfp_t flags, int node) +void *__kmalloc_node_noprof(size_t size, gfp_t flags, int node) { return __do_kmalloc_node(size, flags, node, _RET_IP_); } -EXPORT_SYMBOL(__kmalloc_node); +EXPORT_SYMBOL(__kmalloc_node_noprof); -void *__kmalloc(size_t size, gfp_t flags) +void *__kmalloc_noprof(size_t size, gfp_t flags) { return __do_kmalloc_node(size, flags, NUMA_NO_NODE, _RET_IP_); } -EXPORT_SYMBOL(__kmalloc); +EXPORT_SYMBOL(__kmalloc_noprof); -void *__kmalloc_node_track_caller(size_t size, gfp_t flags, - int node, unsigned long caller) +void *kmalloc_node_track_caller_noprof(size_t size, gfp_t flags, + int node, unsigned long caller) { return __do_kmalloc_node(size, flags, node, caller); } -EXPORT_SYMBOL(__kmalloc_node_track_caller); +EXPORT_SYMBOL(kmalloc_node_track_caller_noprof); -void *kmalloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) +void *kmalloc_trace_noprof(struct kmem_cache *s, gfp_t gfpflags, size_t size) { void *ret = slab_alloc_node(s, NULL, gfpflags, NUMA_NO_NODE, _RET_IP_, size); @@ -4152,9 +4152,9 @@ void *kmalloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) ret = kasan_kmalloc(s, ret, size, gfpflags); return ret; } -EXPORT_SYMBOL(kmalloc_trace); +EXPORT_SYMBOL(kmalloc_trace_noprof); -void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, +void *kmalloc_node_trace_noprof(struct kmem_cache *s, gfp_t gfpflags, int node, size_t size) { void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, size); @@ -4164,7 +4164,7 @@ void *kmalloc_node_trace(struct kmem_cache *s, gfp_t gfpflags, ret = kasan_kmalloc(s, ret, size, gfpflags); return ret; } -EXPORT_SYMBOL(kmalloc_node_trace); +EXPORT_SYMBOL(kmalloc_node_trace_noprof); static noinline void free_to_partial_list( struct kmem_cache *s, struct slab *slab, @@ -4769,8 +4769,8 @@ static int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, #endif /* CONFIG_SLUB_TINY */ /* Note that interrupts must be enabled when calling this function. */ -int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, - void **p) +int kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags, size_t size, + void **p) { int i; struct obj_cgroup *objcg = NULL; @@ -4798,7 +4798,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, return i; } -EXPORT_SYMBOL(kmem_cache_alloc_bulk); +EXPORT_SYMBOL(kmem_cache_alloc_bulk_noprof); /* diff --git a/mm/util.c b/mm/util.c index 669397235787..a79dce7546f1 100644 --- a/mm/util.c +++ b/mm/util.c @@ -115,7 +115,7 @@ char *kstrndup(const char *s, size_t max, gfp_t gfp) EXPORT_SYMBOL(kstrndup); /** - * kmemdup - duplicate region of memory + * kmemdup_noprof - duplicate region of memory * * @src: memory region to duplicate * @len: memory region length @@ -124,16 +124,16 @@ EXPORT_SYMBOL(kstrndup); * Return: newly allocated copy of @src or %NULL in case of error, * result is physically contiguous. Use kfree() to free. */ -void *kmemdup(const void *src, size_t len, gfp_t gfp) +void *kmemdup_noprof(const void *src, size_t len, gfp_t gfp) { void *p; - p = kmalloc_track_caller(len, gfp); + p = kmalloc_node_track_caller_noprof(len, gfp, NUMA_NO_NODE, _RET_IP_); if (p) memcpy(p, src, len); return p; } -EXPORT_SYMBOL(kmemdup); +EXPORT_SYMBOL(kmemdup_noprof); /** * kmemdup_array - duplicate a given array. @@ -594,7 +594,7 @@ unsigned long vm_mmap(struct file *file, unsigned long addr, EXPORT_SYMBOL(vm_mmap); /** - * kvmalloc_node - attempt to allocate physically contiguous memory, but upon + * kvmalloc_node_noprof - attempt to allocate physically contiguous memory, but upon * failure, fall back to non-contiguous (vmalloc) allocation. * @size: size of the request. * @flags: gfp mask for the allocation - must be compatible (superset) with GFP_KERNEL. @@ -609,7 +609,7 @@ EXPORT_SYMBOL(vm_mmap); * * Return: pointer to the allocated memory of %NULL in case of failure */ -void *kvmalloc_node(size_t size, gfp_t flags, int node) +void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) { gfp_t kmalloc_flags = flags; void *ret; @@ -631,7 +631,7 @@ void *kvmalloc_node(size_t size, gfp_t flags, int node) kmalloc_flags &= ~__GFP_NOFAIL; } - ret = kmalloc_node(size, kmalloc_flags, node); + ret = kmalloc_node_noprof(size, kmalloc_flags, node); /* * It doesn't really make sense to fallback to vmalloc for sub page @@ -660,7 +660,7 @@ void *kvmalloc_node(size_t size, gfp_t flags, int node) flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(kvmalloc_node); +EXPORT_SYMBOL(kvmalloc_node_noprof); /** * kvfree() - Free memory. @@ -699,7 +699,7 @@ void kvfree_sensitive(const void *addr, size_t len) } EXPORT_SYMBOL(kvfree_sensitive); -void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) +void *kvrealloc_noprof(const void *p, size_t oldsize, size_t newsize, gfp_t flags) { void *newp; @@ -712,7 +712,7 @@ void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) kvfree(p); return newp; } -EXPORT_SYMBOL(kvrealloc); +EXPORT_SYMBOL(kvrealloc_noprof); /** * __vmalloc_array - allocate memory for a virtually contiguous array. From patchwork Thu Mar 21 16:36:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599183 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E814C54E58 for ; Thu, 21 Mar 2024 16:38:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E26486B00BB; Thu, 21 Mar 2024 12:38:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DB1026B00BC; Thu, 21 Mar 2024 12:38:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB2486B00BD; Thu, 21 Mar 2024 12:38:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9C3526B00BB for ; Thu, 21 Mar 2024 12:38:09 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 78B2D1A1624 for ; Thu, 21 Mar 2024 16:38:09 +0000 (UTC) X-FDA: 81921603498.03.DB03057 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf19.hostedemail.com (Postfix) with ESMTP id 9E9251A0003 for ; Thu, 21 Mar 2024 16:38:07 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=kSzzH6Eh; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3bmL8ZQYKCFsLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3bmL8ZQYKCFsLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039087; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kbJXFr2w8KI4PQ3iTaDylf6laxNigfzl4BgVu2a2cxc=; b=PG4ZW4K/Q6S+vNMy2Wz/2nshJI0UnRgC4j2giV8ESC8pDT4WpL8RyIF6ijFTKeERkuG1dz TTmU3SEN2t1geMgsikFnvU6gPaujkI7+akQST3Pt90oBx9jNxgT5Qd4cALd4AIJlFKpIDA UY24YbNgy+naOIDKoygak7NJX4rifH4= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=kSzzH6Eh; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3bmL8ZQYKCFsLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3bmL8ZQYKCFsLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039087; a=rsa-sha256; cv=none; b=Y2MbucY3NYmJFCP2WLari8l4DtZknb9C+s3T6zrYkkJhmJV7T1BqECyt/5m/jdofv6o7/2 TFGSc6qyhsE/3qWy+uibUW1PTdWIaC2smCgp2RWuqRHdFr5qK72f2Yprz0WVNxWoF1rnWh wVAqh646dzm63D2o531L5F9wAKEOmKY= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60a605154d0so14623377b3.0 for ; Thu, 21 Mar 2024 09:38:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039087; x=1711643887; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kbJXFr2w8KI4PQ3iTaDylf6laxNigfzl4BgVu2a2cxc=; b=kSzzH6EhDsTp6j4E/RGlXDiJPg8Q0P+Tcwufibr3ubaA+Nvb/36hWwHjwNviOEne8P Ms3IwChgiIL3/LF0Ok3N4xf8OyF5PqOI0J7KWWuYLlJonwgs8c4GmfiSTjUSmEui4fYa z3mMHIV0gLjb57lPlr2UcTzuxNc61mg1P1+u+PHP5hUc00T4YnZ/dWwSd3IGmJ8+BknO KDdQNtS5ro22U8OgQM2MaPOH5hpDqU6Rm+W+LpTGJOZZMgRoSjkv3lPsFqTjU5mrTaDV W/V575eRmhIxzKT1Sw3parn6ml6tsepC2dlsbcih5kDtVbeAj5WLtSIwudaxYRpHPRmv pudA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039087; x=1711643887; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kbJXFr2w8KI4PQ3iTaDylf6laxNigfzl4BgVu2a2cxc=; b=sQ2Ho9EUwv8zGUv4ppA/+rfHFH7rpYWcZrhRoayYgnV7XJEvetnmgxi4IjftJ+74u7 O4e5RDLaEedhuGs/hqOBVZYnrYtSZDD7hjG37OWNJUEQnJ/BXbKqUTbpqy9odpYZmZt4 DmcRSQdzJ1hiJmCReNVWKyPr3O6OOMZfpfNUGZbQsOLio1XIk4AWlKf3YSac7NCaccpC lfxXLzr2Ubgeu3WoVDN2gYIMY0ZE1Obtok1lgi6W6e0WeI2bRZCYPzNKtwKoTwxTq3qi XE55zRwPLozJ9fGpklLiwwWnMAGQJIHndzAuPTOlsDm8RRbMDRDTJlaj4N9xdsv4qRu9 tD8Q== X-Forwarded-Encrypted: i=1; AJvYcCVdzRBLxPUX1XO8XMI+4eNBhZ+l5XzLkMyNn9Z6VGzLn8myRGSsZprkaTasFHR8NizNu+VvcQ/Qq+OnJvFJ11Fr8aU= X-Gm-Message-State: AOJu0YyB8FGNDFDm8NOuL4ysIb5WflbOg60sk8bwq93WV0E7EY5AqMY1 dF6BYdYrj2TeFQJNM47hdWRyPTcKmcm9Y9sONEfJYwhogenWQPGbcmn4AhfiXD8GyIry0Th4lQh mWw== X-Google-Smtp-Source: AGHT+IGAFq37Jmasj0HYRGF5DFuiG3xE73e4VFfbf3VLLmYen1YphSWB2duspGO/poHTwQ+f8dcPx9x7Bl8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a81:a1d2:0:b0:610:e44b:acc3 with SMTP id y201-20020a81a1d2000000b00610e44bacc3mr857961ywg.4.1711039086504; Thu, 21 Mar 2024 09:38:06 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:48 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-27-surenb@google.com> Subject: [PATCH v6 26/37] mempool: Hook up to memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Stat-Signature: cbuh769y1q47xqx58pngfk895ack9mqb X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 9E9251A0003 X-HE-Tag: 1711039087-719297 X-HE-Meta: U2FsdGVkX1+4x6NNpT78o3t5PfJDSinsXerUWUPqtJvypsJy24vVJFSdQefQFTAgt9wFKO+mKi0MW4JYudnRfgmD9pvooN9BV/b7mf+ZJo3vxCopan2t1jUB87OuHDwj3oNdi9gWc7UcHQcb4/Zyllx+SHqYr5I2JS1Xw8H1cqfBBBCTlrHyR0p6LKWuXxITiWG/EPe+6XN1R1Xa5KbCY7BHE7zuNVMiqrl4nokfSI4UBnx++Gd0JYoQNI+tC0nvFkOSIYkUg6DWvUg0r76b6JP660I36Q2aUiiC0yBAZjSQCB6xZabKDBhw4K53xMGzsH96+S3bx+wVhz9CkhzNfGrwzmmNhqJ63r6cJt9jznCt1i8eF60xP+RPsYRGtgJleY8nuM0dB8DerpmjuSid/jlJGwn6imsDboEJAA0lYn0dfvSxOYsTTGe6hyYz1Imf0fdx6RKZ30wgSX6RDrZ3hGrNfl51cDb6mdTdyQ9e+FsFoNEudV4lNQaunY45mKqYm1LNffMvLJjB7hmEerHBTuGWT964BnsIl8rWT2FdEzOekd4h3wyC2/r9B4ePnt6RSuQVSFFLCX9ObBqtENDGdplSLrJ+Ffcqq0fOASa/5/HdLSduxT1LdCbMgmvQ1SsR+XcbU6Ju7DJSwKw5s5+MllvWexoe2M9rsVfX6ltJoNzIi28LusFU/YLLcMpdgUl85Lc3LHweIWufOpcTYR2wQMXP8CmAzvt0LC3WGT+NDMyOhk/S1gvmNYrw2Vt7mPqz+89YAKdHvsgnP2qYOr/I5cbBDSzM3jPJnH1WnqPzGXEOyL9pTj6hMpPAiMRxnpimVS54V7yrHDwx6IXxYeJkPDbIwlePOQqCSbgNujZ+/bcztIAEnO8+lhNENYqro1FHl4af39fNnQqtrdUT+ZYjaROHOiv8TwHoK42ArsMnFPhibmYAnaVFDInaf2TPTnGg+UJ9Ii4IwTzsrYgSz2B Aj88+Er/ uS6GWN5P2Uc0tfPoo52Dcs4+Lmxvjl4kpiEvRjV6fTbPycIB+C0YIc1p426abuPOr2tH3BxnaenZzUAjxgXf+kXAH/UXlnkwLRysSbEAdwyH3jsqmI4Lcz7yuz0JQYm4AruIe+jmfJXyjEO5/dSe7pduQZxNP6zsxIjrB60TyIvJ29WVNmvWlTbC3DkfhdrrtdQ0srvG2BJAZAct+q+GT7dYLF7HR5DYUD+m09QQq1YKP5ExbcNJzLUkFgR+5y5DRbZPxfIFPdYyJmRg68IxKgv6tiPlmoMNo0x0yIlB6ekAXWEMbv5jiMeix7ExquO7XK5pwkde7z6B8ImpRfSjZW/eb6rKiMhVv8W65HFoaFhLfOijghry/b/sJnIepKjr62VmpAGcFOYQCdELq04im61WC5QSdQK+XGkYb9JBmZkCzOEdXTFI1yBd+70873xujezUz5c3RkkU0fuueqC4kzY4WB1DgHz00ZZs56WXLStnbE1A8hXubryytZZXaOjgmr8JBndl/qbtg+UWDWpkpcCgQBHxAN+YJb2N07z5N6GLRTYi7YH9jmZerbMg+con3+CIrojNg12l6vVOY4Kw7C+fgS39cD0vlLgAD08t4sQjVoNyTwGIcQZrJWKpKqs2KOwbSgML7m3GM7W/kJthkF1O7tB9s8OgW70vpIiOoUeaeidU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet This adds hooks to mempools for correctly annotating mempool-backed allocations at the correct source line, so they show up correctly in /sys/kernel/debug/allocations. Various inline functions are converted to wrappers so that we can invoke alloc_hooks() in fewer places. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/mempool.h | 73 ++++++++++++++++++++--------------------- mm/mempool.c | 36 ++++++++------------ 2 files changed, 49 insertions(+), 60 deletions(-) diff --git a/include/linux/mempool.h b/include/linux/mempool.h index 16c5cc807ff6..7b151441341b 100644 --- a/include/linux/mempool.h +++ b/include/linux/mempool.h @@ -5,6 +5,8 @@ #ifndef _LINUX_MEMPOOL_H #define _LINUX_MEMPOOL_H +#include +#include #include #include @@ -39,18 +41,32 @@ void mempool_exit(mempool_t *pool); int mempool_init_node(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, gfp_t gfp_mask, int node_id); -int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + +int mempool_init_noprof(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data); +#define mempool_init(...) \ + alloc_hooks(mempool_init_noprof(__VA_ARGS__)) extern mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data); -extern mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, + +extern mempool_t *mempool_create_node_noprof(int min_nr, mempool_alloc_t *alloc_fn, mempool_free_t *free_fn, void *pool_data, gfp_t gfp_mask, int nid); +#define mempool_create_node(...) \ + alloc_hooks(mempool_create_node_noprof(__VA_ARGS__)) + +#define mempool_create(_min_nr, _alloc_fn, _free_fn, _pool_data) \ + mempool_create_node(_min_nr, _alloc_fn, _free_fn, _pool_data, \ + GFP_KERNEL, NUMA_NO_NODE) extern int mempool_resize(mempool_t *pool, int new_min_nr); extern void mempool_destroy(mempool_t *pool); -extern void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) __malloc; + +extern void *mempool_alloc_noprof(mempool_t *pool, gfp_t gfp_mask) __malloc; +#define mempool_alloc(...) \ + alloc_hooks(mempool_alloc_noprof(__VA_ARGS__)) + extern void *mempool_alloc_preallocated(mempool_t *pool) __malloc; extern void mempool_free(void *element, mempool_t *pool); @@ -62,19 +78,10 @@ extern void mempool_free(void *element, mempool_t *pool); void *mempool_alloc_slab(gfp_t gfp_mask, void *pool_data); void mempool_free_slab(void *element, void *pool_data); -static inline int -mempool_init_slab_pool(mempool_t *pool, int min_nr, struct kmem_cache *kc) -{ - return mempool_init(pool, min_nr, mempool_alloc_slab, - mempool_free_slab, (void *) kc); -} - -static inline mempool_t * -mempool_create_slab_pool(int min_nr, struct kmem_cache *kc) -{ - return mempool_create(min_nr, mempool_alloc_slab, mempool_free_slab, - (void *) kc); -} +#define mempool_init_slab_pool(_pool, _min_nr, _kc) \ + mempool_init(_pool, (_min_nr), mempool_alloc_slab, mempool_free_slab, (void *)(_kc)) +#define mempool_create_slab_pool(_min_nr, _kc) \ + mempool_create((_min_nr), mempool_alloc_slab, mempool_free_slab, (void *)(_kc)) /* * a mempool_alloc_t and a mempool_free_t to kmalloc and kfree the @@ -83,17 +90,12 @@ mempool_create_slab_pool(int min_nr, struct kmem_cache *kc) void *mempool_kmalloc(gfp_t gfp_mask, void *pool_data); void mempool_kfree(void *element, void *pool_data); -static inline int mempool_init_kmalloc_pool(mempool_t *pool, int min_nr, size_t size) -{ - return mempool_init(pool, min_nr, mempool_kmalloc, - mempool_kfree, (void *) size); -} - -static inline mempool_t *mempool_create_kmalloc_pool(int min_nr, size_t size) -{ - return mempool_create(min_nr, mempool_kmalloc, mempool_kfree, - (void *) size); -} +#define mempool_init_kmalloc_pool(_pool, _min_nr, _size) \ + mempool_init(_pool, (_min_nr), mempool_kmalloc, mempool_kfree, \ + (void *)(unsigned long)(_size)) +#define mempool_create_kmalloc_pool(_min_nr, _size) \ + mempool_create((_min_nr), mempool_kmalloc, mempool_kfree, \ + (void *)(unsigned long)(_size)) void *mempool_kvmalloc(gfp_t gfp_mask, void *pool_data); void mempool_kvfree(void *element, void *pool_data); @@ -115,16 +117,11 @@ static inline mempool_t *mempool_create_kvmalloc_pool(int min_nr, size_t size) void *mempool_alloc_pages(gfp_t gfp_mask, void *pool_data); void mempool_free_pages(void *element, void *pool_data); -static inline int mempool_init_page_pool(mempool_t *pool, int min_nr, int order) -{ - return mempool_init(pool, min_nr, mempool_alloc_pages, - mempool_free_pages, (void *)(long)order); -} - -static inline mempool_t *mempool_create_page_pool(int min_nr, int order) -{ - return mempool_create(min_nr, mempool_alloc_pages, mempool_free_pages, - (void *)(long)order); -} +#define mempool_init_page_pool(_pool, _min_nr, _order) \ + mempool_init(_pool, (_min_nr), mempool_alloc_pages, \ + mempool_free_pages, (void *)(long)(_order)) +#define mempool_create_page_pool(_min_nr, _order) \ + mempool_create((_min_nr), mempool_alloc_pages, \ + mempool_free_pages, (void *)(long)(_order)) #endif /* _LINUX_MEMPOOL_H */ diff --git a/mm/mempool.c b/mm/mempool.c index 076c736f5f1f..602e6eba68d3 100644 --- a/mm/mempool.c +++ b/mm/mempool.c @@ -240,17 +240,17 @@ EXPORT_SYMBOL(mempool_init_node); * * Return: %0 on success, negative error code otherwise. */ -int mempool_init(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, - mempool_free_t *free_fn, void *pool_data) +int mempool_init_noprof(mempool_t *pool, int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data) { return mempool_init_node(pool, min_nr, alloc_fn, free_fn, pool_data, GFP_KERNEL, NUMA_NO_NODE); } -EXPORT_SYMBOL(mempool_init); +EXPORT_SYMBOL(mempool_init_noprof); /** - * mempool_create - create a memory pool + * mempool_create_node - create a memory pool * @min_nr: the minimum number of elements guaranteed to be * allocated for this pool. * @alloc_fn: user-defined element-allocation function. @@ -265,17 +265,9 @@ EXPORT_SYMBOL(mempool_init); * * Return: pointer to the created memory pool object or %NULL on error. */ -mempool_t *mempool_create(int min_nr, mempool_alloc_t *alloc_fn, - mempool_free_t *free_fn, void *pool_data) -{ - return mempool_create_node(min_nr, alloc_fn, free_fn, pool_data, - GFP_KERNEL, NUMA_NO_NODE); -} -EXPORT_SYMBOL(mempool_create); - -mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, - mempool_free_t *free_fn, void *pool_data, - gfp_t gfp_mask, int node_id) +mempool_t *mempool_create_node_noprof(int min_nr, mempool_alloc_t *alloc_fn, + mempool_free_t *free_fn, void *pool_data, + gfp_t gfp_mask, int node_id) { mempool_t *pool; @@ -291,7 +283,7 @@ mempool_t *mempool_create_node(int min_nr, mempool_alloc_t *alloc_fn, return pool; } -EXPORT_SYMBOL(mempool_create_node); +EXPORT_SYMBOL(mempool_create_node_noprof); /** * mempool_resize - resize an existing memory pool @@ -374,7 +366,7 @@ int mempool_resize(mempool_t *pool, int new_min_nr) EXPORT_SYMBOL(mempool_resize); /** - * mempool_alloc - allocate an element from a specific memory pool + * mempool_alloc_noprof - allocate an element from a specific memory pool * @pool: pointer to the memory pool which was allocated via * mempool_create(). * @gfp_mask: the usual allocation bitmask. @@ -387,7 +379,7 @@ EXPORT_SYMBOL(mempool_resize); * * Return: pointer to the allocated element or %NULL on error. */ -void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) +void *mempool_alloc_noprof(mempool_t *pool, gfp_t gfp_mask) { void *element; unsigned long flags; @@ -454,7 +446,7 @@ void *mempool_alloc(mempool_t *pool, gfp_t gfp_mask) finish_wait(&pool->wait, &wait); goto repeat_alloc; } -EXPORT_SYMBOL(mempool_alloc); +EXPORT_SYMBOL(mempool_alloc_noprof); /** * mempool_alloc_preallocated - allocate an element from preallocated elements @@ -562,7 +554,7 @@ void *mempool_alloc_slab(gfp_t gfp_mask, void *pool_data) { struct kmem_cache *mem = pool_data; VM_BUG_ON(mem->ctor); - return kmem_cache_alloc(mem, gfp_mask); + return kmem_cache_alloc_noprof(mem, gfp_mask); } EXPORT_SYMBOL(mempool_alloc_slab); @@ -580,7 +572,7 @@ EXPORT_SYMBOL(mempool_free_slab); void *mempool_kmalloc(gfp_t gfp_mask, void *pool_data) { size_t size = (size_t)pool_data; - return kmalloc(size, gfp_mask); + return kmalloc_noprof(size, gfp_mask); } EXPORT_SYMBOL(mempool_kmalloc); @@ -610,7 +602,7 @@ EXPORT_SYMBOL(mempool_kvfree); void *mempool_alloc_pages(gfp_t gfp_mask, void *pool_data) { int order = (int)(long)pool_data; - return alloc_pages(gfp_mask, order); + return alloc_pages_noprof(gfp_mask, order); } EXPORT_SYMBOL(mempool_alloc_pages); From patchwork Thu Mar 21 16:36:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599184 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E31B7C54E58 for ; Thu, 21 Mar 2024 16:38:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 266716B0085; Thu, 21 Mar 2024 12:38:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F0026B00BC; Thu, 21 Mar 2024 12:38:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F0E076B00BE; Thu, 21 Mar 2024 12:38:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id DB63D6B00BC for ; Thu, 21 Mar 2024 12:38:11 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id AE3ECA0278 for ; Thu, 21 Mar 2024 16:38:11 +0000 (UTC) X-FDA: 81921603582.10.9318202 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf19.hostedemail.com (Postfix) with ESMTP id D117A1A000D for ; Thu, 21 Mar 2024 16:38:09 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=4Ty7kFCX; spf=pass (imf19.hostedemail.com: domain of 3cGL8ZQYKCF0NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3cGL8ZQYKCF0NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039089; a=rsa-sha256; cv=none; b=zGtqxV6IpNrzCd8S1VI1+06ryf0G7QcmGspecoWbMIfEqYRqBS0CQuTDzAzjr51EjWU2oF 4bgES7Oe4ezCQZPqALt9lq35/vGMoHAqoXMa1WmgOG53ktb4XmUeCoLptTgPxAueHCjtQd UssRCeWekLvCH4DiccN4mONACxMX6SQ= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=4Ty7kFCX; spf=pass (imf19.hostedemail.com: domain of 3cGL8ZQYKCF0NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3cGL8ZQYKCF0NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039089; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DhlVYTL0Ii8S3ZK2BTRjXFWA8z6FsoYkfvkLJwCmND0=; b=zz/CdzJ5y27Gy9jzOb09xEKWH6VYHPC6znGsiiuoC2QwBCD3yo9BXgy2k987GsJ5Yjd9Qy TOBbJUe8BKzcVAa/6KKKxeIJHz3VfDlYspOkq33B2oTQ7JXdJIOqxvdayOdBBFGfG9uqN1 eW1VUEcwa2wyU/tdRgTsYrTuZgPdSKY= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dd1395fd1bfso1868871276.0 for ; Thu, 21 Mar 2024 09:38:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039089; x=1711643889; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=DhlVYTL0Ii8S3ZK2BTRjXFWA8z6FsoYkfvkLJwCmND0=; b=4Ty7kFCXS6+32k0edq6Jh3bOQnQAF2dO8+4LDphQjvw+6IYBIz3bJt4bRpMHfW2Cpj 32gY+UvX0D4UFLBTgFlRbwvM+47PeAzlHO2SGrfzPLDp4z3i1Bi7Lb3Ql6OsRr2VhER9 CROgELONXID+Uzx0XlutXH/fwKIuMuEZtZkNjGy3AjSBPiZz8fMKXU66JmDSKYTexIPl Msdhu0roGSPZeA5SuPuGrnP+oI342o0XT1JZEwmnpF13jSoNAezy2IweXy+XXKdLGo1S oD1wIvMYOhecPLWPXLly1QeQjulD1RggBFhDxkb629Frodb4rXCkBEC69Uk1Jz+aUreB gWGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039089; x=1711643889; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DhlVYTL0Ii8S3ZK2BTRjXFWA8z6FsoYkfvkLJwCmND0=; b=MmL8n8+Qwqmua/M+qHrm5wu8qJEE+FHoOP0UYREsEfNYvgek4PzMnnZqJJhWiZC6Hs q1/O0OEjcIoPa8AQKr2tu7Af1SH/BHrsv+Fg9E65dNO4jkccC9YYZIrgd1Ac7wlsUtnn S6dvbzorfljDORjNcjEL9q9GdcJ0dPM7gMfdoQbiEhzgvZ8ED9pFz8T2G0gGOlEOfjeC g7Gpy/lfaG+VxKjmXm24nrSgjleBCm7jt4GSHJIsS2MxVxdp/jEODDDv87LXeTVLJmk4 J5yagwKB9j8TMX4RGQNrACK8WK1UeKsfau6eXAr8wqwkA5PjP09rInF2YapnXd7uG6OT KO2Q== X-Forwarded-Encrypted: i=1; AJvYcCV76fv94MKeKkLW41gm5GD3Jl4fG/omRkskDVHcpl5P3jrW2qPb2WHMIod27l8Hb5we+u4Jbg/C3AiuJ2lSknNcEBk= X-Gm-Message-State: AOJu0Yw8x0mfbx+EiAAW8pqXzkwCceIqvBHD971mEDExLNi51swMXfZ9 oZKJy2bOSOf47J6sIVWTkpQwIHiE24Vh4lxGJGf1oeZ1A5xC7ehr6Zw7GP9cl1RPeNSKcESQ6RB 3ZQ== X-Google-Smtp-Source: AGHT+IFC+oFCb8HvR8Vt3k92TqGML+O+aM2nK5Ub8xVAQ3egP3eKpX1Sre3j/wPk5Gz2RQOGjx8MvWsloBo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:2689:b0:dcb:e4a2:1ab1 with SMTP id dx9-20020a056902268900b00dcbe4a21ab1mr2382761ybb.11.1711039088572; Thu, 21 Mar 2024 09:38:08 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:49 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-28-surenb@google.com> Subject: [PATCH v6 27/37] mm: percpu: Introduce pcpuobj_ext From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: D117A1A000D X-Stat-Signature: wancai3yiphdiw35eyi3zeyey376pird X-Rspam-User: X-HE-Tag: 1711039089-502926 X-HE-Meta: U2FsdGVkX18WMbcP6jaH3G4VHFsoOav5xc4qWJQk6eb1jTfGXu0UnBd9HsIAJd1orjNkVxS+K0OUaLCAL93SxEDBOGuVzzG5qchlz5pWkY82TPRh8v7TwzTdBxvdnhe8YZUzprdn+8QocLDSGfjA3UG1eEP3RtkuUjuWFQX1aHsLtbQ/H75rfotjJzmborYOQqNDIEan4dSAFpDTpUZq1QJ2tnZBlJe/1CZyg+nu3Vlya+1HVNXf2NvzrcEiac1i0QbkeMmtAXWiNz3LuphpxASoyGDuXJk2IBML+pxq7JWtyaYiSZYW0TmoE5oHtyr78fNwJEX2VqdVmO+/9PI9UZiuLMjZIEh3hbg6sp5uz8esE487Xe6NRyQnjFzXI/JFzwJDlaqBwFDq5Qks9BB8U93hWrUOh3THuJTLYb9e7UmEXoDT/LceTKKiSSFUB7O7THNdv2QTp+gOu6bevCit02nyI6v7i22vzDYYkO1NHBMwbzia+rppMuM7OL2DEqcbxcutOHy5jXO9VlJfjGZE5IsrD0f1X4irrtLVgjWsypZT/bCvPBSClQrFLsJ5575SZopw/9tVkZI3flNnoQfDrSuUc1tqgIw2J1778x486ra8KVTXJItIXPZHKIJO6dJAcRRBPCcApTDT8bpPAiS1ja63oB4D7BvPhG1ZeV7BSdoTTVH2ftDKF8DZ/ZaLiiGt9SudjVJue3afbZfqJ/cgazNoIb0qvLOAbsI7BwK4lMWoKEizrXUgpj1NHIrxJfzetbrGlct/o66NFbilUiJrGSrXGzE0v4KnatxR0CG6STP/vGtbGEtfKnR2y6GWPKWWg7T3HRaKSLSXLgawLWov3pj1HdKFeBZ6sZ8D9Fy8TSIw4KvjYqwFb5g5LHAdtfqT5fQH0D24CW+wEiFxMGmIwNxKUhsP4RZGkeIDDuSWk6nyb3ohb+RtuP8WeQAA80/Dy3/qTnOn0XhWzr5YUPL XM9ab4CT Kqk/CgfvbyJ4XRbfIaObca9NW0nQKu/gBLZLc8cGfz5Q1gYhH3Cx55MqBCTV7HhlYuooF8PnFKpJhFDE5LIRWOVexHcIRN9aNBTA6w7FXI3Kw7KG7J1pFDFSuQJ1e+zgTdjmIIiD2K3WIQyDs/tH/fCjFPmWCNIVNVrQhznK+dtnvqoJwgGDBc96Q4YlIZkSi33K1oR+WGks/twTongVVWuf42DNot3ZK4nZRQ2lKGEl796iP7hmn8brblZIv/JRGRgzb9sbE0XDaWhWECWhAbkkbaTkiZVv8UBROTR5y1/q9vJSMhCJkrx3VGgkLZcDX8b7aaq6LLxzaw25RixyBZkBoymKc7nYPdzIvlHHgGsm1ZFK95Hh59LEw2xeFN1TkhQYzsQfByhQ658epvUQdy7ycK7aWi72NQZ0E04bz9+rj6CYNvbvRp8RqOLGzi7luMwafrKcR6l5rgh1jMMyE7YKEqkDtQIUbhg8GPSOB0awYbWkPVL5N4H3vTPRAMPunksXGlLIIJhxELZT78wBIIoBo2lJYZ9cP7o7dJv6gZpYpq8W2mLVLSo/Zzk7as9uHumr9g5xketI/o6Q2rXIGymzBnp8L3AELYuHF4+mrC3KM8xagdoWQlPbDdjbkrW/J5ZTbUCRQP1Td2R6MUPXJ8jg6Au7IA4obWLEVCO8PvZHFJPwNb7iqGyB+Lwpz7mL9eloFTGedSAv0lYr3GrpyjLeY8zPWZAATj6swNCjO7WYuiZCsW+f/anciWJIf3QDkC+TGCOpRyKHxoFDe/LJ15SayHJIBQrk4aOHS/Ia5ovI0fpABd1NN3JyZuxsQfmWP2ubc7aPvC2VjygvK2CPZAyo4XVRktA23lhdU5y1ojcRVBHqO6jo/1giPLXaYr7h1WLDuy/UJtCfqahqpYAX00s/Fr0yWIIzkT9lvdlQb9LnR/ZN72JO+dVL4FS+tELQC/m970cWacW+w5+z6r/juOEdreA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet Upcoming alloc tagging patches require a place to stash per-allocation metadata. We already do this when memcg is enabled, so this patch generalizes the obj_cgroup * vector in struct pcpu_chunk by creating a pcpu_obj_ext type, which we will be adding to in an upcoming patch - similarly to the previous slabobj_ext patch. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Cc: Andrew Morton Cc: Dennis Zhou Cc: Tejun Heo Cc: Christoph Lameter Cc: linux-mm@kvack.org --- mm/percpu-internal.h | 19 +++++++++++++++++-- mm/percpu.c | 30 +++++++++++++++--------------- 2 files changed, 32 insertions(+), 17 deletions(-) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index cdd0aa597a81..e62d582f4bf3 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -32,6 +32,16 @@ struct pcpu_block_md { int nr_bits; /* total bits responsible for */ }; +struct pcpuobj_ext { +#ifdef CONFIG_MEMCG_KMEM + struct obj_cgroup *cgroup; +#endif +}; + +#ifdef CONFIG_MEMCG_KMEM +#define NEED_PCPUOBJ_EXT +#endif + struct pcpu_chunk { #ifdef CONFIG_PERCPU_STATS int nr_alloc; /* # of allocations */ @@ -64,8 +74,8 @@ struct pcpu_chunk { int end_offset; /* additional area required to have the region end page aligned */ -#ifdef CONFIG_MEMCG_KMEM - struct obj_cgroup **obj_cgroups; /* vector of object cgroups */ +#ifdef NEED_PCPUOBJ_EXT + struct pcpuobj_ext *obj_exts; /* vector of object cgroups */ #endif int nr_pages; /* # of pages served by this chunk */ @@ -74,6 +84,11 @@ struct pcpu_chunk { unsigned long populated[]; /* populated bitmap */ }; +static inline bool need_pcpuobj_ext(void) +{ + return !mem_cgroup_kmem_disabled(); +} + extern spinlock_t pcpu_lock; extern struct list_head *pcpu_chunk_lists; diff --git a/mm/percpu.c b/mm/percpu.c index 4e11fc1e6def..2e5edaad9cc3 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1392,9 +1392,9 @@ static struct pcpu_chunk * __init pcpu_alloc_first_chunk(unsigned long tmp_addr, panic("%s: Failed to allocate %zu bytes\n", __func__, alloc_size); -#ifdef CONFIG_MEMCG_KMEM +#ifdef NEED_PCPUOBJ_EXT /* first chunk is free to use */ - chunk->obj_cgroups = NULL; + chunk->obj_exts = NULL; #endif pcpu_init_md_blocks(chunk); @@ -1463,12 +1463,12 @@ static struct pcpu_chunk *pcpu_alloc_chunk(gfp_t gfp) if (!chunk->md_blocks) goto md_blocks_fail; -#ifdef CONFIG_MEMCG_KMEM - if (!mem_cgroup_kmem_disabled()) { - chunk->obj_cgroups = +#ifdef NEED_PCPUOBJ_EXT + if (need_pcpuobj_ext()) { + chunk->obj_exts = pcpu_mem_zalloc(pcpu_chunk_map_bits(chunk) * - sizeof(struct obj_cgroup *), gfp); - if (!chunk->obj_cgroups) + sizeof(struct pcpuobj_ext), gfp); + if (!chunk->obj_exts) goto objcg_fail; } #endif @@ -1480,7 +1480,7 @@ static struct pcpu_chunk *pcpu_alloc_chunk(gfp_t gfp) return chunk; -#ifdef CONFIG_MEMCG_KMEM +#ifdef NEED_PCPUOBJ_EXT objcg_fail: pcpu_mem_free(chunk->md_blocks); #endif @@ -1498,8 +1498,8 @@ static void pcpu_free_chunk(struct pcpu_chunk *chunk) { if (!chunk) return; -#ifdef CONFIG_MEMCG_KMEM - pcpu_mem_free(chunk->obj_cgroups); +#ifdef NEED_PCPUOBJ_EXT + pcpu_mem_free(chunk->obj_exts); #endif pcpu_mem_free(chunk->md_blocks); pcpu_mem_free(chunk->bound_map); @@ -1646,9 +1646,9 @@ static void pcpu_memcg_post_alloc_hook(struct obj_cgroup *objcg, if (!objcg) return; - if (likely(chunk && chunk->obj_cgroups)) { + if (likely(chunk && chunk->obj_exts)) { obj_cgroup_get(objcg); - chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = objcg; + chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].cgroup = objcg; rcu_read_lock(); mod_memcg_state(obj_cgroup_memcg(objcg), MEMCG_PERCPU_B, @@ -1663,13 +1663,13 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *chunk, int off, size_t size) { struct obj_cgroup *objcg; - if (unlikely(!chunk->obj_cgroups)) + if (unlikely(!chunk->obj_exts)) return; - objcg = chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT]; + objcg = chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].cgroup; if (!objcg) return; - chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = NULL; + chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].cgroup = NULL; obj_cgroup_uncharge(objcg, pcpu_obj_full_size(size)); From patchwork Thu Mar 21 16:36:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40472C6FD1F for ; Thu, 21 Mar 2024 16:38:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F38636B00BC; Thu, 21 Mar 2024 12:38:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EC0F76B00BD; Thu, 21 Mar 2024 12:38:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D11366B00BE; Thu, 21 Mar 2024 12:38:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BE02F6B00BC for ; Thu, 21 Mar 2024 12:38:13 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 94B6780473 for ; Thu, 21 Mar 2024 16:38:13 +0000 (UTC) X-FDA: 81921603666.19.41DBA6F Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf24.hostedemail.com (Postfix) with ESMTP id BA11E18001B for ; Thu, 21 Mar 2024 16:38:11 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zH7SHGhG; spf=pass (imf24.hostedemail.com: domain of 3cmL8ZQYKCF8PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3cmL8ZQYKCF8PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039091; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=krRMCxbEdPvR0gpNvdmt+1GzNi+HHBvoijzlHbN4rmY=; b=yELQTqb8nl0b1MqXs2+ipeiyjEDaaYEX4r5GmtHmt78K/V8DEATF3VvLxrRaivzmLCsAFH pFEdzq8OBJg1V6KVDBDYdNDeURTpXQt8x+hw5xBFX3CD18BkQ+RF3hpS6IgiP4p3U5Ftp7 x6WxvFVT5Ub9E0IoS+Ih41BX+QSQGvQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039091; a=rsa-sha256; cv=none; b=sND3I2MEp5hdpSDLuBLGDR+z/7M/LQItZ5YoeL3CrToKCcDEph8uqGzyu7AtjfrTfOb4m8 S9PFnKqDVVmL1IASQE4Y5ipfCSHTbl5LZclZae5OVhpSi42lYGelNyR04T4/2rqHwlN3Pl Z0WEQ6tYq9GAbBMl32MDLH3xOiWAOK0= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zH7SHGhG; spf=pass (imf24.hostedemail.com: domain of 3cmL8ZQYKCF8PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3cmL8ZQYKCF8PROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc64b659a9cso1797724276.3 for ; Thu, 21 Mar 2024 09:38:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039091; x=1711643891; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=krRMCxbEdPvR0gpNvdmt+1GzNi+HHBvoijzlHbN4rmY=; b=zH7SHGhGcDAXd3tmt034g5Oki9k0m0jDuaXP+Outd/bWt9kMWYGU9Rdzbt++wVq2Dx GMfDneBBJck+bARsho2i2cqiTSjNEd5+yyNd0K4jsHPJC8rkRaLcEcr2PHvdAOKO7d1h pdR8HQjhY8kjFFcvKB7QlyLxj244ZqkhQHERUiFYb/UnzXxvY/pkVedNWMnFh7qYa9nh WYodJF7HqaKd+hSA7TRrIutPJWx88YJfARqu3SfvL3up8zi9/X+qq6HdMZYaUE1PQZqV 9uMQPLPvnFIUI44iXuwxXutW58Wr7wLIvLGD1RWE8FGFxRaNV3JQbcO8FmSmRDjGLXv1 QdtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039091; x=1711643891; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=krRMCxbEdPvR0gpNvdmt+1GzNi+HHBvoijzlHbN4rmY=; b=YKlJuKBozg9cCySDyxR5/8LI0c449NMSaVoIbvq1COMEOXufmZRJXQC4SVyYgWw1y7 JGpTxyqy2aeMoD0IS3MG73coX1yZZkdZ1+nFdMZ9gdLB5bZFgUDRylYR+e7zTxGKEQki jkJ2NdvRDM2QDXfcVXe00dDtiSCuf17w+4V4g6Tv1NUBm0v3KhXwSoLT9eh7mosY0eLN WKiWEZ8RS6q50oUBy0Iu/NW91GaQBPESGcoBO/kiCW6sOrJYcZ7YbappyykYcCDleuOH eh3B4i1b3BGYFaKp4bExAcNd4bWdLxPYmiYQvJrroQ5OjsiHIF4h2u8imfEVoVl3youU 4KMg== X-Forwarded-Encrypted: i=1; AJvYcCUk7ii7K6YupWIT+sKVbm2BU+5+vozTDIDMRhjpYjPY+j079Hesehevx/LpXXWP6lE47GlnUqDroVN0VfvCTjvfQYE= X-Gm-Message-State: AOJu0Yw/xHcHB6CsPCFuoIJLKk1Yc1b0nt7XPUr3TEKdQXJh8GyxpyLs 1vaPFlbM6EhKKeJrvhrif5Uffjv/SMV/tA0TTFoRyfBu7zIu92u35bWkGU/uLGxeWYA5apUPqeF bvQ== X-Google-Smtp-Source: AGHT+IGHRX7538S2HF/3oOa1sNnpVXnzB+i3nUjYB81n0AdHPc2tfmh7wCvtVRvHYQ8dsZZZ2KX9wsSWPB4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:2506:b0:dd1:390a:51e8 with SMTP id dt6-20020a056902250600b00dd1390a51e8mr2395872ybb.10.1711039090742; Thu, 21 Mar 2024 09:38:10 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:50 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-29-surenb@google.com> Subject: [PATCH v6 28/37] mm: percpu: Add codetag reference into pcpuobj_ext From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: BA11E18001B X-Rspam-User: X-Stat-Signature: eiby3fyoubzkf8uagth9keuoa5tszo6i X-Rspamd-Server: rspam03 X-HE-Tag: 1711039091-365728 X-HE-Meta: U2FsdGVkX18+YDAbtwAnINi9rligSpg6R1DGi8XCanTMjiBVdqq4x3MnPJtndpDdYLwa/KYBMc7NvJ6GSPjyLEAW+6tVOtIOlQijdG/aXkqYOnsnVLQhpYiFgZxU6G6HgrbayhWkPqZO70wG+dBK7ApcaAsbYJ6FMkX5UOOlELsbXVq2EAGtfGOu2s233Z40wHQvBYT9gFz5hGIPzrvxEoZNxwvkH7gKtd8n3X9wdfXwHYUUZTY8fSZ9iANM8Nrf37zIJ/4J68IWxgiZu2fIQ1rU0Ok2+PFf6GunITODraYGUX3CS2KqF5Ylp118454IdSa7u/eF2Uriw2bYTeMB72eOTv/dsgfdOFpZAxeKlj3N67dU2K1AyCJ0wzBI1ME8ihXumPa0tnnyEPSAd7p5uFyOICg9I7uPzLPEr9WY0OiOZZ9t6Cb4eDTIPrdgUWZkKTwT1XEPlhcgjTqt3gN13ef9583Z9tH2gXoiml4R2m2PRq04u/PY5S0Z+EF6/U8Ixz+TbFBt2EIto9uZWA/0oA034CuHM8zY8sqTtoCDATdQHIwuXQj77y3D6LaiMvCeMQOOGxCDp9gRx706JjJR6YPfXefZ0gkx1SKHzgy7lauAOUmBLH2OefVq+8tkQLSg8EdrxDFTD6lDMnEYZeTvIxfbdrOpKSHsPNvax0PiDHn3j1ZWDTmGMv4TIKhmDrY/p7lcsxiQQTH+ki045+hLAyThwJMo92GoHCqghfIc1PWEf1h50rLQy6FDIoQy83ZulO+7rpCVyf8qEQzzDVjIzjMrgZooDvzAKE/I8p0gMzV4GaXtUzk6dwukLwp/SIMQze0/ph3VabmM3gQq2YdD9WHE8ivnMwGjNm9M4qA5RdDaVousKy+IfGkvV8KVRtoMH4MzW4uv3KDOgyDRA+8QE+I+9QdpQxMhHXUrW+eiEsLLrwkzUtJIGcnMuTPyKYQUhUAnDWpXaaaSTuoj9uV acj/i6s8 InPuV1aZHsO2ic0kj7eVeyorIA+pjpiu987M6Mgl7z9FILFwbItx6ELQ0Wiw38IJaWooltTds9COPgYD6LWNO9wUB9fuUIpOHO9I9KAlAk4kOLUQPRrQ4ZiL0tFwvx4FlrTbMFEjkkc6mbbVsvvDcY3/b+nlG/7p5KatBdOMlRaFFwoB1EBtxH8134YbyKoq68/m7Ife4uv/4f0/VXg7ERWseW75h59vDgMrG0xUJp6ixhu+yVY81eqkxz86N4QUpCIpuZI4+xjieeC/0Oh+EQqnorZeuZHiaRpWj1ihGGy6/WeCttUEce3SztFMFS9W3VbBRcj/XzwbuJts+KS6dUJYWQF0/18RSnZ+/CDBoslEyVSUjzdx6VTmAlJfzRo4JBH5TKX5cvgEyhhqRWQx4Wal2ATJbLCiFVipwRN0VG7zGp5zP0WMBnI9Yxf9UUa3w8ikBqWi1n1W5JgKfkaT9LRlqeueRUm1bd94J/ks0Ka4ZVdIqTWnwgzuUFpuZAPQDtVr1tY0FccpTL0PUr1mLswwk1N44DIuBA95/59g7vQwbIykh4LV9tmc5H5oQPG2o5TXm3hcFbQrFjy+oP9x36en9UQyhXr0CBCkonhsx6fkI9OmlBET4aIfMEBjJLsvQs7WyRLsE9Iob12Ek/yd6Jh8VlP/bIjZWUohCqvwi0lMVu01GkgFzbZ0Ai63Qp3gMrZpmWO4Wf2cc1pn6A1NjLey5TOXgmV3Cp87G X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet To store codetag for every per-cpu allocation, a codetag reference is embedded into pcpuobj_ext when CONFIG_MEM_ALLOC_PROFILING=y. Hooks to use the newly introduced codetag are added. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- mm/percpu-internal.h | 11 +++++++++-- mm/percpu.c | 26 ++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 2 deletions(-) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index e62d582f4bf3..7e42f0ca3b7b 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -36,9 +36,12 @@ struct pcpuobj_ext { #ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *cgroup; #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + union codetag_ref tag; +#endif }; -#ifdef CONFIG_MEMCG_KMEM +#if defined(CONFIG_MEMCG_KMEM) || defined(CONFIG_MEM_ALLOC_PROFILING) #define NEED_PCPUOBJ_EXT #endif @@ -86,7 +89,11 @@ struct pcpu_chunk { static inline bool need_pcpuobj_ext(void) { - return !mem_cgroup_kmem_disabled(); + if (IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING)) + return true; + if (!mem_cgroup_kmem_disabled()) + return true; + return false; } extern spinlock_t pcpu_lock; diff --git a/mm/percpu.c b/mm/percpu.c index 2e5edaad9cc3..90e9e4004ac9 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1699,6 +1699,32 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *chunk, int off, size_t size) } #endif /* CONFIG_MEMCG_KMEM */ +#ifdef CONFIG_MEM_ALLOC_PROFILING +static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, + size_t size) +{ + if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) { + alloc_tag_add(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, + current->alloc_tag, size); + } +} + +static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t size) +{ + if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) + alloc_tag_sub(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, size); +} +#else +static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, + size_t size) +{ +} + +static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t size) +{ +} +#endif + /** * pcpu_alloc - the percpu allocator * @size: size of area to allocate in bytes From patchwork Thu Mar 21 16:36:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599186 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04099C6FD1F for ; Thu, 21 Mar 2024 16:38:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4AFD46B00BD; Thu, 21 Mar 2024 12:38:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 437F16B00BE; Thu, 21 Mar 2024 12:38:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 28BD46B00BF; Thu, 21 Mar 2024 12:38:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0B75C6B00BD for ; Thu, 21 Mar 2024 12:38:16 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D84288168A for ; Thu, 21 Mar 2024 16:38:15 +0000 (UTC) X-FDA: 81921603750.01.F99D592 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf20.hostedemail.com (Postfix) with ESMTP id 0DF851C001D for ; Thu, 21 Mar 2024 16:38:13 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1YKb+UJq; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3dGL8ZQYKCGERTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3dGL8ZQYKCGERTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039094; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TkCDMNHfK5xLLlYY223eXbm7c73NL7RFva1BQ3rh7WM=; b=FZxR22JHn/U3yw4YBTheFeqOfL9lnzG3TZjTmaUDQVbV+j8+jXi0Hp8sSsWuaMKIRVebPJ OV5C8DanafsSOwPnDAUxgZYPr+QzsI6DHRNO/u4n3jDfdwD6N5CjmkigAhDotkF3v2Z+bL jqFvv9/vqucJSX9SEasY7OgmTcwK0gc= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1YKb+UJq; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3dGL8ZQYKCGERTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3dGL8ZQYKCGERTQDMAFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039094; a=rsa-sha256; cv=none; b=mYMPjVq7mkI62odqpg0pgCeQSkKHcMXF/L5ju+6vm48hJzwfouovUzxNoJjhrMKFcmIFxN tbE0u1LODLkVfY7igSYAujxdOXqwy2BC1XAJXeeEAd4C60SYVrr0DHlhMLPC2HQ/gdeMvb Owome7ho6ugzPh/oc79dBfHcI916DY0= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcbee93a3e1so1861543276.3 for ; Thu, 21 Mar 2024 09:38:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039093; x=1711643893; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TkCDMNHfK5xLLlYY223eXbm7c73NL7RFva1BQ3rh7WM=; b=1YKb+UJqI1WRoUAhazFGjJj57QO2k/qJ2nUEyMnG72UEA0qK3PkHN/M1hB0YOGlMQ9 LvyuZKc1o+5mXkTtt0qkYQK/sNENsa1mWOdXUkehsshVBQFgkfFG/TUZOUiX8dWtSwGB mkGpq3cel0/JifUA+Desrl0mNlLWIcj3en6QPVL3cQazvxfQ+JilhWAshTqwYm+Zk7Jo 1A+z/d8UESlX4iqUqWGWfCe3JlKYGhHQkWjX6Sm7/inUqR4Q9pSsBHfl91ce4BtUtxJK BaxnA1WyYNWaqa0NHFwmvvjNINED5BPVeavLpqpiEdIg5EnmGjhbKfQwOZLyyuky6H4d 4dvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039093; x=1711643893; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TkCDMNHfK5xLLlYY223eXbm7c73NL7RFva1BQ3rh7WM=; b=wydvTy70yNTSV5IJnq1Da78V04TeDqkBphkOacrPLp1w0A5FKb5aMn4M5TC5gr1Qe5 a5JJR4Nq85XzqS7wUewkd01KcL0RLUDGECzMd0WjRyXl62a+A3JGcS9tZotDVKK8Iwoe jVy2Q5qinHe7VBW2DmmRZLWhzME32qB4yr8olD7Oo1f3KwAISXAhz/PflvHLNBc4Hq8F WHLEVDc1xad4Xm1hMqct/cdMH3kXSQoV1D9kFHO+CoEufZdRzOFcvuyC+xT7W8gLAIjl iYH8VZ7UdcWX6FvUKeP8P+cpMKzo7tjNRko2TtmkaOQ2uR7yj1swdmcBqsWO1wICU3Q4 T17Q== X-Forwarded-Encrypted: i=1; AJvYcCXy4qwlgre6iOYQMCGSAvlBxWUFzKtQ5iqWRRx1AQg2iERv3NPGDHNsd0xDbXqPuWsBNMT/ZUgPcOtdFVAVPIOC/SA= X-Gm-Message-State: AOJu0YzsiDKxOcyFY4V6wB9eoU+u3YBFfpnICrrJ4uV8F3yIQnGDasIA 6aJhG0nwhYqHKYQzu1Upf4oQV5cyYcoNsJYmWEQdLpV2hNNd7t2vhz0xZTAYXbujFkn94Q+tN2K cFw== X-Google-Smtp-Source: AGHT+IGzAh9FXKpWdDvxkKjPGrkTEo4AMtUaoBAWJZd0ISNFCDPA2abbhqSo0+lrHGU23eGRJRqbHufH5vQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:1a48:b0:dcc:8be2:7cb0 with SMTP id cy8-20020a0569021a4800b00dcc8be27cb0mr1175240ybb.0.1711039092976; Thu, 21 Mar 2024 09:38:12 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:51 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-30-surenb@google.com> Subject: [PATCH v6 29/37] mm: percpu: enable per-cpu allocation tagging From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 0DF851C001D X-Stat-Signature: nn3ibuuimzd8oa9t63ijo6wff8md3sxg X-HE-Tag: 1711039093-951120 X-HE-Meta: U2FsdGVkX1/C4TpfXTYGAo5AORDsuIIapDaVHSkfMCA4NxpbaNPr790KTix6Dhgsjd+xDP2nN2ZsoBU3DKovM7vhPoY7StF3We9B2aMgKfLB481ZYjwbfl4vsdIJiMKyhsukjsImV2f1HAxKwPzjeJQbn7R7QsvNSyCFgA1kP02ZIFEAOOaD5AUxdy7B30JNxjlPniAm4wCtRwrkZfxb+ndwPEeYgDjQO8uj4XfMC628PF+kB/o/lZmrXSvPfn+OLpYIaCYUM9ln/fp9QIZgy5HpH5GqBfyGNb2L10FWaSC6z+DL8+WLeuaNV6742iEPRrLZdKHNlPcjbuQI7+Fx65i8/s75CxG1DNBPiDOHOqA0pFzqepK5QyL0Sx+rZDJsbEZuUfQMrvx1VEl54yshedDxt15qentM7WbKQY4Wjm6kNsm+Tk8C8pRoWYm/Ogxh1R6gTtD0SY5PVio7/kI9ImoAbK5OP2srk/bYnpbf/45PXrwlRtz/l8Z1sQmYFARazPSnIsy/7XsPXsaiLFRrKg8tuzQC0Dw3tJ2vpHkMETjRzsgNZDu+aGsEQTIzMDQA7eBduhaWUz17uVxD7jPKl+hiMUBWjh/XMS/cSw6d1laECRJqUKhUVDgXpxMvqxzREXMnVFW0vdHyHio8F7dFEuanksNol2E9dx8BIvpaviGMgY+3KyI2BW2MH7iCSRi0WFO2Illq2x48uixIpMtMHTTkbhOjmX5h9WzpH2fqhih5AAmCn1wr9kVuvUSANpEnTIvuMEmdSdP8p77GoIzsQm4UxSxN3DepqQnVwYiRpGo7mgnwrZ0/xsTq4NsmUNkfYXgw5QBtDMIvTKK5v1t+BsAJQFF6SnDam1hT0R2pcWi+fr0nnF/gJerDE6+uoeyqI3pz4yT81j0IMvm5oVvVKQmM+mUrMy86MXf4dYEx9KKbgh8zpZDbdOsFDnljQk/sbuaxMTwMRmEjXn76J31 i93mgUC5 DfG3BCxqK/x4SCadRSbXVKktHUFyIBJn+DarOxdTTupAtgontd1KhESGKbrNYvzSSDQaOE48VUSKMMYsUZgxbZPk2F0AtLseN6RzJzmwGsKoYOXafOnHqD/U2sM5U3wCfNLFrO7vFO05OvqABx6K1JseHV4wkM5xmuGX/Evf8sFL2T1XQt84F4nv1CUZx1olCTjSoxDBmsJtPyNCsECrGPxXgZsflUfrxkSFdQ+mCHuE/nmdSY15WSLmoI7vbLMKcDkIz2+xUzlMBSotbN+7zqpqcewobCD3ariPCNeFI8WAmhjsfqicOcSJ+P30DZ8VELvWrSBCEy78FLFdQboPssoDSor98Ai1UKYA8efOEgr9RBrvXQKimTcDAFImbOqoR25wo04YLXO2inVopYUIjt1D3pXd4urbOUiFVIPhsY9a6sAw8KUUpAPctrScggsbMk6CATDJNcHdtEMmk4UowYPGtzr4yiOWuGw3PAQU+biLP+AEaI42ro0/NYMmo/zVIfU9uo2AtX7E/LqbR1ZaniCWarEkJlgH/jWvyvb7E4RA9AoJa/0/9ujmvdPegMT4kBYmBlByD7foP1+jttuDnUdq7jfL6X8nd5ktBTp1nUjCwV3ZtgyOJbSgbLQM3YT9CzQRsHyGg0k7pRb5a2n/18ueO8O8PZ7oadEn87M5ftQ0V8Bg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Redefine __alloc_percpu, __alloc_percpu_gfp and __alloc_reserved_percpu to record allocations and deallocations done by these functions. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/percpu.h | 23 ++++++++++----- mm/percpu.c | 64 +++++------------------------------------- 2 files changed, 23 insertions(+), 64 deletions(-) diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 62b5eb45bd89..e54921c79c9a 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -2,6 +2,7 @@ #ifndef __LINUX_PERCPU_H #define __LINUX_PERCPU_H +#include #include #include #include @@ -9,6 +10,7 @@ #include #include #include +#include #include @@ -125,7 +127,6 @@ extern int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_t cpu_to_nd_fn); #endif -extern void __percpu *__alloc_reserved_percpu(size_t size, size_t align) __alloc_size(1); extern bool __is_kernel_percpu_address(unsigned long addr, unsigned long *can_addr); extern bool is_kernel_percpu_address(unsigned long addr); @@ -133,14 +134,16 @@ extern bool is_kernel_percpu_address(unsigned long addr); extern void __init setup_per_cpu_areas(void); #endif -extern void __percpu *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp) __alloc_size(1); -extern void __percpu *__alloc_percpu(size_t size, size_t align) __alloc_size(1); -extern void free_percpu(void __percpu *__pdata); +extern void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved, + gfp_t gfp) __alloc_size(1); extern size_t pcpu_alloc_size(void __percpu *__pdata); -DEFINE_FREE(free_percpu, void __percpu *, free_percpu(_T)) - -extern phys_addr_t per_cpu_ptr_to_phys(void *addr); +#define __alloc_percpu_gfp(_size, _align, _gfp) \ + alloc_hooks(pcpu_alloc_noprof(_size, _align, false, _gfp)) +#define __alloc_percpu(_size, _align) \ + alloc_hooks(pcpu_alloc_noprof(_size, _align, false, GFP_KERNEL)) +#define __alloc_reserved_percpu(_size, _align) \ + alloc_hooks(pcpu_alloc_noprof(_size, _align, true, GFP_KERNEL)) #define alloc_percpu_gfp(type, gfp) \ (typeof(type) __percpu *)__alloc_percpu_gfp(sizeof(type), \ @@ -149,6 +152,12 @@ extern phys_addr_t per_cpu_ptr_to_phys(void *addr); (typeof(type) __percpu *)__alloc_percpu(sizeof(type), \ __alignof__(type)) +extern void free_percpu(void __percpu *__pdata); + +DEFINE_FREE(free_percpu, void __percpu *, free_percpu(_T)) + +extern phys_addr_t per_cpu_ptr_to_phys(void *addr); + extern unsigned long pcpu_nr_pages(void); #endif /* __LINUX_PERCPU_H */ diff --git a/mm/percpu.c b/mm/percpu.c index 90e9e4004ac9..dd7eeb370134 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1726,7 +1726,7 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s #endif /** - * pcpu_alloc - the percpu allocator + * pcpu_alloc_noprof - the percpu allocator * @size: size of area to allocate in bytes * @align: alignment of area (max PAGE_SIZE) * @reserved: allocate from the reserved chunk if available @@ -1740,7 +1740,7 @@ static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t s * RETURNS: * Percpu pointer to the allocated area on success, NULL on failure. */ -static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, +void __percpu *pcpu_alloc_noprof(size_t size, size_t align, bool reserved, gfp_t gfp) { gfp_t pcpu_gfp; @@ -1907,6 +1907,8 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, pcpu_memcg_post_alloc_hook(objcg, chunk, off, size); + pcpu_alloc_tag_alloc_hook(chunk, off, size); + return ptr; fail_unlock: @@ -1935,61 +1937,7 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, return NULL; } - -/** - * __alloc_percpu_gfp - allocate dynamic percpu area - * @size: size of area to allocate in bytes - * @align: alignment of area (max PAGE_SIZE) - * @gfp: allocation flags - * - * Allocate zero-filled percpu area of @size bytes aligned at @align. If - * @gfp doesn't contain %GFP_KERNEL, the allocation doesn't block and can - * be called from any context but is a lot more likely to fail. If @gfp - * has __GFP_NOWARN then no warning will be triggered on invalid or failed - * allocation requests. - * - * RETURNS: - * Percpu pointer to the allocated area on success, NULL on failure. - */ -void __percpu *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp) -{ - return pcpu_alloc(size, align, false, gfp); -} -EXPORT_SYMBOL_GPL(__alloc_percpu_gfp); - -/** - * __alloc_percpu - allocate dynamic percpu area - * @size: size of area to allocate in bytes - * @align: alignment of area (max PAGE_SIZE) - * - * Equivalent to __alloc_percpu_gfp(size, align, %GFP_KERNEL). - */ -void __percpu *__alloc_percpu(size_t size, size_t align) -{ - return pcpu_alloc(size, align, false, GFP_KERNEL); -} -EXPORT_SYMBOL_GPL(__alloc_percpu); - -/** - * __alloc_reserved_percpu - allocate reserved percpu area - * @size: size of area to allocate in bytes - * @align: alignment of area (max PAGE_SIZE) - * - * Allocate zero-filled percpu area of @size bytes aligned at @align - * from reserved percpu area if arch has set it up; otherwise, - * allocation is served from the same dynamic area. Might sleep. - * Might trigger writeouts. - * - * CONTEXT: - * Does GFP_KERNEL allocation. - * - * RETURNS: - * Percpu pointer to the allocated area on success, NULL on failure. - */ -void __percpu *__alloc_reserved_percpu(size_t size, size_t align) -{ - return pcpu_alloc(size, align, true, GFP_KERNEL); -} +EXPORT_SYMBOL_GPL(pcpu_alloc_noprof); /** * pcpu_balance_free - manage the amount of free chunks @@ -2328,6 +2276,8 @@ void free_percpu(void __percpu *ptr) spin_lock_irqsave(&pcpu_lock, flags); size = pcpu_free_area(chunk, off); + pcpu_alloc_tag_free_hook(chunk, off, size); + pcpu_memcg_free_hook(chunk, off, size); /* From patchwork Thu Mar 21 16:36:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599187 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBBEBC6FD1F for ; Thu, 21 Mar 2024 16:38:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D63FD6B00BE; Thu, 21 Mar 2024 12:38:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C9EB96B00BF; Thu, 21 Mar 2024 12:38:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA28D6B00C0; Thu, 21 Mar 2024 12:38:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8F1D66B00BE for ; Thu, 21 Mar 2024 12:38:18 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 54BAE1401BB for ; Thu, 21 Mar 2024 16:38:18 +0000 (UTC) X-FDA: 81921603876.07.858D2D1 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf23.hostedemail.com (Postfix) with ESMTP id 758BE140016 for ; Thu, 21 Mar 2024 16:38:16 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=dpISjH2r; spf=pass (imf23.hostedemail.com: domain of 3d2L8ZQYKCGQUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3d2L8ZQYKCGQUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039096; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jUCXlu8AiSNx+hpzzdaqhqf+mzz+Gk6/geVO1sIYVzQ=; b=fk1Ay/NvC1aHlLT6X0+n8QVSAEuaANLaTfGRxAGNLvhcXeH9svkNeRlBLn6ntOg/zD6E7t 19ul9Xw9DFqZGsUecVW2OG41MrYdxo42w5cR5N/B/adwYlXwExV9S8+nMgqtYL7kAVCbUm WulN/zJYkdgwrIHH1mT78WOlLlkhqgA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039096; a=rsa-sha256; cv=none; b=klJZ128dQBUREcS0f/iuHo/A92RymdvkYkMAJ9m1sqEFlu4v2h9lpsvL5FXl5gA1chpaaE H2tFce6PLfEOHxAKMCTf2tvObOfnwNhJ7PNcLGEsc+MFOrSgaWsfcBcycBIWI2E2neuOLy FNHYjfMHxP2LdQ0f5IjYi4ZsBIjZ/zM= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=dpISjH2r; spf=pass (imf23.hostedemail.com: domain of 3d2L8ZQYKCGQUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3d2L8ZQYKCGQUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60a3bb05c9bso19018017b3.1 for ; Thu, 21 Mar 2024 09:38:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039095; x=1711643895; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jUCXlu8AiSNx+hpzzdaqhqf+mzz+Gk6/geVO1sIYVzQ=; b=dpISjH2rH8v2yAyGleyq5nUbQnbHo6khJPV48APtVySyj0mjSfVp6VtYMidsWlb0aS XSWsFStpLS6/Gl9eK7q5zQgH19lhetyfI5Ct4XMdpbWBx9YNPVTbgVMiYsOLwOuhKn+Y xJn6b8NLJrTXK4Yh9k6nQcuSuwgN0OovshDBjb0y6RuuuyXpzCi+FaMG/pKQeSDUC9Rf rnOK6zMK8YUmv8hiA6Ch/BCxdK7/bgbApDa6mjuV57XPaTukt3mgz2ytoLZoqtQFjyvj jgCx11kgyKfWa2Q43JMfMqKs2A40CplpPzTCNZbGwBZIzTpakPhb3jYFSm3TnfAop0FI otNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039095; x=1711643895; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jUCXlu8AiSNx+hpzzdaqhqf+mzz+Gk6/geVO1sIYVzQ=; b=H+vD3mnZmvLrnbS5rfJuJDChLUWrBBfr9LbG7slDbOkl4osl2iWi4D4jbsx433Lmuk 0W5nzataU5vLSkqlDw9dolv87uKxwgPnzVWMw0vVwC7vwz3B57zulJF+vbfxJbSO8Es2 MmCTuuOck68TxxpurtxoyJnDg9RT1mBE8zsZaiXu16dicKK/adlzYuVNC8WAwVRBYQAY jYa7EQs0Q1KQVG+u91/aR9oPS1fPow4P8ZosUIgCNLPBX5sDkWfDEC3XZfWvgveYrdn/ fv+nkRvHFEoc60KtHYhfypguGeqsdolJPLSpfiaJMNuAHcqt0NcTRLz13EuX64arWxVL l9pw== X-Forwarded-Encrypted: i=1; AJvYcCXVBwt4kcsxjqm8jgpnilekOP2C5wFST2ymvL1GgXNV7ItMOAi0FXL6LZf/Pm1Dwf/yw5R9SESxRTJwNOxnwV37jCg= X-Gm-Message-State: AOJu0YxVlqL8Q6gMgqA31SO+KrMz5HNxExRMqywVqlgl1IxN1Crit5lx BveE3Is0qyLxwoBZdSfL812a6O/pAY+i7qdbAuJ8op56tM4Djlpz60kzlLEfp3V9ygQ8JzwV/8d ZQw== X-Google-Smtp-Source: AGHT+IGJd30vTv5gDdTqeklto1K7e4ybq+S8gwYlaB+T5DUdVTgUxg04IJ9eXS9IAccy/+G2x0GWXwkGVtg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a0d:cbc1:0:b0:610:e166:9521 with SMTP id n184-20020a0dcbc1000000b00610e1669521mr1863260ywd.3.1711039095238; Thu, 21 Mar 2024 09:38:15 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:52 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-31-surenb@google.com> Subject: [PATCH v6 30/37] mm: vmalloc: Enable memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 758BE140016 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: s8pe8ua6bba6g69ejcyjqtfiinp5im8a X-HE-Tag: 1711039096-316491 X-HE-Meta: U2FsdGVkX1+sjnGOrU49hOAEZfZKN1o8g5DrHvIRI+S5LuzzD24oZQIOIqxJ5YBnO3LHjjIb0+dO/efIOSnsAKeZTvVcGIa8yEMozHJ8x23U7UNmJh1wH68hNyrPNEDXHzhpGLLd9ckMd35+T4/FP+YAwPrfsmwd7JB7BT25VDMkBtCXVGdNAKMYV6IIW8sFTUIQCx2Uw9cNgu3fyMJQBmtjE2d1gtWOfORSCZlorhpdrFb9fs+guIlBx+mRwEZYVro52CgLalAmgIRlk1pE1bTbKDEkrNwDi/xktJu1ImZsBdQA1uwVed3OV4INU18PN0nJWtq2uZU55MawvfTQQ6dRjGs+LhFQwKbI8Z0VfRNNT/KpHkeFrCMcj4VaSvvSJFJDFYyNkndAannqqEJ6Z/Cfd5PSOpcTrGHinQcnqUtGGVUxN2MRQ06U5HZ/wLNXNgc12+uzWCnrhA+xiOrKou2QPmOfQIbTyvmW9cqtbyd6urJFBeL3+zisZkdYSezuemmGveiSJAtao2dX1UFMrG1VLDauIriGo+GUGpmIPrW1pbHhZbmuchPwXGy5QsvKg3iEdc86/TfeA0nB67RrKyHJdwWfS1T8e5oD3MOAOD+gaEJxRgeR8AcGuzBv7tmeF/B3TfyH0wOHzFEVNlaw/+7FFKxlVkFsJZIJGkl4v1RDQk5gyJZLTqS1pFU+0t6pltLdfxCuRWWHfTYLU++KCM4751BXlCepiWdq2pNzupc+/x1InXBh0Nuv7NTjTfovIXCHNHNRV1AYYjTN9wMaLDEoG+gdlK2sSigRPMqtlJN9NNyzD1C6lgjq6LXv5JNRXBhP5RFg8MBngWtNzRI/I46zfmEC7mSNRjCTpm5eRIbw+2Lh9D8N/TXGh5R4syCuE8DZk09AtFv8gUchX1rSvseSfoWWzy1UN8yoful3K+bY3/lSbUKCYwENqr7hEW0KfbxkNMtCJjYeMtWquJV lsVgYlF+ 1rt4ENcWPxkob1zmuOtXPGDQ/SIAsHidmGrHMsdIcWwKx7dAbo44qgOQ0kGY6bLnkQWIRuWPpOo6yXpsLCGRGD9D5g/T3BGI0u9mhTfweJtOZwY6KoDSm/ddojzYZtj1MrbznuSMigtx5RnAX98BIOQ9v3+A6ZKukEufjpKxdJskhyC0L8rjSNWO301LDYh7/VHFCCfnJP78ijd3PGhyYymDrSBa6kMDOUvmZppm+ccRgl0oUFjaaG5Zr7dwHllB2tDJ01RbIUi/5ncrLJ3gL2kt9AkkPagGRudd2ZU0cY6cSVgva8rrKkf3+y6cICEAHLi+0j4QEb4mxXjnSevOUKd/9E5HGKNcUo7t+Q6TXHZGryLf+8JbNl/v9Op9lO/OdGXocZtjFmuXaGPHoLJbJzDkfeJhh/e8uHXxyZeGS5WOf3lEGFcWkbqB1UkqjPyzjmWnpiVRB3t6W7diPsMWJq250RNz9kBnblWS/R1zAo1QyRW6KUGqu4v3ykXu7b8TgOogLskiPDV+oOK5kdJd4/Y1r+RaPenWQkgH+PH2+eMTx2NVUEii4D6A4soeT7ZMDInA1sBqD4APVbK/WlSIwycWthxlbLZEbg+bka8R1P4guE+L7AecZiEoy7bt5wTnUIAkC59Dc5q1VLJzZBuVuZw4cfmnC/e1mZYw4Ptey2pmInVY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet This wrapps all external vmalloc allocation functions with the alloc_hooks() wrapper, and switches internal allocations to _noprof variants where appropriate, for the new memory allocation profiling feature. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- drivers/staging/media/atomisp/pci/hmm/hmm.c | 2 +- include/linux/vmalloc.h | 60 ++++++++++---- kernel/kallsyms_selftest.c | 2 +- mm/nommu.c | 64 +++++++-------- mm/util.c | 24 +++--- mm/vmalloc.c | 88 ++++++++++----------- 6 files changed, 135 insertions(+), 105 deletions(-) diff --git a/drivers/staging/media/atomisp/pci/hmm/hmm.c b/drivers/staging/media/atomisp/pci/hmm/hmm.c index bb12644fd033..3e2899ad8517 100644 --- a/drivers/staging/media/atomisp/pci/hmm/hmm.c +++ b/drivers/staging/media/atomisp/pci/hmm/hmm.c @@ -205,7 +205,7 @@ static ia_css_ptr __hmm_alloc(size_t bytes, enum hmm_bo_type type, } dev_dbg(atomisp_dev, "pages: 0x%08x (%zu bytes), type: %d, vmalloc %p\n", - bo->start, bytes, type, vmalloc); + bo->start, bytes, type, vmalloc_noprof); return bo->start; diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 98ea90e90439..e4a631ec430b 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -2,6 +2,8 @@ #ifndef _LINUX_VMALLOC_H #define _LINUX_VMALLOC_H +#include +#include #include #include #include @@ -138,26 +140,54 @@ extern unsigned long vmalloc_nr_pages(void); static inline unsigned long vmalloc_nr_pages(void) { return 0; } #endif -extern void *vmalloc(unsigned long size) __alloc_size(1); -extern void *vzalloc(unsigned long size) __alloc_size(1); -extern void *vmalloc_user(unsigned long size) __alloc_size(1); -extern void *vmalloc_node(unsigned long size, int node) __alloc_size(1); -extern void *vzalloc_node(unsigned long size, int node) __alloc_size(1); -extern void *vmalloc_32(unsigned long size) __alloc_size(1); -extern void *vmalloc_32_user(unsigned long size) __alloc_size(1); -extern void *__vmalloc(unsigned long size, gfp_t gfp_mask) __alloc_size(1); -extern void *__vmalloc_node_range(unsigned long size, unsigned long align, +extern void *vmalloc_noprof(unsigned long size) __alloc_size(1); +#define vmalloc(...) alloc_hooks(vmalloc_noprof(__VA_ARGS__)) + +extern void *vzalloc_noprof(unsigned long size) __alloc_size(1); +#define vzalloc(...) alloc_hooks(vzalloc_noprof(__VA_ARGS__)) + +extern void *vmalloc_user_noprof(unsigned long size) __alloc_size(1); +#define vmalloc_user(...) alloc_hooks(vmalloc_user_noprof(__VA_ARGS__)) + +extern void *vmalloc_node_noprof(unsigned long size, int node) __alloc_size(1); +#define vmalloc_node(...) alloc_hooks(vmalloc_node_noprof(__VA_ARGS__)) + +extern void *vzalloc_node_noprof(unsigned long size, int node) __alloc_size(1); +#define vzalloc_node(...) alloc_hooks(vzalloc_node_noprof(__VA_ARGS__)) + +extern void *vmalloc_32_noprof(unsigned long size) __alloc_size(1); +#define vmalloc_32(...) alloc_hooks(vmalloc_32_noprof(__VA_ARGS__)) + +extern void *vmalloc_32_user_noprof(unsigned long size) __alloc_size(1); +#define vmalloc_32_user(...) alloc_hooks(vmalloc_32_user_noprof(__VA_ARGS__)) + +extern void *__vmalloc_noprof(unsigned long size, gfp_t gfp_mask) __alloc_size(1); +#define __vmalloc(...) alloc_hooks(__vmalloc_noprof(__VA_ARGS__)) + +extern void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, unsigned long start, unsigned long end, gfp_t gfp_mask, pgprot_t prot, unsigned long vm_flags, int node, const void *caller) __alloc_size(1); -void *__vmalloc_node(unsigned long size, unsigned long align, gfp_t gfp_mask, +#define __vmalloc_node_range(...) alloc_hooks(__vmalloc_node_range_noprof(__VA_ARGS__)) + +void *__vmalloc_node_noprof(unsigned long size, unsigned long align, gfp_t gfp_mask, int node, const void *caller) __alloc_size(1); -void *vmalloc_huge(unsigned long size, gfp_t gfp_mask) __alloc_size(1); +#define __vmalloc_node(...) alloc_hooks(__vmalloc_node_noprof(__VA_ARGS__)) + +void *vmalloc_huge_noprof(unsigned long size, gfp_t gfp_mask) __alloc_size(1); +#define vmalloc_huge(...) alloc_hooks(vmalloc_huge_noprof(__VA_ARGS__)) + +extern void *__vmalloc_array_noprof(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); +#define __vmalloc_array(...) alloc_hooks(__vmalloc_array_noprof(__VA_ARGS__)) + +extern void *vmalloc_array_noprof(size_t n, size_t size) __alloc_size(1, 2); +#define vmalloc_array(...) alloc_hooks(vmalloc_array_noprof(__VA_ARGS__)) + +extern void *__vcalloc_noprof(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); +#define __vcalloc(...) alloc_hooks(__vcalloc_noprof(__VA_ARGS__)) -extern void *__vmalloc_array(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); -extern void *vmalloc_array(size_t n, size_t size) __alloc_size(1, 2); -extern void *__vcalloc(size_t n, size_t size, gfp_t flags) __alloc_size(1, 2); -extern void *vcalloc(size_t n, size_t size) __alloc_size(1, 2); +extern void *vcalloc_noprof(size_t n, size_t size) __alloc_size(1, 2); +#define vcalloc(...) alloc_hooks(vcalloc_noprof(__VA_ARGS__)) extern void vfree(const void *addr); extern void vfree_atomic(const void *addr); diff --git a/kernel/kallsyms_selftest.c b/kernel/kallsyms_selftest.c index 8a689b4ff4f9..2f84896a7bcb 100644 --- a/kernel/kallsyms_selftest.c +++ b/kernel/kallsyms_selftest.c @@ -82,7 +82,7 @@ static struct test_item test_items[] = { ITEM_FUNC(kallsyms_test_func_static), ITEM_FUNC(kallsyms_test_func), ITEM_FUNC(kallsyms_test_func_weak), - ITEM_FUNC(vmalloc), + ITEM_FUNC(vmalloc_noprof), ITEM_FUNC(vfree), #ifdef CONFIG_KALLSYMS_ALL ITEM_DATA(kallsyms_test_var_bss_static), diff --git a/mm/nommu.c b/mm/nommu.c index 5ec8f44e7ce9..69a6f3b4d156 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -137,28 +137,28 @@ void vfree(const void *addr) } EXPORT_SYMBOL(vfree); -void *__vmalloc(unsigned long size, gfp_t gfp_mask) +void *__vmalloc_noprof(unsigned long size, gfp_t gfp_mask) { /* * You can't specify __GFP_HIGHMEM with kmalloc() since kmalloc() * returns only a logical address. */ - return kmalloc(size, (gfp_mask | __GFP_COMP) & ~__GFP_HIGHMEM); + return kmalloc_noprof(size, (gfp_mask | __GFP_COMP) & ~__GFP_HIGHMEM); } -EXPORT_SYMBOL(__vmalloc); +EXPORT_SYMBOL(__vmalloc_noprof); -void *__vmalloc_node_range(unsigned long size, unsigned long align, +void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, unsigned long start, unsigned long end, gfp_t gfp_mask, pgprot_t prot, unsigned long vm_flags, int node, const void *caller) { - return __vmalloc(size, gfp_mask); + return __vmalloc_noprof(size, gfp_mask); } -void *__vmalloc_node(unsigned long size, unsigned long align, gfp_t gfp_mask, +void *__vmalloc_node_noprof(unsigned long size, unsigned long align, gfp_t gfp_mask, int node, const void *caller) { - return __vmalloc(size, gfp_mask); + return __vmalloc_noprof(size, gfp_mask); } static void *__vmalloc_user_flags(unsigned long size, gfp_t flags) @@ -179,11 +179,11 @@ static void *__vmalloc_user_flags(unsigned long size, gfp_t flags) return ret; } -void *vmalloc_user(unsigned long size) +void *vmalloc_user_noprof(unsigned long size) { return __vmalloc_user_flags(size, GFP_KERNEL | __GFP_ZERO); } -EXPORT_SYMBOL(vmalloc_user); +EXPORT_SYMBOL(vmalloc_user_noprof); struct page *vmalloc_to_page(const void *addr) { @@ -217,13 +217,13 @@ long vread_iter(struct iov_iter *iter, const char *addr, size_t count) * For tight control over page level allocator and protection flags * use __vmalloc() instead. */ -void *vmalloc(unsigned long size) +void *vmalloc_noprof(unsigned long size) { - return __vmalloc(size, GFP_KERNEL); + return __vmalloc_noprof(size, GFP_KERNEL); } -EXPORT_SYMBOL(vmalloc); +EXPORT_SYMBOL(vmalloc_noprof); -void *vmalloc_huge(unsigned long size, gfp_t gfp_mask) __weak __alias(__vmalloc); +void *vmalloc_huge_noprof(unsigned long size, gfp_t gfp_mask) __weak __alias(__vmalloc_noprof); /* * vzalloc - allocate virtually contiguous memory with zero fill @@ -237,14 +237,14 @@ void *vmalloc_huge(unsigned long size, gfp_t gfp_mask) __weak __alias(__vmalloc) * For tight control over page level allocator and protection flags * use __vmalloc() instead. */ -void *vzalloc(unsigned long size) +void *vzalloc_noprof(unsigned long size) { - return __vmalloc(size, GFP_KERNEL | __GFP_ZERO); + return __vmalloc_noprof(size, GFP_KERNEL | __GFP_ZERO); } -EXPORT_SYMBOL(vzalloc); +EXPORT_SYMBOL(vzalloc_noprof); /** - * vmalloc_node - allocate memory on a specific node + * vmalloc_node_noprof - allocate memory on a specific node * @size: allocation size * @node: numa node * @@ -254,14 +254,14 @@ EXPORT_SYMBOL(vzalloc); * For tight control over page level allocator and protection flags * use __vmalloc() instead. */ -void *vmalloc_node(unsigned long size, int node) +void *vmalloc_node_noprof(unsigned long size, int node) { - return vmalloc(size); + return vmalloc_noprof(size); } -EXPORT_SYMBOL(vmalloc_node); +EXPORT_SYMBOL(vmalloc_node_noprof); /** - * vzalloc_node - allocate memory on a specific node with zero fill + * vzalloc_node_noprof - allocate memory on a specific node with zero fill * @size: allocation size * @node: numa node * @@ -272,27 +272,27 @@ EXPORT_SYMBOL(vmalloc_node); * For tight control over page level allocator and protection flags * use __vmalloc() instead. */ -void *vzalloc_node(unsigned long size, int node) +void *vzalloc_node_noprof(unsigned long size, int node) { - return vzalloc(size); + return vzalloc_noprof(size); } -EXPORT_SYMBOL(vzalloc_node); +EXPORT_SYMBOL(vzalloc_node_noprof); /** - * vmalloc_32 - allocate virtually contiguous memory (32bit addressable) + * vmalloc_32_noprof - allocate virtually contiguous memory (32bit addressable) * @size: allocation size * * Allocate enough 32bit PA addressable pages to cover @size from the * page level allocator and map them into contiguous kernel virtual space. */ -void *vmalloc_32(unsigned long size) +void *vmalloc_32_noprof(unsigned long size) { - return __vmalloc(size, GFP_KERNEL); + return __vmalloc_noprof(size, GFP_KERNEL); } -EXPORT_SYMBOL(vmalloc_32); +EXPORT_SYMBOL(vmalloc_32_noprof); /** - * vmalloc_32_user - allocate zeroed virtually contiguous 32bit memory + * vmalloc_32_user_noprof - allocate zeroed virtually contiguous 32bit memory * @size: allocation size * * The resulting memory area is 32bit addressable and zeroed so it can be @@ -301,15 +301,15 @@ EXPORT_SYMBOL(vmalloc_32); * VM_USERMAP is set on the corresponding VMA so that subsequent calls to * remap_vmalloc_range() are permissible. */ -void *vmalloc_32_user(unsigned long size) +void *vmalloc_32_user_noprof(unsigned long size) { /* * We'll have to sort out the ZONE_DMA bits for 64-bit, * but for now this can simply use vmalloc_user() directly. */ - return vmalloc_user(size); + return vmalloc_user_noprof(size); } -EXPORT_SYMBOL(vmalloc_32_user); +EXPORT_SYMBOL(vmalloc_32_user_noprof); void *vmap(struct page **pages, unsigned int count, unsigned long flags, pgprot_t prot) { diff --git a/mm/util.c b/mm/util.c index a79dce7546f1..157b5edcba75 100644 --- a/mm/util.c +++ b/mm/util.c @@ -656,7 +656,7 @@ void *kvmalloc_node_noprof(size_t size, gfp_t flags, int node) * about the resulting pointer, and cannot play * protection games. */ - return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, 1, VMALLOC_START, VMALLOC_END, flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, node, __builtin_return_address(0)); } @@ -715,12 +715,12 @@ void *kvrealloc_noprof(const void *p, size_t oldsize, size_t newsize, gfp_t flag EXPORT_SYMBOL(kvrealloc_noprof); /** - * __vmalloc_array - allocate memory for a virtually contiguous array. + * __vmalloc_array_noprof - allocate memory for a virtually contiguous array. * @n: number of elements. * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -void *__vmalloc_array(size_t n, size_t size, gfp_t flags) +void *__vmalloc_array_noprof(size_t n, size_t size, gfp_t flags) { size_t bytes; @@ -728,18 +728,18 @@ void *__vmalloc_array(size_t n, size_t size, gfp_t flags) return NULL; return __vmalloc(bytes, flags); } -EXPORT_SYMBOL(__vmalloc_array); +EXPORT_SYMBOL(__vmalloc_array_noprof); /** - * vmalloc_array - allocate memory for a virtually contiguous array. + * vmalloc_array_noprof - allocate memory for a virtually contiguous array. * @n: number of elements. * @size: element size. */ -void *vmalloc_array(size_t n, size_t size) +void *vmalloc_array_noprof(size_t n, size_t size) { return __vmalloc_array(n, size, GFP_KERNEL); } -EXPORT_SYMBOL(vmalloc_array); +EXPORT_SYMBOL(vmalloc_array_noprof); /** * __vcalloc - allocate and zero memory for a virtually contiguous array. @@ -747,22 +747,22 @@ EXPORT_SYMBOL(vmalloc_array); * @size: element size. * @flags: the type of memory to allocate (see kmalloc). */ -void *__vcalloc(size_t n, size_t size, gfp_t flags) +void *__vcalloc_noprof(size_t n, size_t size, gfp_t flags) { return __vmalloc_array(n, size, flags | __GFP_ZERO); } -EXPORT_SYMBOL(__vcalloc); +EXPORT_SYMBOL(__vcalloc_noprof); /** - * vcalloc - allocate and zero memory for a virtually contiguous array. + * vcalloc_noprof - allocate and zero memory for a virtually contiguous array. * @n: number of elements. * @size: element size. */ -void *vcalloc(size_t n, size_t size) +void *vcalloc_noprof(size_t n, size_t size) { return __vmalloc_array(n, size, GFP_KERNEL | __GFP_ZERO); } -EXPORT_SYMBOL(vcalloc); +EXPORT_SYMBOL(vcalloc_noprof); struct anon_vma *folio_anon_vma(struct folio *folio) { diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 22aa63f4ef63..b2f2248d85a9 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3507,12 +3507,12 @@ vm_area_alloc_pages(gfp_t gfp, int nid, * but mempolicy wants to alloc memory by interleaving. */ if (IS_ENABLED(CONFIG_NUMA) && nid == NUMA_NO_NODE) - nr = alloc_pages_bulk_array_mempolicy(bulk_gfp, + nr = alloc_pages_bulk_array_mempolicy_noprof(bulk_gfp, nr_pages_request, pages + nr_allocated); else - nr = alloc_pages_bulk_array_node(bulk_gfp, nid, + nr = alloc_pages_bulk_array_node_noprof(bulk_gfp, nid, nr_pages_request, pages + nr_allocated); @@ -3542,9 +3542,9 @@ vm_area_alloc_pages(gfp_t gfp, int nid, break; if (nid == NUMA_NO_NODE) - page = alloc_pages(alloc_gfp, order); + page = alloc_pages_noprof(alloc_gfp, order); else - page = alloc_pages_node(nid, alloc_gfp, order); + page = alloc_pages_node_noprof(nid, alloc_gfp, order); if (unlikely(!page)) { if (!nofail) break; @@ -3601,10 +3601,10 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, /* Please note that the recursion is strictly bounded. */ if (array_size > PAGE_SIZE) { - area->pages = __vmalloc_node(array_size, 1, nested_gfp, node, + area->pages = __vmalloc_node_noprof(array_size, 1, nested_gfp, node, area->caller); } else { - area->pages = kmalloc_node(array_size, nested_gfp, node); + area->pages = kmalloc_node_noprof(array_size, nested_gfp, node); } if (!area->pages) { @@ -3687,7 +3687,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, } /** - * __vmalloc_node_range - allocate virtually contiguous memory + * __vmalloc_node_range_noprof - allocate virtually contiguous memory * @size: allocation size * @align: desired alignment * @start: vm area range start @@ -3714,7 +3714,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, * * Return: the address of the area or %NULL on failure */ -void *__vmalloc_node_range(unsigned long size, unsigned long align, +void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align, unsigned long start, unsigned long end, gfp_t gfp_mask, pgprot_t prot, unsigned long vm_flags, int node, const void *caller) @@ -3843,7 +3843,7 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, } /** - * __vmalloc_node - allocate virtually contiguous memory + * __vmalloc_node_noprof - allocate virtually contiguous memory * @size: allocation size * @align: desired alignment * @gfp_mask: flags for the page level allocator @@ -3861,10 +3861,10 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, * * Return: pointer to the allocated memory or %NULL on error */ -void *__vmalloc_node(unsigned long size, unsigned long align, +void *__vmalloc_node_noprof(unsigned long size, unsigned long align, gfp_t gfp_mask, int node, const void *caller) { - return __vmalloc_node_range(size, align, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, align, VMALLOC_START, VMALLOC_END, gfp_mask, PAGE_KERNEL, 0, node, caller); } /* @@ -3873,15 +3873,15 @@ void *__vmalloc_node(unsigned long size, unsigned long align, * than that. */ #ifdef CONFIG_TEST_VMALLOC_MODULE -EXPORT_SYMBOL_GPL(__vmalloc_node); +EXPORT_SYMBOL_GPL(__vmalloc_node_noprof); #endif -void *__vmalloc(unsigned long size, gfp_t gfp_mask) +void *__vmalloc_noprof(unsigned long size, gfp_t gfp_mask) { - return __vmalloc_node(size, 1, gfp_mask, NUMA_NO_NODE, + return __vmalloc_node_noprof(size, 1, gfp_mask, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(__vmalloc); +EXPORT_SYMBOL(__vmalloc_noprof); /** * vmalloc - allocate virtually contiguous memory @@ -3895,12 +3895,12 @@ EXPORT_SYMBOL(__vmalloc); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc(unsigned long size) +void *vmalloc_noprof(unsigned long size) { - return __vmalloc_node(size, 1, GFP_KERNEL, NUMA_NO_NODE, + return __vmalloc_node_noprof(size, 1, GFP_KERNEL, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc); +EXPORT_SYMBOL(vmalloc_noprof); /** * vmalloc_huge - allocate virtually contiguous memory, allow huge pages @@ -3914,16 +3914,16 @@ EXPORT_SYMBOL(vmalloc); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_huge(unsigned long size, gfp_t gfp_mask) +void *vmalloc_huge_noprof(unsigned long size, gfp_t gfp_mask) { - return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, 1, VMALLOC_START, VMALLOC_END, gfp_mask, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL_GPL(vmalloc_huge); +EXPORT_SYMBOL_GPL(vmalloc_huge_noprof); /** - * vzalloc - allocate virtually contiguous memory with zero fill + * vzalloc_noprof - allocate virtually contiguous memory with zero fill * @size: allocation size * * Allocate enough pages to cover @size from the page level @@ -3935,12 +3935,12 @@ EXPORT_SYMBOL_GPL(vmalloc_huge); * * Return: pointer to the allocated memory or %NULL on error */ -void *vzalloc(unsigned long size) +void *vzalloc_noprof(unsigned long size) { - return __vmalloc_node(size, 1, GFP_KERNEL | __GFP_ZERO, NUMA_NO_NODE, + return __vmalloc_node_noprof(size, 1, GFP_KERNEL | __GFP_ZERO, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vzalloc); +EXPORT_SYMBOL(vzalloc_noprof); /** * vmalloc_user - allocate zeroed virtually contiguous memory for userspace @@ -3951,17 +3951,17 @@ EXPORT_SYMBOL(vzalloc); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_user(unsigned long size) +void *vmalloc_user_noprof(unsigned long size) { - return __vmalloc_node_range(size, SHMLBA, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, SHMLBA, VMALLOC_START, VMALLOC_END, GFP_KERNEL | __GFP_ZERO, PAGE_KERNEL, VM_USERMAP, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_user); +EXPORT_SYMBOL(vmalloc_user_noprof); /** - * vmalloc_node - allocate memory on a specific node + * vmalloc_node_noprof - allocate memory on a specific node * @size: allocation size * @node: numa node * @@ -3973,15 +3973,15 @@ EXPORT_SYMBOL(vmalloc_user); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_node(unsigned long size, int node) +void *vmalloc_node_noprof(unsigned long size, int node) { - return __vmalloc_node(size, 1, GFP_KERNEL, node, + return __vmalloc_node_noprof(size, 1, GFP_KERNEL, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_node); +EXPORT_SYMBOL(vmalloc_node_noprof); /** - * vzalloc_node - allocate memory on a specific node with zero fill + * vzalloc_node_noprof - allocate memory on a specific node with zero fill * @size: allocation size * @node: numa node * @@ -3991,12 +3991,12 @@ EXPORT_SYMBOL(vmalloc_node); * * Return: pointer to the allocated memory or %NULL on error */ -void *vzalloc_node(unsigned long size, int node) +void *vzalloc_node_noprof(unsigned long size, int node) { - return __vmalloc_node(size, 1, GFP_KERNEL | __GFP_ZERO, node, + return __vmalloc_node_noprof(size, 1, GFP_KERNEL | __GFP_ZERO, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(vzalloc_node); +EXPORT_SYMBOL(vzalloc_node_noprof); #if defined(CONFIG_64BIT) && defined(CONFIG_ZONE_DMA32) #define GFP_VMALLOC32 (GFP_DMA32 | GFP_KERNEL) @@ -4011,7 +4011,7 @@ EXPORT_SYMBOL(vzalloc_node); #endif /** - * vmalloc_32 - allocate virtually contiguous memory (32bit addressable) + * vmalloc_32_noprof - allocate virtually contiguous memory (32bit addressable) * @size: allocation size * * Allocate enough 32bit PA addressable pages to cover @size from the @@ -4019,15 +4019,15 @@ EXPORT_SYMBOL(vzalloc_node); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_32(unsigned long size) +void *vmalloc_32_noprof(unsigned long size) { - return __vmalloc_node(size, 1, GFP_VMALLOC32, NUMA_NO_NODE, + return __vmalloc_node_noprof(size, 1, GFP_VMALLOC32, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_32); +EXPORT_SYMBOL(vmalloc_32_noprof); /** - * vmalloc_32_user - allocate zeroed virtually contiguous 32bit memory + * vmalloc_32_user_noprof - allocate zeroed virtually contiguous 32bit memory * @size: allocation size * * The resulting memory area is 32bit addressable and zeroed so it can be @@ -4035,14 +4035,14 @@ EXPORT_SYMBOL(vmalloc_32); * * Return: pointer to the allocated memory or %NULL on error */ -void *vmalloc_32_user(unsigned long size) +void *vmalloc_32_user_noprof(unsigned long size) { - return __vmalloc_node_range(size, SHMLBA, VMALLOC_START, VMALLOC_END, + return __vmalloc_node_range_noprof(size, SHMLBA, VMALLOC_START, VMALLOC_END, GFP_VMALLOC32 | __GFP_ZERO, PAGE_KERNEL, VM_USERMAP, NUMA_NO_NODE, __builtin_return_address(0)); } -EXPORT_SYMBOL(vmalloc_32_user); +EXPORT_SYMBOL(vmalloc_32_user_noprof); /* * Atomically zero bytes in the iterator. From patchwork Thu Mar 21 16:36:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599188 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 022A4CD11C2 for ; Thu, 21 Mar 2024 16:38:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 153CF6B00BF; Thu, 21 Mar 2024 12:38:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 104E56B00C0; Thu, 21 Mar 2024 12:38:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD7736B00C1; Thu, 21 Mar 2024 12:38:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C22286B00BF for ; Thu, 21 Mar 2024 12:38:20 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 70A62120485 for ; Thu, 21 Mar 2024 16:38:20 +0000 (UTC) X-FDA: 81921603960.14.FFA3C9D Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf03.hostedemail.com (Postfix) with ESMTP id 7EBE620005 for ; Thu, 21 Mar 2024 16:38:18 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=PLLW3Rcl; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3eWL8ZQYKCGYWYVIRFKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3eWL8ZQYKCGYWYVIRFKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039098; a=rsa-sha256; cv=none; b=JFNKeJAv/cJK7P7lGhcezZahB+BlSt1gkTi4My2/QfrrqRwYiXD00dG03861JNfhEBhO5N AwkypnqhyqPPz32BfRmKIrva6ADeutp78rC1ellGgJBZMkCJGxC0jOk5wFnSykco4HpjVY 3GblVHUyQCeq/l0DcuNcyQVYV0kJ9i8= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=PLLW3Rcl; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3eWL8ZQYKCGYWYVIRFKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3eWL8ZQYKCGYWYVIRFKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039098; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XaC6gpRUGM+qkcKqV2c0VBwhZNh1AuaCpOBKo907lO0=; b=us+E6Yszr96WYrPlQHyo273+LrrDDaUUsOoxTqHzP6U/XLXc7bqx/U3mB/7kgwYNuec+mp EUujDtaXPHs8Bv/UYvzzcoLrh0VIJhU3lZU5dVyTzcE04o780Ycl6sE7a+MBpfFpYqZdWE 0ZeAKI4tV/0hL10/pD7IpV8yJaYitII= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcf22e5b70bso1903743276.1 for ; Thu, 21 Mar 2024 09:38:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039097; x=1711643897; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=XaC6gpRUGM+qkcKqV2c0VBwhZNh1AuaCpOBKo907lO0=; b=PLLW3RclZeJkerwaRUSNHQE/zg825JUXHN9tEyG8xgG2gbilkz0lYEV/9/tmSEZmbP KZhbk6bgpybIAkCEotiWtxFDGD+gu640+v1ADPIXB/r30zrho4IpA/t/Vd9/v7wKWtvd hAoSv5BAqzchPnsdJZn2hCxUiZhuRRVPZR/dMo3FEZK/iPHjFkgbXMpPxaqhOh+mqoIJ oWMVXaRMEYCMIJFSmp9eaBfoLWT76csoZqOCOQsckKWsaXI4GK4cmr/eaK5J2FrUjCcI 6lGP+JUuW7IDxYGOEnzDMDmUrYfM6D2s9d703Fok0WsIKI1kOOhuaTv7lcf2fNzrvcoG SUng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039097; x=1711643897; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XaC6gpRUGM+qkcKqV2c0VBwhZNh1AuaCpOBKo907lO0=; b=qShf74ul5Ieb3rGoZ+sDTWoDYtwEYKxK1LalajIq+nmXXq0RGPE8zA+adEtACoqd3r m/HM2P0rVWElHkhpfDuxrXYnWAiujyGtJ2O/1Vm6pBpK8UgdHw6rUgiA7YuZvf6rlViT ZQAS0evLE+M62nYkuzgX9tKxXdPj3qO/xl2AUO49S/X/gxEbsSUPShK8G+t2Pq5lsT53 js+OBRRhpZUBVCqzbYkSRCMhTgtmT2pc2e5HyA6KEljmwkiJqPdcNAj8Jx9c2gzj7Vr4 aip6ao3IN2Z6MEU2alZJYeVRvzWjR58BjuvQzEWe+3wTEIgowW/nTsqDbNXwKqjlfhu/ BUUg== X-Forwarded-Encrypted: i=1; AJvYcCVpBg/nI8R5NbWPpfGhF+tnfTZi2+nQuTkupaQ5X2R90KFn5gmkf7WNd2iCU4VIS1la90nk+Uz/m8qg+pcsMomVmzI= X-Gm-Message-State: AOJu0YyaVV0LlmzzGrcA1qaZDkwrv4kRWVsH7sK1dOF7ITBvs2DrTTPX RmAfERV+elcESEG6oFqV3YnsgqpdYUmajiTZ30kBKyWnks6ZlkxTpV/x/miYaOfBP31fGoy+LK4 0sA== X-Google-Smtp-Source: AGHT+IFJZSHudCuIglXlIopkXnNLMb9lH1yhVoF6leBD0jJ9ERT3JCkQE6hJAbySj4mxb56Bo4c8jp1OowQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:1502:b0:dc6:e823:9edb with SMTP id q2-20020a056902150200b00dc6e8239edbmr1143317ybu.12.1711039097346; Thu, 21 Mar 2024 09:38:17 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:53 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-32-surenb@google.com> Subject: [PATCH v6 31/37] rhashtable: Plumb through alloc tag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 7EBE620005 X-Stat-Signature: u96kx4eq4kzakxd6n1uynfq4cxebqkd4 X-HE-Tag: 1711039098-50520 X-HE-Meta: U2FsdGVkX18qOJ3S83o/MBdV4Byk3+XQ3fkgW6aTT3QEs+hfn+H8ivzVSABNlMaqVV1ygWcW7vfPX4C2LO72/gxYNEnNKDmddEwlpPvV+92aIZT3NHUn9G0p+u9rvams5vmJEBzwSa3UVABPkNYQaqHInuVbpKoNm6zVWqFL2aU/xZmSzb7gIVPZ52DSvk1hSjqQPkG6FLOIf06FfxCOrzw8oUt3uWmDf1MZ2Zojnq9MzcRiEm8eVFCQY+J7BPuwK+Lh+cnXWyFpYESZderCOHwy+k1rbjhx/d2pbSO7V40i8y9y59tXA0SFUhgBEWapPV2qK31c7xIKHc+tsQxAYvPWCtfVD2zj7MNdAq0MNxF6nDTq/vdpwZqcJuDQznWSCgohCESBXv5NJum2SWwp7Bvf14cOe2TI1fwjBStG+j/rQkHx5z3THkQsnPM16UQyrocZXLFW9rAUM2zSvwvXPeP50btaGjwrrMQIX6ZIEcZ1WlA+xrFQ+Qgx0xNXhfx22LXWOnATA+52BzIHv7Fc90lU/D5fW84mpJ7p99lWQLaJJ/7eEA6J2e4b4T8GYRa3PumOsMVhPt4kiRNBVBAiv3UBENre+zxCvPhgHmbL/50MSazi8wFLTZOApG60jWAzMs3iRNCDtUxgNVuRfg19rJE+hYV2J+8bx4lLEGmZRq/7814bJNCE6A8zKkuneLcWQtIWrXXxApia8o4Bn/jvbOhXndarhx92rmXMAeqsXw21YNSjWj2N1PyEg+Jpj/sxlo8+iHMtfbRDT4PGrTzGNwnXAmCIXA75frWMZLji9QD8tkT/jeGGEJ82T7/Vs/S17GaKVSHDAsC7Z/d2K/5UOtTb/SMmC9lsdb5XUoJnhg2m2JZTpx6+WgeTLfxGeiZy209aFN1qozGXJtjLKaGQB4jFk8H97YNoW98SI2gGlQCSzbi+/uwioasmrsFaXbzLegbZTKHL+mLlOlHp0lq wHw2+/1o NzR3nKWIraLM2Esu032IFdnJ25uLEZobF4dA9XqhBOA38bxeQ15je4G3tEpcOvDpYWazOzyOy5aMS5OuHWAomQT6bENoJU/2h9GRVJsnbz9pbe4m3/1EnzAQmGtO8rpnGBUeqr5haIcQ9YLoLYIsyHbCnL3+lpQbgwBUQCMt6bS4RY88W/X12pBY9SHmha90qJz784L5zv/8DtGQPoW+fixNofV7cQYkm+oPPTGs9EWkVt7HX6auUtu5oBz6iGknJmrAEAKSR7UuN9/ez/LYUtcyrtpsYrDCYOfMO8AdX6nzYGG4WNya1qTF1G8Wn7HNS9TYFrpbxupfw0dZsdst7tyNz1tqqG1IFPu/XLKt+qtA3ndvqtWQD/ljbTN5+d55WQS4FudQlWS37vs2dwd/brQgSvvU0btYgOGrAxWJLgubiit9+yNDNJNFsB3jd69aQqKaVr+sZvXkW0GiXZH7oGTJJtDU71XIxtIXgwKSCxI4IOXXhgTX/iSPiW6i6Dh3Q/zYYs81Z+OhdLT9OtBItdsekfqPxlYvEHmG1ODIKoBYJnCAMezTcXAHbQqUJVf9HF9yRdnG1hxZdcHkIfMnT/nolM9sdyb/jLJDhSKRvXlE/F/i0I63jgkm9ZHqx4fHB5F9oJCpJs6ylBf9rt0lxCxBu4VfDIn9uJNj0XuurhZ5b5iA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet This gives better memory allocation profiling results; rhashtable allocations will be accounted to the code that initialized the rhashtable. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 3 +++ include/linux/rhashtable-types.h | 11 +++++++++-- lib/rhashtable.c | 28 +++++++++++++++++----------- 3 files changed, 29 insertions(+), 13 deletions(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index bc9b1b99a55b..cf69e037f645 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -141,6 +141,8 @@ static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) ref->ct = NULL; } +#define alloc_tag_record(p) ((p) = current->alloc_tag) + #else /* CONFIG_MEM_ALLOC_PROFILING */ #define DEFINE_ALLOC_TAG(_alloc_tag) @@ -148,6 +150,7 @@ static inline bool mem_alloc_profiling_enabled(void) { return false; } static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, size_t bytes) {} static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) {} +#define alloc_tag_record(p) do {} while (0) #endif /* CONFIG_MEM_ALLOC_PROFILING */ diff --git a/include/linux/rhashtable-types.h b/include/linux/rhashtable-types.h index b6f3797277ff..015c8298bebc 100644 --- a/include/linux/rhashtable-types.h +++ b/include/linux/rhashtable-types.h @@ -9,6 +9,7 @@ #ifndef _LINUX_RHASHTABLE_TYPES_H #define _LINUX_RHASHTABLE_TYPES_H +#include #include #include #include @@ -88,6 +89,9 @@ struct rhashtable { struct mutex mutex; spinlock_t lock; atomic_t nelems; +#ifdef CONFIG_MEM_ALLOC_PROFILING + struct alloc_tag *alloc_tag; +#endif }; /** @@ -127,9 +131,12 @@ struct rhashtable_iter { bool end_of_table; }; -int rhashtable_init(struct rhashtable *ht, +int rhashtable_init_noprof(struct rhashtable *ht, const struct rhashtable_params *params); -int rhltable_init(struct rhltable *hlt, +#define rhashtable_init(...) alloc_hooks(rhashtable_init_noprof(__VA_ARGS__)) + +int rhltable_init_noprof(struct rhltable *hlt, const struct rhashtable_params *params); +#define rhltable_init(...) alloc_hooks(rhltable_init_noprof(__VA_ARGS__)) #endif /* _LINUX_RHASHTABLE_TYPES_H */ diff --git a/lib/rhashtable.c b/lib/rhashtable.c index 6ae2ba8e06a2..35d841cf2b43 100644 --- a/lib/rhashtable.c +++ b/lib/rhashtable.c @@ -130,7 +130,8 @@ static union nested_table *nested_table_alloc(struct rhashtable *ht, if (ntbl) return ntbl; - ntbl = kzalloc(PAGE_SIZE, GFP_ATOMIC); + ntbl = alloc_hooks_tag(ht->alloc_tag, + kmalloc_noprof(PAGE_SIZE, GFP_ATOMIC|__GFP_ZERO)); if (ntbl && leaf) { for (i = 0; i < PAGE_SIZE / sizeof(ntbl[0]); i++) @@ -157,7 +158,8 @@ static struct bucket_table *nested_bucket_table_alloc(struct rhashtable *ht, size = sizeof(*tbl) + sizeof(tbl->buckets[0]); - tbl = kzalloc(size, gfp); + tbl = alloc_hooks_tag(ht->alloc_tag, + kmalloc_noprof(size, gfp|__GFP_ZERO)); if (!tbl) return NULL; @@ -181,7 +183,9 @@ static struct bucket_table *bucket_table_alloc(struct rhashtable *ht, int i; static struct lock_class_key __key; - tbl = kvzalloc(struct_size(tbl, buckets, nbuckets), gfp); + tbl = alloc_hooks_tag(ht->alloc_tag, + kvmalloc_node_noprof(struct_size(tbl, buckets, nbuckets), + gfp|__GFP_ZERO, NUMA_NO_NODE)); size = nbuckets; @@ -975,7 +979,7 @@ static u32 rhashtable_jhash2(const void *key, u32 length, u32 seed) } /** - * rhashtable_init - initialize a new hash table + * rhashtable_init_noprof - initialize a new hash table * @ht: hash table to be initialized * @params: configuration parameters * @@ -1016,7 +1020,7 @@ static u32 rhashtable_jhash2(const void *key, u32 length, u32 seed) * .obj_hashfn = my_hash_fn, * }; */ -int rhashtable_init(struct rhashtable *ht, +int rhashtable_init_noprof(struct rhashtable *ht, const struct rhashtable_params *params) { struct bucket_table *tbl; @@ -1031,6 +1035,8 @@ int rhashtable_init(struct rhashtable *ht, spin_lock_init(&ht->lock); memcpy(&ht->p, params, sizeof(*params)); + alloc_tag_record(ht->alloc_tag); + if (params->min_size) ht->p.min_size = roundup_pow_of_two(params->min_size); @@ -1076,26 +1082,26 @@ int rhashtable_init(struct rhashtable *ht, return 0; } -EXPORT_SYMBOL_GPL(rhashtable_init); +EXPORT_SYMBOL_GPL(rhashtable_init_noprof); /** - * rhltable_init - initialize a new hash list table + * rhltable_init_noprof - initialize a new hash list table * @hlt: hash list table to be initialized * @params: configuration parameters * * Initializes a new hash list table. * - * See documentation for rhashtable_init. + * See documentation for rhashtable_init_noprof. */ -int rhltable_init(struct rhltable *hlt, const struct rhashtable_params *params) +int rhltable_init_noprof(struct rhltable *hlt, const struct rhashtable_params *params) { int err; - err = rhashtable_init(&hlt->ht, params); + err = rhashtable_init_noprof(&hlt->ht, params); hlt->ht.rhlist = true; return err; } -EXPORT_SYMBOL_GPL(rhltable_init); +EXPORT_SYMBOL_GPL(rhltable_init_noprof); static void rhashtable_free_one(struct rhashtable *ht, struct rhash_head *obj, void (*free_fn)(void *ptr, void *arg), From patchwork Thu Mar 21 16:36:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599189 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7400C6FD1F for ; Thu, 21 Mar 2024 16:38:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5E91E6B00C1; Thu, 21 Mar 2024 12:38:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 598BE6B00C2; Thu, 21 Mar 2024 12:38:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 327426B00C3; Thu, 21 Mar 2024 12:38:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1DEC96B00C1 for ; Thu, 21 Mar 2024 12:38:22 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E8E591601AD for ; Thu, 21 Mar 2024 16:38:21 +0000 (UTC) X-FDA: 81921604002.23.77AFF37 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf22.hostedemail.com (Postfix) with ESMTP id 51A5CC0006 for ; Thu, 21 Mar 2024 16:38:20 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=GHAaHRBI; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3e2L8ZQYKCGgYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3e2L8ZQYKCGgYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039100; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iYMC0aOkTaJuIlgenfbozu9BZf0q2gbSOAGaj6P53nM=; b=1WybYPrzexn3gohsvKSUoliKn4uj3FPAIXkyd5jFfxQx69SSNwroGN2orJKBijVMejnibr 2Y6nleNvB9hwLQ+1LnfcHGjAnDWkCOegmORlSE0AlpbEXH+L1lpfSzkDRN8aBdVbL8w2Fq EniSV3g2RHoZok3UgzRy98RVFbimB6Y= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=GHAaHRBI; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3e2L8ZQYKCGgYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3e2L8ZQYKCGgYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039100; a=rsa-sha256; cv=none; b=4gudAPFpN05jIcrLWc3CWThxmQbDuOiZR9Z1AmAxxvd77rYWsxTqjRiWtkSldnxV4+1cr5 zKKpt4Kb4/PtPfDJfH7tbr1h5Un280VSk5jFtqTSdpWA6laZ7I0WWrvlNdawfGhPNitKtN RzNUJpnfJzeY2tNzVuIj2wEeBa0K0KU= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcc15b03287so1686715276.3 for ; Thu, 21 Mar 2024 09:38:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039099; x=1711643899; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=iYMC0aOkTaJuIlgenfbozu9BZf0q2gbSOAGaj6P53nM=; b=GHAaHRBIWh4cHaCkWevDiMuHXFHTMBF9VXE8fPbjZ2qE4jIUeO75ywTuMODfeHItSs Bxt5R26K3HlGCJItLoT21Rzan0tBYhUh+rlCc0D9Ss9AX3ANMAqroH2BVvw14FSY66Ps L/z78Ph4RyIpOC8GJU8jIm0xKNMKrlRitsl13AtMJF8t7kYHy+8pvrLDTAwlKsRLZhm6 3juzwq1a7O9kbmoAK8kgcjANgRvvoxx5p2a/rlqYakpzmL1kTKVkr/W6//YzfrN9I1s7 z7Fg86C+CeRKe63gTxDGRi8eRCJOHyeJa/Q1261ZQBoO9Y78cOXmqol7BNqGAX3uO43g Npfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039099; x=1711643899; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iYMC0aOkTaJuIlgenfbozu9BZf0q2gbSOAGaj6P53nM=; b=To3cNQ0zl5GVJ/T4h8RVy+xsN9WOk1PR1bbXu9YbFYyCMkFRMrxq+aVeHjyXjKz6RR 8LNdfcp/JUvUtNGtGAd0i6JErYb9LVYal1kZsAH8kbd3atxcv9MgOIrT27ECenEItFqI El/joDboIkbba1ObyLjhBDE00akhxrFDaZ7GggvFCKZ9lOKTRGLIiNORfsI3GSjPvDSW cDKWVm/2iY4U+QrQTjuMSECWOSl002uFG69K9qscyRqY74FDL6D525ScVxmk2Ygx/75m P/u6Fwi4XzJ0Puh/TLyzCb3sLqvVwOh4BAIHCgzRKD2jeadB1IHt9bCI2rPVEBfie9xd sOdw== X-Forwarded-Encrypted: i=1; AJvYcCXcXQtLs65O4yVdJ0XK/rMv3XwcdrrW9DhCUYMrimdtVbbWvoVbjYglJQDabzBrF38npuxbS+Sph558nCDC+W4FN80= X-Gm-Message-State: AOJu0Yw42xbfwIe2RVjh38PnKnyrS89spH12SnpG8XlWGWt11gP/5+p8 kAa7KVr1QbsUJYBu7JjwRDM3F0FAwcSpW9DLD5GBCRuvHk9JLlvyUs4yTnvb+uqethrBnRBNSrs kOw== X-Google-Smtp-Source: AGHT+IEl39fT88zAybCKFbgD7Ybs+bIKptMpR9e50EtGQF6WZVLSjiN4xDj0a9gSVDcCEVc03olmfnidgNM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:2009:b0:dcd:c091:e86 with SMTP id dh9-20020a056902200900b00dcdc0910e86mr979486ybb.13.1711039099264; Thu, 21 Mar 2024 09:38:19 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:54 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-33-surenb@google.com> Subject: [PATCH v6 32/37] lib: add memory allocations report in show_mem() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 51A5CC0006 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 8of6q8cxdqdmt8hgwg8okourdbmmn538 X-HE-Tag: 1711039100-957490 X-HE-Meta: U2FsdGVkX1+bE8PlWzOTayt0R8MJp4Wu+RlcbwDdbqTTuaVzkAp5mcWJWGad7ADJT/J6voxAwGsxlrIoL3LIxLf+899Zvv3MR2O69hzqdazxTFfqWS/tw6NLGTlVmnoEI1xEyAWT5ckacc3Rd2odJX5FC3/MTRv/455+irwtubmv8fCffHZTOqjHwCZfJVtNo0rhLnbC2R7Xesr7mVPe8MsszuiU8E/BG9AohdUWiikM5oSHuq0g1lxZ0JuZnZbENEyVWVaacl70h8YPb9VzBKq+eEBB2fIfHgcwIjQTj6l50HCu+CwjxX+eb/SAAOtuApYFfDhcF07t5CgGsa8CehGtr3MnBVg8ADNvJn6/mOW7RYkiNdAIvlmDwenhJIIx3ZHPCDrmDWfwNHCySAw6dfHwitsI3prSa4qIQPRYAEte9ImFLWF8pVFzeZMCjFy6zGtS+aQN5Wh1XBW6LhZXYTIPeYDAyG+k9HZvrwcUXcV2PMPHqip21FNacA+Y7tkHp2NZkxlrp8zEaUqSvpTAWH4AvSKgkZP4a0Pi9It7gYnfOmVPzVpgikM+a8r8tznP8wxqelt87kEf1xZW0O9cQuY3bFEcb6qLkyC2XPELKsQ6twVuAEoXzVUhqfCoWj/Bj+czTx1jpZPhO15S61OgxZWufJ1n27M57zhGCJ+s/QCRzKh3+nvqCBjI51CeqdAqPzgfweCrMb8l5OFoft6AekdyAkcXCpjA00GWgjZCvnaPGpUnFehdoQ5Fr6/PTqSQWWfJ9ZP1fh24cUkAzLgrQg9+8d2HyVI1ADgn3vSdjHBFYVsTU1Q0xpDKH7QnPU6bOAHpsP26WuOJ8G0BUDkaCkNBWCAOBCmJjEgnGXsByOh3IYiq+JjOuIL6llRA9ccKq8ZfQ4glO5L0R9RF23qD2VkS21YRTl9uXCY07g2lQ+t73K7XAzpBAAb7EL5WpcZY/QEFe6IS9IKanj5BHvO vG/YVnPI WqxUwrwJsEq6XTlORuLgP/9L61EnOpjkSbyB7ade4Ht3zEmgox/gtDaWR88bJFB28SF3G0ZpIgGl4vfGhendU1V6H70IqmVzF8iak0PQveGMExpiqo0aEhN7hq1fpSsJVMVZJ10XPsYR57+bFnFhGYCe7vdD39bvm1pGyMesrnCe2SxjXJTzcJYG/f+RIddSwm8m/C87wdThBba1tgo62y1CZBizfcFSx4/1A2LBWqc2nhfn6tqkTyX4pfLh4S7xuRaIUdkgVIeVoi5OhhTZ8OwuIUOfqresCfGFXh/EM7aC4Bekf6DecIQ7VvyE3M60DW4gQfHnXQwVrgSW3MLm/LRtKjKHC85BqcozX9Ilel1aiH+RGsnof8wu/FuM5jmV5MxeG2keZnguSvq/1NqSbaH/pujD3+NI6FTd26mzt4jqd7OuIPrLW3bOUUPxO20DktyQNrXQBRZ+/fBOeaIaN49MH4w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Include allocations in show_mem reports. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka --- include/linux/alloc_tag.h | 7 +++++++ include/linux/codetag.h | 1 + lib/alloc_tag.c | 38 ++++++++++++++++++++++++++++++++++++++ lib/codetag.c | 5 +++++ mm/show_mem.c | 26 ++++++++++++++++++++++++++ 5 files changed, 77 insertions(+) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index cf69e037f645..aefe3c81a1e3 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -30,6 +30,13 @@ struct alloc_tag { #ifdef CONFIG_MEM_ALLOC_PROFILING +struct codetag_bytes { + struct codetag *ct; + s64 bytes; +}; + +size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sleep); + static inline struct alloc_tag *ct_to_alloc_tag(struct codetag *ct) { return container_of(ct, struct alloc_tag, ct); diff --git a/include/linux/codetag.h b/include/linux/codetag.h index bfd0ba5c4185..c2a579ccd455 100644 --- a/include/linux/codetag.h +++ b/include/linux/codetag.h @@ -61,6 +61,7 @@ struct codetag_iterator { } void codetag_lock_module_list(struct codetag_type *cttype, bool lock); +bool codetag_trylock_module_list(struct codetag_type *cttype); struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype); struct codetag *codetag_next_ct(struct codetag_iterator *iter); diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index 617c2fbb6673..e24830c44783 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -86,6 +86,44 @@ static const struct seq_operations allocinfo_seq_op = { .show = allocinfo_show, }; +size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sleep) +{ + struct codetag_iterator iter; + struct codetag *ct; + struct codetag_bytes n; + unsigned int i, nr = 0; + + if (can_sleep) + codetag_lock_module_list(alloc_tag_cttype, true); + else if (!codetag_trylock_module_list(alloc_tag_cttype)) + return 0; + + iter = codetag_get_ct_iter(alloc_tag_cttype); + while ((ct = codetag_next_ct(&iter))) { + struct alloc_tag_counters counter = alloc_tag_read(ct_to_alloc_tag(ct)); + + n.ct = ct; + n.bytes = counter.bytes; + + for (i = 0; i < nr; i++) + if (n.bytes > tags[i].bytes) + break; + + if (i < count) { + nr -= nr == count; + memmove(&tags[i + 1], + &tags[i], + sizeof(tags[0]) * (nr - i)); + nr++; + tags[i] = n; + } + } + + codetag_lock_module_list(alloc_tag_cttype, false); + + return nr; +} + static void __init procfs_init(void) { proc_create_seq("allocinfo", 0444, NULL, &allocinfo_seq_op); diff --git a/lib/codetag.c b/lib/codetag.c index 408062f722ce..5ace625f2328 100644 --- a/lib/codetag.c +++ b/lib/codetag.c @@ -36,6 +36,11 @@ void codetag_lock_module_list(struct codetag_type *cttype, bool lock) up_read(&cttype->mod_lock); } +bool codetag_trylock_module_list(struct codetag_type *cttype) +{ + return down_read_trylock(&cttype->mod_lock) != 0; +} + struct codetag_iterator codetag_get_ct_iter(struct codetag_type *cttype) { struct codetag_iterator iter = { diff --git a/mm/show_mem.c b/mm/show_mem.c index 8dcfafbd283c..bdb439551eef 100644 --- a/mm/show_mem.c +++ b/mm/show_mem.c @@ -423,4 +423,30 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx) #ifdef CONFIG_MEMORY_FAILURE printk("%lu pages hwpoisoned\n", atomic_long_read(&num_poisoned_pages)); #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + { + struct codetag_bytes tags[10]; + size_t i, nr; + + nr = alloc_tag_top_users(tags, ARRAY_SIZE(tags), false); + if (nr) { + pr_notice("Memory allocations:\n"); + for (i = 0; i < nr; i++) { + struct codetag *ct = tags[i].ct; + struct alloc_tag *tag = ct_to_alloc_tag(ct); + struct alloc_tag_counters counter = alloc_tag_read(tag); + + /* Same as alloc_tag_to_text() but w/o intermediate buffer */ + if (ct->modname) + pr_notice("%12lli %8llu %s:%u [%s] func:%s\n", + counter.bytes, counter.calls, ct->filename, + ct->lineno, ct->modname, ct->function); + else + pr_notice("%12lli %8llu %s:%u func:%s\n", + counter.bytes, counter.calls, ct->filename, + ct->lineno, ct->function); + } + } + } +#endif } From patchwork Thu Mar 21 16:36:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599190 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 769E0C54E58 for ; Thu, 21 Mar 2024 16:38:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EE9D26B00C2; Thu, 21 Mar 2024 12:38:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E70866B00C4; Thu, 21 Mar 2024 12:38:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C00736B00C5; Thu, 21 Mar 2024 12:38:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A871A6B00C2 for ; Thu, 21 Mar 2024 12:38:24 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 84E391608D7 for ; Thu, 21 Mar 2024 16:38:24 +0000 (UTC) X-FDA: 81921604128.28.4B19FC1 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf25.hostedemail.com (Postfix) with ESMTP id AB946A002A for ; Thu, 21 Mar 2024 16:38:22 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="vY6Wen6/"; spf=pass (imf25.hostedemail.com: domain of 3fWL8ZQYKCGoacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3fWL8ZQYKCGoacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039102; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WqtLQxxTOcT4NwKA/wnlkHzJUMqCp8K/X9+cNXxjvBQ=; b=LWf0O+G4buT6q0Hgfh/cVqhYzwUgU7yoCymu5Bjb2GFcCnxpg7nkTHZn4+UrxL24kmc1WN CJ32/SSA6qc7L+WwSFpcnGMAmpZJpyUY7Mu3ky5ShAlTTZDnmaOHr1OB6WUdBPbBK+pkW+ vbCNaQ1YEWQxNgOpXE9ChiYowv0IdZA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039102; a=rsa-sha256; cv=none; b=sCgkdCs4dIXFiYLBHmm5hZ+djVMC4AnH+13TNZlYGX9FTsH47CN971/XiVme0sIGhbuS3C Q8dQlGdMhziiNGXvePanHO+jMOvtHO1SE88zdtOFu1KFtvFHRQ2bN0USeAKOuKNhPtNJYI BEJo9xyRTkb6QrxjgK+jcJ47R7VBm1Q= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="vY6Wen6/"; spf=pass (imf25.hostedemail.com: domain of 3fWL8ZQYKCGoacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3fWL8ZQYKCGoacZMVJOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dbf216080f5so1809351276.1 for ; Thu, 21 Mar 2024 09:38:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039102; x=1711643902; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=WqtLQxxTOcT4NwKA/wnlkHzJUMqCp8K/X9+cNXxjvBQ=; b=vY6Wen6/0T2AQDa2tZ5rIPAgcBDQAOofGHIJDYJq/x/R3RSH1JcZCtmwOGqTgDfgQC HjyrPdCr9nB5DoUDHdKrz4IM//Cq5I32MY3dch1EGn4hEod7ahzeEPy/uXnuFeanx5dV HikF0RnIT9v459wbUQFfAerah7tbULzV4nq1HCysMFPz7n9lc+YKgojb0yaLqooBzUgG Gs8x6NW4mIuHlkHVFNGDO+1W7LRzO8UogFbdq46qvxve6U7DyytrF1XuE8/b3u6UbGz2 j7GK7rjWSerkVSsReO9Eo8BFgj+xjtuhWv5UIcW7yXUZcKSKW6WXlDsfElAmtuFR0lH5 a9GQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039102; x=1711643902; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WqtLQxxTOcT4NwKA/wnlkHzJUMqCp8K/X9+cNXxjvBQ=; b=MekXAyxlxNs7qJD8jW10LQoBeVyJLBRoZieMDDOci8DN+Y+rXfC+miDc4MJw5E8I63 rLmB1l2KQTusN1GErKDvEfLxRQSyYyJaQw/eIAoblRTtrdQ6q2HuGi/o9YwQ1WgFHPWv mXbmVrV5Qljv3loOElCLeKjCxY3BY7p2Cl7j/eHu4x+XE69irjrWG11P5Vu+s5jxr24D 8mKvQGP4qrazU9inEOba4uuhEWoLtnKH+fnZkMRZEtV6xQ+nXTRdIJIR1PT+Zglsom3A eDEK1fOg4HrQ40oC+2vOAATj3/Njm3bNOlTxK3nlKpFW/n1bHTRZ0n6XBdYz3zPWppc+ QQZg== X-Forwarded-Encrypted: i=1; AJvYcCVAHbxbqqV8IRrzBpqLSm9WTOS8rXr8eUJSSX4RsTZOTQFC1ZMU0SMYJ/TVGfApNGQtnsLRrc8GbGDj2ly1l13UE5o= X-Gm-Message-State: AOJu0YyM4q6TlRLaXtsSdHUGZPw7GkPvfr8ZM/YSQIvAOTBL02zMnkiU YxuflgA2j+47F7mv+89EBUsm6cJ/jDCpSbubesgZ/Y/KyjwVNhmCz5hYEV6YqVk34Ppk/JCeJ94 e5g== X-Google-Smtp-Source: AGHT+IGROXQhVTEL+/Ecek6yIPftetu6NwNXc2sw4rx8G7tLlUrCGcwRYmXksEw30SynbjVDOeX+ECRwtfM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:2408:b0:dc2:1f34:fac4 with SMTP id dr8-20020a056902240800b00dc21f34fac4mr5790745ybb.2.1711039101565; Thu, 21 Mar 2024 09:38:21 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:55 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-34-surenb@google.com> Subject: [PATCH v6 33/37] codetag: debug: skip objext checking when it's for objext itself From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Stat-Signature: ju8nobm47tgermxzftn9cb67fccbihcx X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: AB946A002A X-Rspam-User: X-HE-Tag: 1711039102-677149 X-HE-Meta: U2FsdGVkX1/LadF5rQ+Qb8CLppSekpt2CVgllazjsr5u4unp+k0OozgmSXL/S8Bv+U4znFIUO+AQmToCxr7tZVJLdfYNGQTZYwfHViSnVGMIsexdXnR3sQHeNLJiVIqxmWQFdcAlfq7YYzh1jn59je0AHaQ9hefFcMadscBVdDiiub5N2CKDxhe2t8gAJ5JKU9G8F2+ESj1vlvc8U/By/tPwdVjsCJFdQSqOR+NhjifEsixWu2hjzBTb3nycg0hVaKeid9OWipOaAKuwIxHGHPR+yp08QZsjHm/1pHzLwlTyZUrLnrJQCn1NTGFUuuqWtkBSHqVJSo9vMg61lSOs2lhYVKZxBeQQsEfGHnZbAFs/10FqVncKel8gDYrn2U9bIt+ZCSJkhBoBuvZMdwgPydQuJD0k+CAk8z283BZEcwFCY47fl2QltH8+LGK2ZtUwaRcFb5uPdt1h1eBSIthxs+x3rOhYgAjWXZpxcGwDoqXw/t7Bi5L4MJdV80zJdFuMLB1/mB/9N2oC6sqJqTwfjVqNCfzgHRZvvNaDrAZkmfhdC1SNUrIy8EXiDBRwUgn0bGjz+dRieV50KzGwY0QVN68CyNYIX/wq9XZgPzgqUXYVKUyCnWXdeXXPBCPK9eDu0fo/xxhX6peTdle3RcV9KS9A+iLT0Tlj3QfLMflCwprmFOB4bZtqzTezCvq53OMQ2K3dJ27XP1pjbL4tnGtDU+4J35k55mrgIIHVjdxFqRh6wzwKsBcy5d4tRXTQb7DMl49kH78hpcsHEmqrZyeb3kXMvOpugdW+HCdNc7qeZs/SjIfnder0eNo8rXcfDSfYcQhyPDZcsyWCrlf2ycIOKxSitju4Tebi125kdglJePXQoGS0ybSmplq3bfY4VCIdLyavjXLMQ48Gx6Yof9mV+KDNyjcNqTcmMCHk9OGQ9nCj+rtVF1fhqEXXcA/eE2nFAYhLzA6BJDKWMWg2ft5 DaGzM7DV oAq4RAetj8419z4CEz3FLEnaSgispCOzVKIJXY1Sjl2NDgxvO+NXu1E/ML4hrvCyZB43es716RrGpTCB6MKY2wo/LuQT8OO9OGd7trmSkHeMUpFNlNagCV3pjR6CwHdDaWbA9foWTgGEJK78NPs7MHD1wRhvyz6/zAUC8mV1/mwhaJ1saO3VS9OjouY1YrenOL4ok1aiwy2BiGWnlsQHgoVXQx4GX3zZD3cOmuqbp8/hM68duuWJzScU5eHAqMsDgoXYF65bizuegPlQYEnzXyZ6v1POdpvyHzBHT//8HMWvMpEpy95esRnWS13DyayNIHXU+zyAXPD8V7JFd3+4ycOI/T62CqglcglHSRaToVzb55ekZAjm4CMIOxZAKOVkUMWTEzS6p9CIW77zMS1inMYEHU9eFHydtmLyXkeekzZbOkFZF8P+P9DwJOcekYkw1PSTHt52aVW6FEX9ELLB2olDn/UZw8vpnU7YEvlYsW4FwqncDKN5rMBheaxGIXQ5My93Csi5FLdnCUkgnw+t6LMBQl7n8OizIgKs/Qu03Vl2/fou41E5sb8jft/KyrO0mfIOfO/RCQvAT8MnvnOCRM+TOlXZ1a4Vawk0jexr4AGfGhh3l0uEL/Ek1949DUosi080izLb/eIcJwRKPhIMppLDhRhoeNvL7c9FxRevBP+e41wA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: objext objects are created with __GFP_NO_OBJ_EXT flag and therefore have no corresponding objext themselves (otherwise we would get an infinite recursion). When freeing these objects their codetag will be empty and when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled this will lead to false warnings. Introduce CODETAG_EMPTY special codetag value to mark allocations which intentionally lack codetag to avoid these warnings. Set objext codetags to CODETAG_EMPTY before freeing to indicate that the codetag is expected to be empty. Signed-off-by: Suren Baghdasaryan --- include/linux/alloc_tag.h | 26 ++++++++++++++++++++++++++ mm/slub.c | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index aefe3c81a1e3..c30e6c944353 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -28,6 +28,27 @@ struct alloc_tag { struct alloc_tag_counters __percpu *counters; } __aligned(8); +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + +#define CODETAG_EMPTY ((void *)1) + +static inline bool is_codetag_empty(union codetag_ref *ref) +{ + return ref->ct == CODETAG_EMPTY; +} + +static inline void set_codetag_empty(union codetag_ref *ref) +{ + if (ref) + ref->ct = CODETAG_EMPTY; +} + +#else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + +static inline bool is_codetag_empty(union codetag_ref *ref) { return false; } + +#endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + #ifdef CONFIG_MEM_ALLOC_PROFILING struct codetag_bytes { @@ -140,6 +161,11 @@ static inline void alloc_tag_sub(union codetag_ref *ref, size_t bytes) if (!ref || !ref->ct) return; + if (is_codetag_empty(ref)) { + ref->ct = NULL; + return; + } + tag = ct_to_alloc_tag(ref->ct); this_cpu_sub(tag->counters->bytes, bytes); diff --git a/mm/slub.c b/mm/slub.c index a05d4daf1efd..de8171603269 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1873,6 +1873,30 @@ static inline enum node_stat_item cache_vmstat_idx(struct kmem_cache *s) #ifdef CONFIG_SLAB_OBJ_EXT +#ifdef CONFIG_MEM_ALLOC_PROFILING_DEBUG + +static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) +{ + struct slabobj_ext *slab_exts; + struct slab *obj_exts_slab; + + obj_exts_slab = virt_to_slab(obj_exts); + slab_exts = slab_obj_exts(obj_exts_slab); + if (slab_exts) { + unsigned int offs = obj_to_index(obj_exts_slab->slab_cache, + obj_exts_slab, obj_exts); + /* codetag should be NULL */ + WARN_ON(slab_exts[offs].ref.ct); + set_codetag_empty(&slab_exts[offs].ref); + } +} + +#else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + +static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) {} + +#endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ + /* * The allocated objcg pointers array is not accounted directly. * Moreover, it should not come from DMA buffer and is not readily @@ -1913,6 +1937,7 @@ static int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, * assign slabobj_exts in parallel. In this case the existing * objcg vector should be reused. */ + mark_objexts_empty(vec); kfree(vec); return 0; } @@ -1929,6 +1954,14 @@ static inline void free_slab_obj_exts(struct slab *slab) if (!obj_exts) return; + /* + * obj_exts was created with __GFP_NO_OBJ_EXT flag, therefore its + * corresponding extension will be NULL. alloc_tag_sub() will throw a + * warning if slab has extensions but the extension of an object is + * NULL, therefore replace NULL with CODETAG_EMPTY to indicate that + * the extension for obj_exts is expected to be NULL. + */ + mark_objexts_empty(obj_exts); kfree(obj_exts); slab->obj_exts = 0; } From patchwork Thu Mar 21 16:36:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A31BC54E58 for ; Thu, 21 Mar 2024 16:38:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC5286B00C4; Thu, 21 Mar 2024 12:38:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BDBAA6B00C5; Thu, 21 Mar 2024 12:38:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A04E66B00C6; Thu, 21 Mar 2024 12:38:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 89FD06B00C4 for ; Thu, 21 Mar 2024 12:38:26 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 6161F120485 for ; Thu, 21 Mar 2024 16:38:26 +0000 (UTC) X-FDA: 81921604212.09.7DA8199 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf16.hostedemail.com (Postfix) with ESMTP id 88B7018000F for ; Thu, 21 Mar 2024 16:38:24 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=i6I0cwro; spf=pass (imf16.hostedemail.com: domain of 3f2L8ZQYKCGwcebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3f2L8ZQYKCGwcebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039104; a=rsa-sha256; cv=none; b=1jbl9Uxi5eAx7BwCLruZA6ukHDgVG2mDOApydhXwidjBUuQk+beh9yrwEGopIv2llux4bv 4IXT7ZVnYLxgbzRbT7i8lvpTN3SP1c1cN1S/3tGXhyYmQrgna+gBnfMP39sZrDvc8iWiTw r5V5jpdkYimqzqSj8BL+vVbIWxUQlgM= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=i6I0cwro; spf=pass (imf16.hostedemail.com: domain of 3f2L8ZQYKCGwcebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3f2L8ZQYKCGwcebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039104; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fkzGAdcAIO9V5Ziif/kwje70ys4aQ86/VyfgELZXTD4=; b=8pE44KknU/dPQGPTtAG6A/9wZfkOJpFr6zeUCmoChMjg9C6t4LBWZZPGQINGBZFG9junqU qRWBiCUKrGZbhS3F59wp3XTLUfmfZe9m6TToJ4kMHYFOvvWmBVrzFMokbOSCcu7ytWSGv/ K43E9oqdHnYQptX7jHgooKsv6WEyZqY= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dcdc3db67f0so2777690276.1 for ; Thu, 21 Mar 2024 09:38:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039104; x=1711643904; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=fkzGAdcAIO9V5Ziif/kwje70ys4aQ86/VyfgELZXTD4=; b=i6I0cwrokHTRSWAl9Xfs4rL8zfUemla8j6nu8wfbWQ2afGyPfKkEvmwf6S9Wf/6p6o czhk6bcCMLJB1/WYChCW5999pKfl4YjEvNXfP1Qf7p4zgBHQC7oc1giAhaDyt2mVyjbU TUXqCbnFalWL3dLqRTUpuDcyZ+coEQ2EKxgnZV/JAD4agjJ8BjsdTt4gq8Ks+d38JbJV YsVr02IdPEeUMl6iNZhLZ0poNoDqItr6csVjFhKztAlCgJt7di2z55osB44sxIf4RLzj XfDsI5vtcUd5z42HyaIb6IeApny5257GFfaeVqv4MNU/cBACBIfDEuOs1O/zHbID1JGl CWHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039104; x=1711643904; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=fkzGAdcAIO9V5Ziif/kwje70ys4aQ86/VyfgELZXTD4=; b=r39Ekw2XQwiFt1LGf07VzuLVJVTuS/U3Z2hoP/zOD5V1gI4Ups5NzMmJNdDKNFAG3W VEo4sjsEsD2rPTvgSEfxJO4jv+pR4RxE8S0WXx+ZXLLEcU2t8X5U4AB6H2LcyDsJRqfl XjGyvhOCFzSnXfXdoFlxIl1LqNm0WJRjEiCnzG0+sXXk5j5jytit6dOQdYEEiuRIaEnw buEhdqprF4hvZFE8oSVP4zEp9cI8DCW2db1AvTNQlJuTgEQtxxUz3fXlbucnhYkC0nSR l3O1P8AAavDBcfGDEGw5Ni7tOG7+UCO2Hg6DWDpki8ZQ2d2YVpdj9mY55rdakGLY5pdf JIew== X-Forwarded-Encrypted: i=1; AJvYcCU76kFSckm0zduLW5n0Y1wmT34So12l2opJ8OowBb/HbS4afTY55PhLa1SnokDjsGoc7UDvy0HacuofhM9h1yKJlqg= X-Gm-Message-State: AOJu0YwyaBHc1nwewbhTD6ziXHYH/VOABqaZnCTRnpQVy+8JNlGIfd+H Vt6fOEs0lrgtwIpVHMif9c3iImlnGV6qtRejOo0LoCoOl5Kjxuhca7paTKw5cSa8BqEG32/E7d7 z3g== X-Google-Smtp-Source: AGHT+IGsZeht1WnRv54WdZCsHHkevoWEcBuLsOMdDTLOZ/BEZ/G/n2nCv8k8gfmNzlHHNHWmAN7X+BiTyog= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a25:dbd2:0:b0:dc6:b7c2:176e with SMTP id g201-20020a25dbd2000000b00dc6b7c2176emr610053ybf.4.1711039103623; Thu, 21 Mar 2024 09:38:23 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:56 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-35-surenb@google.com> Subject: [PATCH v6 34/37] codetag: debug: mark codetags for reserved pages as empty From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 88B7018000F X-Stat-Signature: y9z7we5tsutfxumgt9b31u1ad9pyfugy X-Rspam-User: X-HE-Tag: 1711039104-619664 X-HE-Meta: U2FsdGVkX19EEev7rMNYTgYYpalw+ZOlqSbyIHV4UE4obmR1QwDb9nyCjlEHS4+YFAxOUxuuW2qBLamDvZJJDi5s1y/sMGqqv62+pycz6r1u71gTDuWNIxGMjVGJPiWP3xOaosLLJNqxzMqVsKYphxBZkm6koT17iDO+AtsjtD8WvbVj2hkzbJZlvK55H94JNWnm9N3XgsEmogX0JriNbm35Ae4MghulJELbMReEHodKXQS+W9eceTJf5Inarj0O6TWhdNsshpdWTgDi9Yko4S1MPVDQA1odrMH+y/6IIPcfb92vEzCv3n/9bvs+F8cUByNRhojgplE7l6p3P8oX5XIsAlOVzjInYeGDQ8+EEc8gYLAvP1xMF00212UDe5+/kPdSHNcGcCDqyk9yf21wXmxi2KgAvHE8koJ1jAlqgsuQgi90v1Ge+MGbb7BodQI6DpvHJF7OIXspyUhYL17tI18ng22uLKb8S4tXJ4ku0sxYKLZdGY6u5dajktUKwAEAG2TRTvrErybwBHbXCi1OBDrUx2hVy227e8TjNXqvHYi7jnMK3AiBsrhohovUMPjRXddla9AA9Teveqg2jJvSawbtmyl2q8BJVdsxdfmnFuFDaF/zyExXtd2gZjyoHTkx1qz99YMlzc6K8b+VBhjkDVZbBoojMxBOrOccb25ukBUKTAwFtOxD5WZJjooKtsP8wolJtp6DBkPL0biz5yqyoYFTFgAGSvi72CpDKv8uaeOhqlGYUqU+8nc2rwbmIuw3HXGA3Qepa27eVDJczXTSDhnJ2R6pMkJM24xcY8bF9wgv/e0JOEBYgfOOazWEepfiyaUUEOk3vFAtCudGm4lgMiTYM9WOenlrGWnsutzRupjHzyrKegQsBoJ2X+o3VKtVOcVd5jV+AZwJHBQ9ic+I62ScAdE9HZzjM1Zoa/pB5auG08yO8tiLW6BCy+wKCuTUVZWyYL1++JrzLyhfTHU QCBovf/U uN8OKSR48OBj+pWvH+YEZEBLCYi1iDtt/lsqzvnwyRTRAHqm+mzaI4MDpu7aBk1sa7AAtarOpnjdE3F30KoPuJ6Wzn3LIHvwnBetDc6gzIFQ5PlaxPgLH1dnDinj9hxvcVaUY5wCydu3q48MkyXxfnjnL78GBf3B6UYKhgxdqLMd8piuFCfMI1KRAgEaj3d5y4ifpdwrKL63e/vm1vzstTBvMDOuIK2yqzB+lIqSYvJAxw7FG5Mly6JVD7XIPQOotYwwz8NpXksjYV7Q84LjNtFK5WsGepvFGkkDJ0q3Mt+CB5PWBemelsqGGN55pSU23NmuEFtyJq/KHDBfMpwAo9DEm8gA3XqpUQ/OLoTuUiUycwF2diGp3e7+PPCWE26uzZz9JGK7cqNt3rYcDSZGsjwopFmoaCuX+zVKg3Md3tR+VC/Mi4Ks2VXxSo1xit3tqrQ7UO20u+mSPh+jf6+AfNgMVgt7tG6tvi7v9u1F2NocFxctvUuHi5RBcierI8I+5fAi50dirceBgL+knxcyxFOxPml4JGhZ+PG49MPgxUkyjaktvnANam7bDrXtLeroy5Tf0gWob5kAyQlLroDtB4yZ36aBnIYyF/WRo9hiWEIRM0dq+VZWuQQTKisWsqgEHdSobg6TcrpB2z+/zIGJ2EV/9K/PLdyCcpKGHTMKivDHEchYhCxpBhcVTtDwbAQ97xXAqcdZ7EGK57DNqqXKkf8dndKtIXWqQvhH5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To avoid debug warnings while freeing reserved pages which were not allocated with usual allocators, mark their codetags as empty before freeing. Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook --- include/linux/alloc_tag.h | 1 + include/linux/mm.h | 9 +++++++++ include/linux/pgalloc_tag.h | 2 ++ mm/mm_init.c | 12 +++++++++++- 4 files changed, 23 insertions(+), 1 deletion(-) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index c30e6c944353..100ddf66eb8e 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -46,6 +46,7 @@ static inline void set_codetag_empty(union codetag_ref *ref) #else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ static inline bool is_codetag_empty(union codetag_ref *ref) { return false; } +static inline void set_codetag_empty(union codetag_ref *ref) {} #endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ diff --git a/include/linux/mm.h b/include/linux/mm.h index 8147b1302413..2615aa69c823 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -3132,6 +3133,14 @@ extern void reserve_bootmem_region(phys_addr_t start, /* Free the reserved page into the buddy system, so it gets managed. */ static inline void free_reserved_page(struct page *page) { + if (mem_alloc_profiling_enabled()) { + union codetag_ref *ref = get_page_tag_ref(page); + + if (ref) { + set_codetag_empty(ref); + put_page_tag_ref(ref); + } + } ClearPageReserved(page); init_page_count(page); __free_page(page); diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 50d212330bbb..62d8dad74b37 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -120,6 +120,8 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) #else /* CONFIG_MEM_ALLOC_PROFILING */ +static inline union codetag_ref *get_page_tag_ref(struct page *page) { return NULL; } +static inline void put_page_tag_ref(union codetag_ref *ref) {} static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int nr) {} static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) {} diff --git a/mm/mm_init.c b/mm/mm_init.c index 3e48afcd0faa..c7d6376a180c 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2510,7 +2510,6 @@ void *__init alloc_large_system_hash(const char *tablename, void __init memblock_free_pages(struct page *page, unsigned long pfn, unsigned int order) { - if (IS_ENABLED(CONFIG_DEFERRED_STRUCT_PAGE_INIT)) { int nid = early_pfn_to_nid(pfn); @@ -2522,6 +2521,17 @@ void __init memblock_free_pages(struct page *page, unsigned long pfn, /* KMSAN will take care of these pages. */ return; } + + /* pages were reserved and not allocated */ + if (mem_alloc_profiling_enabled()) { + union codetag_ref *ref = get_page_tag_ref(page); + + if (ref) { + set_codetag_empty(ref); + put_page_tag_ref(ref); + } + } + __free_pages_core(page, order); } From patchwork Thu Mar 21 16:36:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599192 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 199F2C54E58 for ; Thu, 21 Mar 2024 16:39:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1A1A56B00CA; Thu, 21 Mar 2024 12:38:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 151826B00CB; Thu, 21 Mar 2024 12:38:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC0696B00CC; Thu, 21 Mar 2024 12:38:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D090C6B00CA for ; Thu, 21 Mar 2024 12:38:28 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 87EF7A02E6 for ; Thu, 21 Mar 2024 16:38:28 +0000 (UTC) X-FDA: 81921604296.16.44D330F Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf16.hostedemail.com (Postfix) with ESMTP id BAB8E180010 for ; Thu, 21 Mar 2024 16:38:26 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=04alsgHu; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 3gWL8ZQYKCG4egdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3gWL8ZQYKCG4egdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039106; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=f5LdsTMnIqabj8ROFrnDTgTazgUa7GmHleuFcbzHgo8=; b=dIfvJJ0bdu4/xqn0qv3N8cFBJCI59nRo0cBphSd+RossL3Gx45M2SKnLhjqRTOLwxeMUar O1Miann05cnym+1GReE3lo0iAGggHwXoJYUqIx5voY7MmIbaaQbfCoNl644Up46uvwA11o xKCK1acfVVnH00PmcDXw0S+NQOFFOsU= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=04alsgHu; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 3gWL8ZQYKCG4egdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3gWL8ZQYKCG4egdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039106; a=rsa-sha256; cv=none; b=3NqrOujoT82igKou8krX9sJpMr+Of0/a8Em2+Ts190p9i4K1kLVRtt9vwP6MJA3l4O2Ymp YAOweYP7ifcqP4aREn6tMDWoD+BoItaBpdtFDIgCD2/WEr1rKlRYopMJ4v7RQv9wqKgusO TSwCbNPzWmP8pNzSb2zo9oz8JNZvjdw= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60ab69a9e6fso28869637b3.0 for ; Thu, 21 Mar 2024 09:38:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039106; x=1711643906; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=f5LdsTMnIqabj8ROFrnDTgTazgUa7GmHleuFcbzHgo8=; b=04alsgHus8OMwoo9TYwCUt1YdVfV3QMZmGhym5SPRPfX8AZJ0xVQ3pOaO9LCzj/uY+ wREMrjqT7V4B4VsO4hPzMNV1Y+ljCia4w/VJzfKGWBOsNzThsUuffR0YRIUO5y1WRkq8 jsCUL1WCk1Fae3YTD68PKgsdjHeBE49rNcvC6W4/o9HgbetVgrrtMWFZRZd2m28ubZYX /fs/PvnYM6paOU5g6e2b2hyjArRI8rLLfSSEq6c2Xm2CUjwdS7OA6XZaY7+iMfnYSeAq FvlEbCk9JxI2lILhU77GMHLib4tuz5eCoHOrPl7vvMPKhaFLsPyWUNOG1EqoGRIzZJua AW/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039106; x=1711643906; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=f5LdsTMnIqabj8ROFrnDTgTazgUa7GmHleuFcbzHgo8=; b=QTMhtSMBXIseZnYq99Rj84kU72vjXVvE/aWsPzAQyMcq6hjFDx6X92f+3utwYz9nCt g13k8xsKUIIruuzALPwWStV3qZ9kVK75nKXk7ib1luUAMNs174pTWcZGKWZkRKvqjE5N 68wnmrSDKzTRMn931SLh83f6KI+9SMg/9JmDbQbUxi8zZMa2+DXNHo9B4HRp8h1PuaFb JqzYT5yzwBc+oya+9DuXAmcAgDJxvRKA78hyFIMjg0E4n6GyAXn3MN0k5OzTC/ffScBj 9TjKe39AWHrZ+CnaaAWSYvQeGnryJgucLTW4A3dwVvFCUTh/MyUswh3eosDbF5gjTWK8 7mBw== X-Forwarded-Encrypted: i=1; AJvYcCW36/vCKrpRVjELA07IboT0VWBOPbvJELxnRgThCuSDcO0JQ8qN6i8u5r0uZ86cVEstRYAQuXPN/yZO8lY+lOTA2E0= X-Gm-Message-State: AOJu0Yy2K7vntUSf7x1+cPRNCMYa6xAKVFEfzbyU5YoqM6DRBix+DYJn zkn4fXs97dy1gfnPxAHJMBINaKCCryDEPbF3tJDdkCcset1RuuQBg4gxK+B7sO+5TLBk5bxWSU3 qoA== X-Google-Smtp-Source: AGHT+IHcdcDq5+VHUFf2WWlMxzMO+//nklguhi2MOjraFbT9V29sxxC0Or3cfCWp5X/LcbCc/1WTryTy5Uo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a0d:cc41:0:b0:610:dc1b:8e57 with SMTP id o62-20020a0dcc41000000b00610dc1b8e57mr863711ywd.3.1711039105858; Thu, 21 Mar 2024 09:38:25 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:57 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-36-surenb@google.com> Subject: [PATCH v6 35/37] codetag: debug: introduce OBJEXTS_ALLOC_FAIL to mark failed slab_ext allocations From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Stat-Signature: 5zzkdf4zkdowc36d9ski57sx9eyeq6yk X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: BAB8E180010 X-HE-Tag: 1711039106-573316 X-HE-Meta: U2FsdGVkX1+ThYT44ScrEYG5H2x6O7MMOspegY/sJh7A++w6fGZaZBHrI9MetYYgdCz2/qc4//sc3juMAEKOQynfL0JMzEYsw1nDlQpST8eblsIxdRs968at8tOUdRNnJNVpw+5wTAMLDjy+62fgQh1d+ON9OR/ISbplk1Q9rdUJ2UmOJ4cyuci34kJ37NaQOBXer9f8FfIqsfkSwvV4MRLqGatWgadU/oB3fqp9g4vWL/2rFP7tkNq0PRf8V5Wz9ydN/UAF2ZDROqoMhXRFfaRcgxHsxtuq+siJHstGzLgS3BF4J5vIjipkmNDpAXrWDRAU5Ujn08pepomkgQa+1uRFZLN9dp+KifGoViSBDjNWefqrBszgNTiyDjOHQgGA7nebtvQhGof36+HvsUoHbgHiRsyrubT6a26Cu0exYCVt9dUJAHwyq6EYrXZEUYNakNU4YDLzlBsu9YQ/NLZ/N2RkM3zPmajwK49njwCXnGLLri21FDP2O5QnyD4ScQjtN99Nf28Gp9QK1TQ60TgHsvqlc96Dpg6tZbzVr0EGWeldYV9YLSVyI0/5SR3c9lYz7HZ5U/kfj2DfYRsXntZfEuueWtDngMYzESeqoONM9JEee571v9iB6Rtj2OrFndegCfIxPk5awStfyvNfkn1tg8AWFG9+XOikxEI+QiOhXUMRuu7hLQEDvAb2Ti7Hplj8Exv+E0+/G7DazI26YuB+JJIlCq7ab8rBFV+NMa3qAOEN5KeZrAhOxyHtxWBcZ3+uHIGloKma6TvwS24eXJVym9wRHPT1MEecIGEZuGBONsTPtIAH+h2TJI+CeVFO62tgj5i7wWXpVdsNpiBTqdvqaLnbJV6GwVYBvBc4rQNKikd5hrdWjHDOaAt8bA7S0ey+YrcvzMAmFxudfU72QxFeH1ltuQARW7gmq6f9602JgVPKdp/mPXg3W5UT9B6RLJ5bIoxtscpE/C0baSOw2Xn O6RIOeHN REz/zTf48ZvnYgA9kopbBairnqWD6HFZ892Na55tL3myErnAnp/65SawYp4lVhdTtoCDtxkWGsuPNeth6gfjLnINf5Ht7fqgyQWj04K2CO6R7Zs4Y6/y+6xiaLa8+TlwlUtQAxDmOV3ctR+50uioTClAA/eyfvmLOf33cYbYRhl77UnQxskbJJIgwwBLkZ7Xz9WJnw/a3UmdhjL2iU+HtXbd6jB5MR0V7mRzCAZO1n5Grs6yNQZjGZ2JDdosEIx8PzjMHe41YZd1uS2j2YbUAbqNQU15LWbgwR5uYJPMsRjOv+vm+k1Y9ZTe8kW8lCFn+zqBwBMiGfMaHtsvbWZutB67niWqAZV5tzbw/yVWtQHQ1AhS4CPLE9ITEwUDIbVbRX3EqfVG86f/3xpqBL3CCqBoWONmLgoxjuDQhIHYAuR7DBZzlLTwSzcZxkQ5RYUAQuj0cN4e+dSfh1zrX1hvTbuN1BS4D8JgJBht8Q6f8ZfVqE72pQHqhWjYSUJKVtcXWONRxoxPSjUI9RnLVoSdcZyGwlZGO7TTb6F11LFiEUMKpItroX9Q87Or72X9vZVTDBNE95GWnc0uJRupqrC4yAonpTAsf+ctS08lkgr/pdJ1sf7X4V9EJQ5oWj8uqkbL7CdlLYrv9z2ewOykIU88ZSXMqQs9xxl4HZxPxCffqo6rooq6PHAlVo/fAIWrjN7wd8mTIARBHUZI11ui3HLaJ2ugFUFmr+rNTIjNJ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If slabobj_ext vector allocation for a slab object fails and later on it succeeds for another object in the same slab, the slabobj_ext for the original object will be NULL and will be flagged in case when CONFIG_MEM_ALLOC_PROFILING_DEBUG is enabled. Mark failed slabobj_ext vector allocations using a new objext_flags flag stored in the lower bits of slab->obj_exts. When new allocation succeeds it marks all tag references in the same slabobj_ext vector as empty to avoid warnings implemented by CONFIG_MEM_ALLOC_PROFILING_DEBUG checks. Signed-off-by: Suren Baghdasaryan --- include/linux/memcontrol.h | 4 +++- mm/slub.c | 46 ++++++++++++++++++++++++++++++++------ 2 files changed, 42 insertions(+), 8 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 24a6df30be49..8f332b4ae84c 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -366,8 +366,10 @@ enum page_memcg_data_flags { #endif /* CONFIG_MEMCG */ enum objext_flags { + /* slabobj_ext vector failed to allocate */ + OBJEXTS_ALLOC_FAIL = __FIRST_OBJEXT_FLAG, /* the next bit after the last actual flag */ - __NR_OBJEXTS_FLAGS = __FIRST_OBJEXT_FLAG, + __NR_OBJEXTS_FLAGS = (__FIRST_OBJEXT_FLAG << 1), }; #define OBJEXTS_FLAGS_MASK (__NR_OBJEXTS_FLAGS - 1) diff --git a/mm/slub.c b/mm/slub.c index de8171603269..7b68a3451eb9 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1891,9 +1891,33 @@ static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) } } +static inline void mark_failed_objexts_alloc(struct slab *slab) +{ + slab->obj_exts = OBJEXTS_ALLOC_FAIL; +} + +static inline void handle_failed_objexts_alloc(unsigned long obj_exts, + struct slabobj_ext *vec, unsigned int objects) +{ + /* + * If vector previously failed to allocate then we have live + * objects with no tag reference. Mark all references in this + * vector as empty to avoid warnings later on. + */ + if (obj_exts & OBJEXTS_ALLOC_FAIL) { + unsigned int i; + + for (i = 0; i < objects; i++) + set_codetag_empty(&vec[i].ref); + } +} + #else /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ static inline void mark_objexts_empty(struct slabobj_ext *obj_exts) {} +static inline void mark_failed_objexts_alloc(struct slab *slab) {} +static inline void handle_failed_objexts_alloc(unsigned long obj_exts, + struct slabobj_ext *vec, unsigned int objects) {} #endif /* CONFIG_MEM_ALLOC_PROFILING_DEBUG */ @@ -1909,29 +1933,37 @@ static int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, gfp_t gfp, bool new_slab) { unsigned int objects = objs_per_slab(s, slab); - unsigned long obj_exts; - void *vec; + unsigned long new_exts; + unsigned long old_exts; + struct slabobj_ext *vec; gfp &= ~OBJCGS_CLEAR_MASK; /* Prevent recursive extension vector allocation */ gfp |= __GFP_NO_OBJ_EXT; vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, slab_nid(slab)); - if (!vec) + if (!vec) { + /* Mark vectors which failed to allocate */ + if (new_slab) + mark_failed_objexts_alloc(slab); + return -ENOMEM; + } - obj_exts = (unsigned long)vec; + new_exts = (unsigned long)vec; #ifdef CONFIG_MEMCG - obj_exts |= MEMCG_DATA_OBJEXTS; + new_exts |= MEMCG_DATA_OBJEXTS; #endif + old_exts = slab->obj_exts; + handle_failed_objexts_alloc(old_exts, vec, objects); if (new_slab) { /* * If the slab is brand new and nobody can yet access its * obj_exts, no synchronization is required and obj_exts can * be simply assigned. */ - slab->obj_exts = obj_exts; - } else if (cmpxchg(&slab->obj_exts, 0, obj_exts)) { + slab->obj_exts = new_exts; + } else if (cmpxchg(&slab->obj_exts, old_exts, new_exts) != old_exts) { /* * If the slab is already in use, somebody can allocate and * assign slabobj_exts in parallel. In this case the existing From patchwork Thu Mar 21 16:36:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599194 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3314C54E58 for ; Thu, 21 Mar 2024 16:39:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 794506B00CD; Thu, 21 Mar 2024 12:38:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 71DCA6B00CE; Thu, 21 Mar 2024 12:38:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 59AD96B00CF; Thu, 21 Mar 2024 12:38:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 23BCA6B00CE for ; Thu, 21 Mar 2024 12:38:33 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 00EE91401BB for ; Thu, 21 Mar 2024 16:38:32 +0000 (UTC) X-FDA: 81921604506.03.1817699 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf06.hostedemail.com (Postfix) with ESMTP id 2704B18001E for ; Thu, 21 Mar 2024 16:38:29 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=xdbPKwZ1; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 3g2L8ZQYKCHAgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3g2L8ZQYKCHAgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039110; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AW33WX/ioh4EflqxU3ax5xPoUVjwxdtM7w8zUOaZaKk=; b=EzkppOrSb4yDKuHcjmpfByWjjD0hpHJ7u4CM6tJ5qoynBoBH2nRK7HLbby41/OnAh7eoi6 MOgGQK6wLyFMjAMrdJooy+20Nh7YPypwMDe952ad2K2nGtkzQnXVZOahqTi1wR+X//c/sA xF8kfuREzG42Vt48FcbAS8RbnrBysbU= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=xdbPKwZ1; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 3g2L8ZQYKCHAgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3g2L8ZQYKCHAgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039110; a=rsa-sha256; cv=none; b=ZZW04CTiX9/1qHefO1zai+/2soTyKbHWskXBMd/kC3D8JPtpG3sos9T148Tz4eUFSlEKR/ qhGYQ273N3qGMnx8nvq9Cs3wuVIkcBUgbsIIj01ifSXcg1kE6q9Cl5QoHeScwkDWIv5ZD0 xc/0dTwqOGOYpPLGzbIydvPIfQfcr/Y= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-610ed1c0f93so20673097b3.1 for ; Thu, 21 Mar 2024 09:38:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039109; x=1711643909; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AW33WX/ioh4EflqxU3ax5xPoUVjwxdtM7w8zUOaZaKk=; b=xdbPKwZ1zaJfkP+i3s8bCk1sAy7oGjUfhEfYRh0vCiqXU+ev7iZzceAJWw2q6QVLJz lLX3iaWMM2aSUGZTJuCdTPHEtWz8EmgBQJAwqbTEKFi8UJfgHzLOt35cRvhjLpJXV+l+ H7U9rCioo1k4rtz8Uzsq+DfrdzywQ35o8x4OXQoDoB+gpWt19qHxqgu1mXuxOe1QfamV rhosg65dDPWt/UfL+liaizPSl23+TFG81kopplsWGH35kjmrSC4mf+YfPzXKGkuKe6gb ug47zmk/TGIPDja/4q5lDesTJRXS9uqrM8/hfreIULxZWzuucE3jyWNGTwUTMDKsYxxE Wewg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039109; x=1711643909; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AW33WX/ioh4EflqxU3ax5xPoUVjwxdtM7w8zUOaZaKk=; b=cMWTLmEbhCW6gv22Vic5oXlcPpYzeXguab3y0pbcbcb4KbyPB6bDCwzKH1sT2M9EsZ h+og4kaIqqALO75qkt7xqMYP2xLs1Iudv8cvzC5QT6UcjX9g+bffkK04GwWOrG7vPrTH Vc509s1zdz/e4G0gw4Y5UaFcOUG3KA5Wr0Fqsk2tWZVHnIBUwgn8k8+QRt5cHBB+LK2c qu87hrTQZaBOAx6BvbcZUMRV2ZUWafAc70Mszy9arNiA4mAg7OGIpQMMe9Xhhlc+5fEP wjMvCypMkmDmYuX+jDKY62kMUQ0eNB2O9wwT7UZRC8zphe0ugVVYiS0o1UmKptav8z6A SKeg== X-Forwarded-Encrypted: i=1; AJvYcCWJI4AQvEG4p28oQdsB0Pa52QQFFZmE+H0Q4xkhJskgVfY4rELl0cAGovlfvHHPKx/py49F7DbzpC527/w4vVA3Xm0= X-Gm-Message-State: AOJu0YwbqL4rmfXCdcNr+7tbcietCpuaPEaDLb0mZE7kiJWDiegbtzE8 /jKg9Sa+Cdf1LN9QW5HNL35jvIkiqurxQ/NwQh1LKwKEG+cYg0lKcbnNatcsju2DU5FJ0K+Ee+d kdA== X-Google-Smtp-Source: AGHT+IF7IbSbPrud7S4q2CO6A5a//4lNXuqCjaSBMjwnNeD7U+hML/0+WVDue04lWzY5rl3DmStBxkMY2eM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:11cc:b0:dd9:2789:17fb with SMTP id n12-20020a05690211cc00b00dd9278917fbmr720825ybu.3.1711039107919; Thu, 21 Mar 2024 09:38:27 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:58 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-37-surenb@google.com> Subject: [PATCH v6 36/37] MAINTAINERS: Add entries for code tagging and memory allocation profiling From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 2704B18001E X-Stat-Signature: bcf733c3bi8x9bn4yctcqew3ebwscun6 X-HE-Tag: 1711039109-318147 X-HE-Meta: U2FsdGVkX1+AkNqtsJiU8Iod7CtSZWmM/AJuO2h0hAVlhe9TOkjgY1DtXvVWMKVan+KKkQerabIamkk+ANVqBLXLrBd+nBUyMZhFw0rp6mArL7kq32LfRyUTM+OzzMWYfas/f/gk/fju5Wu+xPXUM2PSvjdeTHuufuIuK396cyp9eDQNhNg0O/4eFeq8D1zyR2kN5S+wg+glxahwdGF9sOrBqWV6NpYFGEaggwmneKxSUB0KO9V6iuvZcCK48+oTAvLLI3BySh9YsV1r0wCC2i7wtRjk5h57mrRSdVbxqFmDcO3oW2oTvTeb47dY0NpOS2aJD0qZeYp4S5wFRVmESnK/wntT1tOF2q5bl+ngLulPFEDLAcukI6ohLD2xnB/kJqYWfzHxQATq0Czi2Eou25b5+eud+54LlIDTy8z1b5BPQMQnM6STfXYJaEo+82i/b3wDWV93JivzmEYVABxtpOQ2RtKe5U9jt49Lob0bwsrtWrGaRvX/F9dhV02cneXNPJhD97f/n9Td5jKjcRp50aqSnDMY+UqcsadjQuwJ4AShPuiCPXijLclWGvsEjYlMfWec+3XLB16a/XO+6AUtuS0kqpByjFu7BDV/EevY6fQ7NXcttiJFryAojGna5Ro/qJ/yKR87xLO024jVnVAOQMDpPLqDAcIq3yp03q/tHF912WghLbXsRl58E7gZPM5SPKZosLHTIOfqtphG2peZZKgd5YjOki4VevQG8ryE/MI8IrRnpPS4ddh1yVO3L4POAgaqYrN0ibTE+mbZfP/jUKrSY1Pa/8O2b4xoYueAEeD3bUSDjBF6jy73ot/UPletu8Y5usXjXq5LHUg9Vau0tcUKdI5cCUI+tGgEiS2DhG1SLWPi1uZhiuVJk6gum9T6+hkt8kMXWQDQlZ3G1+Szs8YES2caxuN+7hceczixkisnSD1WkqgTTj0NLBXtk4mJm9+FAFtT5qwdSrPv1iv TA9i6GUx J9LQl7tqZLr++8RDWf75YyBofi/PHAFw8MYknxzkGDNdfHOKf9l0Icb7dyYilO7cwLOjhsbfXKMMxwtrYDezfkJScUFSM+aYFO779KpyKl/xuePincxn4Ksay3/fEw99O6jsaln/4lkhJV3vucaADDF7+zGnALkUooaml4cK6O/gTRINFv8ugQZjPB8q/hvagiraRRskzu/O2LhpaYjx3J9IZe6TWMJQiy14mcdirleOrEnq6S9KEGmE8AX0ZjRe4sMEOfFYstPi6ZZ1hIC0N52lNGMsHMnY82QINtuTVe/QMTLC6ljPmDi26Spn1crw44CsKvgopzmT+ZsQug4owHPjFHFii19iGqxC2tM3JEgOU5QBUqMqw4Ysa2Gta5kHUky6L+Mu99Pq1r278x5xDyhCevM2MWIjbBTGf37kiU9bB0DuUN445TeuxVhDgkgkxy9ylGqDqGGnbgvMPhRdLx3qkE0qPoAosBK6167BQ78aWt30Ht2fiAPCBwOXQeVnIhXZkB9VlFHgXNSJi+sWMu/kg+nKu9FkK/8U6NNAwxibgfvmUfFHxZUiZWWgMcD2VKnz0i9lfK8Q5MmlxOS9h5+BPON6CZXTzR0MOfcQB/4T89Zg8zk1Nw7A6osYZpsMF+l8yPyQ24GZsQPYOBxvveTAWCHXIc8Chhrf8Zmdib7r7nTv1iKraD5vmv+wBAPW0HJef7g/Ar8MVC72+tHcCd15Ipj6s5Wu0KeEJPeyoQgNpgohOA3Vx4IjGdGtosBB+0sIi+/qt9D4raqD8BSR+qFciNPIYADIUacPT+OfltlRQq5rxsD8Lmsmwl4e6G7vtTvZB+RjunVIxsCdLgTrPU2xMErzZCvuReM9QQXfJ6h85qjD6ZIRjXFSFXTiuVNXX7LG4TzxQMyoAi6IrcoCD++AsUqVKr2t3Eu3yk60dVgPfbXAoXhlL/rWKDSDLYYSvkxjKzSofnPWYH6aNECgij07jEmHo mAyKqlEW Ob82FMJ4Zeme8CrIBB3TZHJRJX3aTb+ZVaLwKbSvf97ibO2ojtC3Jg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet The new code & libraries added are being maintained - mark them as such. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan Reviewed-by: Kees Cook --- MAINTAINERS | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index d5f99cc986d1..84c1505bc62a 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -5241,6 +5241,13 @@ S: Supported F: Documentation/process/code-of-conduct-interpretation.rst F: Documentation/process/code-of-conduct.rst +CODE TAGGING +M: Suren Baghdasaryan +M: Kent Overstreet +S: Maintained +F: include/linux/codetag.h +F: lib/codetag.c + COMEDI DRIVERS M: Ian Abbott M: H Hartley Sweeten @@ -14123,6 +14130,16 @@ F: mm/memblock.c F: mm/mm_init.c F: tools/testing/memblock/ +MEMORY ALLOCATION PROFILING +M: Suren Baghdasaryan +M: Kent Overstreet +L: linux-mm@kvack.org +S: Maintained +F: include/linux/alloc_tag.h +F: include/linux/codetag_ctx.h +F: lib/alloc_tag.c +F: lib/pgalloc_tag.c + MEMORY CONTROLLER DRIVERS M: Krzysztof Kozlowski L: linux-kernel@vger.kernel.org From patchwork Thu Mar 21 16:36:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599193 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 141F9C6FD1F for ; Thu, 21 Mar 2024 16:39:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 28D066B00CB; Thu, 21 Mar 2024 12:38:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 216EE6B00CD; Thu, 21 Mar 2024 12:38:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F0D9D6B00CE; Thu, 21 Mar 2024 12:38:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id DACED6B00CB for ; Thu, 21 Mar 2024 12:38:32 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id AEB82C0A2A for ; Thu, 21 Mar 2024 16:38:32 +0000 (UTC) X-FDA: 81921604464.03.81FC081 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf19.hostedemail.com (Postfix) with ESMTP id 188FF1A000C for ; Thu, 21 Mar 2024 16:38:30 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=vuYeiDfv; spf=pass (imf19.hostedemail.com: domain of 3hmL8ZQYKCHMjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3hmL8ZQYKCHMjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039111; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zMtv/wiplx2DxF9seTFx3IcU+R2+PFSDAr6m0+A1ntI=; b=65D0MRstYkcL+gb1YwILciLOtpEQFhVOCqYdeqDVKZH1NEp13v5Ftz3kIfOZH7c7jTRxPO IzsI0ogxT49FO0bzV3ZPCB5x2e76Io7lqRmPZ3b8JJC2m0Cmfn2J2JUI1g/BR15CnfDsrQ +itG5LJDQs5nfBbSgkc8xtemZ0GIr4s= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039111; a=rsa-sha256; cv=none; b=lFDpbBm84cE1xvoiNfmRQ34K9SchiA9EV8xGOgLL3wgutV50kaDsMPxyxGoYppB6JDxcSi GDkTNwelWPphUQdwKCqsOcW7KAN7wtxHTNHXOyCx7Y9kudW6FqrHnvhsepUbU5XNVLUy72 b2DFsgdkABbMQ6AjIzTfRp+Kn9Zinyk= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=vuYeiDfv; spf=pass (imf19.hostedemail.com: domain of 3hmL8ZQYKCHMjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3hmL8ZQYKCHMjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dbf618042daso1830976276.0 for ; Thu, 21 Mar 2024 09:38:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039110; x=1711643910; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zMtv/wiplx2DxF9seTFx3IcU+R2+PFSDAr6m0+A1ntI=; b=vuYeiDfvy3pS+uqkDshpifo1OCZ2vJZNIsg0rCKL0+43nZoK+gAVoOlu73KoF6Mosv E/+14pJR9p+JzEygW5VJg4fHNfUJjhEIpdDg62EUyWJYmOKcb/BiixsVwfZGOC2d7nWF NEmJAEBi4gjpjNjVhdlsPQzNbfyIVKGck37+uprxY3BPu/YCxFpGrT2Afc9e0Pvpk8/2 YMfQ1dlGQgYy2E92BRc5VW5NdaCgyszMt4Sla5PbIBq+8+XUBJZgHBeb2hr8/eCnW8DO +q26xkMnxpLxGFbkqiuiVfjOK0IUjL5m8MLihtuOhdOaKeUrjK9VFcbK3y62NmOpaHca /2RQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039110; x=1711643910; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zMtv/wiplx2DxF9seTFx3IcU+R2+PFSDAr6m0+A1ntI=; b=PQsZkPOWwX3NaCKat332a3I8YGtWh3NzveS013Gc5ukbpW3UHjUhcIZs4XpJR3EuHE F1hn+VUH3y9NO3rwpizs9F3etwNJa9ZZq9SjhIvc7DHy6v2IjCR5CWQaQgc0+yWJWpWS KiBmuMF9SYdBgNWRp4nk1dVK6YqelEf1CMhd8w2BKDLKvNF7IVDvEvYaDTPLQnmi2+KD 7qm1L/6TkVyFJkaZnJg339R+xXfrIvTvIn3i9s+1jfqnr9FSWPMIyJ7urAnDNWLV9CdD BMujJxf7ZSEz9ImzvyL0qLNt53svR8D5JmmkjzV3jkX2akqtk5XjdSZToc1m8FvkFWSK l2Vg== X-Forwarded-Encrypted: i=1; AJvYcCXUtsrNksvJ641MLS3XpztcmcTGbx0V2em0Doyg8Qwh3tosIMFNhR8uEbvxgGE/uFeB0QIXzl9vKSHH7v67vccFM8I= X-Gm-Message-State: AOJu0YxNNge7MNAnFoBGBfrtK6YfykkO54XknbpW8KJLLIpDy4V5yGH1 //h2Kwkuksy3glwxG7UpZ7N156lMLqTZjGcjb1y0IcYJ+aoIxn2GcvqTGVEEbKxvJ6Jm6PLp3bs P0A== X-Google-Smtp-Source: AGHT+IHubMkzttSDdMzbBJ1mYbEtkJMz1sywK2vk2BxZuZG66cHkzs6E2Eyy/Ht6uVFkEMl2Xl2C/1q/RI8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:240e:b0:dc2:5273:53f9 with SMTP id dr14-20020a056902240e00b00dc2527353f9mr1211362ybb.1.1711039110014; Thu, 21 Mar 2024 09:38:30 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:59 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-38-surenb@google.com> Subject: [PATCH v6 37/37] memprofiling: Documentation From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 188FF1A000C X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: ck6ts8iih93h9tm6uqomp1pf58wrpdfp X-HE-Tag: 1711039110-949802 X-HE-Meta: U2FsdGVkX1+UTf0lxmgTADEk/mdV0ayjLWCl7E8L+ZvMarO0D9XPx3wF5Je0cnVf9Kfj5qcd+mc4/eYAnIaPnjzvwrgRRTL3owL1u6lisMlOoVlDdr/6U8/T0/hvXP6dwjRf0SqV3hJlIZimn1RVfU99NGjF/wJOHZHycUPtbUy4JSPEh5SAvMXk7j4OW5ByP/FzDBnvjEYlx5j7r7XPdypaj0jXHYQyE6bjnc9dhMWCOBsgq/ukyKRmWM71t29k3WCRxUEk7jzuqhK9ybAsX6Kq+IlK+jZ2+L4e/c2xmY7Vzmntpu3svBj4jZpe5swXKUa7HZ0aA9hudz/TXqSKnDNzzQhRRjayOCgnYfw3MlWVqPE9dX69Bu+2mTZgALr9maIid4dCwxj0I3RQfDSYzlZj+91YRir0wA7jdNe5vosVqMimvYZ5b9SGb1ndHSoKU8Q0nla9nh8jaOuk5mF9yYw4t4kYxcQjepMU1n6Y6HuTOszDbR5ga0R40LdV+CTmid743bLbB/3icyqixZLQCZKNlVqJdmzeeaR+twjpP20PFBjM5W7F59Jp/KKw0Ud0XfVCEv7e6jl9tJ+FwkWfFD0WtrYhDYlk/E32I6dmX+vyUj+Xq7x/e6G/0HyzdoUgu0Nm2aUid+aiwxwDXO7p287JwhwclL5Z0vTGMd6ofo2zuSyZzRzlDUUU+X9aUYAcFs+A9Jjboe15hnzRMmrvkypGfOW8U7AZUI4bozCjwFYECweKqRBY5nVvAOS/csuH4GkUPb4ukfsZ6LmrriZd/aYFnhTrMt4NNmFGjFkYfRVhr5i6ssSU0kxR8pA6IaDjmGoqEe0E94XH5xsw6LWgHTA1p3jBsAyl9yGCa6UqlqreEtueaNVgYElzFPNND9CUs14+JlUciMTE6BDiYg1x/y+ipr2lvMY00W1H2hEsOMOSSlOp3deL8J0EuuhXKmE69upM6gsPM4NuOg1Jfgo qOV3FMqT cjzAvvfOr9obdZrlz+uhcfNWX+T9Bf9eG3z4zU1gnx3FpFfTevBWB5PXK+5UrJfcHxzPQVMUbqMbL67FOl4WlciAFAE2J+CNIuK9Ii+/xp1Nng5Wl2CVTyqCo2yopiFRnzyKPlkFLAGWpLuoWzI3C8q7kYqpXbZ3fsBRG7jvRPeFSCbEAnp9odLsQZwR9XsYwMe+CQEZadVNlEPnDrXlw5cZ1kQBfGzjIHxthkLvJ46zBcsGZuPZVCkqHDMy2FVp7WAMGHqL2xATfXt92tcONlhvYeFCbj1q8u6gigjd1HlyWMucsKaGysJ/6GXG7BGfDgo8hXOBPUFk2g59me7t8TK1isxizYQtouN5RnQSENZB7QpEjiiXWLm3xXwNA/rgFzn7PMFYQiFNC6tRSKu6fXLhZy6R/lta4qWEw X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet Provide documentation for memory allocation profiling. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- Documentation/mm/allocation-profiling.rst | 100 ++++++++++++++++++++++ Documentation/mm/index.rst | 1 + 2 files changed, 101 insertions(+) create mode 100644 Documentation/mm/allocation-profiling.rst diff --git a/Documentation/mm/allocation-profiling.rst b/Documentation/mm/allocation-profiling.rst new file mode 100644 index 000000000000..d3b733b41ae6 --- /dev/null +++ b/Documentation/mm/allocation-profiling.rst @@ -0,0 +1,100 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=========================== +MEMORY ALLOCATION PROFILING +=========================== + +Low overhead (suitable for production) accounting of all memory allocations, +tracked by file and line number. + +Usage: +kconfig options: +- CONFIG_MEM_ALLOC_PROFILING + +- CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT + +- CONFIG_MEM_ALLOC_PROFILING_DEBUG + adds warnings for allocations that weren't accounted because of a + missing annotation + +Boot parameter: + sysctl.vm.mem_profiling=0|1|never + + When set to "never", memory allocation profiling overhead is minimized and it + cannot be enabled at runtime (sysctl becomes read-only). + When CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y, default value is "1". + When CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n, default value is "never". + +sysctl: + /proc/sys/vm/mem_profiling + +Runtime info: + /proc/allocinfo + +Example output:: + + root@moria-kvm:~# sort -g /proc/allocinfo|tail|numfmt --to=iec + 2.8M 22648 fs/kernfs/dir.c:615 func:__kernfs_new_node + 3.8M 953 mm/memory.c:4214 func:alloc_anon_folio + 4.0M 1010 drivers/staging/ctagmod/ctagmod.c:20 [ctagmod] func:ctagmod_start + 4.1M 4 net/netfilter/nf_conntrack_core.c:2567 func:nf_ct_alloc_hashtable + 6.0M 1532 mm/filemap.c:1919 func:__filemap_get_folio + 8.8M 2785 kernel/fork.c:307 func:alloc_thread_stack_node + 13M 234 block/blk-mq.c:3421 func:blk_mq_alloc_rqs + 14M 3520 mm/mm_init.c:2530 func:alloc_large_system_hash + 15M 3656 mm/readahead.c:247 func:page_cache_ra_unbounded + 55M 4887 mm/slub.c:2259 func:alloc_slab_page + 122M 31168 mm/page_ext.c:270 func:alloc_page_ext + +=================== +Theory of operation +=================== + +Memory allocation profiling builds off of code tagging, which is a library for +declaring static structs (that typically describe a file and line number in +some way, hence code tagging) and then finding and operating on them at runtime, +- i.e. iterating over them to print them in debugfs/procfs. + +To add accounting for an allocation call, we replace it with a macro +invocation, alloc_hooks(), that +- declares a code tag +- stashes a pointer to it in task_struct +- calls the real allocation function +- and finally, restores the task_struct alloc tag pointer to its previous value. + +This allows for alloc_hooks() calls to be nested, with the most recent one +taking effect. This is important for allocations internal to the mm/ code that +do not properly belong to the outer allocation context and should be counted +separately: for example, slab object extension vectors, or when the slab +allocates pages from the page allocator. + +Thus, proper usage requires determining which function in an allocation call +stack should be tagged. There are many helper functions that essentially wrap +e.g. kmalloc() and do a little more work, then are called in multiple places; +we'll generally want the accounting to happen in the callers of these helpers, +not in the helpers themselves. + +To fix up a given helper, for example foo(), do the following: +- switch its allocation call to the _noprof() version, e.g. kmalloc_noprof() + +- rename it to foo_noprof() + +- define a macro version of foo() like so: + + #define foo(...) alloc_hooks(foo_noprof(__VA_ARGS__)) + +It's also possible to stash a pointer to an alloc tag in your own data structures. + +Do this when you're implementing a generic data structure that does allocations +"on behalf of" some other code - for example, the rhashtable code. This way, +instead of seeing a large line in /proc/allocinfo for rhashtable.c, we can +break it out by rhashtable type. + +To do so: +- Hook your data structure's init function, like any other allocation function. + +- Within your init function, use the convenience macro alloc_tag_record() to + record alloc tag in your data structure. + +- Then, use the following form for your allocations: + alloc_hooks_tag(ht->your_saved_tag, kmalloc_noprof(...)) diff --git a/Documentation/mm/index.rst b/Documentation/mm/index.rst index 31d2ac306438..48b9b559ca7b 100644 --- a/Documentation/mm/index.rst +++ b/Documentation/mm/index.rst @@ -26,6 +26,7 @@ see the :doc:`admin guide <../admin-guide/mm/index>`. page_cache shmfs oom + allocation-profiling Legacy Documentation ====================