From patchwork Mon Feb 12 21:39:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13554065 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E1DFC4829E for ; Mon, 12 Feb 2024 21:40:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2BD566B00AF; Mon, 12 Feb 2024 16:40:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 244E66B00B0; Mon, 12 Feb 2024 16:40:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0716D6B00B2; Mon, 12 Feb 2024 16:40:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E5DF86B00AF for ; Mon, 12 Feb 2024 16:40:33 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9A9C1802C7 for ; Mon, 12 Feb 2024 21:40:33 +0000 (UTC) X-FDA: 81784471146.03.5A3C702 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf28.hostedemail.com (Postfix) with ESMTP id B980DC000F for ; Mon, 12 Feb 2024 21:40:31 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Rux7IPi6; spf=pass (imf28.hostedemail.com: domain of 3TpDKZQYKCM0BDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3TpDKZQYKCM0BDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707774031; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=X07z1RjJe29sLMh5N/vKSXU8uipRKYcH64glAbq+0r0=; b=5xtnMUdlgkzotomJvX17HBFK3kPN7tAP0cPHTqTplWc6p9Pg/y293PDZMnGYER4+nESuct ehrjAmiq13dPm4wnKyLenWQHZ0x7iHHka/tRGNk6VSm580kNBGijxThb0AVRSMO79a5IhN tuyB2DVRH/6TnFJfUb5LDXY3xxIyYvo= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Rux7IPi6; spf=pass (imf28.hostedemail.com: domain of 3TpDKZQYKCM0BDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3TpDKZQYKCM0BDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707774031; a=rsa-sha256; cv=none; b=GLZdTaCj4jtxiu+YBTf0EX9EwiCvzOV8h3txrKYQcIeS3/O+U7yNV9QRH1h/7r+X3j5YvX w4ksc9vPhp+LcoHZ+5k44gePdy80517ZYrKsow/CgddvNUM4zQZmUxzZWSSiwCIxAdIhAU cc08eNL/0DRFu4OLKYC/p6IDb4REy/E= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc743cc50a6so306078276.2 for ; Mon, 12 Feb 2024 13:40:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707774031; x=1708378831; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=X07z1RjJe29sLMh5N/vKSXU8uipRKYcH64glAbq+0r0=; b=Rux7IPi6jVnPBccGrYhBpFdq4sdl7YDFkCHwoKWMUKC0FyP1Up5MsxAR+dSa98L55x PjLYRU5OUrF7IlBt57A9n0SrvrOvKzy1CBl7sTdofzjGA1lBp8MyAa32CAG4djsUgAGw ObNEdXopAZCyiadTW7Ko2PyFLY0zzOdfkq4/laMVkc3T2cYJzpY8T9WZq0E89IAUegn9 MdwpGbiv3qbOSj7zQvmJBz4d2a+fijNIOr0vpbB6YODNkPltY9G/MP2zBLbuvEhRbWoc p09FlS37aWFcLE4oaE1xnGtjm+WW6j4Ine4K4gQhL0IL89ss4AKrnWYFwPklWpZrwyQq 2pdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707774031; x=1708378831; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=X07z1RjJe29sLMh5N/vKSXU8uipRKYcH64glAbq+0r0=; b=CvwLtKcfo7Tnwld+LtPw11YAi15WknCsaTj8uUTYGmTGfSD37FurpAzcP2igzQgtf+ xzJOeEFwt8g/TpqJxDczynqixniikWVaCiqApd9TXG0sJjW8CC+Bqz2g76CNHmK9udVg Inr7UE2oe2Q8a4ZPvl+oCngdOTKrRbKge9tUEqQfWp+kCPn18WVEVRYlm74Mf+blEmpK c3x0VRo3dnIcn5ok1uGqvQw0b1yDT/sVnm8DQYuH7bmfmsNttdO1oCVWG/0GTY6nxlhA Re+0PvI1sc1wxn5d/eDtg+M4F6ZZTTaUFVzgVGe3GQiHQHiw2bkqe9fIS9Pm/G5/XWlr skcg== X-Forwarded-Encrypted: i=1; AJvYcCV7YeaCufxo6HdD17uLdJIfHzPr9O9EOmF+E5mtlW+4LfwzxJiyJVO6/KxO7eY0HXvg7itX326vhnAOKGYCga2M9fU= X-Gm-Message-State: AOJu0Yw6vvPsNOVKJwh3TQIx54tTd5buR4FjwOucn+OOC7ucA8JXha2A JtTqU4x0aatRsk9Y+lDCXNSE+qOmcWICj2xaxx9AtjU9byf9w2Triltsq94RKYZnbIxaEFZCoer SJQ== X-Google-Smtp-Source: AGHT+IE+9v5iTSehfGXvFLsyFgeYeTOkKmnm1/9Fi71HMokK15QIurDQiuu9x1J7ZwyZ5mBIdOmgWIrIEcg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b848:2b3f:be49:9cbc]) (user=surenb job=sendgmr) by 2002:a05:6902:709:b0:dc7:59d9:7b46 with SMTP id k9-20020a056902070900b00dc759d97b46mr291099ybt.3.1707774030642; Mon, 12 Feb 2024 13:40:30 -0800 (PST) Date: Mon, 12 Feb 2024 13:39:13 -0800 In-Reply-To: <20240212213922.783301-1-surenb@google.com> Mime-Version: 1.0 References: <20240212213922.783301-1-surenb@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240212213922.783301-28-surenb@google.com> Subject: [PATCH v3 27/35] mm: percpu: Add codetag reference into pcpuobj_ext From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: B980DC000F X-Rspam-User: X-Stat-Signature: fgj815setmzf9hte7rtjuobnosmupery X-Rspamd-Server: rspam01 X-HE-Tag: 1707774031-921104 X-HE-Meta: U2FsdGVkX1+07mIXc8V5ZK76ZNgbF7QnThDlo8d1AW02K33XbcfwNy2eTOkcEq6QPcNBZtH+ufHvtgUDPjG9biHJa1P+2Iloow9OfkdL4g3zFkhPNh5EaUyNErgNJcuqBkJrDBWB7lqK5sGO5URxBm8Z711ydIJ0hK/sK8IcGYkbIotz33awxDBfJdZYmvjlu5NUAaPJaKiImjN88bdPUWPod5OH+Z0efFpsj00tduEsQlkpE33o4sxczLcbjVqaDuLeQHNfyV9ixvBfYYVrCiqfgnWZAPV7ac0hnveJzXL6LOMWnzgtAoiLd1oRbO+WXnhVEoyAfkJibl8/p5fUzhmZlpd5y6W0EvMamVtBidSPutSJh189tjKZjOYql69NhOkkPdk8ptCaRJN4iQrJlFojdrPekn8Jyka4GlJI0eBvAeT46DMq4ZD7pdKsuGnLlfldgCp0DNv1n/ydModikxxAFx0QFCIeEDEH27zZZisMFXwYGMj1Q3vBuIYSnmD2JfupI6EjzUEWMWiA8RNTmmheM6/NvVpkMEXb4cM8vxMfwlSWSRLp9dn7xcG0UwNlAKAVhVD6+mGTBS3JJuAWRSnd58DsHoWU1KueKty/V6K/3zmmOAzGvYl5lWM7SHfQyBUmkIXtNtUjrrQriEGhYS0NxWruO2wmYpj3Z98clhsFahuoWEbNm4YHvVd/llAFuObLWgTNSPU0ADQBY5oi5pcnSAoqX2gvVUYizjGv9MMQeBXz7usVxEhKEcmF347sOXEIC4GXNK8HMaYZ7NdR/iIBs3aMCMdgJ0wOGPfmypbZvB4CNtfV3KlSmvThqrk6EWvuqw+Y8qlilxhzNLpYYe8IxN9wTFNIX2AThssss/j6RpD/qz3sX9iFQeTknav3gavhLBpXr9PI7VOmrZrQhvcrqgawGxFKiQyNx2ydCETJQ7sGgRpPNPoHEQ1hyKIdgkg4KmH21o/f6ARMpcx 0QRSPRcc rxNZ1BERLHruwrfXsmdAGo+lQGrhJTonXeHf19nKiYYnqXOrRoTa7PQ4PKnwJWnOdInHfVZyW8VCNcdgDGpbJn0+2PconVwey2J5VyAQS/aqnf10Xg3/OgRz4OjxTlIY9r+KQf3Tp6sOzeWn9zDTR1e+Cm5ovRYwB+cMFmw9BLi6orFXqlIAWYzgJU7EC37yaPQNQFAItxIlzGDFnhr9tU+P2Q1oTUT7rduk7OXscKf2Tgv595oZHpVaGuUhWutulW1Cr+vUgbpAM4bot1keGfTb4f2++SclpAIFJVzMa25pBfGAz2Oh9WamL7yM7aR7My0oyqmZZwMmHVz5FuiGE23GxTDS8TczkHc6umilSn9MrEjY7jvZ9P3Rz1J6b4Dc6JH2vxdkZvch2XtigEbWl9UXbfuSg4z85M2uLnZcHEHJf5H90co99iz6uDDD2s4gB42+LfE5rViTInYWOYXPECG83k0ZBJ7C5ALiAAJe4lumd5f3cj5fcCf1NS5kLIXImdxzlughfJMRHs9G/rLoieNnqg8KAAukZkoMNbVXywBF5HATKJWqZAXtP0sfIxSDKHCR3bcb9jVgrYDdrL74VyiJ8+rGA4gImRruxcr2UekduR/Rs+A07LqsUTPp/k4qOYRwtk/u9RNrIWG4FRqYPNDGbjNqgWiFxUM+6obBCnHjxoPAlM2giFQN6BtNr18P9V0AVnx1qDy4VkKq70WZu/L1WCw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kent Overstreet To store codetag for every per-cpu allocation, a codetag reference is embedded into pcpuobj_ext when CONFIG_MEM_ALLOC_PROFILING=y. Hooks to use the newly introduced codetag are added. Signed-off-by: Kent Overstreet Signed-off-by: Suren Baghdasaryan --- mm/percpu-internal.h | 11 +++++++++-- mm/percpu.c | 26 ++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 2 deletions(-) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index e62d582f4bf3..7e42f0ca3b7b 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -36,9 +36,12 @@ struct pcpuobj_ext { #ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *cgroup; #endif +#ifdef CONFIG_MEM_ALLOC_PROFILING + union codetag_ref tag; +#endif }; -#ifdef CONFIG_MEMCG_KMEM +#if defined(CONFIG_MEMCG_KMEM) || defined(CONFIG_MEM_ALLOC_PROFILING) #define NEED_PCPUOBJ_EXT #endif @@ -86,7 +89,11 @@ struct pcpu_chunk { static inline bool need_pcpuobj_ext(void) { - return !mem_cgroup_kmem_disabled(); + if (IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING)) + return true; + if (!mem_cgroup_kmem_disabled()) + return true; + return false; } extern spinlock_t pcpu_lock; diff --git a/mm/percpu.c b/mm/percpu.c index 2e5edaad9cc3..578531ea1f43 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1699,6 +1699,32 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *chunk, int off, size_t size) } #endif /* CONFIG_MEMCG_KMEM */ +#ifdef CONFIG_MEM_ALLOC_PROFILING +static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, + size_t size) +{ + if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) { + alloc_tag_add(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, + current->alloc_tag, size); + } +} + +static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t size) +{ + if (mem_alloc_profiling_enabled() && likely(chunk->obj_exts)) + alloc_tag_sub_noalloc(&chunk->obj_exts[off >> PCPU_MIN_ALLOC_SHIFT].tag, size); +} +#else +static void pcpu_alloc_tag_alloc_hook(struct pcpu_chunk *chunk, int off, + size_t size) +{ +} + +static void pcpu_alloc_tag_free_hook(struct pcpu_chunk *chunk, int off, size_t size) +{ +} +#endif + /** * pcpu_alloc - the percpu allocator * @size: size of area to allocate in bytes