From patchwork Thu Apr 10 21:05:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 14047212 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21674C369A6 for ; Thu, 10 Apr 2025 21:05:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3E0B828013A; Thu, 10 Apr 2025 17:05:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 38E61280138; Thu, 10 Apr 2025 17:05:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 27DC828013A; Thu, 10 Apr 2025 17:05:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0C316280138 for ; Thu, 10 Apr 2025 17:05:51 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B65A41CD198 for ; Thu, 10 Apr 2025 21:05:51 +0000 (UTC) X-FDA: 83319366102.02.3DFE951 Received: from out-185.mta0.migadu.com (out-185.mta0.migadu.com [91.218.175.185]) by imf12.hostedemail.com (Postfix) with ESMTP id E237F40004 for ; Thu, 10 Apr 2025 21:05:49 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=QwXVB+97; spf=pass (imf12.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.185 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744319150; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=UySO3eXUO6+JFC4kGqzkj2HoZuMo1TTGk0FfxlKFpMM=; b=mNAk+OOZFnTbNNqAfZEHc6qSoMSpDJD1dZGxJwlNBsZox7uWDNsrzjsMuaSV179ynZkTtG i26+VQmQWFdlPnAwlpezcgFs6GVUAqXAzss+0VGhrKvZHsUL6T+x1+U11SpTIawyPP9Lhh b9CMKeQpCdOqNKRbO0VgvpsR4nulme4= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=QwXVB+97; spf=pass (imf12.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.185 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744319150; a=rsa-sha256; cv=none; b=pxnd31grWDxYHemIU4bCyvFRnfiBu2FtncdqEAKYvMVxz0pcPTqDCmlf4Y8VWLCqIJYRhe AAwM7ofem7LWjFFJc7Cx5wPUG0JvR/p6z2zO7sDR/VUJA6baJbRVswDCsMciAcY+AC+pzA UtPjQU12wo5LRJMK474Gey5Sq54fRYE= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1744319146; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=UySO3eXUO6+JFC4kGqzkj2HoZuMo1TTGk0FfxlKFpMM=; b=QwXVB+9720cVeGPvm4RweqZqY9uk03RwZuPPIbJi7yo8oYE6rDz+86RbN2cTFoK3jKl8cL 6/QRY1wMQgGsSOZKJb2zfioVFhO73LVDHn3DtWkCGkjnv3VwKsBwIHRkshP7oLRRIQ8Ez7 4dcvuHhxmVe/p1M8dZSSootgyomMGcc= From: Shakeel Butt To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Yosry Ahmed , Waiman Long , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH] memcg: no refill for offlined objcg Date: Thu, 10 Apr 2025 14:05:35 -0700 Message-ID: <20250410210535.1005312-1-shakeel.butt@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: E237F40004 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: o8d9zzkjz3w54qqocyxqndzu9yxx8gt5 X-HE-Tag: 1744319149-248180 X-HE-Meta: U2FsdGVkX1+RL53x2PKBvMRD96NvrRvor/7ScgC4ii9XRQM4sgIyGQnbRYr8gWAHxmQv/p/0sFqKxPnsJ275mLF2nIUhLiCwL7aHdwSqv6FwqcpOhg9WpnOHflc2NepEPmbt3KpKDFMUf2blsoKj+wE8mItCbB4ka5sjwY1xHtJ8m1iTqIgPVGHLecCnD8JftdnC9IXPyiC1QV/VJYe5XAri6eyMnBmUz+5xct2PRdYGSUOi6NngtiwS9nWo9YQz+dqjLQeQcXGezo92Vel6caRbSJMG63MRFXhD/pxZEpq3MO3khPwI8XDZiPESXKF0roA+1H1h8KXB7SYY4nTSjyGo2xk/5A/ZYYiGnCISUuc70iSWLdFJ6awDOpQ6sEYaHm6LjVWWncq4Ka92gPFkakJ7RVh4KssRdmkz/x4mQv0lDkCjLXkAPHs2QtDEAz74SR6hT+E8eWb1VxNvQwOL/s3zuwrzevmnQ2PHaX5WGfvBagzNiavlJKRSUAcGeTArEjuzSx8WwAKgwYNz5czn+KXX/EPWtsHpimbgbS3R2zjMvB3O20KjHdviaaIhdMIZpijIrRT992ePvVfoQsCfPlAjYNzF2V1tQsifp/NDlUblWShBQ1h0HiZdCnyb6wXnyAh0T/H7DtAFcGMmIsPu0hZ4ZeqWK8IwxLomx8FdF0QXVeBFmlEqg7AnOru1HmPJeEh/hTylT8fXpCBl0OodChXN7JneKLAceQjg6TzLTRyLvJxF6zxCHkfBHbwr08mf4z70dr12ZxL/+969lR6gphAP879k9+Ff/TpKgrYgXWbsMLqs2cAuKFVFJ5r8014DKVCRhEMVHwB/eM+M/YGplSYbHIpD24lprojpAfux/O3dt7BNH8zAmUuyL5BOn8vDiNgdMJniiTSC6iusxdRG+BeSQ1R8evu6bakq3NPZ09NshW+I2cxeX7jQmqK+iR6JDAsx6fOJWG+LsJ3SU31 uSu3Cwzr znuLdBxeq1wHfGjFL2PidlWlyYHpiTp2oHesaeVEqqnM7NajGK/Gtx4XUs0dzC1yDzB+m+3BMVai5NwdJVJ3BNwFej4njg1DRjErwUniFl0fdu2tJfH1f0ZUViYuQ1/HpGyF53qcyRYlK/5UO3eS7Y2jR24U05He97qN9HHUPu/YRyS8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In our fleet, we are observing refill_obj_stock() spending a lot of cpu in obj_cgroup_get() and on further inspection it seems like the given objcg is offlined and the kernel has to take the slow path i.e. atomic operations for objcg reference counting. Other than expensive atomic operations, refilling stock of an offlined objcg is a waster as there will not be new allocations for the offlined objcg. In addition, refilling triggers flush of the previous objcg which might be used in future. So, let's just avoid refilling the stock with the offlined objcg. Signed-off-by: Shakeel Butt --- mm/memcontrol.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 2178a051bd09..23c62ae6a8c6 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2474,6 +2474,17 @@ static inline void __mod_objcg_mlstate(struct obj_cgroup *objcg, rcu_read_unlock(); } +static inline void mod_objcg_mlstate(struct obj_cgroup *objcg, + struct pglist_data *pgdat, + enum node_stat_item idx, int nr) +{ + unsigned long flags; + + local_irq_save(flags); + __mod_objcg_mlstate(objcg, pgdat, idx, nr); + local_irq_restore(flags); +} + static __always_inline struct mem_cgroup *mem_cgroup_from_obj_folio(struct folio *folio, void *p) { @@ -2925,6 +2936,13 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, unsigned long flags; unsigned int nr_pages = 0; + if (unlikely(percpu_ref_is_dying(&objcg->refcnt))) { + atomic_add(nr_bytes, &objcg->nr_charged_bytes); + if (pgdat) + mod_objcg_mlstate(objcg, pgdat, idx, nr_acct); + return; + } + local_lock_irqsave(&memcg_stock.stock_lock, flags); stock = this_cpu_ptr(&memcg_stock);