From patchwork Tue Dec 10 02:39:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13900697 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC68CE7717D for ; Tue, 10 Dec 2024 02:40:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 39B9B6B00E0; Mon, 9 Dec 2024 21:40:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 325BD6B00E2; Mon, 9 Dec 2024 21:40:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1024A6B00E3; Mon, 9 Dec 2024 21:40:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id DBD466B00E0 for ; Mon, 9 Dec 2024 21:40:02 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 68D5B80210 for ; Tue, 10 Dec 2024 02:40:02 +0000 (UTC) X-FDA: 82877493804.25.56D5A48 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf02.hostedemail.com (Postfix) with ESMTP id C999780008 for ; Tue, 10 Dec 2024 02:39:19 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Wu0mFc+6; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733798380; a=rsa-sha256; cv=none; b=mAI6PyowayzJ/2uOwRblKX65tcdB9A7+i2vp/6kLyXZ+3ZMxiAxmcik7WAzC88QG4ATMeC D2HhXAF+tRGjbskfEz0PHFH7dchlUJRpbfSthwLtYiF22TJthOhas8lyOZQsvOCNPYploA 0pVEwlt1jUPlA2FD+vriTwNuYAixCl4= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Wu0mFc+6; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733798380; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wo7sUpYgRQrXUd48YB6NzK0xhu80+vJLM/DolMWSp9w=; b=5DOOk4VWhzyCMXZFi3WF7iFw1GjUzHCkr8FSw6a+5zK/tegwgGNxjFDsZeTLOIDYtA9Ctg 9fqYbuqz9Ug4Ptal0HFjwV6+z0XfMKFUNN0iACkLEDut/3xmWF50GdcfQLbhnYYsGxiMGi FPT2qWR0NAUSdnQ08n++PfqCS8dX2ZI= Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-7fc41b4c78bso2970861a12.3 for ; Mon, 09 Dec 2024 18:40:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733798399; x=1734403199; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wo7sUpYgRQrXUd48YB6NzK0xhu80+vJLM/DolMWSp9w=; b=Wu0mFc+6WmDd6L0TNTDdMxvcNFoaaOXkb9e+Hrr9jbwuHknKI0Y2bHdtomJIZ85X40 8wu2iC8J+XXFobd1UdPKoY/PmU2Hau6pJ+9K2hdLx6hzitnghEp/D1tRZ0xNyC4/08RR iaxfk1MssqI1xPwZJqlJGaKCQwiD34EFSXhypvNCnbjcIY/wsAP5TbV7892oMev6wxK0 4eLLQ2UrIY/oirGhjN1UaRS2O3mp5IE1hn7IzwNeTEJldnwYRKbV7BsWR1/LhdonMCzA a5e/yW3mwQwqI/XPQ5gwTKuqeiJ97CrdPJrd32PKAWhOf7VgmlfNlIJ1wvjbZ82wWpf3 ZjtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733798399; x=1734403199; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wo7sUpYgRQrXUd48YB6NzK0xhu80+vJLM/DolMWSp9w=; b=k68bm5OflAcJN2cQzj/HTSGiKCy14EOlC4njItm0TanGvJwd58lhLqE92hIA2Z4kwa dmshDIcKhPTUOLMYiEPkY15yCvJJheYfuzUS/FLSGbgPAIFaX4RkDQlguvahPDkxXMba YuXLjojZaYtJdOPS9N+7FP4L4owFu0j2N0lWN2q4VAtU1ZWTDD5VH6u7SDWVk31Uf84O LRPBx6SZStWejEYnYN/UyL/dKXdnYs5jnHNkNpWHVa6Nc+p3UmHMRzZ0qpCpSVgP/ruU bJh3ZA0GdMLuUYH5L2RL4mOXiEgLy5A0mgxSPuoCpRzJdZKSXQCbpVszq+mgl9BHQHQR pWzA== X-Forwarded-Encrypted: i=1; AJvYcCUNjAI2DNkOmn9YDnV0FLUDF9NqlhqA7f/R4jHdNDN1GFHyLZmrOP8VHdA/yfzrylumrkWjsJWOFw==@kvack.org X-Gm-Message-State: AOJu0YxuW/X2kIGmDQApq18jqod1hxt5LGABINOOldcbY6qjAtffB0F7 6hBXExBH+w28ZObDrkCR22FdPKXSyjZKMxs876lT11Nmljg/mDuC X-Gm-Gg: ASbGnctRiF2yTY1/84udLHdER68i5+TRfGpbW34HC1xfNp53bApzDN7f4HHJ6pqTXpl kXGReR5mv+jntd0fwK5UB+uS4fAn8Xvoy7XjZniRJmHLxIPRsu8oONBqS+LaZ/FiMzJsYTrt1Em mGWqTfgKyQc+KWDzNeGV23LWg2K8mJ0yq75yza0WdoiKphpvzpUXpYBnuqgzYM1XrzJ3Llofr9E QYv7QHYAb20S+dEO/dpkRwuSjJJxoXmkgVE316SEoJ8Rc7/IzGw62Ay4p4aVOxQ36tfN+kfE4ns CbSf9w== X-Google-Smtp-Source: AGHT+IFtiAsGhIVFD64KUeuc/HqOnxB2YRGY24DKyXhrRpv8mSEuSX8pUEmuRY3vKJ8avlYCY8VlLQ== X-Received: by 2002:a05:6a21:890d:b0:1e1:9fef:e96a with SMTP id adf61e73a8af0-1e1b1a79c40mr3694721637.6.1733798399161; Mon, 09 Dec 2024 18:39:59 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:83b0]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7fd560985e0sm1471732a12.79.2024.12.09.18.39.57 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 09 Dec 2024 18:39:58 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v2 4/6] memcg: Add __GFP_TRYLOCK support. Date: Mon, 9 Dec 2024 18:39:34 -0800 Message-Id: <20241210023936.46871-5-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20241210023936.46871-1-alexei.starovoitov@gmail.com> References: <20241210023936.46871-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Stat-Signature: b5f77unk4onkmu3ksemjiew3kmm4xd9f X-Rspam-User: X-Rspamd-Queue-Id: C999780008 X-Rspamd-Server: rspam08 X-HE-Tag: 1733798359-13189 X-HE-Meta: U2FsdGVkX19oXhuJuKqV9YNaW0LzGFI8Ru07+5NoFIbl8AYMikAZb9jE623PkZLJ1e/CcxnCtWoVH/LnXV0+0bSuwG+nn2ElSNMmLaPiUzEvl/M/yFY9Pu69fq51g4hC9DbFt6EHExkqbzyDVio8aSD7Kdc4vZXTvAJEeM9prnQg+LFb3vsbr0c/H9+ojNRz3Ur46C7xwurnoxSiLyS+RoBtmeptM5poNiihCgDSEWbItg5l2GylLSarF1R3P3amN8g1j3JI3Ya+Fn9IzC11HkFHjMTRWhC4uRwHkMTIC7pjgrDHWhPHnjMVRyLTShdeamgvZvUIRnxEcu9AvCKTCFprhFb92ghN0CRZPp3c8EwS/ObLJ/Y/TSNzeEKAflGPO5WLog/C7PQnOxAsL7Y8A5pqiT0lddm16Wo0akIDeForsyYoIfC8MkihLzCI/1eObPxYvxdzlUoWW3H6Pab4IEh+721Uhip1ROBnfAb/ZuwCAcrJROypvDtRvSBQxlmuHCa4txjFThnOm38UV8eZlBBY998KouE6H+dnPVYYoolRMA2/wqIuGe+Lw+cLo91bWVVhQ2zrzn3QvBoM844lB8PpBjPVeR4dpHnYM8rHTdu3ZDwQmFzpf4wn0XFfHqCLfQeIomhwJXL5aeoPhpvp/7czpz9smkruS5Va/PGJrg7fT/7dfj9X2WkI6VAmPNSN81CLj/953AzYrjh3Fw68yyuZ8VbN05Y42GQiPuKnaqVYPAKd/P2qQSjZ7JuA+Q1IY7BO1xhU6NYci2TO2m6HnvXsIOBET2vhOgOT1p9mE/qUIdYR0x8BZMOu/GDjH3vCxEMrlKjL2axeM1rNzz1TL+k4Gz6P5t3HPjoy5/tEMvweOtRNQbOLFlFkceS0kd8mxtK764RxVWFfwmrdjmtKxTP7soxhbG4iSO2jRTOhVCh72ekWIXW2HWcoB10DB+kBVF1f6aeTeioqZ2e0VOg C+TJoj92 6xsbLIzihkfKg2qTRDMchkzWtVxaQ1L5aU+nMZ8mwe1YwjuHaEvX/5jgSPAIYMdWjj9cvnIvcctKtlk7/HjV8m1d2HjEhZylbm3DOy+mbr8ZKWmXs+sARbRFFsxEQ9hgK1LmqZW5mkK44xdzCf2dBdGeoGVFal4zE5MhhctsV0m2uPIr5zQ668uOJaLgDmS/+/52yr/u6IkB/LTi36Ls8JJbqV2HNOMMKmve+wCNQfeS+zLmCZ5yE13yrG3ZH+X9EjIJeL/CAaMeestPkAwqlBeqtcSk91jgtwmo/juqwU1ulubfBHivQwIX2xkrAUjwU/yEYf4Arlhn3z/E3UEC+IQsIpg2qjBkE1UThdjmq10xfQoMMcXC110L3X+q7fKXBLIEdiiFfYDwoQHmoMuSzzPrd/PveU5PXd8OPo6KFk5qYiCn+ahKIUOTbOcpQm+0D0Nn3soaaP/ajleWEPDxh8cFNDEJgZHNR1pKa1+olC5+xMcJ8TAAYOSPfcA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.390122, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Teach memcg to operate under __GFP_TRYLOCK conditions when spinning locks cannot be used. The end result is __memcg_kmem_charge_page() and __memcg_kmem_uncharge_page() become lockless. Signed-off-by: Alexei Starovoitov --- mm/memcontrol.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7b3503d12aaf..459f35f15819 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1756,7 +1756,8 @@ static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, * * returns true if successful, false otherwise. */ -static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) +static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages, + gfp_t gfp_mask) { struct memcg_stock_pcp *stock; unsigned int stock_pages; @@ -1766,7 +1767,11 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) if (nr_pages > MEMCG_CHARGE_BATCH) return ret; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + if (gfp_mask & __GFP_TRYLOCK) + return ret; + local_lock_irqsave(&memcg_stock.stock_lock, flags); + } stock = this_cpu_ptr(&memcg_stock); stock_pages = READ_ONCE(stock->nr_pages); @@ -1851,7 +1856,15 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) { unsigned long flags; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + /* + * In !RT local_trylock_irqsave() always succeeds. + * In case of unlikely failure to lock percpu stock_lock in RT + * uncharge memcg directly. + */ + mem_cgroup_cancel_charge(memcg, nr_pages); + return; + } __refill_stock(memcg, nr_pages); local_unlock_irqrestore(&memcg_stock.stock_lock, flags); } @@ -2196,7 +2209,7 @@ int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, unsigned long pflags; retry: - if (consume_stock(memcg, nr_pages)) + if (consume_stock(memcg, nr_pages, gfp_mask)) return 0; if (!do_memsw_account() ||