From patchwork Wed Dec 18 03:07:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13912977 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C8E0E77187 for ; Wed, 18 Dec 2024 03:07:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 345996B0093; Tue, 17 Dec 2024 22:07:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F5976B0095; Tue, 17 Dec 2024 22:07:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16F546B0096; Tue, 17 Dec 2024 22:07:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id EBB6F6B0093 for ; Tue, 17 Dec 2024 22:07:47 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6C4DD42E24 for ; Wed, 18 Dec 2024 03:07:47 +0000 (UTC) X-FDA: 82906593126.09.8A1BA7E Received: from mail-ot1-f45.google.com (mail-ot1-f45.google.com [209.85.210.45]) by imf17.hostedemail.com (Postfix) with ESMTP id C5D0740004 for ; Wed, 18 Dec 2024 03:07:21 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=lUHKGoGY; spf=pass (imf17.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.45 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734491236; a=rsa-sha256; cv=none; b=txTLCBa/lsOAgA0mrjL+iBNQ0rBO00qBuMYCD37f9g3vS5Qt9RvcUMT2wXOXnhr4D9X4XX db+CtL9pyTC0tB4mY+C6TmTNEB/EeJ4C13lYMcbrFD0FV/lbTc6XOS5INCpbCsquZ6RgWS nhNg0tMnyP1LO9PIJsjld/AG/KwAO9w= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=lUHKGoGY; spf=pass (imf17.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.45 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734491236; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/7W5s6nR1eKMiZGmAkj0lhGxTA+TFwBoEtTYrZWxIj4=; b=8ajmAgA4mEMuZVoBuDVEuDKIGGHqbhRgLIlVog0WIAjTkuNfpJkpVEjpq0/wmaxNDQmZbI BKnpyqxc2neQYNBXKsrADwEFJ1a0j4al9Cfi0dclPOy2BWn56UzduupCPHV+gs0dPtRl83 sLsY+cwE2S6h/entoWqzmVq67wIybIc= Received: by mail-ot1-f45.google.com with SMTP id 46e09a7af769-71e2764aa46so3714448a34.2 for ; Tue, 17 Dec 2024 19:07:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734491264; x=1735096064; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/7W5s6nR1eKMiZGmAkj0lhGxTA+TFwBoEtTYrZWxIj4=; b=lUHKGoGYphXDqiOfXdoPh5vg46kqg5eUrkKZ5wl9mX/D8ZYM2lPkuQuQgqpF2goR67 69WTxx4Nv//TFBmXuFzTKehfiiAqgmCwOdQTQl9hiVx/qWKjKoCEopjxI1KyuHvsfMrl Zi/25m9P6cjB9dcsagTtga9d+adHPH9zzgyqchWMdXFtAkP6Mvs+eZZ0DrN1YFXVUqlO LrnbqNPs/WI5hkmmOyJ/39yCYTGDH5IYqZf/+IPooCLq7yTcT+Z3HjSY/ulajQ0TG3b/ G8efpG8Qgg4dh5mYsMgP5LLPflsdXdahDbo8I6up3ed5L2MLRk1lkpmiBVyjl6c5uHRW xevw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734491264; x=1735096064; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/7W5s6nR1eKMiZGmAkj0lhGxTA+TFwBoEtTYrZWxIj4=; b=sQnSgI6L97ldBz7ML/2v1OEzQUu72d1e5pjnL/XBqQbQlbnkTSlKXRhpT0HhrVs97r f0LGwHHsSTwqR6XsM7gljKUpUWtkFLWoiZhcS1ZgHuqUsRXSaLplKrCg0R+xSe9QC6ro hmEWl26EBcVnr/XoWn6Nznt5kWXstR2CjwAXyYJI+paoEeF0xuOyi/JDwcHMwUefqXG3 11shij+9hRbH100X9S3ZBuitF5J8KJTLLZ0ajFRTCE1XcpxKutzvgnYEJDfUOgeRvn0o BXPxbJQZED1F7acawqp4o7VoZ5qW4u5XwpKtfQp7ozz7/yQ04eKEXC+ar2U1uO9d5zsO uaTA== X-Forwarded-Encrypted: i=1; AJvYcCUv2HASiZK3Ip9D8bZO/nVgpUL96if0bCTmOsw9WMLMWudHDbbfausjME8KSQzq691u5caJ6q/yVw==@kvack.org X-Gm-Message-State: AOJu0Yx9rYci6/63h1IYzCb+iDUzZPsauBTT555jL3NZ9lvAuHhg5BAu nMaRW7FuLLfi+hLYxThyALZNhOyovNe8h4iDWX9nmpI+M5i778mh X-Gm-Gg: ASbGncvOe53ijRnfF6km3h7mquOH26O+IM7g5m4jQpVUJjInQQ0eoPlR10wsi1nvJj3 POOK4+BX/xNzTp0F0s2uGG1aWTMMOP+MScLa+SJ4zn8qWHsSrJZYC5OCSxym7MZpL8JzlXbCEgO JiszNcdbG1jsKqbCuLJgveSQ7EM+oeHzOeujlV/DMDJVOGl2NrDIczFjGctACjZX6LBGB63U1oa o2H+CrVEwta4+9VwXtBuckCMu7JxQ9FVOhkwzEcSdgy/a7vrDYfM08RuPnWUw== X-Google-Smtp-Source: AGHT+IH2tfKmT4WhT188FuyTuGvwz4iinrO9XQia8eG+MuqgHEUEQ+qNAXFpX68m9XBO2n0bhRDyxw== X-Received: by 2002:a05:6830:7308:b0:71e:904:6aed with SMTP id 46e09a7af769-71fb757a1afmr792576a34.10.1734491264502; Tue, 17 Dec 2024 19:07:44 -0800 (PST) Received: from localhost ([2a03:2880:12ff:2::]) by smtp.gmail.com with ESMTPSA id 46e09a7af769-71e4835630esm2481271a34.27.2024.12.17.19.07.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Dec 2024 19:07:43 -0800 (PST) From: alexei.starovoitov@gmail.com To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v3 4/6] memcg: Use trylock to access memcg stock_lock. Date: Tue, 17 Dec 2024 19:07:17 -0800 Message-ID: <20241218030720.1602449-5-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241218030720.1602449-1-alexei.starovoitov@gmail.com> References: <20241218030720.1602449-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: C5D0740004 X-Stat-Signature: og6kb5p4umn1m9z66stxddbgwihf9ccb X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1734491241-931576 X-HE-Meta: U2FsdGVkX1+FG3boiuMhi0yhiNTScS566hRHXOdx6MAqGIAUykQe+P/HUz6mTXyZFXBCQha/sufvfZM0xBFTiu0Jf8fu11IjilRuWiBQjqCtbOgdFBDVDaReAo0ibbDTzicBHBKO4PdCgEhxO0/mvn1LKwRQLNW3vGLxPfcB9wqFcisyMiQKjjbE2j69qVskwHYcO29Qp257zrHqZjJfdTSrkXT8uJS3NMD0vJ9v1DY4aFQqyGD306wSbNbhGn8RiGSoNdTb+LSTyQdvMEYXPu65Dt8lfuoD26IzBhZ08PDVsLKNURdM+gFnmht6ydWs2Ug9OA/ADQ6wcCKEjmXdkHhzHMuD3SLi+wEvPAWZ12VCGKBMLjfrRisVVEPfRTqU/gRTqUrmDPRpXu739lwOrgpT3UCIeYXrZ2xK46rwiJmZvQEewu3Is4bz5yf45GRJLy0t/fZStMgmm6JlNzk8p2i6K3staYm/gycrUGmrDI8/4xMzfNeBSvi6Tcssc5Ty1kzljQM+oBz3A2jNTxrmNxqgx8VqwoNJqk6kuPntJXNdqHnBaxB5Gxp2Rs57bf0XiIZppJ4HPFEMy0Y++lNdGO7V/gcpzk3ths5zwuEmHjXOLs+aSNCbVCDAR0aV148of5QTMPpoiN7WLd0BzXdyMz6qG1RY7h8lEC0EncCImIA6UDeruC6uClMm5JP+FnMuDqxON0luuZM0/KU8eam+2EN+peHnG3V4PH9kLHlUKBrvtLrptmXV1kzGpPFdDj/s5zCfm6q47OqebtSirXzrm1xOAT7CqGF5EtEdS+aWkar37n7x8ihlORT1ZN4T4SeUyE5NxaU4pKcdp7wjg/J56Qr5DFDTOC0TXh0p3BFfeGgbHwG4XMO2+pp+/t+ABwC9jogysLsxhttG31S8fEG2SjkGJAL54jOmQX6yXiHF3K2dIpQ2dwVJulVoA/pFKZRnI7lRXye//e+QwRg7Fgg 4PklYmBg VDTEKUdxj+ZS8qDc8KCHx5AVSYI1RjE4g/bNuwndvxVmmjMg19emr/Ac3+WFy6hocidDzRPqa+aZyqTFx3lTF0v/J2khNCrshsF5wUKg6Nhys0+b5wWsB5LeJzqA6eLTkaxAdAYC6dGYMbOgZdegPPVKsrmUZmuSzowqW7+UvT6k5Wy99CPIDS4s1LPdd34mS6yYZ6xnO0URdS5JATMr3NN+QaHBriozrlkom7Qvq3qNUOGTk8m44mdwY0Uz1M3VXIEKmkJnTIXP37zI357az4n/IgUwB539BD83KqeRQlmpkfx7Xu7JWct74DsCs2J/9QZU4A8BjMFiBtjKdp/igz15U8tOn/q2FC4szuHRD9+ecoebbJ9bIdSCBkevrTjeIwgNjZYmujsVdBLw1it9ZxmFJ6F7BtZx8KhPEHQhYfF/TRd1E0VhzMwc2oez+FFtzzwowsS6A0x2094hJvMo0s6yPsQS94RlNIhQGAB862b0WvCet7xVuOOjcpg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.010465, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Teach memcg to operate under trylock conditions when spinning locks cannot be used. The end result is __memcg_kmem_charge_page() and __memcg_kmem_uncharge_page() are safe to use from any context in RT and !RT. In !RT the NMI handler may fail to trylock stock_lock. In RT hard IRQ and NMI handlers will not attempt to trylock. Signed-off-by: Alexei Starovoitov --- mm/memcontrol.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7b3503d12aaf..f168d223375f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1756,7 +1756,8 @@ static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, * * returns true if successful, false otherwise. */ -static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) +static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages, + gfp_t gfp_mask) { struct memcg_stock_pcp *stock; unsigned int stock_pages; @@ -1766,7 +1767,11 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) if (nr_pages > MEMCG_CHARGE_BATCH) return ret; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + if (gfp_mask & __GFP_TRYLOCK) + return ret; + local_lock_irqsave(&memcg_stock.stock_lock, flags); + } stock = this_cpu_ptr(&memcg_stock); stock_pages = READ_ONCE(stock->nr_pages); @@ -1851,7 +1856,14 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) { unsigned long flags; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + /* + * In case of unlikely failure to lock percpu stock_lock + * uncharge memcg directly. + */ + mem_cgroup_cancel_charge(memcg, nr_pages); + return; + } __refill_stock(memcg, nr_pages); local_unlock_irqrestore(&memcg_stock.stock_lock, flags); } @@ -2196,7 +2208,7 @@ int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, unsigned long pflags; retry: - if (consume_stock(memcg, nr_pages)) + if (consume_stock(memcg, nr_pages, gfp_mask)) return 0; if (!do_memsw_account() ||