From patchwork Sun Jun 19 15:50:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12886753 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17F2FCCA47A for ; Sun, 19 Jun 2022 15:50:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7F1926B0093; Sun, 19 Jun 2022 11:50:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7A2BA8D0006; Sun, 19 Jun 2022 11:50:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 61ABE6B0096; Sun, 19 Jun 2022 11:50:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 4C2716B0093 for ; Sun, 19 Jun 2022 11:50:46 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 237E612076A for ; Sun, 19 Jun 2022 15:50:46 +0000 (UTC) X-FDA: 79595423292.30.E329EAB Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf29.hostedemail.com (Postfix) with ESMTP id B9E66120099 for ; Sun, 19 Jun 2022 15:50:45 +0000 (UTC) Received: by mail-pj1-f42.google.com with SMTP id go6so3354567pjb.0 for ; Sun, 19 Jun 2022 08:50:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pbJcR0JCgnnBinC8hqBQ8PqrT65xvYVhQt3SCbh4L30=; b=gL00DEY/pyx6xquZdsMWkhU03MRL9mGCjCzvdQLvuCKP+h0F+pUJpIDb/Fok05wmTd 2+CfJ8htUDSpBIWHUdOUWZDWShYErmlvKY3bZtamer8ejyjCB/hXwyjExzv6synKSlRs 8IQp9bwHIZhN1pxPrkMCIbL9fzBCKegJ3aAxkT1SuGFDU6Upkuza0BdySmGk/VIQC3nz RiYOA7kjCPEnstqkceUyef3q7WKD5xMbvP/9EPQ5yRxnxpt13eziLgEEHUKqzjxM8QwR HANv+Klf9VzXVYgF6saPC9cZFvotG47okLI1KN63Koldz7qt8xAz+8tuWfWejJNZl5YA 12JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pbJcR0JCgnnBinC8hqBQ8PqrT65xvYVhQt3SCbh4L30=; b=lFCZf3tf7RC/ixXdyj6knJrXUVcukHPSA2V8I9BgM43QtBpCtTM5pHsb3sMXueHQOh KYwgGMYitDnJEPxk5kqPbws8OPNtSS2zzFRUlV3duAOnKKp2Z5auWtUFwz+UJEngJPGG MWcfWCAOLL2NLw4tALqGzY42GMDX45YJBdYnq2rX1ju2llOxlulUzq0rhvNHv+iaEcFn V8466AYOjxNmsk9wd2x8Zthltq5dAQemh81Xlpz+w83McQH3EmgM9hLuOSqqRYCxoHsA 3fchRQwPv8Zh1Crf9N+dwqm22sYqawqAqiK+AovsdBcndimI64JVun7pj1ZGqeudo7fk CVlA== X-Gm-Message-State: AJIora9zWsZmiYYen8T5EKG6DllFpTVXjO05pJ7ZqbtBjCIsCcXDYzbH X6jGVnPbeNnJcidCcF7v4E0= X-Google-Smtp-Source: AGRyM1sijBY2x0OKk+fpSwQADGBG+54XL0sh/Ig4fH5KnHiRM32aoKLh3IZrhd4oeJ52rbEU+RGMGw== X-Received: by 2002:a17:90a:740e:b0:1e2:a631:87e6 with SMTP id a14-20020a17090a740e00b001e2a63187e6mr21950473pjg.115.1655653844671; Sun, 19 Jun 2022 08:50:44 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:2b24:5400:4ff:fe09:b144]) by smtp.gmail.com with ESMTPSA id z10-20020a1709027e8a00b001690a7df347sm6381761pla.96.2022.06.19.08.50.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jun 2022 08:50:43 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, quentin@isovalent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 01/10] mm, memcg: Add a new helper memcg_should_recharge() Date: Sun, 19 Jun 2022 15:50:23 +0000 Message-Id: <20220619155032.32515-2-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220619155032.32515-1-laoar.shao@gmail.com> References: <20220619155032.32515-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="gL00DEY/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655653845; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pbJcR0JCgnnBinC8hqBQ8PqrT65xvYVhQt3SCbh4L30=; b=UiXjpi0MeZi+/m2Dcs3zUvFGPdWaed4KM1Gye3IhSIuMDa+sKqbUYPSc/dpvvND9pos98i 82UpAEJEx1EmHGV3uVo/m7kSP5UkWl0O9UEGaYTZ2pQ18ufPnZkMMOYzidYu8YOLAZwPIa tfOQJX13Sl4+ggmGtRreGKH96MDYugg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655653845; a=rsa-sha256; cv=none; b=UCbd1IkCHFyHgJk/vgOsXPds5cUPjxUhhOcV0rPaFVfaphN47Ah5v6mzeX9EUzbarXkLPk UHnmtQ0NWBjcoTxbnOO/FX23Vujon2d0Pfhok1qcUfM2WwxPvh/2w9idfpg3cTI1uTFnHk 6xzqYXH8dtfx/9MtoJHdWsqFavEpeSc= X-Stat-Signature: kprpeo8jx41mb4t7jg7kyq8kd33nppfq X-Rspamd-Queue-Id: B9E66120099 X-Rspam-User: Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="gL00DEY/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com X-Rspamd-Server: rspam10 X-HE-Tag: 1655653845-484493 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently we only recharge pages from an offline memcg or reparented memcg. In the future we may explicitly introduce a need_recharge state for the memcg which should be recharged. Signed-off-by: Yafang Shao --- include/linux/memcontrol.h | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 9ecead1042b9..cf074156c6ac 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1755,6 +1755,18 @@ static inline void count_objcg_event(struct obj_cgroup *objcg, rcu_read_unlock(); } +static inline bool memcg_need_recharge(struct mem_cgroup *memcg) +{ + if (!memcg || memcg == root_mem_cgroup) + return false; + /* + * Currently we only recharge pages from an offline memcg, + * in the future we may explicitly introduce a need_recharge + * state for the memcg which should be recharged. + */ + return memcg->kmemcg_id == memcg->id.id ? false : true; +} + #else static inline bool mem_cgroup_kmem_disabled(void) { @@ -1806,6 +1818,11 @@ static inline void count_objcg_event(struct obj_cgroup *objcg, { } +static inline bool memcg_need_recharge(struct mem_cgroup *memcg) +{ + return false; +} + #endif /* CONFIG_MEMCG_KMEM */ #if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_ZSWAP) From patchwork Sun Jun 19 15:50:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12886754 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5293C433EF for ; Sun, 19 Jun 2022 15:50:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4E41B8D0007; Sun, 19 Jun 2022 11:50:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 494DC8D0006; Sun, 19 Jun 2022 11:50:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E7ED8D0007; Sun, 19 Jun 2022 11:50:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1694B8D0006 for ; Sun, 19 Jun 2022 11:50:48 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id E49F680689 for ; Sun, 19 Jun 2022 15:50:47 +0000 (UTC) X-FDA: 79595423334.13.27D9B5B Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by imf09.hostedemail.com (Postfix) with ESMTP id 9B72F1400A8 for ; Sun, 19 Jun 2022 15:50:47 +0000 (UTC) Received: by mail-pj1-f47.google.com with SMTP id x1-20020a17090abc8100b001ec7f8a51f5so4624473pjr.0 for ; Sun, 19 Jun 2022 08:50:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=s0+f5gHUcy4bOB49vr1foYREOsrIfIZFQCWb5SF1+Ms=; b=PosIsDNdyLPWrPtg4PrzsAVZoAXqL2g++Mki03DgUE/qzNvRlT8FvxmWGhIXyf6B2T S+SGm6A/MYaN4Zn5mylTRYZhOedr+gMVV9L3mcYuPJNxgmFgdPDfdhawpUff+Y38Jwya Wiw7zb7ZXfLw1RfhEv7uJtWYkhVtM/nG07kn9Q4PDgpVkpXzuZGjfwVCYD1PCstIVplC 3otx8kHv/cT0JpSNR9jDnw9x6LkkKJEHOLWrGwz/f1QXVprH7lWzBKcAqLQvUO0ZpdFa dez3HpGdPF7mH0pBtpM1IEsYhUhO1zgdUcgI4KeKXlM58/nk/5Lr5fsT/kzxW3ErZPBD 4EqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=s0+f5gHUcy4bOB49vr1foYREOsrIfIZFQCWb5SF1+Ms=; b=VNJ6TDR8QU8dNrZXigaHhaP2Udf287ZA1KwgPVtPkUfc8l6CaqXDTQpxA7J8jAyb1Z 77YT4dFouL5f0rCsnNiFTtDsYSkoBlzmZMnrtXG0a4ht04oGg37hvc35SZWUJVWmVCNt 8Hq4AQPAeD9ilMk1Qop11YR1PT16b/OY/aQfopnQ3VoVWvborjxhlFJAnMTIIR2IgXDf /BGN7sFdaMpO5YPNPdo6qm2E/0AfMcr69GSZ1XSAYyM+6Gf66w/bi6eID75YMnhWMTN8 hjWqGFyFSz5mRPjBe1A7sGZ46d19LtDkpOdqhTKLLTOVyOBpuOz6v3ZMdbnJSbCYsE4O jTjQ== X-Gm-Message-State: AJIora/iIMBbDywgJ0CP0tkkQsBMoYWZXzQyWMxVs3k5uEs40DY0I4yd BR1Z/kV3WxQB1QXAwfU47Ds= X-Google-Smtp-Source: AGRyM1sgtmrxoQfyITPlN/cfCgNmFWaIyPAqg6YjtKB9WTJgFrN1BLBYKOAWHzXeDwKE2MMfkvG0wA== X-Received: by 2002:a17:902:7d8e:b0:162:22ff:495b with SMTP id a14-20020a1709027d8e00b0016222ff495bmr19627141plm.1.1655653846678; Sun, 19 Jun 2022 08:50:46 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:2b24:5400:4ff:fe09:b144]) by smtp.gmail.com with ESMTPSA id z10-20020a1709027e8a00b001690a7df347sm6381761pla.96.2022.06.19.08.50.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jun 2022 08:50:45 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, quentin@isovalent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 02/10] bpftool: Show memcg info of bpf map Date: Sun, 19 Jun 2022 15:50:24 +0000 Message-Id: <20220619155032.32515-3-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220619155032.32515-1-laoar.shao@gmail.com> References: <20220619155032.32515-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655653847; a=rsa-sha256; cv=none; b=tuX5/AMBuldmaIjidt/RRe135YJQVjFEMqfhR4515BVOnghQVuMDzZ4LcqryTP2DNplbQK PCEwfICIJRnK5ZeBASXtPl2bGjMNivKcUac0/+sz5OCGQUlcJIdiX3Rz8j+1gBfC1OFsSu fFZmMWP052ZI5yOGqZc9l4WWYIR+SWE= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=PosIsDNd; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf09.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655653847; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=s0+f5gHUcy4bOB49vr1foYREOsrIfIZFQCWb5SF1+Ms=; b=qcjHufjVnLHVbhG5z1YH7Y/Qs2zHjcbvxAwPssB8bHSwri0f2RzZAEKLH1c1UVzMs3I9Tj 2dj+TWRVxWhJVWcHg1BDxqKVUapbFD2erlv/xjL1ZXFYq4fRQ3Xd/R2ylCMgXwarYNUgBs /1ZK4NQVjKELXuZJbD1wkK4VECPsTI8= Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=PosIsDNd; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf09.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com X-Rspam-User: X-Stat-Signature: ti5n4k9im5kx8mork95mn493jb113gfp X-Rspamd-Queue-Id: 9B72F1400A8 X-Rspamd-Server: rspam08 X-HE-Tag: 1655653847-474782 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: bpf map can be charged to a memcg, so we'd better show the memcg info to do better bpf memory management. This patch adds a new field "memcg_state" to show whether a bpf map is charged to a memcg and whether the memcg is offlined. Currently it has three values, 0 : belongs to root memcg -1 : its original memcg is not online now 1 : its original memcg is online For instance, $ bpftool map show 2: array name iterator.rodata flags 0x480 key 4B value 98B max_entries 1 memlock 4096B btf_id 240 frozen memcg_state 0 3: hash name calico_failsafe flags 0x1 key 4B value 1B max_entries 65535 memlock 524288B memcg_state 1 6: lru_hash name access_record flags 0x0 key 8B value 24B max_entries 102400 memlock 3276800B btf_id 256 memcg_state -1 Signed-off-by: Yafang Shao --- include/uapi/linux/bpf.h | 4 +++- kernel/bpf/syscall.c | 11 +++++++++++ tools/bpf/bpftool/map.c | 2 ++ tools/include/uapi/linux/bpf.h | 4 +++- 4 files changed, 19 insertions(+), 2 deletions(-) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index e81362891596..f2f658e224a7 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -6092,7 +6092,9 @@ struct bpf_map_info { __u32 btf_id; __u32 btf_key_type_id; __u32 btf_value_type_id; - __u32 :32; /* alignment pad */ + __s8 memcg_state; + __s8 :8; /* alignment pad */ + __u16 :16; /* alignment pad */ __u64 map_extra; } __attribute__((aligned(8))); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 7d5af5b99f0d..d4659d58d288 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -4168,6 +4168,17 @@ static int bpf_map_get_info_by_fd(struct file *file, } info.btf_vmlinux_value_type_id = map->btf_vmlinux_value_type_id; +#ifdef CONFIG_MEMCG_KMEM + if (map->memcg) { + struct mem_cgroup *memcg = map->memcg; + + if (memcg == root_mem_cgroup) + info.memcg_state = 0; + else + info.memcg_state = memcg_need_recharge(memcg) ? -1 : 1; + } +#endif + if (bpf_map_is_dev_bound(map)) { err = bpf_map_offload_info_fill(&info, map); if (err) diff --git a/tools/bpf/bpftool/map.c b/tools/bpf/bpftool/map.c index 38b6bc9c26c3..ba68512e83aa 100644 --- a/tools/bpf/bpftool/map.c +++ b/tools/bpf/bpftool/map.c @@ -525,6 +525,7 @@ static int show_map_close_json(int fd, struct bpf_map_info *info) jsonw_end_array(json_wtr); } + jsonw_int_field(json_wtr, "memcg_state", info->memcg_state); emit_obj_refs_json(refs_table, info->id, json_wtr); jsonw_end_object(json_wtr); @@ -615,6 +616,7 @@ static int show_map_close_plain(int fd, struct bpf_map_info *info) if (frozen) printf("%sfrozen", info->btf_id ? " " : ""); + printf("\n\tmemcg_state %d", info->memcg_state); emit_obj_refs_plain(refs_table, info->id, "\n\tpids "); printf("\n"); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index e81362891596..f2f658e224a7 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -6092,7 +6092,9 @@ struct bpf_map_info { __u32 btf_id; __u32 btf_key_type_id; __u32 btf_value_type_id; - __u32 :32; /* alignment pad */ + __s8 memcg_state; + __s8 :8; /* alignment pad */ + __u16 :16; /* alignment pad */ __u64 map_extra; } __attribute__((aligned(8))); From patchwork Sun Jun 19 15:50:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12886755 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D42F7C433EF for ; Sun, 19 Jun 2022 15:50:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5A7A68D0008; Sun, 19 Jun 2022 11:50:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 530D78D0006; Sun, 19 Jun 2022 11:50:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3AA4A8D0008; Sun, 19 Jun 2022 11:50:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 293378D0006 for ; Sun, 19 Jun 2022 11:50:50 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id F002AA7A for ; Sun, 19 Jun 2022 15:50:49 +0000 (UTC) X-FDA: 79595423418.26.623180C Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by imf22.hostedemail.com (Postfix) with ESMTP id 9BA72C000A for ; Sun, 19 Jun 2022 15:50:49 +0000 (UTC) Received: by mail-pj1-f47.google.com with SMTP id 73-20020a17090a0fcf00b001eaee69f600so8134914pjz.1 for ; Sun, 19 Jun 2022 08:50:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WaE/WLiaH5o5ki3TY2BjGi6nEsbC9ev3Bayiew634jE=; b=lg4hLmrvx1aAxFRgX2lydGh2RYsS+K8/vHGdBT3GPJM2V/iumEKMc9+bns60t1/KuB J1TuHcq8XADBIaFEVzyfvThx9pJUKJuJOKByFLFtB3cvZod6/1K+3pO5KBTrbMAriRSp 0aA84OhCbo1cxE5mjGwVS8OjVkf8uyLxSG96ru3xWftJyjaRYeDWc//IT0jYdBNtRhPn WcloU6qqNmA+04BwAXt7Tyjzl/yvXQUfBqUwQO+iOJ6GZB91CgEPWKZXrsjb0Cphc5pp f7Z0OMdpQ6lWNQjBQSjPrk9BqXMbRzM5/bOul+57dEAkay+yUKbMpZ9MsZadWBKuXs7W AYJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WaE/WLiaH5o5ki3TY2BjGi6nEsbC9ev3Bayiew634jE=; b=QAvGdInjRJxQgO6om8qjD5iQrYjn6pVYZMSlOS0Gr4A0sFEwdb4pD/y4is+6lDA/7K IhSdhIuKKoZRXx+8PE/z/cEtgnsYVIacaxITUoPpTBCpf2OL/VftfJuVUvSSWFjvVhSN sjqBNu5+Ow5pF8xGIRtJbYZphIXclVIwzMswa3niG4AVt1ePp7KLPt7M5TO8adgGNZqW ZC+cH4noq/nyW6KNq2xI35Q7bDuKM3I8m1hHa8vL/dHvRQZV1/nwHRACHNGTdsmsJFXe iwSmpfVJupluWjkcCEGDZgiKSY4XXlo1pgpaH1Gu1heQ32VDpgoRIwkgeV7zETjovj7v PBqQ== X-Gm-Message-State: AJIora/s3JCNVZpMMjJ0Id/brU0pMsHq1k7TiSx3Ddr7v7YbXtcOeKD4 EMotMgDm+PILTAO0cVNnZGE= X-Google-Smtp-Source: AGRyM1tw+4VoyM5S+fxLBlskF4kAB2xCBsbZchmj1iffm8dIiAU8+M1iozEcYYLSnyvhMrmeJK+cAw== X-Received: by 2002:a17:903:11cd:b0:167:90e6:5d83 with SMTP id q13-20020a17090311cd00b0016790e65d83mr19355523plh.136.1655653848680; Sun, 19 Jun 2022 08:50:48 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:2b24:5400:4ff:fe09:b144]) by smtp.gmail.com with ESMTPSA id z10-20020a1709027e8a00b001690a7df347sm6381761pla.96.2022.06.19.08.50.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jun 2022 08:50:47 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, quentin@isovalent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 03/10] mm, memcg: Add new helper obj_cgroup_from_current() Date: Sun, 19 Jun 2022 15:50:25 +0000 Message-Id: <20220619155032.32515-4-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220619155032.32515-1-laoar.shao@gmail.com> References: <20220619155032.32515-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655653849; a=rsa-sha256; cv=none; b=LGKr16bhPG8OgB7yR/Nr9R4XM55GJLdCdiIA355D26aokMmv5aKJOjhajFBkHHU/MrtCjs Trbh41zMZXp6crIjCwGSaXjIfHdt01wjk/kFXuTxLIwGeryviLXCyk2cRKeTJ7g287CQGp WvekK4RgYeyamH4o4XdnNPvpcHPILpU= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=lg4hLmrv; spf=pass (imf22.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655653849; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WaE/WLiaH5o5ki3TY2BjGi6nEsbC9ev3Bayiew634jE=; b=Y6LEWofv+WWUUA+gOpTLgBygSi2mwkWO9yGViXGvTmqJwKGWYu2hOGSMEpgrrhjU4SXkt2 kNDDmre57cwjDQKcf9U3mzHYkWHTOmnMSHEs4YNURGakjFukYTEZ9P0uIMfK4lw2iyCylv uI0qHrB1dK0jW1ZcDuaIsQ0moDdQXjM= X-Stat-Signature: sd8btri5u1xqj3kwkmakid438463zqmg X-Rspamd-Queue-Id: 9BA72C000A Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=lg4hLmrv; spf=pass (imf22.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam07 X-Rspam-User: X-HE-Tag: 1655653849-72757 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The difference between get_obj_cgroup_from_current() and obj_cgroup_from_current() is that the later one doesn't add objcg's refcnt. Signed-off-by: Yafang Shao --- include/linux/memcontrol.h | 1 + mm/memcontrol.c | 24 ++++++++++++++++++++++++ 2 files changed, 25 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index cf074156c6ac..402b42670bcd 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1703,6 +1703,7 @@ bool mem_cgroup_kmem_disabled(void); int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order); void __memcg_kmem_uncharge_page(struct page *page, int order); +struct obj_cgroup *obj_cgroup_from_current(void); struct obj_cgroup *get_obj_cgroup_from_current(void); struct obj_cgroup *get_obj_cgroup_from_page(struct page *page); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index abec50f31fe6..350a7849dac3 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2950,6 +2950,30 @@ struct obj_cgroup *get_obj_cgroup_from_page(struct page *page) return objcg; } +__always_inline struct obj_cgroup *obj_cgroup_from_current(void) +{ + struct obj_cgroup *objcg = NULL; + struct mem_cgroup *memcg; + + if (memcg_kmem_bypass()) + return NULL; + + rcu_read_lock(); + if (unlikely(active_memcg())) + memcg = active_memcg(); + else + memcg = mem_cgroup_from_task(current); + + for (; memcg != root_mem_cgroup; memcg = parent_mem_cgroup(memcg)) { + objcg = rcu_dereference(memcg->objcg); + if (objcg) + break; + } + rcu_read_unlock(); + + return objcg; +} + static void memcg_account_kmem(struct mem_cgroup *memcg, int nr_pages) { mod_memcg_state(memcg, MEMCG_KMEM, nr_pages); From patchwork Sun Jun 19 15:50:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12886756 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7E60C433EF for ; Sun, 19 Jun 2022 15:50:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6DEFE8D0009; Sun, 19 Jun 2022 11:50:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 66A588D0006; Sun, 19 Jun 2022 11:50:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5089F8D0009; Sun, 19 Jun 2022 11:50:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3B22D8D0006 for ; Sun, 19 Jun 2022 11:50:52 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 114A033812 for ; Sun, 19 Jun 2022 15:50:52 +0000 (UTC) X-FDA: 79595423544.07.87FBCD5 Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) by imf06.hostedemail.com (Postfix) with ESMTP id A69591800A4 for ; Sun, 19 Jun 2022 15:50:51 +0000 (UTC) Received: by mail-pj1-f48.google.com with SMTP id a11-20020a17090acb8b00b001eca0041455so651449pju.1 for ; Sun, 19 Jun 2022 08:50:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fG2YxPMk3XHSDo57as/MEaodXX0vf3EFBzxExOHzp94=; b=XkgPPFKVR0Jb570fcBCjFqP4oTrs6/NzA9SwpN40QvOkpRrkdVm4xtW+wtc1OeLYnQ 7n5ljaFkzV62y5SctU2XLyM5Ncv2PvQXvXlTPFHLGngn5hRYSNnk/hAML04kw5vaftoC kicAFGjW8BsQmffGKtlas6ScJlQuTDHwrwdPMjdRMG4+MbIEI/+4yBkDRHeRmPwSA9pr ZFBtd8SEVoAv/qVyE5F76lxq6R1Jd47JM6IooznrhE3yduAUu+K8GZYPIRZc5jleBog6 Z2XpWHdYxSbHdu+VIYBfO0TJAdtxROQ77z4rPcz4rXDjpZ0kS+mzu8bj5jSq4/PuEIPu a4HQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fG2YxPMk3XHSDo57as/MEaodXX0vf3EFBzxExOHzp94=; b=0YHEbjjzyYsSpD8PSWVPVIUSEugs5GQIGQ8dYdSN9HWf31P9eKYnhhfN7iQSGeuOpL N+8rPTaplDGXku7beNtmqiPQK+MlZVgyvSIRVviil2H8MKu5TtiZhT9duhI0lXjddT/h ONEBw13PWDA+rhN4SlaDlicg9Ep1ie7batv9eqmez000tHVe/91ckb4fESoKnTQqsGFH OZ+vk0q5qSiSz7iXhatLL1OU3mmvgtERFFqM5ky+TVJ9Ke6NOyuBI4FZwwzPnnGz7Wfz Bug1CtJbSPDdLyFiieZL7wymtuqS82bh335bb8L98BnSXGWCdnC/dljNp3BQCoG/aCDo x5SQ== X-Gm-Message-State: AJIora8sQpf+RXwZJhcnN1wlGXzQ+kj+qcQG4F1I8UcsWRFneXQen5nq oIZUWylqJGekyf9XwtiZpr0= X-Google-Smtp-Source: AGRyM1vxAU7OTmDElw6CQ4IfMkyUasemfaxdeD46XcwgZFgN1qsd8XEuFJv7/HuK+g+5Vt/uKY9xag== X-Received: by 2002:a17:90b:4d11:b0:1e8:436b:a9cc with SMTP id mw17-20020a17090b4d1100b001e8436ba9ccmr33074721pjb.40.1655653850770; Sun, 19 Jun 2022 08:50:50 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:2b24:5400:4ff:fe09:b144]) by smtp.gmail.com with ESMTPSA id z10-20020a1709027e8a00b001690a7df347sm6381761pla.96.2022.06.19.08.50.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jun 2022 08:50:49 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, quentin@isovalent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 04/10] mm, memcg: Make obj_cgroup_{charge, uncharge}_pages public Date: Sun, 19 Jun 2022 15:50:26 +0000 Message-Id: <20220619155032.32515-5-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220619155032.32515-1-laoar.shao@gmail.com> References: <20220619155032.32515-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655653851; a=rsa-sha256; cv=none; b=r8ZglZg6rX3wtkw9xGqv3/cz8RD0u3NpIFuFXO4/ZaVtV/Ro2gSxgVcEx6h25QkEiBxtqO 3iWu5vXwyuvLBW8VjS2GdWAFgUF5O+2APgZWAw554H8MDERQbfqqMN8EFRZfIvgYatgVeS aVhsdZlhVkSdNoEhmgsUZ++jSnmJJ+s= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=XkgPPFKV; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf06.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.48 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655653851; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fG2YxPMk3XHSDo57as/MEaodXX0vf3EFBzxExOHzp94=; b=NeYOrk6ZhT75rI0+wAhkP+kR8Yv294IjsN4Rm1SlSrq9ExDQrQWpMSZ545M1+cWKg2JItj mSt8APM1H9+s4MxWJOWyFkQYfhGGJ11XLTaWR1h9cE8y25jQzQM/N/IawrK+mYLGGggIQW YxIBOW8EY3kaj1FobCf8yxYiVd9uvuo= Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=XkgPPFKV; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf06.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.48 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com X-Rspam-User: X-Stat-Signature: k4ki1kets977upnqzm5f5ywbt6edbcp4 X-Rspamd-Queue-Id: A69591800A4 X-Rspamd-Server: rspam08 X-HE-Tag: 1655653851-561497 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make these two helpers public for later use. Signed-off-by: Yafang Shao --- include/linux/memcontrol.h | 4 ++++ mm/memcontrol.c | 11 ++++------- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 402b42670bcd..ec4637687d6a 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1709,6 +1709,10 @@ struct obj_cgroup *get_obj_cgroup_from_page(struct page *page); int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size); void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size); +int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp, + unsigned int nr_pages); +void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg, + unsigned int nr_pages); extern struct static_key_false memcg_kmem_enabled_key; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 350a7849dac3..0ba321afba3b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -260,9 +260,6 @@ bool mem_cgroup_kmem_disabled(void) return cgroup_memory_nokmem; } -static void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg, - unsigned int nr_pages); - static void obj_cgroup_release(struct percpu_ref *ref) { struct obj_cgroup *objcg = container_of(ref, struct obj_cgroup, refcnt); @@ -2991,8 +2988,8 @@ static void memcg_account_kmem(struct mem_cgroup *memcg, int nr_pages) * @objcg: object cgroup to uncharge * @nr_pages: number of pages to uncharge */ -static void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg, - unsigned int nr_pages) +void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg, + unsigned int nr_pages) { struct mem_cgroup *memcg; @@ -3012,8 +3009,8 @@ static void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg, * * Returns 0 on success, an error code on failure. */ -static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp, - unsigned int nr_pages) +int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp, + unsigned int nr_pages) { struct mem_cgroup *memcg; int ret; From patchwork Sun Jun 19 15:50:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12886757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 486A4C43334 for ; Sun, 19 Jun 2022 15:50:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A61F18D000A; Sun, 19 Jun 2022 11:50:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9EC5D8D0006; Sun, 19 Jun 2022 11:50:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8180C8D000A; Sun, 19 Jun 2022 11:50:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6AF338D0006 for ; Sun, 19 Jun 2022 11:50:54 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3E9F5349F1 for ; Sun, 19 Jun 2022 15:50:54 +0000 (UTC) X-FDA: 79595423628.09.0B97EEE Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by imf28.hostedemail.com (Postfix) with ESMTP id C6949C0009 for ; Sun, 19 Jun 2022 15:50:53 +0000 (UTC) Received: by mail-pj1-f52.google.com with SMTP id cv13so5477093pjb.4 for ; Sun, 19 Jun 2022 08:50:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Nnh4T4SgR/gW+b4BUtX1sa65JkL4/iwCexxQ6cEBZbc=; b=lq1nH/bYPC7duqDClETgEfZ7OE6F+xkpRcqocCJ/GrXekZW96wX4D7QC6CL0SDMQdb +FtNyN0D5mFC/MpqRtckmw2IGiaIue5QZVxiT3LG2ig1yEV/rMfSIW30ZRqkBs50a5x3 5zC+YcMCSjZjk0kZSWD3lqERf7kOjQg3EYRqTnOKVOmKB0IqOiR+27y5FXRnkPaOB+MY RFGpBS8GBE1JNuB7EdwrfI/7sNVIpZvMyWwzW4boOiU8pq4vCQ9fEIq+tm43dfAEhAKW EclyeeUGkwPoimSQxiZ1VUd5+ZHtzKleSs9So4zgUWqecwReWqilJIzbUGrD8vJzxaW6 XXrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Nnh4T4SgR/gW+b4BUtX1sa65JkL4/iwCexxQ6cEBZbc=; b=QEJwSg7OmKZnlHeMmkNwHWjQll2D07qk6MGdKsLAdPZjv8jtF3uh2fqOrkvxNoEvv2 hHOL49AJn2qrwXht71prZAYYtEuJcg2QoP3eXR/ZeXGiGmRDYpuG2CqWgC5zGX4QyidG Q6G2wF91IfZ/E2xLzBlxvCKbdzTL+qElFduqXv/vYLx5E7FR0u+xyPHKQFS67ElkPPur 7b4WXNG3ez7RXcjqGhrEccuFy0o0/hNA3FlNmiN9VOqv0uHKWVJIWygCSyXZKu2RuYvH EbqU40ZGteZ2Wd244TnHdgOxBzSXUg9m/1lPP9Ie4TGPLmPcft9k0GaphVV/0BD2FtZw GXhQ== X-Gm-Message-State: AJIora/pOtDUmCsvxsx/M1PqmcvD6buEdDITamf3/19y65f3ShSfU9Pp u5P4POBsAjZBtF2WxSmCG71ydOkEn8F7QklFlW8= X-Google-Smtp-Source: AGRyM1u7nD+7X0UghJbd7/MLvgdOfAhOJLKxy2iGHckStcQblFZ3VTGbP3YeL6rPM8st9nF1xiP8zg== X-Received: by 2002:a17:902:9041:b0:16a:aef:7b84 with SMTP id w1-20020a170902904100b0016a0aef7b84mr10956835plz.124.1655653852872; Sun, 19 Jun 2022 08:50:52 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:2b24:5400:4ff:fe09:b144]) by smtp.gmail.com with ESMTPSA id z10-20020a1709027e8a00b001690a7df347sm6381761pla.96.2022.06.19.08.50.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jun 2022 08:50:52 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, quentin@isovalent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 05/10] mm: Add helper to recharge kmalloc'ed address Date: Sun, 19 Jun 2022 15:50:27 +0000 Message-Id: <20220619155032.32515-6-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220619155032.32515-1-laoar.shao@gmail.com> References: <20220619155032.32515-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655653853; a=rsa-sha256; cv=none; b=rFADXts7PcEpQ58Y16Ce2iKVJYIjQ93hC5iMy0HYKLcDhhyNF6V5qqvDhLSEmQggTHGvzR nAFJntkaUg/9bonAHEmqxN3lRZ+E3oxjJL1ZI7P250nEgk+Ac3jFB4RyzkIp3iluJN9U4b 0+S0HKqvOJDWW1rwtCRreRhEebcuGQY= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="lq1nH/bY"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.52 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655653853; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Nnh4T4SgR/gW+b4BUtX1sa65JkL4/iwCexxQ6cEBZbc=; b=ehgyqgpkU+JQ1dbajAk/3cSlLfRm1K1m6wChAPbfAj4dwvgaa4nKObqnNV4ssQ+gGtq3oV UfdDxlRS+oXrz5xdEmaf42hd08Vyipg5TyH8T/zFvHuWw54Kt9JsueMSWG4s3Ke+d6ii9T OeMN0DUiFHt4eOugRSWDvW3P1m1/QM0= X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: C6949C0009 X-Stat-Signature: 5nps1y83c66q7hcq6aw8n683hgzi1cdc X-Rspam-User: Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="lq1nH/bY"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.52 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com X-HE-Tag: 1655653853-533593 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch introduces a helper to recharge the corresponding pages of a given kmalloc'ed address. The recharge is divided into three steps, - pre charge to the new memcg To make sure once we uncharge from the old memcg, we can always charge to the new memcg succeesfully. If we can't pre charge to the new memcg, we won't allow it to be uncharged from the old memcg. - uncharge from the old memcg After pre charge to the new memcg, we can uncharge from the old memcg. - post charge to the new memcg Modify the counters of the new memcg. Sometimes we may want to recharge many kmalloc'ed addresses to the same memcg, in that case we should pre charge all these addresses first, then do the uncharge and finnally do the post charge. But it may happens that after succeesfully pre charge some address we fail to pre charge a new address, then we have to cancel the finished pre charge, so charge err is introduced for this purpose. Signed-off-by: Yafang Shao --- include/linux/slab.h | 17 ++++++ mm/slab.c | 85 +++++++++++++++++++++++++++++ mm/slob.c | 7 +++ mm/slub.c | 125 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 234 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index 0fefdf528e0d..18ab30aa8fe8 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -194,6 +194,23 @@ bool kmem_valid_obj(void *object); void kmem_dump_obj(void *object); #endif +/* + * The recharge will be separated into three steps: + * MEMCG_KMEM_PRE_CHARGE : pre charge to the new memcg + * MEMCG_KMEM_UNCHARGE : uncharge from the old memcg + * MEMCG_KMEM_POST_CHARGE : post charge to the new memcg + * and an error handler: + * MEMCG_KMEM_CHARGE_ERR : in pre charge state, we may succeed to + * charge some objp's but fail to charge + * a new one, then in this case we should + * uncharge the already charged objp's. + */ +#define MEMCG_KMEM_PRE_CHARGE 0 +#define MEMCG_KMEM_UNCHARGE 1 +#define MEMCG_KMEM_POST_CHARGE 2 +#define MEMCG_KMEM_CHARGE_ERR 3 +bool krecharge(const void *objp, int step); + /* * Some archs want to perform DMA into kmalloc caches and need a guaranteed * alignment larger than the alignment of a 64-bit integer. diff --git a/mm/slab.c b/mm/slab.c index f8cd00f4ba13..4795014edd30 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3798,6 +3798,91 @@ void kfree(const void *objp) } EXPORT_SYMBOL(kfree); +bool krecharge(const void *objp, int step) +{ + void *object = (void *)objp; + struct obj_cgroup *objcg_old; + struct obj_cgroup *objcg_new; + struct obj_cgroup **objcgs; + struct kmem_cache *s; + struct slab *slab; + unsigned long flags; + unsigned int off; + + WARN_ON(!in_task()); + + if (unlikely(ZERO_OR_NULL_PTR(objp))) + return true; + + if (!memcg_kmem_enabled()) + return true; + + local_irq_save(flags); + s = virt_to_cache(objp); + if (!s) + goto out; + + if (!(s->flags & SLAB_ACCOUNT)) + goto out; + + slab = virt_to_slab(object); + if (!slab) + goto out; + + objcgs = slab_objcgs(slab); + if (!objcgs) + goto out; + + off = obj_to_index(s, slab, object); + objcg_old = objcgs[off]; + if (!objcg_old && step != MEMCG_KMEM_POST_CHARGE) + goto out; + + /* + * The recharge can be separated into three steps, + * 1. Pre charge to the new memcg + * 2. Uncharge from the old memcg + * 3. Charge to the new memcg + */ + switch (step) { + case MEMCG_KMEM_PRE_CHARGE: + /* Pre recharge */ + objcg_new = get_obj_cgroup_from_current(); + WARN_ON(!objcg_new); + if (obj_cgroup_charge(objcg_new, GFP_KERNEL, obj_full_size(s))) { + obj_cgroup_put(objcg_new); + local_irq_restore(flags); + return false; + } + break; + case MEMCG_KMEM_UNCHARGE: + /* Uncharge from the old memcg */ + obj_cgroup_uncharge(objcg_old, obj_full_size(s)); + objcgs[off] = NULL; + mod_objcg_state(objcg_old, slab_pgdat(slab), cache_vmstat_idx(s), + -obj_full_size(s)); + obj_cgroup_put(objcg_old); + break; + case MEMCG_KMEM_POST_CHARGE: + /* Charge to the new memcg */ + objcg_new = obj_cgroup_from_current(); + objcgs[off] = objcg_new; + mod_objcg_state(objcg_new, slab_pgdat(slab), cache_vmstat_idx(s), obj_full_size(s)); + break; + case MEMCG_KMEM_CHARGE_ERR: + objcg_new = obj_cgroup_from_current(); + obj_cgroup_uncharge(objcg_new, obj_full_size(s)); + obj_cgroup_put(objcg_new); + break; + } + +out: + local_irq_restore(flags); + + return true; +} +EXPORT_SYMBOL(krecharge); + /* * This initializes kmem_cache_node or resizes various caches for all nodes. */ diff --git a/mm/slob.c b/mm/slob.c index f47811f09aca..6d68ad57b4a2 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -574,6 +574,13 @@ void kfree(const void *block) } EXPORT_SYMBOL(kfree); +/* kmemcg is no supported for SLOB */ +bool krecharge(const void *block, int step) +{ + return true; +} +EXPORT_SYMBOL(krecharge); + /* can't use ksize for kmem_cache_alloc memory, only kmalloc */ size_t __ksize(const void *block) { diff --git a/mm/slub.c b/mm/slub.c index e5535020e0fd..ef6475ed6407 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4556,6 +4556,131 @@ void kfree(const void *x) } EXPORT_SYMBOL(kfree); +bool krecharge(const void *x, int step) +{ + void *object = (void *)x; + struct obj_cgroup *objcg_old; + struct obj_cgroup *objcg_new; + struct obj_cgroup **objcgs; + struct kmem_cache *s; + struct folio *folio; + struct slab *slab; + unsigned int off; + + WARN_ON(!in_task()); + + if (!memcg_kmem_enabled()) + return true; + + if (unlikely(ZERO_OR_NULL_PTR(x))) + return true; + + folio = virt_to_folio(x); + if (unlikely(!folio_test_slab(folio))) { + unsigned int order = folio_order(folio); + struct page *page; + + switch (step) { + case MEMCG_KMEM_PRE_CHARGE: + objcg_new = get_obj_cgroup_from_current(); + WARN_ON(!objcg_new); + /* Try charge current memcg */ + if (obj_cgroup_charge_pages(objcg_new, GFP_KERNEL, + 1 << order)) { + obj_cgroup_put(objcg_new); + return false; + } + break; + case MEMCG_KMEM_UNCHARGE: + /* Uncharge folio memcg */ + objcg_old = __folio_objcg(folio); + page = folio_page(folio, 0); + WARN_ON(!objcg_old); + obj_cgroup_uncharge_pages(objcg_old, 1 << order); + mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE_B, + -(PAGE_SIZE << order)); + page->memcg_data = 0; + obj_cgroup_put(objcg_old); + break; + case MEMCG_KMEM_POST_CHARGE: + /* Set current memcg to folio page */ + objcg_new = obj_cgroup_from_current(); + page = folio_page(folio, 0); + page->memcg_data = (unsigned long)objcg_new | MEMCG_DATA_KMEM; + mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE_B, + -(PAGE_SIZE << order)); + break; + case MEMCG_KMEM_CHARGE_ERR: + objcg_new = obj_cgroup_from_current(); + obj_cgroup_uncharge_pages(objcg_new, 1 << order); + obj_cgroup_put(objcg_new); + break; + } + return true; + } + + slab = folio_slab(folio); + if (!slab) + return true; + + s = slab->slab_cache; + if (!(s->flags & SLAB_ACCOUNT)) + return true; + + objcgs = slab_objcgs(slab); + if (!objcgs) + return true; + off = obj_to_index(s, slab, object); + objcg_old = objcgs[off]; + /* In step MEMCG_KMEM_UNCHARGE, the objcg will set to NULL. */ + if (!objcg_old && step != MEMCG_KMEM_POST_CHARGE) + return true; + + /* + * The recharge can be separated into three steps, + * 1. Pre charge to the new memcg + * 2. Uncharge from the old memcg + * 3. Charge to the new memcg + */ + switch (step) { + case MEMCG_KMEM_PRE_CHARGE: + /* + * Before uncharge from the old memcg, we must pre charge the new memcg + * first, to make sure it always succeed to recharge to the new memcg + * after uncharge from the old memcg. + */ + objcg_new = get_obj_cgroup_from_current(); + WARN_ON(!objcg_new); + if (obj_cgroup_charge(objcg_new, GFP_KERNEL, obj_full_size(s))) { + obj_cgroup_put(objcg_new); + return false; + } + break; + case MEMCG_KMEM_UNCHARGE: + /* Uncharge from old memcg */ + obj_cgroup_uncharge(objcg_old, obj_full_size(s)); + objcgs[off] = NULL; + mod_objcg_state(objcg_old, slab_pgdat(slab), cache_vmstat_idx(s), + -obj_full_size(s)); + obj_cgroup_put(objcg_old); + break; + case MEMCG_KMEM_POST_CHARGE: + /* Charge to the new memcg */ + objcg_new = obj_cgroup_from_current(); + objcgs[off] = objcg_new; + mod_objcg_state(objcg_new, slab_pgdat(slab), cache_vmstat_idx(s), obj_full_size(s)); + break; + case MEMCG_KMEM_CHARGE_ERR: + objcg_new = obj_cgroup_from_current(); + obj_cgroup_uncharge(objcg_new, obj_full_size(s)); + obj_cgroup_put(objcg_new); + break; + } + + return true; +} +EXPORT_SYMBOL(krecharge); + #define SHRINK_PROMOTE_MAX 32 /* From patchwork Sun Jun 19 15:50:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12886758 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67D30C43334 for ; Sun, 19 Jun 2022 15:50:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EF6598D000B; Sun, 19 Jun 2022 11:50:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EA6388D0006; Sun, 19 Jun 2022 11:50:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD22A8D000B; Sun, 19 Jun 2022 11:50:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B94C68D0006 for ; Sun, 19 Jun 2022 11:50:56 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8A9CBAF9 for ; Sun, 19 Jun 2022 15:50:56 +0000 (UTC) X-FDA: 79595423712.26.CF588B4 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) by imf02.hostedemail.com (Postfix) with ESMTP id 2CBD480017 for ; Sun, 19 Jun 2022 15:50:55 +0000 (UTC) Received: by mail-pg1-f170.google.com with SMTP id r66so2239087pgr.2 for ; Sun, 19 Jun 2022 08:50:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FQMzfPqgezzDzrRv8wh1613ZP98YLQGz+eOS2SGIRLM=; b=S2PD9ftCpBG9QGAr4mB7ADu83tpdlow4NteqA06BGj1jvJIQVzAOTvJlSYYGTshREd ytbN//cR3Cko4oTkb4pBv5/BrHknENhFGZctfFs0Qi1/gVnhHshITRo1VcjHMDlYkLn6 nZ3e4fCEeR34zHBeLzGBpoEVYP7KOIBifF/zcGFh54GRLKBN9XKlyMZoWPEt5v7MQ7/t GdNhDv+p9RGJnaOAvYSAUiRKkdPlY4JTGSPMPzwRG3AzdhoyAe4NXxQ0Fmt8SEnI0ll2 NYEpYVFgVOk6U7AMLGJXT5kvhZ9DtorIopbsQdHbxVGh0iEbAVqltUkvZBYgmFYdvaUz Ey2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FQMzfPqgezzDzrRv8wh1613ZP98YLQGz+eOS2SGIRLM=; b=Wy3XKdi/WbBy+1QGs2D33o9EB0TPvbXOXbXDdX2CMDmCzRwQQEILTiTFVUCpwAIUHa LXSZhyA5Z7waxnmJhI36ewoTJTaADP7348s02HZK9vhE6b4hALe0jZkv5MT3cGpS4/hL sTogBANBx7KR+UuHoq8mleYaRDNu+xA/r5vm6heGBGRi6I0cBMs+pzJiuIuarzvGVoME NNmEy2Vp1UGIEh4avx7lBjhRnCJRmORlK2FON5K6LplvYuDM/96NnDkGpg2FHkpY4hTw UCn7ZO87aigrxsXJ4kCWb4JKD1BOIQUOdIccUIgMayLR+XdZSNejsbZkXrrfGjZNx/12 V0lw== X-Gm-Message-State: AJIora8hPtSKRdH1F7OEiwlg34DUrfPOHBqBsaZw0ocuaPUuApFVrz3h I2iygbFeXXv4fzp77ZGh5Es= X-Google-Smtp-Source: AGRyM1sefUlfO+kxoEWF8nnzpyg1oWx0iO3gGkduXWLQM7oyJzs209FUxi9aQRx8YGBdp74a/WvUWw== X-Received: by 2002:a63:130f:0:b0:401:ce98:24eb with SMTP id i15-20020a63130f000000b00401ce9824ebmr18468553pgl.217.1655653855042; Sun, 19 Jun 2022 08:50:55 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:2b24:5400:4ff:fe09:b144]) by smtp.gmail.com with ESMTPSA id z10-20020a1709027e8a00b001690a7df347sm6381761pla.96.2022.06.19.08.50.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jun 2022 08:50:54 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, quentin@isovalent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 06/10] mm: Add helper to recharge vmalloc'ed address Date: Sun, 19 Jun 2022 15:50:28 +0000 Message-Id: <20220619155032.32515-7-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220619155032.32515-1-laoar.shao@gmail.com> References: <20220619155032.32515-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655653856; a=rsa-sha256; cv=none; b=N6v1ESLLYT/94UQCHT4AOj3HCM2JLTXnYfSaOIpLfS/+lbbpC/pc3RGj2YuAeqtqs2Ii6M U9luSmET03Aygiwgz9+XN3SpLSVie2FdipWI+0DipE5Nm46wLtpWvDrwWBKLgmCb7pg8vY srtfZQOYN9aWrPrkMXAmC3ziUcm7wnU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655653856; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FQMzfPqgezzDzrRv8wh1613ZP98YLQGz+eOS2SGIRLM=; b=8oZbDeol1D7laESYlwy+Nk/scCwyUdQTP35fLW/xMpNJs1DIXoZxfBn3cm8fpu993tWwri p+z0tma3NGotnsxI0FlVhh13nfcTYt6hwmk7j6L1W5NHTu5aqkV13F1NRkOKSFTBxv8Q5s 33LcHqmNezza7/g/mtsLlwS3aBuhPQA= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=S2PD9ftC; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=S2PD9ftC; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 2CBD480017 X-Stat-Signature: ycwfzpswrw8sz7jqdc6yfcz18e9uetib X-HE-Tag: 1655653855-765617 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch introduces a helper to recharge the corresponding pages of a given vmalloc'ed address. It is similar with how to recharge a kmalloced'ed address. Signed-off-by: Yafang Shao --- include/linux/slab.h | 1 + include/linux/vmalloc.h | 2 + mm/util.c | 9 +++++ mm/vmalloc.c | 87 +++++++++++++++++++++++++++++++++++++++++ 4 files changed, 99 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index 18ab30aa8fe8..e8fb0f6a3660 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -794,6 +794,7 @@ extern void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flag __alloc_size(3); extern void kvfree(const void *addr); extern void kvfree_sensitive(const void *addr, size_t len); +bool kvrecharge(const void *addr, int step); unsigned int kmem_cache_size(struct kmem_cache *s); void __init kmem_cache_init_late(void); diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 096d48aa3437..37c6d0e7b8d5 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -162,6 +162,8 @@ extern void *vcalloc(size_t n, size_t size) __alloc_size(1, 2); extern void vfree(const void *addr); extern void vfree_atomic(const void *addr); +bool vrecharge(const void *addr, int step); +void vuncharge(const void *addr); extern void *vmap(struct page **pages, unsigned int count, unsigned long flags, pgprot_t prot); diff --git a/mm/util.c b/mm/util.c index 0837570c9225..312c05e83132 100644 --- a/mm/util.c +++ b/mm/util.c @@ -656,6 +656,15 @@ void kvfree(const void *addr) } EXPORT_SYMBOL(kvfree); +bool kvrecharge(const void *addr, int step) +{ + if (is_vmalloc_addr(addr)) + return vrecharge(addr, step); + + return krecharge(addr, step); +} +EXPORT_SYMBOL(kvrecharge); + /** * kvfree_sensitive - Free a data object containing sensitive information. * @addr: address of the data object to be freed. diff --git a/mm/vmalloc.c b/mm/vmalloc.c index effd1ff6a4b4..7da6e429a45f 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2745,6 +2745,93 @@ void vfree(const void *addr) } EXPORT_SYMBOL(vfree); +bool vrecharge(const void *addr, int step) +{ + struct obj_cgroup *objcg_new; + unsigned int page_order; + struct vm_struct *area; + struct folio *folio; + int i; + + WARN_ON(!in_task()); + + if (!addr) + return true; + + area = find_vm_area(addr); + if (unlikely(!area)) + return true; + + page_order = vm_area_page_order(area); + + switch (step) { + case MEMCG_KMEM_PRE_CHARGE: + for (i = 0; i < area->nr_pages; i += 1U << page_order) { + struct page *page = area->pages[i]; + + WARN_ON(!page); + objcg_new = get_obj_cgroup_from_current(); + WARN_ON(!objcg_new); + if (obj_cgroup_charge_pages(objcg_new, GFP_KERNEL, + 1 << page_order)) + goto out_pre; + cond_resched(); + } + break; + case MEMCG_KMEM_UNCHARGE: + for (i = 0; i < area->nr_pages; i += 1U << page_order) { + struct page *page = area->pages[i]; + struct obj_cgroup *objcg_old; + + WARN_ON(!page); + folio = page_folio(page); + WARN_ON(!folio_memcg_kmem(folio)); + objcg_old = __folio_objcg(folio); + + obj_cgroup_uncharge_pages(objcg_old, 1 << page_order); + /* mod memcg from page */ + mod_memcg_state(page_memcg(page), MEMCG_VMALLOC, + -(1U << page_order)); + page->memcg_data = 0; + obj_cgroup_put(objcg_old); + cond_resched(); + } + break; + case MEMCG_KMEM_POST_CHARGE: + objcg_new = obj_cgroup_from_current(); + for (i = 0; i < area->nr_pages; i += 1U << page_order) { + struct page *page = area->pages[i]; + + page->memcg_data = (unsigned long)objcg_new | MEMCG_DATA_KMEM; + /* mod memcg from current */ + mod_memcg_state(page_memcg(page), MEMCG_VMALLOC, + 1U << page_order); + + } + break; + case MEMCG_KMEM_CHARGE_ERR: + objcg_new = obj_cgroup_from_current(); + for (i = 0; i < area->nr_pages; i += 1U << page_order) { + obj_cgroup_uncharge_pages(objcg_new, 1 << page_order); + obj_cgroup_put(objcg_new); + cond_resched(); + } + break; + } + + return true; + +out_pre: + for (; i > 0; i -= 1U << page_order) { + obj_cgroup_uncharge_pages(objcg_new, 1 << page_order); + obj_cgroup_put(objcg_new); + cond_resched(); + } + + return false; +} +EXPORT_SYMBOL(vrecharge); + /** * vunmap - release virtual mapping obtained by vmap() * @addr: memory base address From patchwork Sun Jun 19 15:50:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12886759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 582F2C43334 for ; Sun, 19 Jun 2022 15:50:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ED0F78D000C; Sun, 19 Jun 2022 11:50:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E58738D0006; Sun, 19 Jun 2022 11:50:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF8A58D000C; Sun, 19 Jun 2022 11:50:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BA1758D0006 for ; Sun, 19 Jun 2022 11:50:58 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 9267D60967 for ; Sun, 19 Jun 2022 15:50:58 +0000 (UTC) X-FDA: 79595423796.23.08772DE Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf28.hostedemail.com (Postfix) with ESMTP id 2EB46C0009 for ; Sun, 19 Jun 2022 15:50:58 +0000 (UTC) Received: by mail-pl1-f171.google.com with SMTP id h1so7731739plf.11 for ; Sun, 19 Jun 2022 08:50:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8axVjIKJzoKX7/vYAqMF/TYbFLVxAIKs18CKF3m+oFs=; b=PuPpsRhrY7YHspdQWWpU1xoiN04UvGnMB4vC75wdnvBWiS+Y1ytnUMJqu3Ho9YMv7j yUVPddPMbPZG1XjtCjIyGWJoikOcQ0GatJvErGEg6QrKCQj3dvmPOZSUtiXTU+dFvoSp YcIX9PwU4tdeb+QuUi7/vdBpJBPcFXrepnvYHjd/drtSfgthuwT/jXJ9dp48KqVK1jGQ L78qryWdc++QJK17LRHW7muc+RPiMWMgaJ4tfBg+wB226QbRz/Vy32swaf+QXSEiOP6S DXPQMqrQ0zysXkG87o3+oTm4V7pbmv2EYd17jJv4SCOll+2gRmAW4MAkIbBXsxPkuf++ IiPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8axVjIKJzoKX7/vYAqMF/TYbFLVxAIKs18CKF3m+oFs=; b=dCdhTR6jXFsjOiZG8c1U8s7sCGoMKFyehuzGUwV89JeyWL8hQl63BW5i09EgCGJiPT ep+MpAQCGv2C6R0IKwN06Bw1IQX/QYwf1OzNNuFTn1/5b3AeqSagD7F6zSQvN5iV5pzl 4ghYt4I2eh1wI7ui7uVCTmV+4zOUuqSQRggBDZBwJAEHaPv0Dfk51S0YxJl8/5bAqK/2 w0g2GVMvVfHrsBXdlbdVHHSUlHJFGQlQ8MMY1eeYhc8U58OQiknCnZwgB2iSNKerzx0q zjlH4SNdNr6HNdlXPxhSsghutqBRSieAej4YR6PcT+ThNThcnBLIEpHu7+UkK9mUx5Ne N2Ag== X-Gm-Message-State: AJIora8pl4AFrvhHtY+HAHvjyFFyDJ2VScThQzo7VzqNlUPKnxOE+kFZ PTLZJNbncr+vOCLBDF3wgIY= X-Google-Smtp-Source: AGRyM1tWFLmDZ1xZ4c7PpIxT4CG5PqHtIf/jPAzTTqfH8RQEghLLoAKdBEQZ9RMRcWjmMK8GqJOEtA== X-Received: by 2002:a17:902:7686:b0:168:de55:8c45 with SMTP id m6-20020a170902768600b00168de558c45mr19518921pll.129.1655653857305; Sun, 19 Jun 2022 08:50:57 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:2b24:5400:4ff:fe09:b144]) by smtp.gmail.com with ESMTPSA id z10-20020a1709027e8a00b001690a7df347sm6381761pla.96.2022.06.19.08.50.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jun 2022 08:50:56 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, quentin@isovalent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 07/10] mm: Add helper to recharge percpu address Date: Sun, 19 Jun 2022 15:50:29 +0000 Message-Id: <20220619155032.32515-8-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220619155032.32515-1-laoar.shao@gmail.com> References: <20220619155032.32515-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655653858; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8axVjIKJzoKX7/vYAqMF/TYbFLVxAIKs18CKF3m+oFs=; b=AuEHW+lG4lrHlvO83QPDmCJjyyEvQBDs4vmmyd0kdvoJHTQSZeC/ccPPZCd2+N0Z0C5dsR d8ES+jDCtIX2rFJCcZVQcBVx3Q9MGqPZzE/FitCuf++ibQrpHbrctrH+Vk1Nb4scxs3VyG Hd/drPzyJorYDX70cwYhDoXKchk/Nn0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655653858; a=rsa-sha256; cv=none; b=QGntzomlUBCEdisi8gyYLGDiXHtwoz7X/SniF/++1IfQFtcFGZXislKcBnhMkJW//seCxh ICxcdlGV8A1b/fU20YMTOHU/C9yV8mCiGMd4C1FsYic1pNsVTUNT6m0LwQYgX2A0+O1zbX IdoE0V7tHAdsROvg9+D5thwV2wTv3gU= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=PuPpsRhr; spf=pass (imf28.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: c74w5bn9uj3btcu4uk4zmaazdupxy8kj Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=PuPpsRhr; spf=pass (imf28.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Queue-Id: 2EB46C0009 X-Rspamd-Server: rspam02 X-Rspam-User: X-HE-Tag: 1655653858-895542 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch introduces a helper to recharge the corresponding pages of a given percpu address. It is similar with how to recharge a kmalloc'ed address. Signed-off-by: Yafang Shao --- include/linux/percpu.h | 1 + mm/percpu.c | 98 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 99 insertions(+) diff --git a/include/linux/percpu.h b/include/linux/percpu.h index f1ec5ad1351c..e88429410179 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -128,6 +128,7 @@ extern void __init setup_per_cpu_areas(void); extern void __percpu *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp) __alloc_size(1); extern void __percpu *__alloc_percpu(size_t size, size_t align) __alloc_size(1); extern void free_percpu(void __percpu *__pdata); +bool recharge_percpu(void __percpu *__pdata, int step); extern phys_addr_t per_cpu_ptr_to_phys(void *addr); #define alloc_percpu_gfp(type, gfp) \ diff --git a/mm/percpu.c b/mm/percpu.c index 3633eeefaa0d..fd81f4d79f2f 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -2310,6 +2310,104 @@ void free_percpu(void __percpu *ptr) } EXPORT_SYMBOL_GPL(free_percpu); +#ifdef CONFIG_MEMCG_KMEM +bool recharge_percpu(void __percpu *ptr, int step) +{ + int bit_off, off, bits, size, end; + struct obj_cgroup *objcg_old; + struct obj_cgroup *objcg_new; + struct pcpu_chunk *chunk; + unsigned long flags; + void *addr; + + WARN_ON(!in_task()); + + if (!ptr) + return true; + + addr = __pcpu_ptr_to_addr(ptr); + spin_lock_irqsave(&pcpu_lock, flags); + chunk = pcpu_chunk_addr_search(addr); + off = addr - chunk->base_addr; + objcg_old = chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT]; + if (!objcg_old && step != MEMCG_KMEM_POST_CHARGE) { + spin_unlock_irqrestore(&pcpu_lock, flags); + return true; + } + + bit_off = off / PCPU_MIN_ALLOC_SIZE; + /* find end index */ + end = find_next_bit(chunk->bound_map, pcpu_chunk_map_bits(chunk), + bit_off + 1); + bits = end - bit_off; + size = bits * PCPU_MIN_ALLOC_SIZE; + + switch (step) { + case MEMCG_KMEM_PRE_CHARGE: + objcg_new = get_obj_cgroup_from_current(); + WARN_ON(!objcg_new); + if (obj_cgroup_charge(objcg_new, GFP_KERNEL, + size * num_possible_cpus())) { + obj_cgroup_put(objcg_new); + spin_unlock_irqrestore(&pcpu_lock, flags); + return false; + } + break; + case MEMCG_KMEM_UNCHARGE: + obj_cgroup_uncharge(objcg_old, size * num_possible_cpus()); + rcu_read_lock(); + mod_memcg_state(obj_cgroup_memcg(objcg_old), MEMCG_PERCPU_B, + -(size * num_possible_cpus())); + rcu_read_unlock(); + chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = NULL; + obj_cgroup_put(objcg_old); + break; + case MEMCG_KMEM_POST_CHARGE: + rcu_read_lock(); + chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = obj_cgroup_from_current(); + mod_memcg_state(mem_cgroup_from_task(current), MEMCG_PERCPU_B, + (size * num_possible_cpus())); + rcu_read_unlock(); + break; + case MEMCG_KMEM_CHARGE_ERR: + /* + * In case fail to charge to the new one in the pre charge state, + * for example, we have pre-charged one memcg successfully but fail + * to pre-charge the second memcg, then we should uncharge the first + * memcg. + */ + objcg_new = obj_cgroup_from_current(); + obj_cgroup_uncharge(objcg_new, size * num_possible_cpus()); + obj_cgroup_put(objcg_new); + rcu_read_lock(); + mod_memcg_state(obj_cgroup_memcg(objcg_new), MEMCG_PERCPU_B, + -(size * num_possible_cpus())); + rcu_read_unlock(); + + break; + } + + spin_unlock_irqrestore(&pcpu_lock, flags); + + return true; +} +EXPORT_SYMBOL(recharge_percpu); + +#else /* CONFIG_MEMCG_KMEM */ + +bool charge_percpu(void __percpu *ptr, bool charge) +{ + return true; +} +EXPORT_SYMBOL(charge_percpu); + +void uncharge_percpu(void __percpu *ptr) +{ +} +EXPORT_SYMBOL(uncharge_percpu); + +#endif /* CONFIG_MEMCG_KMEM */ + bool __is_kernel_percpu_address(unsigned long addr, unsigned long *can_addr) { #ifdef CONFIG_SMP From patchwork Sun Jun 19 15:50:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12886760 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF8B4CCA47A for ; Sun, 19 Jun 2022 15:51:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 890048D000D; Sun, 19 Jun 2022 11:51:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 77B408D0006; Sun, 19 Jun 2022 11:51:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 558098D000D; Sun, 19 Jun 2022 11:51:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 301438D0006 for ; Sun, 19 Jun 2022 11:51:00 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1402633808 for ; Sun, 19 Jun 2022 15:51:00 +0000 (UTC) X-FDA: 79595423880.23.A193C6C Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by imf28.hostedemail.com (Postfix) with ESMTP id B1FCCC0002 for ; Sun, 19 Jun 2022 15:50:59 +0000 (UTC) Received: by mail-pj1-f52.google.com with SMTP id cv13so5477093pjb.4 for ; Sun, 19 Jun 2022 08:50:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wupU3s6HDHFwU6sdzvBeIV5oXRSNq7jyoLqRbCJxYac=; b=ouUobzlqmq8S/Azfyps3usSVZ5UvHYpdVP1vu+gtd0yfCYf+Bzj6oDe+t+ss7KLBSf 49dFjjsYRk7VFsjhV3FODdJpDko/alFqgElQMsTdQ7rE3prZj2MIR3wBldkL29tmg8hY WdHG53fBco4KiIraG/2iAZgtIVMOvMVToF33Rx2Q9+oc4GQX/3Q5aq8Wo8P4nysP+w5e 2eUFBCQxRzJk2HamKYvCj0plIjvc0hhDGIC8v68RlNWj6n0rlOJa48Ff1QOKAuzhCW93 gS4S70/+InEZ+SY8uBkQpCQ9xW4bcbu79d4WCk/f6OGain7VFBSaQrtbwWMxyWF6uwKH 17fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wupU3s6HDHFwU6sdzvBeIV5oXRSNq7jyoLqRbCJxYac=; b=wm/UCnPs3bjWQutYF5aVBR5WzglgEo9D/Wxk6ILlPpUEFkqzYmoQk4lyPgy88NoDVO YmxtjvqAh7HDSaOr76tys6GpURQVWSRSSPCIcTtmn2wKDEVZTTjFM2Fzu8qDTrl5i2cx UfHv1G3TNiwQzUFp2aZCPoLoFrjIPtiiNXGynlmA6EyOmtmZQgojg1h0v/LHy1bIs+WW b5X7O7DLaDFsB/nf6ddytuA+29NOKowu+651kkNHw2jnSQFfKuKdZHbhBFEB0ShONKXY KF8oul7kAFrASsPPapzaGUHrm6sPpDH5TBPF8j/Q/U4LWyJDGCzwVdk13CDqO2o0ZR+b iK1Q== X-Gm-Message-State: AJIora9O6Uzu/wg1kg1PGG+PXkGbrwdlU7spEL0Fm8cGzwCYKzTjSdNq 47Mr6MWrGETj84asDrn/f0Y= X-Google-Smtp-Source: AGRyM1vEp6+rL+Wouq6U3Uc3dHqIACyRg+BpnCyHuM0NN5vXImRir63qLU15/bkknG3MjEs3GQ/aPw== X-Received: by 2002:a17:902:bb86:b0:169:caf:895c with SMTP id m6-20020a170902bb8600b001690caf895cmr16028267pls.13.1655653859261; Sun, 19 Jun 2022 08:50:59 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:2b24:5400:4ff:fe09:b144]) by smtp.gmail.com with ESMTPSA id z10-20020a1709027e8a00b001690a7df347sm6381761pla.96.2022.06.19.08.50.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jun 2022 08:50:58 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, quentin@isovalent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 08/10] bpf: Recharge memory when reuse bpf map Date: Sun, 19 Jun 2022 15:50:30 +0000 Message-Id: <20220619155032.32515-9-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220619155032.32515-1-laoar.shao@gmail.com> References: <20220619155032.32515-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655653859; a=rsa-sha256; cv=none; b=VfTgBuA+doud5TAy38aFt1hG9BjnOZ4+QmNBCOjG1beDYpBWvv7papUaV/cBk217nufunN lduildqbPER3IuQaIIuy9R1NYo4p39Wha4OXXyWEyxUpZIXepX/3LokRUGrNfhkGB0n2N3 Pu/UiNxeSGvTIgJ97rpRdkQ6SHmV7Po= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ouUobzlq; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.52 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655653859; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wupU3s6HDHFwU6sdzvBeIV5oXRSNq7jyoLqRbCJxYac=; b=LICXkpYF+fS3qqcNVppias9ZiL6zG8eJmOx1gY585BHfePAQD120mYJD2l56PbRCKn5Nz8 DVqzir1GSxU+fwmhK+JD6Bb2aXEbNYxyu2hb7dOKXvDFY3QfumieDg3vlD4InQFzydjrKk hbf3chp52fgtyFN6HSIzZYMfQNcQNz4= X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: B1FCCC0002 X-Stat-Signature: xhpfbde8j8mexpogfzrfmog1s14pyq11 X-Rspam-User: Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ouUobzlq; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.52 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com X-HE-Tag: 1655653859-450706 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When we reuse a pinned bpf map, if it belongs to a memcg which needs to be recharged, we will uncharge the pages of this bpf map from its original memcg and then charge its pages to the current memcg. We have to explicitly tell the kernel if it is a reuse path as the kernel can't detect it intelligently. That can be done in libbpf, then the user code don't need to be changed. Signed-off-by: Yafang Shao --- include/linux/bpf.h | 2 ++ include/uapi/linux/bpf.h | 2 +- kernel/bpf/syscall.c | 10 ++++++++++ tools/include/uapi/linux/bpf.h | 2 +- tools/lib/bpf/libbpf.c | 2 +- 5 files changed, 15 insertions(+), 3 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 0edd7d2c0064..b18a30e70507 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -152,6 +152,8 @@ struct bpf_map_ops { bpf_callback_t callback_fn, void *callback_ctx, u64 flags); + bool (*map_memcg_recharge)(struct bpf_map *map); + /* BTF id of struct allocated by map_alloc */ int *map_btf_id; diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index f2f658e224a7..ffbe15c1c8c6 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -6093,7 +6093,7 @@ struct bpf_map_info { __u32 btf_key_type_id; __u32 btf_value_type_id; __s8 memcg_state; - __s8 :8; /* alignment pad */ + __s8 memcg_recharge; __u16 :16; /* alignment pad */ __u64 map_extra; } __attribute__((aligned(8))); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index d4659d58d288..8817c40275f3 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -4170,12 +4170,22 @@ static int bpf_map_get_info_by_fd(struct file *file, #ifdef CONFIG_MEMCG_KMEM if (map->memcg) { + size_t offset = offsetof(struct bpf_map_info, memcg_recharge); struct mem_cgroup *memcg = map->memcg; + char recharge; if (memcg == root_mem_cgroup) info.memcg_state = 0; else info.memcg_state = memcg_need_recharge(memcg) ? -1 : 1; + + if (copy_from_user(&recharge, (char __user *)uinfo + offset, sizeof(char))) + return -EFAULT; + + if (recharge && memcg_need_recharge(memcg)) { + if (map->ops->map_memcg_recharge) + map->ops->map_memcg_recharge(map); + } } #endif diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index f2f658e224a7..ffbe15c1c8c6 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -6093,7 +6093,7 @@ struct bpf_map_info { __u32 btf_key_type_id; __u32 btf_value_type_id; __s8 memcg_state; - __s8 :8; /* alignment pad */ + __s8 memcg_recharge; __u16 :16; /* alignment pad */ __u64 map_extra; } __attribute__((aligned(8))); diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 49e359cd34df..f0eb67c983d8 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -4488,7 +4488,7 @@ int bpf_map__set_autocreate(struct bpf_map *map, bool autocreate) int bpf_map__reuse_fd(struct bpf_map *map, int fd) { - struct bpf_map_info info = {}; + struct bpf_map_info info = {.memcg_recharge = 1}; __u32 len = sizeof(info); int new_fd, err; char *new_name; From patchwork Sun Jun 19 15:50:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12886761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40C5BC43334 for ; Sun, 19 Jun 2022 15:51:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CD3588D000E; Sun, 19 Jun 2022 11:51:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C5C418D0006; Sun, 19 Jun 2022 11:51:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AAEF48D000E; Sun, 19 Jun 2022 11:51:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 98BD08D0006 for ; Sun, 19 Jun 2022 11:51:02 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 736F03461F for ; Sun, 19 Jun 2022 15:51:02 +0000 (UTC) X-FDA: 79595423964.24.6EF6508 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by imf26.hostedemail.com (Postfix) with ESMTP id 1E935140096 for ; Sun, 19 Jun 2022 15:51:01 +0000 (UTC) Received: by mail-pj1-f54.google.com with SMTP id k5-20020a17090a404500b001e8875e6242so8109923pjg.5 for ; Sun, 19 Jun 2022 08:51:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cwCrlHiRqBK3TQXbqbFOzQOTzh9qCJzbmrIFhJw7nPY=; b=gG5XX7PqlXa2X73375TmgCa4Z4pHjtha1wTFLorvCMGCCG36m1IsgvAHVnj/+kxo0i bWjyjchH3NmfZsW8qb5crT1xlS7txgjFZBq6XHUtzm5v7+d5b1Jfe7SBqP3xa2NivqE3 F7nM5p8L3hCk7zsN+8OlXjO49kBQox6Pjc22I8Nfqgo9hUW/SzxD8+SCypdijeEFIQcz 9zyiIDcWGRXH1RgYtszLyCKiIQZRixcRQZs+NkJIGA8ynrZxUXI3prYBc1yQQlqwKI3v sYHP1GcXC93i3jEvjKVjPrOrB8b6Ukbxq90/8tq+kV1wG7hFeFgZ7l+K62H6HbTNaP9u PXdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cwCrlHiRqBK3TQXbqbFOzQOTzh9qCJzbmrIFhJw7nPY=; b=IxL4/oJAOv59e8CGcemNUszGPYN6S2g57GIE67sIJxxEdUKcKht13cmffHmwEyUxT6 CnEdeh2g9sJaxsyCer3mNisK6HfGZVZf2Kk/lqSMpO+uukGOyaaHgrjgCt7i403iwdPN mVBa0M9MFIoy+7JQuLIxRVR901g/E9Mfb9sIIt7nAdfOZjv1EMJpAKPWxyoUSffRV62w 3vgiS4K+bUibKQXGyWXV+LqCVQ9VyB+9ARXPGW4Uw9ADXOMmZm7Ulg1lfsKArs5FM25m PRGI5O7xL6edUd/m5XvIc6F3JZgy7k7gEV0/mokJU7mlia6vuQO2L1YrNran52tkXl6Y Ibrw== X-Gm-Message-State: AJIora/2UZC2NOX1sNdrI/Iq27NS9CNAOtaDeTRKVSxz9rcsZB1s6+qY x3pbTngbSFTPS4gywZyqox4= X-Google-Smtp-Source: AGRyM1vAxtpulJ0ehxCQm0vNdQXfcu7UMbEBF0DvHPLPdUMcTffedr4blykuzh6+o2/NsoGPcNcR7w== X-Received: by 2002:a17:90a:408f:b0:1d1:d1ba:2abb with SMTP id l15-20020a17090a408f00b001d1d1ba2abbmr32595817pjg.152.1655653861217; Sun, 19 Jun 2022 08:51:01 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:2b24:5400:4ff:fe09:b144]) by smtp.gmail.com with ESMTPSA id z10-20020a1709027e8a00b001690a7df347sm6381761pla.96.2022.06.19.08.50.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jun 2022 08:51:00 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, quentin@isovalent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 09/10] bpf: Make bpf_map_{save, release}_memcg public Date: Sun, 19 Jun 2022 15:50:31 +0000 Message-Id: <20220619155032.32515-10-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220619155032.32515-1-laoar.shao@gmail.com> References: <20220619155032.32515-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655653862; a=rsa-sha256; cv=none; b=PWkHdgYOB+aJv9IfHMz9IV01bNftexWLXC16wVymtKVVHk1pQ4M55vgIGb6TkWgZvwbgoB XhVGbNK5Tpcy34BdwhYYYx1gfvD1aR+EpDrLQ0eVpJAUZq0+SCLCpzY1GHuPCJ+UzVqgoF iO3a3/anpdZLBhAUWdBJCGo9++FhBL8= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=gG5XX7Pq; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf26.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.54 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655653862; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cwCrlHiRqBK3TQXbqbFOzQOTzh9qCJzbmrIFhJw7nPY=; b=klgpj4sxmrtKwfA4pxylJWvlrm8J3ltazW7HAXZZEwpD6bZjGh1fu9FKwlgwoDI/+ewvUL rDS/r5jF86P6Y13VvwxM0axe71YiRdxZc7i5HSQ51pjRM9uFW+ZvwthoGqFmN4f1fMQIN3 WRxgctBLrRygq12ycLOgF6NPb3Zs9LM= X-Rspamd-Queue-Id: 1E935140096 X-Rspam-User: Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=gG5XX7Pq; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf26.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.54 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com X-Rspamd-Server: rspam06 X-Stat-Signature: ndjdtq8ia4eaa5u7kmcjawepsubpxj64 X-HE-Tag: 1655653861-184825 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: These two helpers will be used in map specific files later. Signed-off-by: Yafang Shao --- include/linux/bpf.h | 21 +++++++++++++++++++++ kernel/bpf/syscall.c | 19 ------------------- 2 files changed, 21 insertions(+), 19 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index b18a30e70507..a0f21d4382ff 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -27,6 +27,7 @@ #include #include #include +#include struct bpf_verifier_env; struct bpf_verifier_log; @@ -248,6 +249,26 @@ struct bpf_map { bool frozen; /* write-once; write-protected by freeze_mutex */ }; +#ifdef CONFIG_MEMCG_KMEM +static inline void bpf_map_save_memcg(struct bpf_map *map) +{ + map->memcg = get_mem_cgroup_from_mm(current->mm); +} + +static inline void bpf_map_release_memcg(struct bpf_map *map) +{ + mem_cgroup_put(map->memcg); +} +#else +static inline void bpf_map_save_memcg(struct bpf_map *map) +{ +} + +static inline void bpf_map_release_memcg(struct bpf_map *map) +{ +} +#endif + static inline bool map_value_has_spin_lock(const struct bpf_map *map) { return map->spin_lock_off >= 0; diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 8817c40275f3..5159b97d1064 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -417,16 +417,6 @@ void bpf_map_free_id(struct bpf_map *map, bool do_idr_lock) } #ifdef CONFIG_MEMCG_KMEM -static void bpf_map_save_memcg(struct bpf_map *map) -{ - map->memcg = get_mem_cgroup_from_mm(current->mm); -} - -static void bpf_map_release_memcg(struct bpf_map *map) -{ - mem_cgroup_put(map->memcg); -} - void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, int node) { @@ -464,15 +454,6 @@ void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, return ptr; } - -#else -static void bpf_map_save_memcg(struct bpf_map *map) -{ -} - -static void bpf_map_release_memcg(struct bpf_map *map) -{ -} #endif static int bpf_map_kptr_off_cmp(const void *a, const void *b) From patchwork Sun Jun 19 15:50:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12886762 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18205C433EF for ; Sun, 19 Jun 2022 15:51:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AAFB88D000F; Sun, 19 Jun 2022 11:51:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A60018D0006; Sun, 19 Jun 2022 11:51:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 901398D000F; Sun, 19 Jun 2022 11:51:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7F5DB8D0006 for ; Sun, 19 Jun 2022 11:51:09 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 609CD34A3D for ; Sun, 19 Jun 2022 15:51:09 +0000 (UTC) X-FDA: 79595424258.10.5E96B31 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf29.hostedemail.com (Postfix) with ESMTP id 17D6012000A for ; Sun, 19 Jun 2022 15:51:03 +0000 (UTC) Received: by mail-pl1-f176.google.com with SMTP id m2so492213plx.3 for ; Sun, 19 Jun 2022 08:51:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QAskb+5yNIJsaldLr50aWhyLhYS5r5QAog9O7C8QAr0=; b=KVtvmyag+k7GB/fCkk8AqkzTGDWXZtmlQ+PT8NnSmeq2wzVE6RIirW/ZsVSL0WWl65 r/F0p777dkMTZPtEYW0XSiRRz5FeDX1WCndPeYgV5jvIuU4HH1HmzNOprr3x38QJih41 /DVbvpKY2WZujqIpOlmgwxyc+85XJbvqSIsK1n2vY4Jo67si1AG5+OHzdizLQQEHW700 uWHc1PExNAaByKT6ttxpzOxXL+OeK0CLn1S7pL0mI73iHpByUDSEephJTCx7J1VJeacY TTjatlqZe7ts+Vs2hs68NR/gdLC+OZnyxmvFQ2Hx3FZhcKdTpBFYsXDEU2cpSj5g5/6l UeHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QAskb+5yNIJsaldLr50aWhyLhYS5r5QAog9O7C8QAr0=; b=uHnBfszCygN6CaZstG0IXAos0ozLgbNg8EdWhMNoWiiACimGU0kX5wV2OZn3UCh+jR 64NLspiUiqR8yySKbpw+f0QjbPl6j+aMSQYab4bIQJTWhg2fV5fKlAge4rcyCEt5Kf+y Im8+xuPX8215B+VXHB6LFRFhGQezTQZUBOixppU9cKR604zslaNrETJIC3hnxjZu36Ea NOANbi5L93KPArAQqNAhR5aeUKOjWHWUf1vy6X4bJ7JNVgcBxYVF8wWsc3ytA2Maz/5y G5birqf+BKTS35JJI7dMPyBAPaimB4xqd0GA+T94eSHRSUhR4ctOPaC+RMyk0kawX/+N sgtg== X-Gm-Message-State: AJIora9G3bKfLP4Y3lIia47KiXs6q/VOB4tpOimJKlBrQOYzTCwexs2G DNme74ysErH2FVfUPx81yyvCc/5OHHQolDJUISU= X-Google-Smtp-Source: AGRyM1tZi5Zt0G1PIz0scQfO4Nr+NU/o7ujG423ZsaI0GC2fomWJFdewsZiWfRB2CETXpaFwWsKygw== X-Received: by 2002:a17:902:b597:b0:168:d8ce:4a63 with SMTP id a23-20020a170902b59700b00168d8ce4a63mr19745970pls.57.1655653863216; Sun, 19 Jun 2022 08:51:03 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:2b24:5400:4ff:fe09:b144]) by smtp.gmail.com with ESMTPSA id z10-20020a1709027e8a00b001690a7df347sm6381761pla.96.2022.06.19.08.51.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jun 2022 08:51:02 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, quentin@isovalent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 10/10] bpf: Support recharge for hash map Date: Sun, 19 Jun 2022 15:50:32 +0000 Message-Id: <20220619155032.32515-11-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220619155032.32515-1-laoar.shao@gmail.com> References: <20220619155032.32515-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655653864; a=rsa-sha256; cv=none; b=yjcvzGy32VUthjQkQMbUhYDpv7MK66S5nmtUF3PCNyL6npZcA6dGz+/0WiWzqcmE3cRXvv 2Xl5lreEAhuS7+zMSFcakc/LbXma90CSmpQm4CKaojeErmVUEOebMq6YiveCaE2rslRD+j XEmYZ8iMdFYZt884LMyhL0MHGCstD+Y= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=KVtvmyag; spf=pass (imf29.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655653864; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QAskb+5yNIJsaldLr50aWhyLhYS5r5QAog9O7C8QAr0=; b=obcSZ0YW/nPHNNuqLYajDzaYtFIZQOlx9xCIhlASZXqGDBOt9enUJxypAhGJJpuhU6hmYx n7hLcFD1DpUbkT7Vl1yvLOUKxiUg6i0AYar0Ja1oWgeUrvhuairNFXaVEq7RM1qb+00MXs 2/982OBeHB/9sHhKtBhZ7sinzZAMO+c= X-Stat-Signature: b3ocojrsmkjinhk4dazggpax5fgix48o X-Rspamd-Queue-Id: 17D6012000A Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=KVtvmyag; spf=pass (imf29.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam07 X-Rspam-User: X-HE-Tag: 1655653863-199894 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch introduces a helper to recharge pages of a hash map. We have already known how the hash map is allocated and freed, we can also know how to charge and uncharge the hash map. Firstly, we need to pre charge to the new memcg, if the pre charge successes then we uncharge from the old memcg. Finnaly we do the post charge to the new memcg, in which we will modify the counter in memcgs. Signed-off-by: Yafang Shao --- kernel/bpf/hashtab.c | 74 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 17fb69c0e0dc..fe61976262ee 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -11,6 +11,7 @@ #include #include #include +#include #include "percpu_freelist.h" #include "bpf_lru_list.h" #include "map_in_map.h" @@ -1499,6 +1500,75 @@ static void htab_map_free(struct bpf_map *map) kfree(htab); } +#ifdef CONFIG_MEMCG_KMEM +static bool htab_map_memcg_recharge(struct bpf_map *map) +{ + struct bpf_htab *htab = container_of(map, struct bpf_htab, map); + struct mem_cgroup *old = map->memcg; + int i; + + /* + * Although the bpf map's offline memcg has been reparented, there + * is still reference on it, so it is safe to be accessed. + */ + if (!old) + return false; + + /* Pre charge to the new memcg */ + if (!krecharge(htab, MEMCG_KMEM_PRE_CHARGE)) + return false; + + if (!kvrecharge(htab->buckets, MEMCG_KMEM_PRE_CHARGE)) + goto out_k; + + if (!recharge_percpu(htab->extra_elems, MEMCG_KMEM_PRE_CHARGE)) + goto out_kv; + + for (i = 0; i < HASHTAB_MAP_LOCK_COUNT; i++) { + if (!recharge_percpu(htab->map_locked[i], MEMCG_KMEM_PRE_CHARGE)) + goto out_p; + } + + /* Uncharge from the old memcg. */ + krecharge(htab, MEMCG_KMEM_UNCHARGE); + kvrecharge(htab->buckets, MEMCG_KMEM_UNCHARGE); + recharge_percpu(htab->extra_elems, MEMCG_KMEM_UNCHARGE); + for (i = 0; i < HASHTAB_MAP_LOCK_COUNT; i++) + recharge_percpu(htab->map_locked[i], MEMCG_KMEM_UNCHARGE); + + /* Release the old memcg */ + bpf_map_release_memcg(map); + + /* Post charge to the new memcg */ + krecharge(htab, MEMCG_KMEM_POST_CHARGE); + kvrecharge(htab->buckets, MEMCG_KMEM_POST_CHARGE); + recharge_percpu(htab->extra_elems, MEMCG_KMEM_POST_CHARGE); + for (i = 0; i < HASHTAB_MAP_LOCK_COUNT; i++) + recharge_percpu(htab->map_locked[i], MEMCG_KMEM_POST_CHARGE); + + /* Save the new memcg */ + bpf_map_save_memcg(map); + + return true; + +out_p: + for (; i > 0; i--) + recharge_percpu(htab->map_locked[i], MEMCG_KMEM_CHARGE_ERR); + recharge_percpu(htab->extra_elems, MEMCG_KMEM_CHARGE_ERR); +out_kv: + kvrecharge(htab->buckets, MEMCG_KMEM_CHARGE_ERR); +out_k: + krecharge(htab, MEMCG_KMEM_CHARGE_ERR); + + return false; +} +#else +static bool htab_map_memcg_recharge(struct bpf_map *map) +{ + return true; +} +#endif + static void htab_map_seq_show_elem(struct bpf_map *map, void *key, struct seq_file *m) { @@ -2152,6 +2222,7 @@ const struct bpf_map_ops htab_map_ops = { .map_alloc_check = htab_map_alloc_check, .map_alloc = htab_map_alloc, .map_free = htab_map_free, + .map_memcg_recharge = htab_map_memcg_recharge, .map_get_next_key = htab_map_get_next_key, .map_release_uref = htab_map_free_timers, .map_lookup_elem = htab_map_lookup_elem, @@ -2172,6 +2243,7 @@ const struct bpf_map_ops htab_lru_map_ops = { .map_alloc_check = htab_map_alloc_check, .map_alloc = htab_map_alloc, .map_free = htab_map_free, + .map_memcg_recharge = htab_map_memcg_recharge, .map_get_next_key = htab_map_get_next_key, .map_release_uref = htab_map_free_timers, .map_lookup_elem = htab_lru_map_lookup_elem, @@ -2325,6 +2397,7 @@ const struct bpf_map_ops htab_percpu_map_ops = { .map_alloc_check = htab_map_alloc_check, .map_alloc = htab_map_alloc, .map_free = htab_map_free, + .map_memcg_recharge = htab_map_memcg_recharge, .map_get_next_key = htab_map_get_next_key, .map_lookup_elem = htab_percpu_map_lookup_elem, .map_lookup_and_delete_elem = htab_percpu_map_lookup_and_delete_elem, @@ -2344,6 +2417,7 @@ const struct bpf_map_ops htab_lru_percpu_map_ops = { .map_alloc_check = htab_map_alloc_check, .map_alloc = htab_map_alloc, .map_free = htab_map_free, + .map_memcg_recharge = htab_map_memcg_recharge, .map_get_next_key = htab_map_get_next_key, .map_lookup_elem = htab_lru_percpu_map_lookup_elem, .map_lookup_and_delete_elem = htab_lru_percpu_map_lookup_and_delete_elem,