From patchwork Thu Aug 18 14:31:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12948098 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37CD4C00140 for ; Thu, 18 Aug 2022 22:39:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C48068E0001; Thu, 18 Aug 2022 18:39:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BF7518D0002; Thu, 18 Aug 2022 18:39:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ABF948E0001; Thu, 18 Aug 2022 18:39:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9E0E38D0002 for ; Thu, 18 Aug 2022 18:39:23 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7018912202A for ; Thu, 18 Aug 2022 22:39:23 +0000 (UTC) X-FDA: 79814181006.16.CA34094 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by imf28.hostedemail.com (Postfix) with ESMTP id C88D1C995A for ; Thu, 18 Aug 2022 22:08:08 +0000 (UTC) Received: by mail-pf1-f178.google.com with SMTP id y141so2740510pfb.7 for ; Thu, 18 Aug 2022 15:08:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=7uI5yxQ2IficxCafBouZLQeDm3f9EVu1uBy3AFKC6gc=; b=efDyAXQ6g+bmM4mO4s9FEnX/G1io8w/Sb/p1agZBS8OYl+4qzSpqCOAmCv3KAH464o yqXHdmbv31M0qS0KPcZ+AfpgUlngTmPnP1urpMeb2TYvCM7sALpSLDshkSIVSeLrLRoF 9QRaSQEbHv1F0yKOy1EsV9/6nosGOpeu+xYijhMJZB5j2CwYWyjafJJPw/2wgoPOAmX9 LusMGoVPGPBLhQCbR4KB1vFQXYKGb7ro+K46rcWNOd67jWtcnaicVx8TbJZBi/u+VF8e xfkjVg6/1RxyOP6+vO8uXbp3X4dJAawiUTncBQeH7ADAUdm0lWBNB61oEHpzTJH5pQ32 5q8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=7uI5yxQ2IficxCafBouZLQeDm3f9EVu1uBy3AFKC6gc=; b=d9Y41AX7d6sM99XebDNBExdy7x6kF3ODD5Lk5IDtpSc9sTGOKZW7rweEHH3IUtE3Gp AT7E6SV6P+605N/MYnCNSA423FSOyVb9GGt0pQMoF6/q/nvJ9U+2bri5zY9APL57VVwj S7/xGPYIqRM9DDVszfvKEdLx/IWkI+MsTqDVbv/81gYITI3Kmin136nq8/F0Brg4yC77 LnYt9eYm+vEsR1/G48ID7oC1eucygAjo22XxK7VT5e4Jz617C8GzncGiky6biEyE+ho8 vOFWyAkV6kLsX/y/C+fieAEDzQ6G1Isr6pPBd4m0NzYXHShXPt2hKKdnYEXwHRWSTfxl Eghw== X-Gm-Message-State: ACgBeo0Na88t+bi4A2FCWPfunLc0q4j7HSkkaDUAoHz9OMIVo+9SJEuw 4j+dca+BDL37SnBgNdhHzCw7vxTsrquSICLN X-Google-Smtp-Source: AA6agR4cpUjkd3sgji2FqH6ByueAGCTReVusgTDZMfiRqYfoSDWjRgIK3pIyCMwyH5XK6YNRGb4yrg== X-Received: by 2002:a63:d652:0:b0:41c:45e9:abee with SMTP id d18-20020a63d652000000b0041c45e9abeemr2763443pgj.110.1660833098487; Thu, 18 Aug 2022 07:31:38 -0700 (PDT) Received: from vultr.guest ([45.32.72.237]) by smtp.gmail.com with ESMTPSA id h5-20020a63f905000000b003fdc16f5de2sm1379124pgi.15.2022.08.18.07.31.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Aug 2022 07:31:37 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao , Yosry Ahmed Subject: [PATCH bpf-next v2 01/12] cgroup: Update the comment on cgroup_get_from_fd Date: Thu, 18 Aug 2022 14:31:07 +0000 Message-Id: <20220818143118.17733-2-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220818143118.17733-1-laoar.shao@gmail.com> References: <20220818143118.17733-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660860489; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7uI5yxQ2IficxCafBouZLQeDm3f9EVu1uBy3AFKC6gc=; b=XYCQlhl1zvQsci9ea7pPo0ST2VSunyCYFp2Q/y0fuymXVb+PZvFGX3ZQwqls9c7EQ+2GWn C863gDZhmIuRfYBl6z6Se8SL+UUwJRZEls5Ef3543C/MgyQPp22g8D8Y8kTK6Yz6MU+jsj oBq6/7iSHYlUJG3/3fd9tQ7/XFgx4S8= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=efDyAXQ6; spf=pass (imf28.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660860489; a=rsa-sha256; cv=none; b=rShsN3wpQmi154LkWQaNCcUzxMYO4iUS57fXZ/cVwq1BsJDfZxGqW9laivAe5OLgN2zdCw Kk2YjAA8AOMsdgVEzCcCFv9JdF93fDx9Zu5bqEI2jgudMjg9X/wEDP81IqE8BbxC3DQLlE pAynDFrK7jojOs+18dAH+Ip/MXFY4qQ= X-Stat-Signature: xq1rnq4hwu97zehwxz5cioji7axw6y7u X-Rspam-User: Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=efDyAXQ6; spf=pass (imf28.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: C88D1C995A X-HE-Tag: 1660860488-97785 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: After commit f3a2aebdd6fb ("cgroup: enable cgroup_get_from_file() on cgroup1") we can open a cgroup1 dir as well. So let's update the comment. Cc: Yosry Ahmed Cc: Hao Luo Cc: Tejun Heo Signed-off-by: Yafang Shao --- kernel/cgroup/cgroup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index 5f4502a..b7d2e55 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -6625,7 +6625,7 @@ struct cgroup *cgroup_get_from_path(const char *path) /** * cgroup_get_from_fd - get a cgroup pointer from a fd - * @fd: fd obtained by open(cgroup2_dir) + * @fd: fd obtained by open(cgroup_dir) * * Find the cgroup from a fd which should be obtained * by opening a cgroup directory. Returns a pointer to the From patchwork Thu Aug 18 14:31:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12948093 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B92CEC00140 for ; Thu, 18 Aug 2022 22:35:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 568978E0002; Thu, 18 Aug 2022 18:35:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F1068D0002; Thu, 18 Aug 2022 18:35:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3441B8E0002; Thu, 18 Aug 2022 18:35:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 241C78D0002 for ; Thu, 18 Aug 2022 18:35:41 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id AEE631C797F for ; Thu, 18 Aug 2022 22:35:40 +0000 (UTC) X-FDA: 79814171640.03.B6E03D4 Received: by imf22.hostedemail.com (Postfix, from userid 200) id 87AF9C9B5A; Thu, 18 Aug 2022 16:28:06 +0000 (UTC) Received: from mail-oa1-f53.google.com (mail-oa1-f53.google.com [209.85.160.53]) by imf22.hostedemail.com (Postfix) with ESMTP id C30B0C577C for ; Thu, 18 Aug 2022 16:27:00 +0000 (UTC) Received: by mail-oa1-f53.google.com with SMTP id 586e51a60fabf-11c896b879bso2361517fac.3 for ; Thu, 18 Aug 2022 09:26:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=wg2OeZwYg3Kz1nqXHwyNkbleHxYXQ5NXoZNmpCP8Llk=; b=R/So91AyylEMxXSEpTKpKv01qBGi6xOrsO4noHScLMrE+YO98wrVs063FgNuxiln9E dEOV2CN+RE5tNE1z5ZmERPvt6JN3yJolI3nO4D7eoOwHzHR3S3GwYf8gAP+Zp3dVkc1h Z6OgTgdGTt8p3N2cLkdNOMMqY7MRDmGpFg/mZ7MdHtfXixdsajHvr/B5hsZe4b+cNEpC c55C3O6Uh+Yd6+uu0O6olhxAZYTjxgcFlVzx7dW+WaNGl7QbddUZQSwdmV01ba77Jhdd HJ0cHxk3jJ7BaszZQWwQXr5U4CksixminXvUIPa/uoHfx9aVA/VbkVu91OGPozK0RZSh u7rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=wg2OeZwYg3Kz1nqXHwyNkbleHxYXQ5NXoZNmpCP8Llk=; b=Zc9owiTpjRbfNXcKeOUv8FYJ3OFVsrkytjH4JHt5oCUyFw7abpMQ2zDJ3pabaI1v1U Yl6IrXYWqU/nCU54rFlvmlv8fSQ258gMB3d+RJZWhEgCIp8dpwgQ7tSl6cABRzBDUYE7 hSs5iBKDqv2x5Ki4NvUfktCYsTA5H1j4WVRmko3+uV6tZmWfHpCo8qNZU8TaYa9y4IID RYd/z/Kpfgn4FV/R89i0SU368/RV23XBpkS5qX4csxnjzD1oPLhIPqGOGlZIWP9lVl9O K0VSEFA4csnK2mgRi2txy4SyKTyHSOgl4R7Yrms55K3mt3cRkMPaRvmMtrPUlUhQm/td jTLA== X-Gm-Message-State: ACgBeo0FxKhBVRzZ2UEcpf517BuYkZLLkzAEtJ4hycoG+1RN06GSYPUU 55I+c3/2a7Esj8zDb+xVifUPqfManD4qElvy X-Google-Smtp-Source: AA6agR7OEELPz4uROPaJyiezq+3PyARpNOx7by3N5XsqNh2UAUDM9ERvl5GthWrBFRaYDeIe9ZSj0Q== X-Received: by 2002:a17:90b:3889:b0:1f5:88cd:350d with SMTP id mu9-20020a17090b388900b001f588cd350dmr9116900pjb.9.1660833102811; Thu, 18 Aug 2022 07:31:42 -0700 (PDT) Received: from vultr.guest ([45.32.72.237]) by smtp.gmail.com with ESMTPSA id h5-20020a63f905000000b003fdc16f5de2sm1379124pgi.15.2022.08.18.07.31.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Aug 2022 07:31:41 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v2 02/12] bpf: Introduce new helper bpf_map_put_memcg() Date: Thu, 18 Aug 2022 14:31:08 +0000 Message-Id: <20220818143118.17733-3-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220818143118.17733-1-laoar.shao@gmail.com> References: <20220818143118.17733-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660840083; a=rsa-sha256; cv=none; b=pRR37AkHiDHSCzWrobyj3kdZl17jPX0XAZA5MQhQwKuKVyghEfRlfagVgnKtbEJRN3ouM5 2zlT2lvyT9izC7gcn/L+Geic/7gnqhiz4WvPu32kIUxC+4vOxALJbdqTRG0Ms9ZNCjOjyX q4srCFBzXa5WDU6hRHaS4y5IzUD3xZ0= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="R/So91Ay"; spf=pass (imf22.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.53 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660840083; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wg2OeZwYg3Kz1nqXHwyNkbleHxYXQ5NXoZNmpCP8Llk=; b=KO56zUzAFHnjOmUZ/ii1so6gTQf6SroEH6+swuVfWKVoPxRGodczw48LdrrUBu64/wVClI QiVtxPaUHxdUVgwxN/RsCCx37lFSGUpNtIi1crvuqEXJjvSq0EgzkD4TFyItzXCY3Xxrx4 +PN/kbgHi+GddksG5MqWPwzmsP/MEg0= X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: C30B0C577C Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="R/So91Ay"; spf=pass (imf22.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.53 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: 4edxuouyirgcio6ojsdc43wubtxi7ipn X-Rspam-User: X-HE-Tag: 1660840020-121301 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace the open-coded mem_cgroup_put() with a new helper bpf_map_put_memcg(). That could make it more clear. Signed-off-by: Yafang Shao --- kernel/bpf/syscall.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 83c7136..2f18ae2 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -441,6 +441,11 @@ static struct mem_cgroup *bpf_map_get_memcg(const struct bpf_map *map) return root_mem_cgroup; } +static void bpf_map_put_memcg(struct mem_cgroup *memcg) +{ + mem_cgroup_put(memcg); +} + void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, int node) { @@ -451,7 +456,7 @@ void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, old_memcg = set_active_memcg(memcg); ptr = kmalloc_node(size, flags | __GFP_ACCOUNT, node); set_active_memcg(old_memcg); - mem_cgroup_put(memcg); + bpf_map_put_memcg(memcg); return ptr; } @@ -465,7 +470,7 @@ void *bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags) old_memcg = set_active_memcg(memcg); ptr = kzalloc(size, flags | __GFP_ACCOUNT); set_active_memcg(old_memcg); - mem_cgroup_put(memcg); + bpf_map_put_memcg(memcg); return ptr; } @@ -480,7 +485,7 @@ void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, old_memcg = set_active_memcg(memcg); ptr = __alloc_percpu_gfp(size, align, flags | __GFP_ACCOUNT); set_active_memcg(old_memcg); - mem_cgroup_put(memcg); + bpf_map_put_memcg(memcg); return ptr; } From patchwork Thu Aug 18 14:31:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12948112 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C047C00140 for ; Thu, 18 Aug 2022 23:01:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E4C4F8D0003; Thu, 18 Aug 2022 19:01:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DD5108D0002; Thu, 18 Aug 2022 19:01:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C4F008D0003; Thu, 18 Aug 2022 19:01:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B23248D0002 for ; Thu, 18 Aug 2022 19:01:06 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9169941884 for ; Thu, 18 Aug 2022 23:01:06 +0000 (UTC) X-FDA: 79814235732.27.26CDF7D Received: by imf27.hostedemail.com (Postfix, from userid 200) id 2E9FD41C78; Thu, 18 Aug 2022 16:13:15 +0000 (UTC) Received: from mail-qv1-f43.google.com (mail-qv1-f43.google.com [209.85.219.43]) by imf27.hostedemail.com (Postfix) with ESMTP id 21E50444AE for ; Thu, 18 Aug 2022 16:13:04 +0000 (UTC) Received: by mail-qv1-f43.google.com with SMTP id d1so1550589qvs.0 for ; Thu, 18 Aug 2022 09:13:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=51S9dT2G3aXePCnrE/PN1X8bOnyFVMn8Ux5epyAQWQw=; b=Snljo/+katzYm1S8ojYVqCaSaKESK1ljndsaJlzs3c8AhsrzYbRv9I6T1bQBxxgppR 7T6YPv9CGg2pRiOTwx3jg27WxnB2YzkNqvaGBFMBO3Fw6+lCfz4SR3H/Cpmf/xlPbMaw J2aNdyXVOC5eliet6WkG0q/QNdHtRprClQoOLyotQNi7Vi2ynk2mepc4SAZtc773bHyh o9ykJMmMhaQz8teEky3Vke6Xn8Ppy5/K3jtvJXBpc2ssuO0anMN7Dpuwo312FLtSlW6B 3vZ57EgRgMqllC8fRv5gzJQZS4151OfxP0rQE/n+geleYz4CO38s8n3Uo8ai/1Jevdf8 ipTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=51S9dT2G3aXePCnrE/PN1X8bOnyFVMn8Ux5epyAQWQw=; b=2LzImwlRHA+6XRG4HNhHnG3WldStBomjYR9qzc+cF/7pQAZYMCB+s1S+IAfY2d4eja 2jdunmmAT05IGIcXeol9Bbe2htITLzoyQ++sjUVkZphX9zZFINk2UmIOFD0nONEIKvae pSoBb1PUzi6vTBSuoEvAEZJEIMJOkX8drLDT3UBMM17BsKKXfhuvz50hm7t6NCALd2FE yomD4D8AtI7c0mj33bnFBlr0Q6+DS7vR5OgQFSV4mbB+6Asor2CqCFomd+RzlTAVYAgG M5FfgP04c+YSv6PjI//hG0ashHyQSxKok69ZoIcx4Nn+KJsVw789EzRfJurbNlsw2FO1 sJlQ== X-Gm-Message-State: ACgBeo3h1NG6SJforB4C+D0XVH/FiVPUurjjG/6wGXP7NAZKVj3e8Gsx N8y9LWDmZzcFERQIzX0T2VNWKs9aWTHSsgwQ X-Google-Smtp-Source: AA6agR4/IcIPwFl8NjZf201BmyQdHaE6fvOSIZxi9N2e4RlcPtMrgMGi0rFoxp19D9oRoi7uY7hWEg== X-Received: by 2002:a05:6a00:3691:b0:535:d465:45c5 with SMTP id dw17-20020a056a00369100b00535d46545c5mr2313132pfb.30.1660833105886; Thu, 18 Aug 2022 07:31:45 -0700 (PDT) Received: from vultr.guest ([45.32.72.237]) by smtp.gmail.com with ESMTPSA id h5-20020a63f905000000b003fdc16f5de2sm1379124pgi.15.2022.08.18.07.31.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Aug 2022 07:31:45 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v2 03/12] bpf: Define bpf_map_{get,put}_memcg for !CONFIG_MEMCG_KMEM Date: Thu, 18 Aug 2022 14:31:09 +0000 Message-Id: <20220818143118.17733-4-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220818143118.17733-1-laoar.shao@gmail.com> References: <20220818143118.17733-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660839191; a=rsa-sha256; cv=none; b=kIgU2Ede1epMmncELbvSx7q01GpIsHtWCtUjtLy4WSu2xk1SZFYiNo0qol7pFgMIlfYAhl KwzfpDJ4ge3wQf+GBoSa2As0H4Nsu/vPI1m5ep9Auvdut++rlCF7sdh/PMjz6mEGHkmoZr UtZ5btll96RI1kJ5oJ99bkHXtvFJT6g= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="Snljo/+k"; spf=pass (imf27.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.219.43 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660839191; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=51S9dT2G3aXePCnrE/PN1X8bOnyFVMn8Ux5epyAQWQw=; b=GBXGMR61hhVUHBbeZ5dxta1HMy7mEtJ6BOiQDKWWGOIQ7/NI7HXFEoazd/R5OHIcLWxq4l 42ZvXUoQtQL/fS5ZeGa7tPQ9wl8RDkk/0gXHWGE7xpmUrnPLxNLZN2pVshYvCqxkDTbiKL sQHOkpRn/B5LKBGHNc/iE/ObmMu1zuE= Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="Snljo/+k"; spf=pass (imf27.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.219.43 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Rspamd-Queue-Id: 21E50444AE X-Rspamd-Server: rspam04 X-Stat-Signature: w3m7fj9xmn1cbw47xkhcu3656qa9ug3q X-HE-Tag: 1660839184-665994 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We can use this helper when CONFIG_MEMCG_KMEM or CONFIG_MEMCG is not set. It also moves bpf_map_{get,put}_memcg into include/linux/bpf.h, so these two helpers can be used in other source files. Signed-off-by: Yafang Shao --- include/linux/bpf.h | 26 ++++++++++++++++++++++++++ include/linux/memcontrol.h | 10 ++++++++++ kernel/bpf/syscall.c | 13 ------------- 3 files changed, 36 insertions(+), 13 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index a627a02..ded7d23 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -27,6 +27,7 @@ #include #include #include +#include struct bpf_verifier_env; struct bpf_verifier_log; @@ -2572,4 +2573,29 @@ static inline void bpf_cgroup_atype_get(u32 attach_btf_id, int cgroup_atype) {} static inline void bpf_cgroup_atype_put(int cgroup_atype) {} #endif /* CONFIG_BPF_LSM */ +#ifdef CONFIG_MEMCG_KMEM +static inline struct mem_cgroup *bpf_map_get_memcg(const struct bpf_map *map) +{ + if (map->objcg) + return get_mem_cgroup_from_objcg(map->objcg); + + return root_mem_cgroup; +} + +static inline void bpf_map_put_memcg(struct mem_cgroup *memcg) +{ + mem_cgroup_put(memcg); +} + +#else +static inline struct mem_cgroup *bpf_map_get_memcg(const struct bpf_map *map) +{ + return root_memcg(); +} + +static inline void bpf_map_put_memcg(struct mem_cgroup *memcg) +{ +} +#endif + #endif /* _LINUX_BPF_H */ diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 9ecead1..2f0a611 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -361,6 +361,11 @@ struct mem_cgroup { extern struct mem_cgroup *root_mem_cgroup; +static inline struct mem_cgroup *root_memcg(void) +{ + return root_mem_cgroup; +} + enum page_memcg_data_flags { /* page->memcg_data is a pointer to an objcgs vector */ MEMCG_DATA_OBJCGS = (1UL << 0), @@ -1138,6 +1143,11 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, #define MEM_CGROUP_ID_SHIFT 0 #define MEM_CGROUP_ID_MAX 0 +static inline struct mem_cgroup *root_memcg(void) +{ + return NULL; +} + static inline struct mem_cgroup *folio_memcg(struct folio *folio) { return NULL; diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 2f18ae2..19c3a81 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -433,19 +433,6 @@ static void bpf_map_release_memcg(struct bpf_map *map) obj_cgroup_put(map->objcg); } -static struct mem_cgroup *bpf_map_get_memcg(const struct bpf_map *map) -{ - if (map->objcg) - return get_mem_cgroup_from_objcg(map->objcg); - - return root_mem_cgroup; -} - -static void bpf_map_put_memcg(struct mem_cgroup *memcg) -{ - mem_cgroup_put(memcg); -} - void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, int node) { From patchwork Thu Aug 18 14:31:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12948196 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1E1BC00140 for ; Thu, 18 Aug 2022 23:55:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 427EA8D0003; Thu, 18 Aug 2022 19:55:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3D7808D0001; Thu, 18 Aug 2022 19:55:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2789E8D0003; Thu, 18 Aug 2022 19:55:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 1ADBF8D0001 for ; Thu, 18 Aug 2022 19:55:38 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E7D87C093C for ; Thu, 18 Aug 2022 23:55:37 +0000 (UTC) X-FDA: 79814373114.14.3EBF0AD Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) by imf08.hostedemail.com (Postfix) with ESMTP id BECF8160A4A for ; Thu, 18 Aug 2022 23:51:48 +0000 (UTC) Received: by mail-pj1-f48.google.com with SMTP id bf22so3112884pjb.4 for ; Thu, 18 Aug 2022 16:51:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=ZuUr0Lmb/ZOxk9PY5AfEZVtv5LOeYubF7wGlzoETmHE=; b=d1Dmx23BuvVjJGEdABIJqux0UqGU7kR857GG+nUy3p5Gl+JFQRPnacYR5C1E11Ofv+ VQzRJXyagtlUyX2PsvUhvwbBEGI5jCo8K1MXP9YC9jrYwQ1ql3E23yGsomtt3aLtLkyG 6jlsslfX7hAUQnLnftrWYWKn1ckbPQdwKng7PSX/hzD0DU1ebls3NA0vJ28f+kCAbJzx Dnbnw+RjYZUrzVphtx6OF+YrhmHnSp3Z5NGEX6Pky5rHNvrD7tahg66442kAz7OJTR+Q Xfox1U4JtTXiJmuNH/IzeI1IAXmH9WYKpr67CkHaEi1FWvYNxiP6kgVw4yJMKC3uCG38 /OMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=ZuUr0Lmb/ZOxk9PY5AfEZVtv5LOeYubF7wGlzoETmHE=; b=5XrubypFsiZToS7j640cOOsrZUWTf+gXluAdJ0WXx4BgDMniWMGiBLMR04iHndbW0e mcYhRe1P548eXsiTDFGmI/KwFKXVng5jjJ8V3hf3hN9WuHoOur64rPBPlAZ6OT6ZstK9 zVfs2/R4fLoejx0nAQI1xM/fFJycnAWpR8z94InmCTPcEwk3zfzkwhZJfQEVtHDMAbpv OL1Y9+/+Gy3sRh1FpaKLYE57smwFPOTjt+pWHUIrO0H9r/jRxcp50cyxcqb3MeC2xb85 Ex0EpVH+OKWIC7hziZ79Rm6QD9T/LxQ9DIRwXAgJ4Rm+ucpRJGNZ7YQU1zESRny54q3m 2aUw== X-Gm-Message-State: ACgBeo0UkICXBfOmEP3BoqKn/f0JvkUFAzA2OGZfiRCOfbzRek9emnBn 4k8bopcMFtrnR4pYLaLwUMy0aXsgKnFmsoP2 X-Google-Smtp-Source: AA6agR7c8t/UNHEBNrrIWCnH4P2UoAcPiCKseHMm/BYgU8xvPNvxWD6o9PWbAC1k+VmB22rXaSPvww== X-Received: by 2002:a17:902:a5c5:b0:16f:1e31:daab with SMTP id t5-20020a170902a5c500b0016f1e31daabmr2876148plq.82.1660833111438; Thu, 18 Aug 2022 07:31:51 -0700 (PDT) Received: from vultr.guest ([45.32.72.237]) by smtp.gmail.com with ESMTPSA id h5-20020a63f905000000b003fdc16f5de2sm1379124pgi.15.2022.08.18.07.31.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Aug 2022 07:31:50 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v2 04/12] bpf: Call bpf_map_init_from_attr() immediately after map creation Date: Thu, 18 Aug 2022 14:31:10 +0000 Message-Id: <20220818143118.17733-5-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220818143118.17733-1-laoar.shao@gmail.com> References: <20220818143118.17733-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660866708; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZuUr0Lmb/ZOxk9PY5AfEZVtv5LOeYubF7wGlzoETmHE=; b=pyutS/+ad3Xd2EJ76HXvqqp37COPk3qZCNYTLqBl/K84JACjRs/ESvQ/Yihfc7S+yAR/0f WnncjJ3l1QdscFPOjwGuMaooKre9okcwzs7MQQVl0Oug1C22bp6/G6Gl22rwkdtIaUbXmR E9J9wXW5P1aK7tL2tdHKKSYv+LEehVg= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=d1Dmx23B; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf08.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.48 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660866708; a=rsa-sha256; cv=none; b=HJ188JvFYdLDErX/M2ROM+rWc2mUdqAJ98fGsm4qimwPbpoNyRpbBlk5Bijx/l8ADYUL3X +4uCVee3tBT02Y4N6eqVioa4hlkdkol3VCJYO3TSUGL/+iOHAz3cmkjdkzaUDDJR6CYj6s PQ9hf+iFuxOG8BNHgzd6ZeGPK6xbGyU= Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=d1Dmx23B; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf08.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.48 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: BECF8160A4A X-Stat-Signature: 1re76rix8q7g4qqm5ai1pbgzgfpdgno6 X-Rspam-User: X-HE-Tag: 1660866708-432598 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In order to make all other map related memory allocations been allocated after memcg is saved in the map, we should save the memcg immediately after map creation. But the map is created in bpf_map_area_alloc(), within which we can't get the related bpf_map (except with a pointer casting which may be error prone), so we can do it in bpf_map_init_from_attr(), which is used by all bpf maps. bpf_map_init_from_attr() is executed immediately after bpf_map_area_alloc() for almost all bpf maps except bpf_struct_ops, devmap and hashmap, so this patch changes these three maps. In the future we will change the return type of bpf_map_init_from_attr() from void to int for error cases, so put it immediately after bpf_map_area_alloc() will make it eary to handle the error case. Signed-off-by: Yafang Shao --- kernel/bpf/bpf_struct_ops.c | 2 +- kernel/bpf/devmap.c | 5 ++--- kernel/bpf/hashtab.c | 4 ++-- 3 files changed, 5 insertions(+), 6 deletions(-) diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index 84b2d9d..36f24f8 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -624,6 +624,7 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) st_map->st_ops = st_ops; map = &st_map->map; + bpf_map_init_from_attr(map, attr); st_map->uvalue = bpf_map_area_alloc(vt->size, NUMA_NO_NODE); st_map->links = @@ -637,7 +638,6 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) mutex_init(&st_map->lock); set_vm_flush_reset_perms(st_map->image); - bpf_map_init_from_attr(map, attr); return map; } diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index f9a87dc..20decc7 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -127,9 +127,6 @@ static int dev_map_init_map(struct bpf_dtab *dtab, union bpf_attr *attr) */ attr->map_flags |= BPF_F_RDONLY_PROG; - - bpf_map_init_from_attr(&dtab->map, attr); - if (attr->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { dtab->n_buckets = roundup_pow_of_two(dtab->map.max_entries); @@ -167,6 +164,8 @@ static struct bpf_map *dev_map_alloc(union bpf_attr *attr) if (!dtab) return ERR_PTR(-ENOMEM); + bpf_map_init_from_attr(&dtab->map, attr); + err = dev_map_init_map(dtab, attr); if (err) { bpf_map_area_free(dtab); diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 8392f7f..48dc04c 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -499,10 +499,10 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) if (!htab) return ERR_PTR(-ENOMEM); - lockdep_register_key(&htab->lockdep_key); - bpf_map_init_from_attr(&htab->map, attr); + lockdep_register_key(&htab->lockdep_key); + if (percpu_lru) { /* ensure each CPU's lru list has >=1 elements. * since we are at it, make each lru list has the same From patchwork Thu Aug 18 14:31:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12948070 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C13AC00140 for ; Thu, 18 Aug 2022 22:23:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F014E8D0003; Thu, 18 Aug 2022 18:23:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB1138D0002; Thu, 18 Aug 2022 18:23:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D2AB68D0003; Thu, 18 Aug 2022 18:23:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C31E48D0002 for ; Thu, 18 Aug 2022 18:23:23 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 945FBA1A28 for ; Thu, 18 Aug 2022 22:23:23 +0000 (UTC) X-FDA: 79814140686.16.CAC14F0 Received: from mail-oa1-f42.google.com (mail-oa1-f42.google.com [209.85.160.42]) by imf05.hostedemail.com (Postfix) with ESMTP id 0CCD9105C2D for ; Thu, 18 Aug 2022 22:14:42 +0000 (UTC) Received: by mail-oa1-f42.google.com with SMTP id 586e51a60fabf-11ba6e79dd1so3317465fac.12 for ; Thu, 18 Aug 2022 15:14:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=uJuEbaEmBO8baBGpsUUDvamw98XhcvJRwwJ2WD0RIKw=; b=a32b18qRUiOvyUNSMksCfLkNXdnc7+RSMEY3p3IUko8sb/JQoiH2hxEMjoZycrvXnS 0iC2w6VtQMhmRB1jHZlmC1CjR8zwB959WEhgCy3pucrqEp/PSzKQvqRMS3YsQfa4/SKP sIZ4tVv9TxKTlopZKiypxfutt1uWjpMbfuFbBCtX5M7/3t4cdR/H9XtCa2ok6v6cocGD UHv7ks6YOSLlraiamjEQrgJLooUm3IVy6UyD3MtBaqNwMje8LyzhSNTu4eUjCh90HGyQ V3wnqES6JRfW+br157liVGS0sesB/gXndHwy3lB7xFswmNrqfAGYxV6WUt1w33X9qXL4 lPBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=uJuEbaEmBO8baBGpsUUDvamw98XhcvJRwwJ2WD0RIKw=; b=GT1v28nNzD2Nmlx14yJj/cow7nBU2snHZt7VOHKYycRGnroj8riaucBZfGB8k0CMmC kSlZJU1LNXh+8KOcAEnCzWxftQIjh5sWeWxo0Fe1NDMQM95xYjEUSK6IqEik1pufbAb/ jfVziAGo0LyMJ5Qbg8VFQ+bFzAhO1gY0Epvp4gt9puuxb+0QPWriq4kVpiNrOEDIbkDd Wify3NY9EC09DAv/L0x4VED+3HS383NNm/gcQlTzw3nvHzs+n5r5aw0dEKFvBRkl2ux/ CxGL+Xu0iOJCmCLQSHP3uXGxxjeIYveg8BnhtUq+k8tkiaQ6FfHWuGyDs17U1oerFMb4 zQqQ== X-Gm-Message-State: ACgBeo1Y8sODHFgfJi4riRTu6CT4T5fSXRXPlz0SdcPdaYG0FYyuBDqe 4pX3vRjJwoF9Qy5EkWTYiW7h7hDxmrHw2hsB X-Google-Smtp-Source: AA6agR4dAEPSIrjJLVed0oJ5tPjhwG4QUWaDBvIeFaeBMGQUBcy2h7+UJGNA2nNEPs40mtDFB4sijA== X-Received: by 2002:a17:90b:5096:b0:1fa:59f6:a442 with SMTP id rt22-20020a17090b509600b001fa59f6a442mr9062340pjb.81.1660833114252; Thu, 18 Aug 2022 07:31:54 -0700 (PDT) Received: from vultr.guest ([45.32.72.237]) by smtp.gmail.com with ESMTPSA id h5-20020a63f905000000b003fdc16f5de2sm1379124pgi.15.2022.08.18.07.31.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Aug 2022 07:31:53 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v2 05/12] bpf: Save memcg in bpf_map_init_from_attr() Date: Thu, 18 Aug 2022 14:31:11 +0000 Message-Id: <20220818143118.17733-6-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220818143118.17733-1-laoar.shao@gmail.com> References: <20220818143118.17733-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660860884; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uJuEbaEmBO8baBGpsUUDvamw98XhcvJRwwJ2WD0RIKw=; b=EcCXWj4l8AxWqhr3VmE3f3AouXGU8lXm23/BST46FqGumywXGkIekHxfFFMoFjEoE6zimF 4djirpc4bikcdMc61dwwn93zaNfRVPUJNzq/TBJ3hux5+t8s/PWA8/xEIQU7iguEpbTT1v cec1q57UwNiLeE2InIA4DCJQ1jojVMs= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=a32b18qR; spf=pass (imf05.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.42 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660860884; a=rsa-sha256; cv=none; b=EUdkzlrsKhprOiHwSB3NyYOPG5wtx+0S/FP28YgW3DNEVhXqKYuNte085l1Xjyy3CmGxbZ pBmNyawTKasBXIy8C7b0/3sfNJK4z4sbHwuQ3sQHHIidoINbHyZJZ06MWIRLzS3YWza9b6 exSfZvP61Id+tv9kZiLp7jaXGxbzb+Q= Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=a32b18qR; spf=pass (imf05.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.42 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Rspamd-Queue-Id: 0CCD9105C2D X-Rspamd-Server: rspam02 X-Stat-Signature: p3j6bgpek5m4iazzfk9mza5jkndpcyas X-HE-Tag: 1660860882-446685 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move bpf_map_save_memcg() into bpf_map_init_from_attr(), then all other map related memory allocation will be allocated after saving the memcg. And then we can get memcg from the map in the followup memory allocation. To pair with this change, bpf_map_release_memcg() is moved into bpf_map_area_free(). A new parameter struct bpf_map is introduced into bpf_map_area_free() for this purpose. Signed-off-by: Yafang Shao --- include/linux/bpf.h | 2 +- kernel/bpf/arraymap.c | 8 +++--- kernel/bpf/bloom_filter.c | 2 +- kernel/bpf/bpf_local_storage.c | 4 +-- kernel/bpf/bpf_struct_ops.c | 6 ++--- kernel/bpf/cpumap.c | 6 ++--- kernel/bpf/devmap.c | 8 +++--- kernel/bpf/hashtab.c | 10 +++---- kernel/bpf/local_storage.c | 2 +- kernel/bpf/lpm_trie.c | 2 +- kernel/bpf/offload.c | 4 +-- kernel/bpf/queue_stack_maps.c | 2 +- kernel/bpf/reuseport_array.c | 2 +- kernel/bpf/ringbuf.c | 8 +++--- kernel/bpf/stackmap.c | 8 +++--- kernel/bpf/syscall.c | 60 ++++++++++++++++++++++-------------------- net/core/sock_map.c | 12 ++++----- net/xdp/xskmap.c | 2 +- 18 files changed, 76 insertions(+), 72 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index ded7d23..6f85137 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1637,7 +1637,7 @@ struct bpf_prog *bpf_prog_get_type_dev(u32 ufd, enum bpf_prog_type type, void bpf_map_put(struct bpf_map *map); void *bpf_map_area_alloc(u64 size, int numa_node); void *bpf_map_area_mmapable_alloc(u64 size, int numa_node); -void bpf_map_area_free(void *base); +void bpf_map_area_free(void *base, struct bpf_map *map); bool bpf_map_write_active(const struct bpf_map *map); void bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr); int generic_map_lookup_batch(struct bpf_map *map, diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index d3e734b..9ce4d1b 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -147,7 +147,7 @@ static struct bpf_map *array_map_alloc(union bpf_attr *attr) array->elem_size = elem_size; if (percpu && bpf_array_alloc_percpu(array)) { - bpf_map_area_free(array); + bpf_map_area_free(array, &array->map); return ERR_PTR(-ENOMEM); } @@ -430,9 +430,9 @@ static void array_map_free(struct bpf_map *map) bpf_array_free_percpu(array); if (array->map.map_flags & BPF_F_MMAPABLE) - bpf_map_area_free(array_map_vmalloc_addr(array)); + bpf_map_area_free(array_map_vmalloc_addr(array), map); else - bpf_map_area_free(array); + bpf_map_area_free(array, map); } static void array_map_seq_show_elem(struct bpf_map *map, void *key, @@ -774,7 +774,7 @@ static void fd_array_map_free(struct bpf_map *map) for (i = 0; i < array->map.max_entries; i++) BUG_ON(array->ptrs[i] != NULL); - bpf_map_area_free(array); + bpf_map_area_free(array, map); } static void *fd_array_map_lookup_elem(struct bpf_map *map, void *key) diff --git a/kernel/bpf/bloom_filter.c b/kernel/bpf/bloom_filter.c index b9ea539..e59064d 100644 --- a/kernel/bpf/bloom_filter.c +++ b/kernel/bpf/bloom_filter.c @@ -168,7 +168,7 @@ static void bloom_map_free(struct bpf_map *map) struct bpf_bloom_filter *bloom = container_of(map, struct bpf_bloom_filter, map); - bpf_map_area_free(bloom); + bpf_map_area_free(bloom, map); } static void *bloom_map_lookup_elem(struct bpf_map *map, void *key) diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index 4ee2e72..77e075b 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -582,7 +582,7 @@ void bpf_local_storage_map_free(struct bpf_local_storage_map *smap, synchronize_rcu(); kvfree(smap->buckets); - bpf_map_area_free(smap); + bpf_map_area_free(smap, &smap->map); } int bpf_local_storage_map_alloc_check(union bpf_attr *attr) @@ -623,7 +623,7 @@ struct bpf_local_storage_map *bpf_local_storage_map_alloc(union bpf_attr *attr) smap->buckets = kvcalloc(sizeof(*smap->buckets), nbuckets, GFP_USER | __GFP_NOWARN | __GFP_ACCOUNT); if (!smap->buckets) { - bpf_map_area_free(smap); + bpf_map_area_free(smap, &smap->map); return ERR_PTR(-ENOMEM); } diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index 36f24f8..9fb8ad1 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -577,10 +577,10 @@ static void bpf_struct_ops_map_free(struct bpf_map *map) if (st_map->links) bpf_struct_ops_map_put_progs(st_map); - bpf_map_area_free(st_map->links); + bpf_map_area_free(st_map->links, NULL); bpf_jit_free_exec(st_map->image); - bpf_map_area_free(st_map->uvalue); - bpf_map_area_free(st_map); + bpf_map_area_free(st_map->uvalue, NULL); + bpf_map_area_free(st_map, map); } static int bpf_struct_ops_map_alloc_check(union bpf_attr *attr) diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index b5ba34d..7de2ae6 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -118,7 +118,7 @@ static struct bpf_map *cpu_map_alloc(union bpf_attr *attr) return &cmap->map; free_cmap: - bpf_map_area_free(cmap); + bpf_map_area_free(cmap, &cmap->map); return ERR_PTR(err); } @@ -622,8 +622,8 @@ static void cpu_map_free(struct bpf_map *map) /* bq flush and cleanup happens after RCU grace-period */ __cpu_map_entry_replace(cmap, i, NULL); /* call_rcu */ } - bpf_map_area_free(cmap->cpu_map); - bpf_map_area_free(cmap); + bpf_map_area_free(cmap->cpu_map, NULL); + bpf_map_area_free(cmap, map); } /* Elements are kept alive by RCU; either by rcu_read_lock() (from syscall) or diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 20decc7..3268ce7 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -168,7 +168,7 @@ static struct bpf_map *dev_map_alloc(union bpf_attr *attr) err = dev_map_init_map(dtab, attr); if (err) { - bpf_map_area_free(dtab); + bpf_map_area_free(dtab, &dtab->map); return ERR_PTR(err); } @@ -221,7 +221,7 @@ static void dev_map_free(struct bpf_map *map) } } - bpf_map_area_free(dtab->dev_index_head); + bpf_map_area_free(dtab->dev_index_head, NULL); } else { for (i = 0; i < dtab->map.max_entries; i++) { struct bpf_dtab_netdev *dev; @@ -236,10 +236,10 @@ static void dev_map_free(struct bpf_map *map) kfree(dev); } - bpf_map_area_free(dtab->netdev_map); + bpf_map_area_free(dtab->netdev_map, NULL); } - bpf_map_area_free(dtab); + bpf_map_area_free(dtab, &dtab->map); } static int dev_map_get_next_key(struct bpf_map *map, void *key, void *next_key) diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 48dc04c..1b3653d 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -290,7 +290,7 @@ static void htab_free_elems(struct bpf_htab *htab) cond_resched(); } free_elems: - bpf_map_area_free(htab->elems); + bpf_map_area_free(htab->elems, NULL); } /* The LRU list has a lock (lru_lock). Each htab bucket has a lock @@ -576,10 +576,10 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) free_map_locked: for (i = 0; i < HASHTAB_MAP_LOCK_COUNT; i++) free_percpu(htab->map_locked[i]); - bpf_map_area_free(htab->buckets); + bpf_map_area_free(htab->buckets, NULL); free_htab: lockdep_unregister_key(&htab->lockdep_key); - bpf_map_area_free(htab); + bpf_map_area_free(htab, &htab->map); return ERR_PTR(err); } @@ -1492,11 +1492,11 @@ static void htab_map_free(struct bpf_map *map) bpf_map_free_kptr_off_tab(map); free_percpu(htab->extra_elems); - bpf_map_area_free(htab->buckets); + bpf_map_area_free(htab->buckets, NULL); for (i = 0; i < HASHTAB_MAP_LOCK_COUNT; i++) free_percpu(htab->map_locked[i]); lockdep_unregister_key(&htab->lockdep_key); - bpf_map_area_free(htab); + bpf_map_area_free(htab, map); } static void htab_map_seq_show_elem(struct bpf_map *map, void *key, diff --git a/kernel/bpf/local_storage.c b/kernel/bpf/local_storage.c index 098cf33..c705d66 100644 --- a/kernel/bpf/local_storage.c +++ b/kernel/bpf/local_storage.c @@ -345,7 +345,7 @@ static void cgroup_storage_map_free(struct bpf_map *_map) WARN_ON(!RB_EMPTY_ROOT(&map->root)); WARN_ON(!list_empty(&map->list)); - bpf_map_area_free(map); + bpf_map_area_free(map, _map); } static int cgroup_storage_delete_elem(struct bpf_map *map, void *key) diff --git a/kernel/bpf/lpm_trie.c b/kernel/bpf/lpm_trie.c index d833496..fd99360 100644 --- a/kernel/bpf/lpm_trie.c +++ b/kernel/bpf/lpm_trie.c @@ -609,7 +609,7 @@ static void trie_free(struct bpf_map *map) } out: - bpf_map_area_free(trie); + bpf_map_area_free(trie, map); } static int trie_get_next_key(struct bpf_map *map, void *_key, void *_next_key) diff --git a/kernel/bpf/offload.c b/kernel/bpf/offload.c index 13e4efc..c9941a9 100644 --- a/kernel/bpf/offload.c +++ b/kernel/bpf/offload.c @@ -404,7 +404,7 @@ struct bpf_map *bpf_map_offload_map_alloc(union bpf_attr *attr) err_unlock: up_write(&bpf_devs_lock); rtnl_unlock(); - bpf_map_area_free(offmap); + bpf_map_area_free(offmap, &offmap->map); return ERR_PTR(err); } @@ -428,7 +428,7 @@ void bpf_map_offload_map_free(struct bpf_map *map) up_write(&bpf_devs_lock); rtnl_unlock(); - bpf_map_area_free(offmap); + bpf_map_area_free(offmap, map); } int bpf_map_offload_lookup_elem(struct bpf_map *map, void *key, void *value) diff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c index 8a5e060..f2ec0c4 100644 --- a/kernel/bpf/queue_stack_maps.c +++ b/kernel/bpf/queue_stack_maps.c @@ -92,7 +92,7 @@ static void queue_stack_map_free(struct bpf_map *map) { struct bpf_queue_stack *qs = bpf_queue_stack(map); - bpf_map_area_free(qs); + bpf_map_area_free(qs, map); } static int __queue_map_get(struct bpf_map *map, void *value, bool delete) diff --git a/kernel/bpf/reuseport_array.c b/kernel/bpf/reuseport_array.c index e2618fb..594cdb0 100644 --- a/kernel/bpf/reuseport_array.c +++ b/kernel/bpf/reuseport_array.c @@ -146,7 +146,7 @@ static void reuseport_array_free(struct bpf_map *map) * Once reaching here, all sk->sk_user_data is not * referencing this "array". "array" can be freed now. */ - bpf_map_area_free(array); + bpf_map_area_free(array, map); } static struct bpf_map *reuseport_array_alloc(union bpf_attr *attr) diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index b483aea..74dd8dc 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -116,7 +116,7 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) err_free_pages: for (i = 0; i < nr_pages; i++) __free_page(pages[i]); - bpf_map_area_free(pages); + bpf_map_area_free(pages, NULL); return NULL; } @@ -172,7 +172,7 @@ static struct bpf_map *ringbuf_map_alloc(union bpf_attr *attr) rb_map->rb = bpf_ringbuf_alloc(attr->max_entries, rb_map->map.numa_node); if (!rb_map->rb) { - bpf_map_area_free(rb_map); + bpf_map_area_free(rb_map, &rb_map->map); return ERR_PTR(-ENOMEM); } @@ -190,7 +190,7 @@ static void bpf_ringbuf_free(struct bpf_ringbuf *rb) vunmap(rb); for (i = 0; i < nr_pages; i++) __free_page(pages[i]); - bpf_map_area_free(pages); + bpf_map_area_free(pages, NULL); } static void ringbuf_map_free(struct bpf_map *map) @@ -199,7 +199,7 @@ static void ringbuf_map_free(struct bpf_map *map) rb_map = container_of(map, struct bpf_ringbuf_map, map); bpf_ringbuf_free(rb_map->rb); - bpf_map_area_free(rb_map); + bpf_map_area_free(rb_map, map); } static void *ringbuf_map_lookup_elem(struct bpf_map *map, void *key) diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index 1adbe67..042b7d2 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -62,7 +62,7 @@ static int prealloc_elems_and_freelist(struct bpf_stack_map *smap) return 0; free_elems: - bpf_map_area_free(smap->elems); + bpf_map_area_free(smap->elems, NULL); return err; } @@ -120,7 +120,7 @@ static struct bpf_map *stack_map_alloc(union bpf_attr *attr) put_buffers: put_callchain_buffers(); free_smap: - bpf_map_area_free(smap); + bpf_map_area_free(smap, &smap->map); return ERR_PTR(err); } @@ -648,9 +648,9 @@ static void stack_map_free(struct bpf_map *map) { struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map); - bpf_map_area_free(smap->elems); + bpf_map_area_free(smap->elems, NULL); pcpu_freelist_destroy(&smap->freelist); - bpf_map_area_free(smap); + bpf_map_area_free(smap, map); put_callchain_buffers(); } diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 19c3a81..01d8d4a 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -293,6 +293,34 @@ static int bpf_map_copy_value(struct bpf_map *map, void *key, void *value, return err; } +#ifdef CONFIG_MEMCG_KMEM +static void bpf_map_save_memcg(struct bpf_map *map) +{ + /* Currently if a map is created by a process belonging to the root + * memory cgroup, get_obj_cgroup_from_current() will return NULL. + * So we have to check map->objcg for being NULL each time it's + * being used. + */ + map->objcg = get_obj_cgroup_from_current(); +} + +static void bpf_map_release_memcg(struct bpf_map *map) +{ + if (map->objcg) + obj_cgroup_put(map->objcg); +} + +#else +static void bpf_map_save_memcg(struct bpf_map *map) +{ +} + +static void bpf_map_release_memcg(struct bpf_map *map) +{ +} + +#endif + /* Please, do not use this function outside from the map creation path * (e.g. in map update path) without taking care of setting the active * memory cgroup (see at bpf_map_kmalloc_node() for example). @@ -344,8 +372,10 @@ void *bpf_map_area_mmapable_alloc(u64 size, int numa_node) return __bpf_map_area_alloc(size, numa_node, true); } -void bpf_map_area_free(void *area) +void bpf_map_area_free(void *area, struct bpf_map *map) { + if (map) + bpf_map_release_memcg(map); kvfree(area); } @@ -363,6 +393,7 @@ static u32 bpf_map_flags_retain_permanent(u32 flags) void bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr) { + bpf_map_save_memcg(map); map->map_type = attr->map_type; map->key_size = attr->key_size; map->value_size = attr->value_size; @@ -417,22 +448,6 @@ void bpf_map_free_id(struct bpf_map *map, bool do_idr_lock) } #ifdef CONFIG_MEMCG_KMEM -static void bpf_map_save_memcg(struct bpf_map *map) -{ - /* Currently if a map is created by a process belonging to the root - * memory cgroup, get_obj_cgroup_from_current() will return NULL. - * So we have to check map->objcg for being NULL each time it's - * being used. - */ - map->objcg = get_obj_cgroup_from_current(); -} - -static void bpf_map_release_memcg(struct bpf_map *map) -{ - if (map->objcg) - obj_cgroup_put(map->objcg); -} - void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, int node) { @@ -477,14 +492,6 @@ void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, return ptr; } -#else -static void bpf_map_save_memcg(struct bpf_map *map) -{ -} - -static void bpf_map_release_memcg(struct bpf_map *map) -{ -} #endif static int bpf_map_kptr_off_cmp(const void *a, const void *b) @@ -605,7 +612,6 @@ static void bpf_map_free_deferred(struct work_struct *work) security_bpf_map_free(map); kfree(map->off_arr); - bpf_map_release_memcg(map); /* implementation dependent freeing, map_free callback also does * bpf_map_free_kptr_off_tab, if needed. */ @@ -1154,8 +1160,6 @@ static int map_create(union bpf_attr *attr) if (err) goto free_map_sec; - bpf_map_save_memcg(map); - err = bpf_map_new_fd(map, f_flags); if (err < 0) { /* failed to allocate fd. diff --git a/net/core/sock_map.c b/net/core/sock_map.c index d0c4338..e8f414a 100644 --- a/net/core/sock_map.c +++ b/net/core/sock_map.c @@ -52,7 +52,7 @@ static struct bpf_map *sock_map_alloc(union bpf_attr *attr) sizeof(struct sock *), stab->map.numa_node); if (!stab->sks) { - bpf_map_area_free(stab); + bpf_map_area_free(stab, &stab->map); return ERR_PTR(-ENOMEM); } @@ -360,8 +360,8 @@ static void sock_map_free(struct bpf_map *map) /* wait for psock readers accessing its map link */ synchronize_rcu(); - bpf_map_area_free(stab->sks); - bpf_map_area_free(stab); + bpf_map_area_free(stab->sks, NULL); + bpf_map_area_free(stab, map); } static void sock_map_release_progs(struct bpf_map *map) @@ -1106,7 +1106,7 @@ static struct bpf_map *sock_hash_alloc(union bpf_attr *attr) return &htab->map; free_htab: - bpf_map_area_free(htab); + bpf_map_area_free(htab, &htab->map); return ERR_PTR(err); } @@ -1158,8 +1158,8 @@ static void sock_hash_free(struct bpf_map *map) /* wait for psock readers accessing its map link */ synchronize_rcu(); - bpf_map_area_free(htab->buckets); - bpf_map_area_free(htab); + bpf_map_area_free(htab->buckets, NULL); + bpf_map_area_free(htab, map); } static void *sock_hash_lookup_sys(struct bpf_map *map, void *key) diff --git a/net/xdp/xskmap.c b/net/xdp/xskmap.c index acc8e52..5abb87e 100644 --- a/net/xdp/xskmap.c +++ b/net/xdp/xskmap.c @@ -90,7 +90,7 @@ static void xsk_map_free(struct bpf_map *map) struct xsk_map *m = container_of(map, struct xsk_map, map); synchronize_net(); - bpf_map_area_free(m); + bpf_map_area_free(m, map); } static int xsk_map_get_next_key(struct bpf_map *map, void *key, void *next_key) From patchwork Thu Aug 18 14:31:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12947882 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FC5FC28B2B for ; Thu, 18 Aug 2022 21:01:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C8AE8D0003; Thu, 18 Aug 2022 17:01:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 89E9E8D0002; Thu, 18 Aug 2022 17:01:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 741468E0001; Thu, 18 Aug 2022 17:01:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 631918D0002 for ; Thu, 18 Aug 2022 17:01:09 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 433E01C7F2F for ; Thu, 18 Aug 2022 21:01:09 +0000 (UTC) X-FDA: 79813933458.19.38C6A49 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by imf31.hostedemail.com (Postfix) with ESMTP id 3284520F3F for ; Thu, 18 Aug 2022 20:50:59 +0000 (UTC) Received: by mail-pf1-f178.google.com with SMTP id y141so2597016pfb.7 for ; Thu, 18 Aug 2022 13:51:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=QGQz0k6TbsAnAJbmJctQ4nS+LAE6qyUHyo74uqh9bAo=; b=Y2IslTLwaCFiHWqssLCfkPf+GGGOThTb2gxHQ79SNgpleMW8s4rGovLpPXzuR2pnq4 LwEdC329T4CN+tdTQZBQukw+i521mkCLJDVAjqbYz31H5XsQo6tsZWSjRz23E41w7pxQ 5OGbzZaYJz255Zg+9bOLI6U1pnduB5nu2p47FCd79gNvhz5vTNj5wetgJPcpnw/b+pHo ypXJIQJadBkoTnCyZPDJL3/fLIynj6cyay52sAlPvmb4vAsPkBRdSOnYqr4A0cKTLWBo JZ14gSbNC2WUV4vQlXkQ0QPudDeQBbXAmrQ78mxhGjIPmy70UW7AUOAATwtZEIinSrqy vIDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=QGQz0k6TbsAnAJbmJctQ4nS+LAE6qyUHyo74uqh9bAo=; b=wEyjpLH3PGfbuimsxdgPZ+ybpPDg4Ogozyx3B+oH+2MMWZ9hum/dlsycwwW3xFhZGM HpEQkUm2F7MoeumxTG3WRGqGTZDihC/Xv1S1Zz03vCkxbzmJbbtP11ZJ3OcugHRWN8mt L/+uOz93IKJpHK3uFPxlClMqscVu/wcKXfwrlcCVy3LNa/pXtXAa4g/hii7f78JqGJ1L 8R5FE/qoCrJNatx504oqfdc4np2oI8yrX+z8fh870Ytb0Wq/GVNblvufoX+UnQMdhPKy G0oKx/lZmftQTDRK9oNQNtjJlBMLxpghRtty2v5gabFa7gSGgGpr2A0sC5NIUJC0Je49 GIEg== X-Gm-Message-State: ACgBeo1viH4AyvqaiVeyvdXMwRn9J3qzJlCKcwbH+Llc7SjWnbprGESD wA7+2svNer5w/F+gQm7B8FfoAyYU2u/iZq52 X-Google-Smtp-Source: AA6agR72O7XCUYbrdl0F9W19fLbiqcTPDipxkOWq0k58kGZZmD+rdPMbQu98Agw1M7MO/RJWdnjVJg== X-Received: by 2002:a63:415:0:b0:422:6de7:d9b7 with SMTP id 21-20020a630415000000b004226de7d9b7mr2715524pge.432.1660833116970; Thu, 18 Aug 2022 07:31:56 -0700 (PDT) Received: from vultr.guest ([45.32.72.237]) by smtp.gmail.com with ESMTPSA id h5-20020a63f905000000b003fdc16f5de2sm1379124pgi.15.2022.08.18.07.31.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Aug 2022 07:31:56 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v2 06/12] bpf: Use scoped-based charge in bpf_map_area_alloc Date: Thu, 18 Aug 2022 14:31:12 +0000 Message-Id: <20220818143118.17733-7-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220818143118.17733-1-laoar.shao@gmail.com> References: <20220818143118.17733-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660855861; a=rsa-sha256; cv=none; b=Bm/RhN2Y7OUej2smBq6TMyTP/r3KA7zXlDpz4plZPbkwHuYVwdyjpz1ei/S5vlcLS29Oln YZPsiOtaCLKVHbcnnbiAMVi1cWA6y71ImRdyWpRJldA1IEWfdzl4ponPuXO2RJFXV8ygxw kholIYB7h5tis5n81JN9cDME57jng00= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Y2IslTLw; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf31.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660855861; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QGQz0k6TbsAnAJbmJctQ4nS+LAE6qyUHyo74uqh9bAo=; b=gt3zN7PLYATg0kMZrsVUIBdVv6FXaiOWaoq7Th3gd1jqfwzl0Mzry/mdFeKNVCa4GKvATx 4BIMGN2CfIKv9nfofCrB0O2fluDNM0th/uulOciqAgzO98bIn+84wkcNQQwkiJJx+gCI6G GyDeXoe4KUCSHGLa/dsBeKGHCGmnCf4= X-Stat-Signature: 8sfae1rngi748r3bj7k4kwhc3fj1tsqu X-Rspamd-Queue-Id: 3284520F3F Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Y2IslTLw; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf31.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com X-Rspam-User: X-Rspamd-Server: rspam06 X-HE-Tag: 1660855859-702848 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently bpf_map_area_alloc() is used to allocate a container of struct bpf_map or members in this container. To distinguish the map creation and the other case, a new parameter struct bpf_map is added into bpf_map_area_alloc(). Then for the non-map-creation case, we could get the memcg from the map instead of using the current memcg. Signed-off-by: Yafang Shao --- include/linux/bpf.h | 2 +- kernel/bpf/arraymap.c | 2 +- kernel/bpf/bloom_filter.c | 2 +- kernel/bpf/bpf_local_storage.c | 2 +- kernel/bpf/bpf_struct_ops.c | 6 +++--- kernel/bpf/cpumap.c | 5 +++-- kernel/bpf/devmap.c | 13 ++++++++----- kernel/bpf/hashtab.c | 8 +++++--- kernel/bpf/local_storage.c | 2 +- kernel/bpf/lpm_trie.c | 2 +- kernel/bpf/offload.c | 2 +- kernel/bpf/queue_stack_maps.c | 2 +- kernel/bpf/reuseport_array.c | 2 +- kernel/bpf/ringbuf.c | 15 +++++++++------ kernel/bpf/stackmap.c | 5 +++-- kernel/bpf/syscall.c | 16 ++++++++++++++-- net/core/sock_map.c | 10 ++++++---- net/xdp/xskmap.c | 2 +- 18 files changed, 61 insertions(+), 37 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 6f85137..066f351d 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1635,7 +1635,7 @@ struct bpf_prog *bpf_prog_get_type_dev(u32 ufd, enum bpf_prog_type type, struct bpf_map * __must_check bpf_map_inc_not_zero(struct bpf_map *map); void bpf_map_put_with_uref(struct bpf_map *map); void bpf_map_put(struct bpf_map *map); -void *bpf_map_area_alloc(u64 size, int numa_node); +void *bpf_map_area_alloc(u64 size, int numa_node, struct bpf_map *map); void *bpf_map_area_mmapable_alloc(u64 size, int numa_node); void bpf_map_area_free(void *base, struct bpf_map *map); bool bpf_map_write_active(const struct bpf_map *map); diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index 9ce4d1b..80974c5 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -135,7 +135,7 @@ static struct bpf_map *array_map_alloc(union bpf_attr *attr) array = data + PAGE_ALIGN(sizeof(struct bpf_array)) - offsetof(struct bpf_array, value); } else { - array = bpf_map_area_alloc(array_size, numa_node); + array = bpf_map_area_alloc(array_size, numa_node, NULL); } if (!array) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/bloom_filter.c b/kernel/bpf/bloom_filter.c index e59064d..6691f79 100644 --- a/kernel/bpf/bloom_filter.c +++ b/kernel/bpf/bloom_filter.c @@ -142,7 +142,7 @@ static struct bpf_map *bloom_map_alloc(union bpf_attr *attr) } bitset_bytes = roundup(bitset_bytes, sizeof(unsigned long)); - bloom = bpf_map_area_alloc(sizeof(*bloom) + bitset_bytes, numa_node); + bloom = bpf_map_area_alloc(sizeof(*bloom) + bitset_bytes, numa_node, NULL); if (!bloom) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index 77e075b..67ab249 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -610,7 +610,7 @@ struct bpf_local_storage_map *bpf_local_storage_map_alloc(union bpf_attr *attr) unsigned int i; u32 nbuckets; - smap = bpf_map_area_alloc(sizeof(*smap), NUMA_NO_NODE); + smap = bpf_map_area_alloc(sizeof(*smap), NUMA_NO_NODE, NULL); if (!smap) return ERR_PTR(-ENOMEM); bpf_map_init_from_attr(&smap->map, attr); diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index 9fb8ad1..37ba5c0 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -618,7 +618,7 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) */ (vt->size - sizeof(struct bpf_struct_ops_value)); - st_map = bpf_map_area_alloc(st_map_size, NUMA_NO_NODE); + st_map = bpf_map_area_alloc(st_map_size, NUMA_NO_NODE, NULL); if (!st_map) return ERR_PTR(-ENOMEM); @@ -626,10 +626,10 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) map = &st_map->map; bpf_map_init_from_attr(map, attr); - st_map->uvalue = bpf_map_area_alloc(vt->size, NUMA_NO_NODE); + st_map->uvalue = bpf_map_area_alloc(vt->size, NUMA_NO_NODE, map); st_map->links = bpf_map_area_alloc(btf_type_vlen(t) * sizeof(struct bpf_links *), - NUMA_NO_NODE); + NUMA_NO_NODE, map); st_map->image = bpf_jit_alloc_exec(PAGE_SIZE); if (!st_map->uvalue || !st_map->links || !st_map->image) { bpf_struct_ops_map_free(map); diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index 7de2ae6..b593157 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -97,7 +97,7 @@ static struct bpf_map *cpu_map_alloc(union bpf_attr *attr) attr->map_flags & ~BPF_F_NUMA_NODE) return ERR_PTR(-EINVAL); - cmap = bpf_map_area_alloc(sizeof(*cmap), NUMA_NO_NODE); + cmap = bpf_map_area_alloc(sizeof(*cmap), NUMA_NO_NODE, NULL); if (!cmap) return ERR_PTR(-ENOMEM); @@ -112,7 +112,8 @@ static struct bpf_map *cpu_map_alloc(union bpf_attr *attr) /* Alloc array for possible remote "destination" CPUs */ cmap->cpu_map = bpf_map_area_alloc(cmap->map.max_entries * sizeof(struct bpf_cpu_map_entry *), - cmap->map.numa_node); + cmap->map.numa_node, + &cmap->map); if (!cmap->cpu_map) goto free_cmap; diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 3268ce7..807a4cd 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -89,12 +89,13 @@ struct bpf_dtab { static LIST_HEAD(dev_map_list); static struct hlist_head *dev_map_create_hash(unsigned int entries, - int numa_node) + int numa_node, + struct bpf_map *map) { int i; struct hlist_head *hash; - hash = bpf_map_area_alloc((u64) entries * sizeof(*hash), numa_node); + hash = bpf_map_area_alloc((u64) entries * sizeof(*hash), numa_node, map); if (hash != NULL) for (i = 0; i < entries; i++) INIT_HLIST_HEAD(&hash[i]); @@ -136,7 +137,8 @@ static int dev_map_init_map(struct bpf_dtab *dtab, union bpf_attr *attr) if (attr->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { dtab->dev_index_head = dev_map_create_hash(dtab->n_buckets, - dtab->map.numa_node); + dtab->map.numa_node, + &dtab->map); if (!dtab->dev_index_head) return -ENOMEM; @@ -144,7 +146,8 @@ static int dev_map_init_map(struct bpf_dtab *dtab, union bpf_attr *attr) } else { dtab->netdev_map = bpf_map_area_alloc((u64) dtab->map.max_entries * sizeof(struct bpf_dtab_netdev *), - dtab->map.numa_node); + dtab->map.numa_node, + &dtab->map); if (!dtab->netdev_map) return -ENOMEM; } @@ -160,7 +163,7 @@ static struct bpf_map *dev_map_alloc(union bpf_attr *attr) if (!capable(CAP_NET_ADMIN)) return ERR_PTR(-EPERM); - dtab = bpf_map_area_alloc(sizeof(*dtab), NUMA_NO_NODE); + dtab = bpf_map_area_alloc(sizeof(*dtab), NUMA_NO_NODE, NULL); if (!dtab) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 1b3653d..e9a6d2c4 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -332,7 +332,8 @@ static int prealloc_init(struct bpf_htab *htab) num_entries += num_possible_cpus(); htab->elems = bpf_map_area_alloc((u64)htab->elem_size * num_entries, - htab->map.numa_node); + htab->map.numa_node, + &htab->map); if (!htab->elems) return -ENOMEM; @@ -495,7 +496,7 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) struct bpf_htab *htab; int err, i; - htab = bpf_map_area_alloc(sizeof(*htab), NUMA_NO_NODE); + htab = bpf_map_area_alloc(sizeof(*htab), NUMA_NO_NODE, NULL); if (!htab) return ERR_PTR(-ENOMEM); @@ -534,7 +535,8 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) err = -ENOMEM; htab->buckets = bpf_map_area_alloc(htab->n_buckets * sizeof(struct bucket), - htab->map.numa_node); + htab->map.numa_node, + &htab->map); if (!htab->buckets) goto free_htab; diff --git a/kernel/bpf/local_storage.c b/kernel/bpf/local_storage.c index c705d66..fcc7ece 100644 --- a/kernel/bpf/local_storage.c +++ b/kernel/bpf/local_storage.c @@ -313,7 +313,7 @@ static struct bpf_map *cgroup_storage_map_alloc(union bpf_attr *attr) /* max_entries is not used and enforced to be 0 */ return ERR_PTR(-EINVAL); - map = bpf_map_area_alloc(sizeof(struct bpf_cgroup_storage_map), numa_node); + map = bpf_map_area_alloc(sizeof(struct bpf_cgroup_storage_map), numa_node, NULL); if (!map) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/lpm_trie.c b/kernel/bpf/lpm_trie.c index fd99360..3d329ae 100644 --- a/kernel/bpf/lpm_trie.c +++ b/kernel/bpf/lpm_trie.c @@ -558,7 +558,7 @@ static struct bpf_map *trie_alloc(union bpf_attr *attr) attr->value_size > LPM_VAL_SIZE_MAX) return ERR_PTR(-EINVAL); - trie = bpf_map_area_alloc(sizeof(*trie), NUMA_NO_NODE); + trie = bpf_map_area_alloc(sizeof(*trie), NUMA_NO_NODE, NULL); if (!trie) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/offload.c b/kernel/bpf/offload.c index c9941a9..87c59da 100644 --- a/kernel/bpf/offload.c +++ b/kernel/bpf/offload.c @@ -372,7 +372,7 @@ struct bpf_map *bpf_map_offload_map_alloc(union bpf_attr *attr) attr->map_type != BPF_MAP_TYPE_HASH) return ERR_PTR(-EINVAL); - offmap = bpf_map_area_alloc(sizeof(*offmap), NUMA_NO_NODE); + offmap = bpf_map_area_alloc(sizeof(*offmap), NUMA_NO_NODE, NULL); if (!offmap) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c index f2ec0c4..bf57e45 100644 --- a/kernel/bpf/queue_stack_maps.c +++ b/kernel/bpf/queue_stack_maps.c @@ -74,7 +74,7 @@ static struct bpf_map *queue_stack_map_alloc(union bpf_attr *attr) size = (u64) attr->max_entries + 1; queue_size = sizeof(*qs) + size * attr->value_size; - qs = bpf_map_area_alloc(queue_size, numa_node); + qs = bpf_map_area_alloc(queue_size, numa_node, NULL); if (!qs) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/reuseport_array.c b/kernel/bpf/reuseport_array.c index 594cdb0..52c7e77 100644 --- a/kernel/bpf/reuseport_array.c +++ b/kernel/bpf/reuseport_array.c @@ -158,7 +158,7 @@ static struct bpf_map *reuseport_array_alloc(union bpf_attr *attr) return ERR_PTR(-EPERM); /* allocate all map elements and zero-initialize them */ - array = bpf_map_area_alloc(struct_size(array, ptrs, attr->max_entries), numa_node); + array = bpf_map_area_alloc(struct_size(array, ptrs, attr->max_entries), numa_node, NULL); if (!array) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index 74dd8dc..5eb7820 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -59,7 +59,8 @@ struct bpf_ringbuf_hdr { u32 pg_off; }; -static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) +static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node, + struct bpf_map *map) { const gfp_t flags = GFP_KERNEL_ACCOUNT | __GFP_RETRY_MAYFAIL | __GFP_NOWARN | __GFP_ZERO; @@ -89,7 +90,7 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) * user-space implementations significantly. */ array_size = (nr_meta_pages + 2 * nr_data_pages) * sizeof(*pages); - pages = bpf_map_area_alloc(array_size, numa_node); + pages = bpf_map_area_alloc(array_size, numa_node, map); if (!pages) return NULL; @@ -127,11 +128,12 @@ static void bpf_ringbuf_notify(struct irq_work *work) wake_up_all(&rb->waitq); } -static struct bpf_ringbuf *bpf_ringbuf_alloc(size_t data_sz, int numa_node) +static struct bpf_ringbuf *bpf_ringbuf_alloc(size_t data_sz, int numa_node, + struct bpf_map *map) { struct bpf_ringbuf *rb; - rb = bpf_ringbuf_area_alloc(data_sz, numa_node); + rb = bpf_ringbuf_area_alloc(data_sz, numa_node, map); if (!rb) return NULL; @@ -164,13 +166,14 @@ static struct bpf_map *ringbuf_map_alloc(union bpf_attr *attr) return ERR_PTR(-E2BIG); #endif - rb_map = bpf_map_area_alloc(sizeof(*rb_map), NUMA_NO_NODE); + rb_map = bpf_map_area_alloc(sizeof(*rb_map), NUMA_NO_NODE, NULL); if (!rb_map) return ERR_PTR(-ENOMEM); bpf_map_init_from_attr(&rb_map->map, attr); - rb_map->rb = bpf_ringbuf_alloc(attr->max_entries, rb_map->map.numa_node); + rb_map->rb = bpf_ringbuf_alloc(attr->max_entries, rb_map->map.numa_node, + &rb_map->map); if (!rb_map->rb) { bpf_map_area_free(rb_map, &rb_map->map); return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index 042b7d2..9440fab 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -49,7 +49,8 @@ static int prealloc_elems_and_freelist(struct bpf_stack_map *smap) int err; smap->elems = bpf_map_area_alloc(elem_size * smap->map.max_entries, - smap->map.numa_node); + smap->map.numa_node, + &smap->map); if (!smap->elems) return -ENOMEM; @@ -100,7 +101,7 @@ static struct bpf_map *stack_map_alloc(union bpf_attr *attr) return ERR_PTR(-E2BIG); cost = n_buckets * sizeof(struct stack_map_bucket *) + sizeof(*smap); - smap = bpf_map_area_alloc(cost, bpf_map_attr_numa_node(attr)); + smap = bpf_map_area_alloc(cost, bpf_map_attr_numa_node(attr), NULL); if (!smap) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 01d8d4a..02ce7e9 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -362,9 +362,21 @@ static void *__bpf_map_area_alloc(u64 size, int numa_node, bool mmapable) flags, numa_node, __builtin_return_address(0)); } -void *bpf_map_area_alloc(u64 size, int numa_node) +void *bpf_map_area_alloc(u64 size, int numa_node, struct bpf_map *map) { - return __bpf_map_area_alloc(size, numa_node, false); + struct mem_cgroup *memcg, *old_memcg; + void *ptr; + + if (!map) + return __bpf_map_area_alloc(size, numa_node, false); + + memcg = bpf_map_get_memcg(map); + old_memcg = set_active_memcg(memcg); + ptr = __bpf_map_area_alloc(size, numa_node, false); + set_active_memcg(old_memcg); + bpf_map_put_memcg(memcg); + + return ptr; } void *bpf_map_area_mmapable_alloc(u64 size, int numa_node) diff --git a/net/core/sock_map.c b/net/core/sock_map.c index e8f414a..2b3b24e 100644 --- a/net/core/sock_map.c +++ b/net/core/sock_map.c @@ -41,7 +41,7 @@ static struct bpf_map *sock_map_alloc(union bpf_attr *attr) attr->map_flags & ~SOCK_CREATE_FLAG_MASK) return ERR_PTR(-EINVAL); - stab = bpf_map_area_alloc(sizeof(*stab), NUMA_NO_NODE); + stab = bpf_map_area_alloc(sizeof(*stab), NUMA_NO_NODE, NULL); if (!stab) return ERR_PTR(-ENOMEM); @@ -50,7 +50,8 @@ static struct bpf_map *sock_map_alloc(union bpf_attr *attr) stab->sks = bpf_map_area_alloc((u64) stab->map.max_entries * sizeof(struct sock *), - stab->map.numa_node); + stab->map.numa_node, + &stab->map); if (!stab->sks) { bpf_map_area_free(stab, &stab->map); return ERR_PTR(-ENOMEM); @@ -1076,7 +1077,7 @@ static struct bpf_map *sock_hash_alloc(union bpf_attr *attr) if (attr->key_size > MAX_BPF_STACK) return ERR_PTR(-E2BIG); - htab = bpf_map_area_alloc(sizeof(*htab), NUMA_NO_NODE); + htab = bpf_map_area_alloc(sizeof(*htab), NUMA_NO_NODE, NULL); if (!htab) return ERR_PTR(-ENOMEM); @@ -1093,7 +1094,8 @@ static struct bpf_map *sock_hash_alloc(union bpf_attr *attr) htab->buckets = bpf_map_area_alloc(htab->buckets_num * sizeof(struct bpf_shtab_bucket), - htab->map.numa_node); + htab->map.numa_node, + &htab->map); if (!htab->buckets) { err = -ENOMEM; goto free_htab; diff --git a/net/xdp/xskmap.c b/net/xdp/xskmap.c index 5abb87e..beb11fd 100644 --- a/net/xdp/xskmap.c +++ b/net/xdp/xskmap.c @@ -75,7 +75,7 @@ static struct bpf_map *xsk_map_alloc(union bpf_attr *attr) numa_node = bpf_map_attr_numa_node(attr); size = struct_size(m, xsk_map, attr->max_entries); - m = bpf_map_area_alloc(size, numa_node); + m = bpf_map_area_alloc(size, numa_node, NULL); if (!m) return ERR_PTR(-ENOMEM); From patchwork Thu Aug 18 14:31:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12947991 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE376C00140 for ; Thu, 18 Aug 2022 21:28:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C9B08D0002; Thu, 18 Aug 2022 17:28:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 379688D0001; Thu, 18 Aug 2022 17:28:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 219ED8D0002; Thu, 18 Aug 2022 17:28:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0EC478D0001 for ; Thu, 18 Aug 2022 17:28:50 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C202841A54 for ; Thu, 18 Aug 2022 21:28:49 +0000 (UTC) X-FDA: 79814003178.04.CB739F5 Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42]) by imf01.hostedemail.com (Postfix) with ESMTP id 35C67405AE for ; Thu, 18 Aug 2022 21:14:20 +0000 (UTC) Received: by mail-qv1-f42.google.com with SMTP id y4so2104547qvp.3 for ; Thu, 18 Aug 2022 14:14:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=wfVdiCm3ki6hY+6umh5GbzZ8nNKi0kDY1+Bd1ej/jM8=; b=RUKuxfm9E43E5qFQxMfS7R7LCmH7AoY/Gxkx5z0NnN217RoW+idxLKZwqN214SOPzH R7piyA5YeKysCY6mM/eUv7rthB/rLsOzSkIiO6t4LEipXfMIXU3WwDO0ARd+SR3DQWU1 tkAyiSTiSkiobEoeVQCt/ZI9GvL1pe3bJtnh8JJxazxASGrh8rnB1apZ+QqSExE7b2kC qD5Wc/SjTr44YPyHd3iQ7qvxVOILbekjFOJ7t7XHJTssYPcUuTdvujBMfI6eyq6W+J+q KvIEc61DmQwriVen4GenWzpC5f0TiuwN6owAG2711o9IGRTeXtqEAn03bgqrgWggFrhR pVPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=wfVdiCm3ki6hY+6umh5GbzZ8nNKi0kDY1+Bd1ej/jM8=; b=2dnaYBl/XgrFM3CQsVyPOodRuCb1roSI0ZPbfXgApM1lpPFMzKqp6J6U02APUW6FXV WlidYijxMLtxrnLMU/0Qvypj3XHMYVn11e5E/SjtnLfnBp8iVLpaeVY9mB7TKaujijZa +7Ijxy+Q7R2B1nta4DjCtWdMYSeDe4Rm0jhGwXUd/eAX2GKXqbR+wMi5L4Np6C5ewtBk 7ZAgqa0ah7ctrQymEOIPeGaTcZEhhPBBVLPoADaC9wIm+lktYoFFBjLBSChQ/5WP3k9z mY8EYP8m2Kn0hmqdye8Vei2rToe4UOzh2TUS6gYlFlml/O9bPqF+AwkpdQPw3v4LOqtY 7+LQ== X-Gm-Message-State: ACgBeo0uonxECV8j9YaV8u/xGtfSehKIgo2o2XtWD81xiIwJa/RBxDmB jWNC+R0uTKNLJm/8OaMNunWPyPw2lnPgSWjP X-Google-Smtp-Source: AA6agR7BKXRJhkd3R6jnBGYeXTOD46Z2q1NB8JbJ2YahjboeHT+Ec+2dxbjICtwclbkbfExrPr+dDw== X-Received: by 2002:a05:6a00:1709:b0:52e:677b:702b with SMTP id h9-20020a056a00170900b0052e677b702bmr3231480pfc.19.1660833119614; Thu, 18 Aug 2022 07:31:59 -0700 (PDT) Received: from vultr.guest ([45.32.72.237]) by smtp.gmail.com with ESMTPSA id h5-20020a63f905000000b003fdc16f5de2sm1379124pgi.15.2022.08.18.07.31.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Aug 2022 07:31:58 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao , Andrii Nakryiko Subject: [PATCH bpf-next v2 07/12] bpf: Introduce new helpers bpf_ringbuf_pages_{alloc,free} Date: Thu, 18 Aug 2022 14:31:13 +0000 Message-Id: <20220818143118.17733-8-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220818143118.17733-1-laoar.shao@gmail.com> References: <20220818143118.17733-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660857262; a=rsa-sha256; cv=none; b=rqT3hOmbpznR8HdgYAlTR4zU8mMKE+GbDw6Mn6YZnlUc9D/uGed1cYiYQOYFME/Lb6StTd nAZoRhQhU/4RRYpGugHskXlaygRO+kXOwG1ezFzusUpGVu9R/+1f/8EeA0WYkhjhwkanOF iq5hdr23dkI4+bvLMsQZtfFtoiu58W8= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=RUKuxfm9; spf=pass (imf01.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.219.42 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660857262; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wfVdiCm3ki6hY+6umh5GbzZ8nNKi0kDY1+Bd1ej/jM8=; b=5UbIBqReOG++39G1JuviBbS5GM4vHnSsa/MbsKzQNvEN473nW3m7c1s2Ud8Gf1aiTNWvlP wSZ7QITod9huxPTN6uJw+k+q69jEZPkHS3fnGcZDWHbPENP6OpHXfF1YL6UIEvv3nZeSV+ gwSEOhgFdoglBPwkWc6AWBASMzWBxAM= X-Rspamd-Server: rspam11 Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=RUKuxfm9; spf=pass (imf01.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.219.42 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Stat-Signature: x4xgfqegty67ygn39tbhqkcjeeb4gbnx X-Rspamd-Queue-Id: 35C67405AE X-HE-Tag: 1660857260-592292 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allocate pages related memory into the new helper bpf_ringbuf_pages_alloc(), then it can be handled as a single unit. Suggested-by: Andrii Nakryiko Signed-off-by: Yafang Shao Acked-by: Andrii Nakryiko --- kernel/bpf/ringbuf.c | 80 ++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 56 insertions(+), 24 deletions(-) diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index 5eb7820..1e7284c 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -59,6 +59,57 @@ struct bpf_ringbuf_hdr { u32 pg_off; }; +static void bpf_ringbuf_pages_free(struct page **pages, int nr_pages) +{ + int i; + + for (i = 0; i < nr_pages; i++) + __free_page(pages[i]); + bpf_map_area_free(pages, NULL); +} + +static struct page **bpf_ringbuf_pages_alloc(struct bpf_map *map, + int nr_meta_pages, + int nr_data_pages, + int numa_node, + const gfp_t flags) +{ + int nr_pages = nr_meta_pages + nr_data_pages; + struct mem_cgroup *memcg, *old_memcg; + struct page **pages, *page; + int array_size; + int i; + + memcg = bpf_map_get_memcg(map); + old_memcg = set_active_memcg(memcg); + array_size = (nr_meta_pages + 2 * nr_data_pages) * sizeof(*pages); + pages = bpf_map_area_alloc(array_size, numa_node, NULL); + if (!pages) + goto err; + + for (i = 0; i < nr_pages; i++) { + page = alloc_pages_node(numa_node, flags, 0); + if (!page) { + nr_pages = i; + goto err_free_pages; + } + pages[i] = page; + if (i >= nr_meta_pages) + pages[nr_data_pages + i] = page; + } + set_active_memcg(old_memcg); + bpf_map_put_memcg(memcg); + + return pages; + +err_free_pages: + bpf_ringbuf_pages_free(pages, nr_pages); +err: + set_active_memcg(old_memcg); + bpf_map_put_memcg(memcg); + return NULL; +} + static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node, struct bpf_map *map) { @@ -67,10 +118,8 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node, int nr_meta_pages = RINGBUF_PGOFF + RINGBUF_POS_PAGES; int nr_data_pages = data_sz >> PAGE_SHIFT; int nr_pages = nr_meta_pages + nr_data_pages; - struct page **pages, *page; struct bpf_ringbuf *rb; - size_t array_size; - int i; + struct page **pages; /* Each data page is mapped twice to allow "virtual" * continuous read of samples wrapping around the end of ring @@ -89,22 +138,11 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node, * when mmap()'ed in user-space, simplifying both kernel and * user-space implementations significantly. */ - array_size = (nr_meta_pages + 2 * nr_data_pages) * sizeof(*pages); - pages = bpf_map_area_alloc(array_size, numa_node, map); + pages = bpf_ringbuf_pages_alloc(map, nr_meta_pages, nr_data_pages, + numa_node, flags); if (!pages) return NULL; - for (i = 0; i < nr_pages; i++) { - page = alloc_pages_node(numa_node, flags, 0); - if (!page) { - nr_pages = i; - goto err_free_pages; - } - pages[i] = page; - if (i >= nr_meta_pages) - pages[nr_data_pages + i] = page; - } - rb = vmap(pages, nr_meta_pages + 2 * nr_data_pages, VM_MAP | VM_USERMAP, PAGE_KERNEL); if (rb) { @@ -114,10 +152,6 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node, return rb; } -err_free_pages: - for (i = 0; i < nr_pages; i++) - __free_page(pages[i]); - bpf_map_area_free(pages, NULL); return NULL; } @@ -188,12 +222,10 @@ static void bpf_ringbuf_free(struct bpf_ringbuf *rb) * to unmap rb itself with vunmap() below */ struct page **pages = rb->pages; - int i, nr_pages = rb->nr_pages; + int nr_pages = rb->nr_pages; vunmap(rb); - for (i = 0; i < nr_pages; i++) - __free_page(pages[i]); - bpf_map_area_free(pages, NULL); + bpf_ringbuf_pages_free(pages, nr_pages); } static void ringbuf_map_free(struct bpf_map *map) From patchwork Thu Aug 18 14:31:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12947816 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94C25C32773 for ; Thu, 18 Aug 2022 20:20:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 055C28D0003; Thu, 18 Aug 2022 16:20:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 005098D0002; Thu, 18 Aug 2022 16:20:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE8598D0003; Thu, 18 Aug 2022 16:20:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D13CA8D0002 for ; Thu, 18 Aug 2022 16:20:50 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id AEA5B1A1F1C for ; Thu, 18 Aug 2022 20:20:50 +0000 (UTC) X-FDA: 79813831860.09.BE4F8CA Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf22.hostedemail.com (Postfix) with ESMTP id DADEDC136B for ; Thu, 18 Aug 2022 20:12:14 +0000 (UTC) Received: by mail-pl1-f177.google.com with SMTP id m2so2418823pls.4 for ; Thu, 18 Aug 2022 13:12:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=6MxaJVkVqzqq1Kknlzxc81EzIkRTYeOwRhc2SFg8N/c=; b=UwVG/VSP0500nvEMvFgAFhuzl7jOoNgmsUjHA03AL7Loq4gte3g3tEw92bhEq4R3/I 9BO87zC4T+n1Isqcj6XWQaRGJh9dVeNMeVJHqNXReWmH1FFppfSnTUWTSsii0f4j8Mrf fGspLs9apHqZWT6hmOW66VERvD64fONYTuCqCPRfLJttOt8GaR/XXzexgSDul6/v8Qzc 1s22qNFwBX3yTFKyIH6vIVN7bV7rvuxaShBZH4Xkp/O0XDbckHuH/goksJJndgMlSnA/ 0RjflanMCjl9/cLUXSUkd7M15JXVuotakDNoBl5I76iaQJTxRzCgDXb1/eXTG13AGKqP 3ThA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=6MxaJVkVqzqq1Kknlzxc81EzIkRTYeOwRhc2SFg8N/c=; b=bManadum2yklEogMmy9PwP5R+bO8umx+F3tFgNjxeoXFO9tmyI6QAbWe3TS5XT16/w 2iqP5OkWLXn4u17Dp0epEcGCZruSR6r8wUVrta29KYMNjDruK9oEZUsVScjIPN0JWCwq ldZmYc9OSZPg12aD3S3beARdf9/OIPf3GeWcEUkPPgylHvYSg5b/0Ym4yQhZsIvaMOCI 5SEAk4D7YvQtUbtxgLwewBASuH8mlWW1zzjI/zFkxDyiamhZ5civg/seZ4xcCOZGfCRJ t1/tkSecaYTmoX+TxWoOr43BxOaxSZxNbaxwNSvLSfjmZH36nJNNqhNThPDGmj+WOZuA 835w== X-Gm-Message-State: ACgBeo2qJJlWLgVHy9yEhnRIFZOfcGbVYAgHmNLo4+04GePg/6gSsOac J3GbyBcLoIMtg2DK75JIM7n574oUbN/IANp/ X-Google-Smtp-Source: AA6agR72M8P6pJtUCoBgO4ikpegsRLjp2RJjn9maPzCVOcuC6aNzZXvVy3iAP3lFIy0mTaP+56rrqQ== X-Received: by 2002:a17:902:8e8c:b0:171:2a36:e390 with SMTP id bg12-20020a1709028e8c00b001712a36e390mr3080551plb.77.1660833122298; Thu, 18 Aug 2022 07:32:02 -0700 (PDT) Received: from vultr.guest ([45.32.72.237]) by smtp.gmail.com with ESMTPSA id h5-20020a63f905000000b003fdc16f5de2sm1379124pgi.15.2022.08.18.07.31.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Aug 2022 07:32:01 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v2 08/12] bpf: Use bpf_map_kzalloc in arraymap Date: Thu, 18 Aug 2022 14:31:14 +0000 Message-Id: <20220818143118.17733-9-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220818143118.17733-1-laoar.shao@gmail.com> References: <20220818143118.17733-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660853535; a=rsa-sha256; cv=none; b=DONcL7EWAoxceR6dz6puYutEV6opy0EDbRO06YLgcgoUemPMiqpnxPSo7DAz/uqn7pVBOM MCMtIyz29zYxVav3A3jGb2tjDh9CbRA/gLePAs/zTT4UK6c8KLCB91N4nBoxIhpraZqcNQ K13O9su3DbZCZrjNB0FMXkeTQGjN/Ro= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="UwVG/VSP"; spf=pass (imf22.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660853535; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6MxaJVkVqzqq1Kknlzxc81EzIkRTYeOwRhc2SFg8N/c=; b=erWl8ozZn0Iet36YF5U5DI26SA+tOsBzQ4AX2HRcN97MCzHPFT6UjHXh9U+B07JD9eotFw efF7NMx8IRV6pf/BezhXOU4Gzg2IeZDtdOU/L1oU/OShCz6g4UcrDMKLEv2uW/BQfgFq2y sWk09FxGpWgebB5/fnQFbRyJ2WGPlOU= X-Rspamd-Server: rspam11 Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="UwVG/VSP"; spf=pass (imf22.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Stat-Signature: y6ysm1oxqt3jhs71biwwc4834f4o4ga3 X-Rspamd-Queue-Id: DADEDC136B X-HE-Tag: 1660853534-462956 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allocates memory after map creation, then we can use the generic helper bpf_map_kzalloc() instead of the open-coded kzalloc(). Signed-off-by: Yafang Shao --- kernel/bpf/arraymap.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index 80974c5..3039832 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -1090,20 +1090,20 @@ static struct bpf_map *prog_array_map_alloc(union bpf_attr *attr) struct bpf_array_aux *aux; struct bpf_map *map; - aux = kzalloc(sizeof(*aux), GFP_KERNEL_ACCOUNT); - if (!aux) + map = array_map_alloc(attr); + if (IS_ERR(map)) return ERR_PTR(-ENOMEM); + aux = bpf_map_kzalloc(map, sizeof(*aux), GFP_KERNEL); + if (!aux) { + array_map_free(map); + return ERR_PTR(-ENOMEM); + } + INIT_WORK(&aux->work, prog_array_map_clear_deferred); INIT_LIST_HEAD(&aux->poke_progs); mutex_init(&aux->poke_mutex); - map = array_map_alloc(attr); - if (IS_ERR(map)) { - kfree(aux); - return map; - } - container_of(map, struct bpf_array, map)->aux = aux; aux->map = map; From patchwork Thu Aug 18 14:31:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12947863 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 613B9C00140 for ; Thu, 18 Aug 2022 20:38:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EB3ED8D0003; Thu, 18 Aug 2022 16:37:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E630B8D0002; Thu, 18 Aug 2022 16:37:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D05108D0003; Thu, 18 Aug 2022 16:37:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C3CF38D0002 for ; Thu, 18 Aug 2022 16:37:59 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 8EC151421F3 for ; Thu, 18 Aug 2022 20:37:59 +0000 (UTC) X-FDA: 79813875078.06.01BC84F Received: from mail-io1-f46.google.com (mail-io1-f46.google.com [209.85.166.46]) by imf05.hostedemail.com (Postfix) with ESMTP id C4C8A10049F for ; Thu, 18 Aug 2022 20:16:17 +0000 (UTC) Received: by mail-io1-f46.google.com with SMTP id y187so1948323iof.0 for ; Thu, 18 Aug 2022 13:16:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=JWj3ghaLi18D37Utgpu0sbreSBl9gVi+HG8XRXJ1k4E=; b=J/vjjRSdspqfz5QQqVSBxlG3cZRAEs5vE1YwK2jl25Xu07tjrYiNRrxlucPklmoR9x fBJUXAQ/LT3QR42GeO3UqrnqRp1RSdz+Swira1oRSwzSExCLMH+CuRCFEcRhvHrFW78v tng9ceDAB9ew2ydHzWZnZxvNtW2T8poiW5TUemW/mvFECm1CyXNstuUVpnJyVVd0JQ42 YAHYbedJSYa9NAsf76auiydFa3uO2+AC0S3Uw0DnpnJXy7daYP7XQjFs//HOX0+9aAIU MXoS+DnB6GhRcPCFe5Xzh+tzSuYHsA0ibH1Y2orQGs1+05k1JUR/i+F72OtwdTOj61Wn 9JGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=JWj3ghaLi18D37Utgpu0sbreSBl9gVi+HG8XRXJ1k4E=; b=CU0XdFFeNBGv8e6hXwNAUOtFwD7pe9KnPIZkHUcQmvKgG5YdHAonY/Tzs3YQy0J6Vo JUpViIJmcoPpoIZn6MwNT+ACRBfPdUTo1BFlpZducLLL53oDCycW+Mnu7lZadpmC78PV 2lXL49UufLtggWiddiOERNtiTIybx7bZFVHjLkbBE5w33IdSgfDtBEKO15E+hoyxM15S Q9XiQUDBy9KP3sMHa3t0Qspe84gRZCYmOmstFDZyUv8DjzXQyBlUMUn9nAlAhqskkqPW uEChT5M85Fgfw29qm3g3u0X2qBg3YB9QyP/yvrM/T7FSFTx3/tLF5mFAXfksnAxxGiqs RhuQ== X-Gm-Message-State: ACgBeo02VOtnbar/HDPcpE2/p9AbMyjAWSE72rP2gctzyTlUu84HXNx4 iCB1JwFk6fPLuz3ZRJAkdBCGubeOaQ357V21 X-Google-Smtp-Source: AA6agR6tlBl2i+jwjIYUJY+noBYDNAFEC5gq9HwJ1ZBb4KG+9KIyTOYDLgt8RG6tQbIPRV4tNA0M8g== X-Received: by 2002:a63:e90c:0:b0:422:5ab9:99d6 with SMTP id i12-20020a63e90c000000b004225ab999d6mr2697130pgh.394.1660833125123; Thu, 18 Aug 2022 07:32:05 -0700 (PDT) Received: from vultr.guest ([45.32.72.237]) by smtp.gmail.com with ESMTPSA id h5-20020a63f905000000b003fdc16f5de2sm1379124pgi.15.2022.08.18.07.32.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Aug 2022 07:32:04 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v2 09/12] bpf: Use bpf_map_kvcalloc in bpf_local_storage Date: Thu, 18 Aug 2022 14:31:15 +0000 Message-Id: <20220818143118.17733-10-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220818143118.17733-1-laoar.shao@gmail.com> References: <20220818143118.17733-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660853778; a=rsa-sha256; cv=none; b=SXve7AmGDWBq9JmSO+x7l/apmzpiQbDvAKYcmFjqfv0xpIYLqX0c/G30nYqZJJpd+K/a0E 6OFr0/XZRCCARrZhl6dEkugqxlsGqjy6PvNphb+aFYNGpmy0JRqvFLL2azKGeyaiwlxiUU NhfiP9vwZZRBBr1vZ+xTLFejYh7eEGQ= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="J/vjjRSd"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf05.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.166.46 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660853778; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JWj3ghaLi18D37Utgpu0sbreSBl9gVi+HG8XRXJ1k4E=; b=b64bSxk4f8Majidm/6QY2qNE8FX3rBta1K44Vz67aBp5XdQFUuU6++FL58cCRpqVeQzRzE JepJwUT5cUiUexBh9Xap0+eBvBr4UP3E3txIQlu2fl1wJ0KWZOeFzG+sbCLrcDo+GPYf8X cIcd6AG/2wS2WPeXzBtIJoCzuWfgmkw= X-Stat-Signature: biwf45epchtuemmk1kyqp9xpqqu6mg8m X-Rspamd-Queue-Id: C4C8A10049F Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="J/vjjRSd"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf05.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.166.46 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com X-Rspam-User: X-Rspamd-Server: rspam06 X-HE-Tag: 1660853777-731612 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce new helper bpf_map_kvcalloc() for this memory allocation. Signed-off-by: Yafang Shao --- include/linux/bpf.h | 8 ++++++++ kernel/bpf/bpf_local_storage.c | 4 ++-- kernel/bpf/syscall.c | 15 +++++++++++++++ 3 files changed, 25 insertions(+), 2 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 066f351d..f95a2df 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1656,6 +1656,8 @@ int generic_map_delete_batch(struct bpf_map *map, void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, int node); void *bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags); +void *bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size, + gfp_t flags); void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, size_t align, gfp_t flags); #else @@ -1672,6 +1674,12 @@ void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, return kzalloc(size, flags); } +static inline void * +bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size, gfp_t flags) +{ + return kvcalloc(n, size, flags); +} + static inline void __percpu * bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, size_t align, gfp_t flags) diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index 67ab249..71d5bf1 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -620,8 +620,8 @@ struct bpf_local_storage_map *bpf_local_storage_map_alloc(union bpf_attr *attr) nbuckets = max_t(u32, 2, nbuckets); smap->bucket_log = ilog2(nbuckets); - smap->buckets = kvcalloc(sizeof(*smap->buckets), nbuckets, - GFP_USER | __GFP_NOWARN | __GFP_ACCOUNT); + smap->buckets = bpf_map_kvcalloc(&smap->map, sizeof(*smap->buckets), + nbuckets, GFP_USER | __GFP_NOWARN); if (!smap->buckets) { bpf_map_area_free(smap, &smap->map); return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 02ce7e9..2bd3bcf 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -489,6 +489,21 @@ void *bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags) return ptr; } +void *bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size, + gfp_t flags) +{ + struct mem_cgroup *memcg, *old_memcg; + void *ptr; + + memcg = bpf_map_get_memcg(map); + old_memcg = set_active_memcg(memcg); + ptr = kvcalloc(n, size, flags | __GFP_ACCOUNT); + set_active_memcg(old_memcg); + bpf_map_put_memcg(memcg); + + return ptr; +} + void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, size_t align, gfp_t flags) { From patchwork Thu Aug 18 14:31:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12947994 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F7C6C00140 for ; Thu, 18 Aug 2022 21:32:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EA1FA8D0002; Thu, 18 Aug 2022 17:32:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E4FEF8D0001; Thu, 18 Aug 2022 17:32:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF09D8D0002; Thu, 18 Aug 2022 17:32:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BCE718D0001 for ; Thu, 18 Aug 2022 17:32:35 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 958E7140671 for ; Thu, 18 Aug 2022 21:32:35 +0000 (UTC) X-FDA: 79814012670.18.8EDFF17 Received: by imf24.hostedemail.com (Postfix, from userid 200) id F0B2618003D; Thu, 18 Aug 2022 15:43:04 +0000 (UTC) Received: from mail-il1-f173.google.com (mail-il1-f173.google.com [209.85.166.173]) by imf24.hostedemail.com (Postfix) with ESMTP id 02A24180719 for ; Thu, 18 Aug 2022 15:42:57 +0000 (UTC) Received: by mail-il1-f173.google.com with SMTP id l15so996480ilf.4 for ; Thu, 18 Aug 2022 08:42:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=SdEsGdLtUTnGTQlmpkcqbLnnsMN1enZs7HVXG4Feokc=; b=iM7+xYE3ZpaZKh2j/9NZNfU6WfXX1qnhHz90ontLLo/0kh1EMI2GgT3hsaLXjYuqB+ Bc+3EgV3QlYwmbQSgP6DF4xyLZM3XeV8u8WuYlvq9gCHoYixiOEju6GCoB9t8bIaV/h0 8FIPglxdIU0iKsIttK5lqBYDQPvSCT6m7EJ3UB3fKymtdyT2oCkbs3nW8AuR5HmU6c4U plqqyHQwuiSWQJN/q+zwHtXRUvp4HuPyDXIB+i0qM1xVXS6Uwz7nDD7ePd9S4T3uG5Lr XMmyq9x38HyRsbixPhyelgKdWfrTLNyNj+YPuYZOVsEtiDUSthZTWGolXni2ZmEiLLAD tSiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=SdEsGdLtUTnGTQlmpkcqbLnnsMN1enZs7HVXG4Feokc=; b=cFqSg8BeDf/pEhwieFQ9gj3MZQ3SiExyQoz45X4zWKHLnwk3+EQwNMkJhtRU7kpCBA 3D0wNDWQOP3BaW9LtcYjw7qHlWU/jP9wygXEWy2bGmTM3W8gL0U8/xdLlmSWHGSgIj6F mgl8IsOZG+R2q39YrEx09EoWCEQrokHpBdc6j4H9RKW4C3QQUYmLmPZJm+pev95dlCrB vgruQRacFDDjpLPwBmY2eIu+s6puhsLQBWyEjcAFsDCOFd7jIFlic5wAzZr7N7iTpjhv a/OCDaQ3cltRNq7ji5YMU+2kFUBrJpoKHy4xNAedsF33DUT5psSET/gUkPZPGoSwREZ+ e7Ag== X-Gm-Message-State: ACgBeo3gctEeGEfctjsaaAwvNwwthdaIbkameBeZYn9eojyyX6js8hO1 T+nZcdi4oQNA38AsPbloo+mYHtYSOWVv2AVn X-Google-Smtp-Source: AA6agR6tGbyVsMQyI/lXKdwbyTP8XOf52SpT7zfV868FB9HDcvT6Z5ZuEY1X02mmUFcf4qavA1RCrg== X-Received: by 2002:a63:2208:0:b0:429:9444:85be with SMTP id i8-20020a632208000000b00429944485bemr2660287pgi.236.1660833128367; Thu, 18 Aug 2022 07:32:08 -0700 (PDT) Received: from vultr.guest ([45.32.72.237]) by smtp.gmail.com with ESMTPSA id h5-20020a63f905000000b003fdc16f5de2sm1379124pgi.15.2022.08.18.07.32.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Aug 2022 07:32:07 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v2 10/12] mm, memcg: Add new helper get_obj_cgroup_from_cgroup Date: Thu, 18 Aug 2022 14:31:16 +0000 Message-Id: <20220818143118.17733-11-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220818143118.17733-1-laoar.shao@gmail.com> References: <20220818143118.17733-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660837383; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SdEsGdLtUTnGTQlmpkcqbLnnsMN1enZs7HVXG4Feokc=; b=qJmL93NnL5zX5lSKoB1Btx7WQImdI1yTsSCL5e+9wi63/cWC0vKvMZ3JeB73zoQyfBAfkT b4OQVDvqhY+bTGxaIAYCUP36DCZPqVOwwyczpZfxpujN9S/GjXUKvANWCPfAlVqBoj68N+ /ZfRXP8yuyVVxT6FFI15jhS+fBBRWXk= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=iM7+xYE3; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf24.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.166.173 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660837383; a=rsa-sha256; cv=none; b=ViNqkvz/7ysxxa5kOMRP6cN34Xie0Z5bW9J7Iv7zsDJpZ75DtMu8TFrxf4MXu0Nla7gYHd X4SFD1zWniLNnROgWZ8xOoYBoOhY8skr97a/VnIaPopWm3KOa2+euKXSYa6Do//zZjA4bf DX48VntpDyOGKXgL6PRQPBfqZLVlXHU= X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 02A24180719 Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=iM7+xYE3; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf24.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.166.173 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com X-Rspam-User: X-Stat-Signature: roaius3h58qqanuizty8onar8fmhw1zy X-HE-Tag: 1660837377-729775 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We want to open a cgroup directory and pass the fd into kernel, and then in the kernel we can charge the allocated memory into the open cgroup if it has valid memory subsystem. In the bpf subsystem, the opened cgroup will be store as a struct obj_cgroup pointer, so a new helper get_obj_cgroup_from_cgroup() is introduced. It also add a comment on why the helper __get_obj_cgroup_from_memcg() must be protected by rcu read lock. Signed-off-by: Yafang Shao --- include/linux/memcontrol.h | 1 + mm/memcontrol.c | 47 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 2f0a611..901a921 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1713,6 +1713,7 @@ static inline void set_shrinker_bit(struct mem_cgroup *memcg, int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order); void __memcg_kmem_uncharge_page(struct page *page, int order); +struct obj_cgroup *get_obj_cgroup_from_cgroup(struct cgroup *cgrp); struct obj_cgroup *get_obj_cgroup_from_current(void); struct obj_cgroup *get_obj_cgroup_from_page(struct page *page); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 618c366..0409cc4 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2895,6 +2895,14 @@ struct mem_cgroup *mem_cgroup_from_obj(void *p) return page_memcg_check(folio_page(folio, 0)); } +/* + * Pls. note that the memg->objcg can be freed after it is deferenced, + * that can happen when the memcg is changed from online to offline. + * So this helper must be protected by read rcu lock. + * + * After obj_cgroup_tryget(), it is safe to use the objcg outside of the rcu + * read-side critical section. + */ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) { struct obj_cgroup *objcg = NULL; @@ -2908,6 +2916,45 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) return objcg; } +static struct obj_cgroup *get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) +{ + struct obj_cgroup *objcg; + + if (memcg_kmem_bypass()) + return NULL; + + rcu_read_lock(); + objcg = __get_obj_cgroup_from_memcg(memcg); + rcu_read_unlock(); + return objcg; +} + +struct obj_cgroup *get_obj_cgroup_from_cgroup(struct cgroup *cgrp) +{ + struct cgroup_subsys_state *css; + struct mem_cgroup *memcg; + struct obj_cgroup *objcg; + + rcu_read_lock(); + css = rcu_dereference(cgrp->subsys[memory_cgrp_id]); + if (!css || !css_tryget_online(css)) { + rcu_read_unlock(); + return ERR_PTR(-EINVAL); + } + rcu_read_unlock(); + + memcg = mem_cgroup_from_css(css); + if (!memcg) { + css_put(css); + return ERR_PTR(-EINVAL); + } + + objcg = get_obj_cgroup_from_memcg(memcg); + css_put(css); + + return objcg; +} + __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) { struct obj_cgroup *objcg = NULL; From patchwork Thu Aug 18 14:31:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12948192 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63FDDC00140 for ; Thu, 18 Aug 2022 23:47:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BC5C18D0002; Thu, 18 Aug 2022 19:47:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B4F328D0001; Thu, 18 Aug 2022 19:47:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 979508D0002; Thu, 18 Aug 2022 19:47:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 820268D0001 for ; Thu, 18 Aug 2022 19:47:44 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 450AA14014E for ; Thu, 18 Aug 2022 23:47:44 +0000 (UTC) X-FDA: 79814353248.29.2F81E4C Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf31.hostedemail.com (Postfix) with ESMTP id 17841250ED for ; Thu, 18 Aug 2022 23:32:11 +0000 (UTC) Received: by mail-pl1-f174.google.com with SMTP id u22so2739477plq.12 for ; Thu, 18 Aug 2022 16:32:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=Vt3B1B2xA9UnavRf/lLTg9rVQPFR8dNDUq2oJHhoN6w=; b=n3Innaui3HRKiBQRFol3saxsQBriZpyrVHcrf3ZaokVPTDqaCPmdRetYE3fsOEwFOH /XcLDSud/O5Qj2YCIl0COAWJOUeDis98JL6Rfa5GvtmHVCLEd+CAzRjQze2UH0OoSGZx pCAJEjdtM0GU6Zcjaa+gy4wARXquYBGwEJo+Sh2iRjUQ9TTwr7hIZDYg8f5gOGukVco9 SVAGlsVkK1bSFAoraBEK90o3jel3oLMoX4z3yf5lElgiCUZu4VAwZC4CFDkEpO+qYWAG ovYrU07xdFn1NT6gODYEYxjeeofRq0zDDA0tMSAf4nLStNI1/CIduH6TlsBFhW0Bu4Do 9cug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=Vt3B1B2xA9UnavRf/lLTg9rVQPFR8dNDUq2oJHhoN6w=; b=b2H7LmguLk9zb25cOeNLeN/ayvQsR3UwXQS2WbAI++5IaCkEjaBE3oKd80ODAphWfC YQmTfIIsl/vu+fXXfz4dLhXoAenGJvL5ZhM9OJUyzvpSvyCTvVZVEA39pNQNpPpUwEOE pmxrPWHjkzRFoNqAwukI8vM+22wYeoJC3+R6PhU5gYcy5TMpp6u1E+1Eth7cKgFdNMsa FiUvJJ8eAcHQGAfYlHYzprrDJ3b2KDG+rt8q4S7c8VCld6Min6W/VDl3OYakh8TUhCUY +UjxYMZsFRnYbDCskHhLRsdCmnZIXRNDtTZc9Yi8aJfk4OiJ+UALYUIOll3DhzgXBJgt E5FA== X-Gm-Message-State: ACgBeo3CkpWUquwoljeO5sowBagy3QtUAYCfuzSCzXsJpsZsMFOaUfCL QypqBhCZ8C0FyHDPBHxPQME928tTlruvSZOM X-Google-Smtp-Source: AA6agR5EXaH76tMFhRJJyEMzhXf/v1unPqYtNdNO+cnXN0MCFFOHCNFLxZi6nmgpeRkORqwHo5ZvVg== X-Received: by 2002:a05:6a00:2a05:b0:52e:6c05:9b84 with SMTP id ce5-20020a056a002a0500b0052e6c059b84mr3183685pfb.77.1660833132805; Thu, 18 Aug 2022 07:32:12 -0700 (PDT) Received: from vultr.guest ([45.32.72.237]) by smtp.gmail.com with ESMTPSA id h5-20020a63f905000000b003fdc16f5de2sm1379124pgi.15.2022.08.18.07.32.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Aug 2022 07:32:11 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v2 11/12] bpf: Add return value for bpf_map_init_from_attr Date: Thu, 18 Aug 2022 14:31:17 +0000 Message-Id: <20220818143118.17733-12-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220818143118.17733-1-laoar.shao@gmail.com> References: <20220818143118.17733-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660865533; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Vt3B1B2xA9UnavRf/lLTg9rVQPFR8dNDUq2oJHhoN6w=; b=Lv3CmtPeW88UKZfLrSdw2YEV2otCOPCqLnlZ0Ky27eGEYJdOPV05FHElqtsX8IonCTCYfl nNFlte4SS7gq0XrD75zFZfuPE4IUohCyj938ROdPu+6EpFUdCxxY7mYCWMNZ3jNf4eMd5K nfcdByBJScdo2SvF6tmc7dZczP0iKfk= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=n3Innaui; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf31.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660865533; a=rsa-sha256; cv=none; b=y9cgXzL66hN8+vsVFH1RzEcqUNfnQB266ewRhLxDTJfWFtsvCXiySUbaw8pO5hefyPAK9u ArMLuh0AF1k4S5pQWxjocepmMDuqDsJamsv16Mo6Pg/n7ii4Ae9sZYjPBf3hK2Yo6O8JdK QYUPhkyjiCUsAYcTbKYI6RCdsxq4PWw= X-Stat-Signature: o94rzzn1tkjd965q53jrtdi5gsido38j X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 17841250ED Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=n3Innaui; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf31.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com X-Rspam-User: X-HE-Tag: 1660865531-785385 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add return value for bpf_map_init_from_attr() to indicate whether it init successfully. This is a preparation of the followup patch. Signed-off-by: Yafang Shao --- include/linux/bpf.h | 2 +- kernel/bpf/arraymap.c | 8 +++++++- kernel/bpf/bloom_filter.c | 7 ++++++- kernel/bpf/bpf_local_storage.c | 7 ++++++- kernel/bpf/bpf_struct_ops.c | 7 ++++++- kernel/bpf/cpumap.c | 6 +++++- kernel/bpf/devmap.c | 6 +++++- kernel/bpf/hashtab.c | 6 +++++- kernel/bpf/local_storage.c | 7 ++++++- kernel/bpf/lpm_trie.c | 8 +++++++- kernel/bpf/offload.c | 6 +++++- kernel/bpf/queue_stack_maps.c | 7 ++++++- kernel/bpf/reuseport_array.c | 7 ++++++- kernel/bpf/ringbuf.c | 7 ++++++- kernel/bpf/syscall.c | 4 +++- net/core/sock_map.c | 8 +++++++- net/xdp/xskmap.c | 8 +++++++- 17 files changed, 94 insertions(+), 17 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index f95a2df..0c3534c 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1639,7 +1639,7 @@ struct bpf_prog *bpf_prog_get_type_dev(u32 ufd, enum bpf_prog_type type, void *bpf_map_area_mmapable_alloc(u64 size, int numa_node); void bpf_map_area_free(void *base, struct bpf_map *map); bool bpf_map_write_active(const struct bpf_map *map); -void bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr); +int bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr); int generic_map_lookup_batch(struct bpf_map *map, const union bpf_attr *attr, union bpf_attr __user *uattr); diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index 3039832..28d0310 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -85,6 +85,7 @@ static struct bpf_map *array_map_alloc(union bpf_attr *attr) bool bypass_spec_v1 = bpf_bypass_spec_v1(); u64 array_size, mask64; struct bpf_array *array; + int err; elem_size = round_up(attr->value_size, 8); @@ -143,7 +144,12 @@ static struct bpf_map *array_map_alloc(union bpf_attr *attr) array->map.bypass_spec_v1 = bypass_spec_v1; /* copy mandatory map attributes */ - bpf_map_init_from_attr(&array->map, attr); + err = bpf_map_init_from_attr(&array->map, attr); + if (err) { + bpf_map_area_free(array, NULL); + return ERR_PTR(err); + } + array->elem_size = elem_size; if (percpu && bpf_array_alloc_percpu(array)) { diff --git a/kernel/bpf/bloom_filter.c b/kernel/bpf/bloom_filter.c index 6691f79..be64227 100644 --- a/kernel/bpf/bloom_filter.c +++ b/kernel/bpf/bloom_filter.c @@ -93,6 +93,7 @@ static struct bpf_map *bloom_map_alloc(union bpf_attr *attr) u32 bitset_bytes, bitset_mask, nr_hash_funcs, nr_bits; int numa_node = bpf_map_attr_numa_node(attr); struct bpf_bloom_filter *bloom; + int err; if (!bpf_capable()) return ERR_PTR(-EPERM); @@ -147,7 +148,11 @@ static struct bpf_map *bloom_map_alloc(union bpf_attr *attr) if (!bloom) return ERR_PTR(-ENOMEM); - bpf_map_init_from_attr(&bloom->map, attr); + err = bpf_map_init_from_attr(&bloom->map, attr); + if (err) { + bpf_map_area_free(bloom, NULL); + return ERR_PTR(err); + } bloom->nr_hash_funcs = nr_hash_funcs; bloom->bitset_mask = bitset_mask; diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index 71d5bf1..b82d124 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -609,11 +609,16 @@ struct bpf_local_storage_map *bpf_local_storage_map_alloc(union bpf_attr *attr) struct bpf_local_storage_map *smap; unsigned int i; u32 nbuckets; + int err; smap = bpf_map_area_alloc(sizeof(*smap), NUMA_NO_NODE, NULL); if (!smap) return ERR_PTR(-ENOMEM); - bpf_map_init_from_attr(&smap->map, attr); + err = bpf_map_init_from_attr(&smap->map, attr); + if (err) { + bpf_map_area_free(&smap->map, NULL); + return ERR_PTR(err); + } nbuckets = roundup_pow_of_two(num_possible_cpus()); /* Use at least 2 buckets, select_bucket() is undefined behavior with 1 bucket */ diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index 37ba5c0..7cfbabc 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -598,6 +598,7 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) struct bpf_struct_ops_map *st_map; const struct btf_type *t, *vt; struct bpf_map *map; + int err; if (!bpf_capable()) return ERR_PTR(-EPERM); @@ -624,7 +625,11 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) st_map->st_ops = st_ops; map = &st_map->map; - bpf_map_init_from_attr(map, attr); + err = bpf_map_init_from_attr(map, attr); + if (err) { + bpf_map_area_free(st_map, NULL); + return ERR_PTR(err); + } st_map->uvalue = bpf_map_area_alloc(vt->size, NUMA_NO_NODE, map); st_map->links = diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index b593157..e672e62 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -101,7 +101,11 @@ static struct bpf_map *cpu_map_alloc(union bpf_attr *attr) if (!cmap) return ERR_PTR(-ENOMEM); - bpf_map_init_from_attr(&cmap->map, attr); + err = bpf_map_init_from_attr(&cmap->map, attr); + if (err) { + bpf_map_area_free(cmap, NULL); + return ERR_PTR(err); + } /* Pre-limit array size based on NR_CPUS, not final CPU check */ if (cmap->map.max_entries > NR_CPUS) { diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 807a4cd..10f038d 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -167,7 +167,11 @@ static struct bpf_map *dev_map_alloc(union bpf_attr *attr) if (!dtab) return ERR_PTR(-ENOMEM); - bpf_map_init_from_attr(&dtab->map, attr); + err = bpf_map_init_from_attr(&dtab->map, attr); + if (err) { + bpf_map_area_free(dtab, NULL); + return ERR_PTR(err); + } err = dev_map_init_map(dtab, attr); if (err) { diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index e9a6d2c4..f059ae0 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -500,7 +500,11 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) if (!htab) return ERR_PTR(-ENOMEM); - bpf_map_init_from_attr(&htab->map, attr); + err = bpf_map_init_from_attr(&htab->map, attr); + if (err) { + bpf_map_area_free(htab, NULL); + return ERR_PTR(err); + } lockdep_register_key(&htab->lockdep_key); diff --git a/kernel/bpf/local_storage.c b/kernel/bpf/local_storage.c index fcc7ece..1901195 100644 --- a/kernel/bpf/local_storage.c +++ b/kernel/bpf/local_storage.c @@ -287,6 +287,7 @@ static struct bpf_map *cgroup_storage_map_alloc(union bpf_attr *attr) __u32 max_value_size = BPF_LOCAL_STORAGE_MAX_VALUE_SIZE; int numa_node = bpf_map_attr_numa_node(attr); struct bpf_cgroup_storage_map *map; + int err; /* percpu is bound by PCPU_MIN_UNIT_SIZE, non-percu * is the same as other local storages. @@ -318,7 +319,11 @@ static struct bpf_map *cgroup_storage_map_alloc(union bpf_attr *attr) return ERR_PTR(-ENOMEM); /* copy mandatory map attributes */ - bpf_map_init_from_attr(&map->map, attr); + err = bpf_map_init_from_attr(&map->map, attr); + if (err) { + bpf_map_area_free(map, NULL); + return ERR_PTR(err); + } spin_lock_init(&map->lock); map->root = RB_ROOT; diff --git a/kernel/bpf/lpm_trie.c b/kernel/bpf/lpm_trie.c index 3d329ae..38d7b00 100644 --- a/kernel/bpf/lpm_trie.c +++ b/kernel/bpf/lpm_trie.c @@ -543,6 +543,7 @@ static int trie_delete_elem(struct bpf_map *map, void *_key) static struct bpf_map *trie_alloc(union bpf_attr *attr) { struct lpm_trie *trie; + int err; if (!bpf_capable()) return ERR_PTR(-EPERM); @@ -563,7 +564,12 @@ static struct bpf_map *trie_alloc(union bpf_attr *attr) return ERR_PTR(-ENOMEM); /* copy mandatory map attributes */ - bpf_map_init_from_attr(&trie->map, attr); + err = bpf_map_init_from_attr(&trie->map, attr); + if (err) { + bpf_map_area_free(trie, NULL); + return ERR_PTR(err); + } + trie->data_size = attr->key_size - offsetof(struct bpf_lpm_trie_key, data); trie->max_prefixlen = trie->data_size * 8; diff --git a/kernel/bpf/offload.c b/kernel/bpf/offload.c index 87c59da..dba7ed2 100644 --- a/kernel/bpf/offload.c +++ b/kernel/bpf/offload.c @@ -376,7 +376,11 @@ struct bpf_map *bpf_map_offload_map_alloc(union bpf_attr *attr) if (!offmap) return ERR_PTR(-ENOMEM); - bpf_map_init_from_attr(&offmap->map, attr); + err = bpf_map_init_from_attr(&offmap->map, attr); + if (err) { + bpf_map_area_free(offmap, NULL); + return ERR_PTR(err); + } rtnl_lock(); down_write(&bpf_devs_lock); diff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c index bf57e45..f231897 100644 --- a/kernel/bpf/queue_stack_maps.c +++ b/kernel/bpf/queue_stack_maps.c @@ -70,6 +70,7 @@ static struct bpf_map *queue_stack_map_alloc(union bpf_attr *attr) int numa_node = bpf_map_attr_numa_node(attr); struct bpf_queue_stack *qs; u64 size, queue_size; + int err; size = (u64) attr->max_entries + 1; queue_size = sizeof(*qs) + size * attr->value_size; @@ -78,7 +79,11 @@ static struct bpf_map *queue_stack_map_alloc(union bpf_attr *attr) if (!qs) return ERR_PTR(-ENOMEM); - bpf_map_init_from_attr(&qs->map, attr); + err = bpf_map_init_from_attr(&qs->map, attr); + if (err) { + bpf_map_area_free(qs, NULL); + return ERR_PTR(err); + } qs->size = size; diff --git a/kernel/bpf/reuseport_array.c b/kernel/bpf/reuseport_array.c index 52c7e77..4da969b 100644 --- a/kernel/bpf/reuseport_array.c +++ b/kernel/bpf/reuseport_array.c @@ -153,6 +153,7 @@ static struct bpf_map *reuseport_array_alloc(union bpf_attr *attr) { int numa_node = bpf_map_attr_numa_node(attr); struct reuseport_array *array; + int err; if (!bpf_capable()) return ERR_PTR(-EPERM); @@ -163,7 +164,11 @@ static struct bpf_map *reuseport_array_alloc(union bpf_attr *attr) return ERR_PTR(-ENOMEM); /* copy mandatory map attributes */ - bpf_map_init_from_attr(&array->map, attr); + err = bpf_map_init_from_attr(&array->map, attr); + if (err) { + bpf_map_area_free(array, NULL); + return ERR_PTR(err); + } return &array->map; } diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index 1e7284c..3edefd3 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -185,6 +185,7 @@ static struct bpf_ringbuf *bpf_ringbuf_alloc(size_t data_sz, int numa_node, static struct bpf_map *ringbuf_map_alloc(union bpf_attr *attr) { struct bpf_ringbuf_map *rb_map; + int err; if (attr->map_flags & ~RINGBUF_CREATE_FLAG_MASK) return ERR_PTR(-EINVAL); @@ -204,7 +205,11 @@ static struct bpf_map *ringbuf_map_alloc(union bpf_attr *attr) if (!rb_map) return ERR_PTR(-ENOMEM); - bpf_map_init_from_attr(&rb_map->map, attr); + err = bpf_map_init_from_attr(&rb_map->map, attr); + if (err) { + bpf_map_area_free(rb_map, NULL); + return ERR_PTR(err); + } rb_map->rb = bpf_ringbuf_alloc(attr->max_entries, rb_map->map.numa_node, &rb_map->map); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 2bd3bcf..5f5cade4 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -403,7 +403,7 @@ static u32 bpf_map_flags_retain_permanent(u32 flags) return flags & ~(BPF_F_RDONLY | BPF_F_WRONLY); } -void bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr) +int bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr) { bpf_map_save_memcg(map); map->map_type = attr->map_type; @@ -413,6 +413,8 @@ void bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr) map->map_flags = bpf_map_flags_retain_permanent(attr->map_flags); map->numa_node = bpf_map_attr_numa_node(attr); map->map_extra = attr->map_extra; + + return 0; } static int bpf_map_alloc_id(struct bpf_map *map) diff --git a/net/core/sock_map.c b/net/core/sock_map.c index 2b3b24e..766a260 100644 --- a/net/core/sock_map.c +++ b/net/core/sock_map.c @@ -31,6 +31,7 @@ static int sock_map_prog_update(struct bpf_map *map, struct bpf_prog *prog, static struct bpf_map *sock_map_alloc(union bpf_attr *attr) { struct bpf_stab *stab; + int err; if (!capable(CAP_NET_ADMIN)) return ERR_PTR(-EPERM); @@ -45,7 +46,12 @@ static struct bpf_map *sock_map_alloc(union bpf_attr *attr) if (!stab) return ERR_PTR(-ENOMEM); - bpf_map_init_from_attr(&stab->map, attr); + err = bpf_map_init_from_attr(&stab->map, attr); + if (err) { + bpf_map_area_free(stab, NULL); + return ERR_PTR(err); + } + raw_spin_lock_init(&stab->lock); stab->sks = bpf_map_area_alloc((u64) stab->map.max_entries * diff --git a/net/xdp/xskmap.c b/net/xdp/xskmap.c index beb11fd..8e7a5a6 100644 --- a/net/xdp/xskmap.c +++ b/net/xdp/xskmap.c @@ -63,6 +63,7 @@ static struct bpf_map *xsk_map_alloc(union bpf_attr *attr) struct xsk_map *m; int numa_node; u64 size; + int err; if (!capable(CAP_NET_ADMIN)) return ERR_PTR(-EPERM); @@ -79,7 +80,12 @@ static struct bpf_map *xsk_map_alloc(union bpf_attr *attr) if (!m) return ERR_PTR(-ENOMEM); - bpf_map_init_from_attr(&m->map, attr); + err = bpf_map_init_from_attr(&m->map, attr); + if (err) { + bpf_map_area_free(m, NULL); + return ERR_PTR(err); + } + spin_lock_init(&m->lock); return &m->map; From patchwork Thu Aug 18 14:31:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12948195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E495C00140 for ; Thu, 18 Aug 2022 23:55:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6E6A18D0002; Thu, 18 Aug 2022 19:55:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 696248D0001; Thu, 18 Aug 2022 19:55:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5366B8D0002; Thu, 18 Aug 2022 19:55:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 44BEC8D0001 for ; Thu, 18 Aug 2022 19:55:24 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0FF201C5D35 for ; Thu, 18 Aug 2022 23:55:24 +0000 (UTC) X-FDA: 79814372568.25.1A2086F Received: from mail-oa1-f52.google.com (mail-oa1-f52.google.com [209.85.160.52]) by imf08.hostedemail.com (Postfix) with ESMTP id 9107D160649 for ; Thu, 18 Aug 2022 23:53:29 +0000 (UTC) Received: by mail-oa1-f52.google.com with SMTP id 586e51a60fabf-11c5ee9bf43so3557725fac.5 for ; Thu, 18 Aug 2022 16:53:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=64R3bxHMvv5+V0T/PDUTVz1Uk0VIs+H8eHbhqMNvmEQ=; b=EX8ou0RTkSODLqaqCgCGl9TJKOPxlpSZvuMeBQIXCinFHIrXw9t+gJjdaOl1mdSXVT FIAy96Vr2dV3mTuOnnHpKPLUiP9iI2Of1ABEgqlQg49YBMVR4yPV9qLCh5X0Fcn9hc+9 cl543oOVxhHqJyYddxl+ATvDR2FjVNUBymnWUayB/yeDtZtpbqdoWXZWgoZnRR5/ceUg l6ajY8CycQHl70bSisAS55nOgS1iLAsM/DWjfN8EiJj9GAYQvLcJsjEZooxZ0KR84cQc iu8CpiidQdVIJI3fOVPgpk6h2wy/3yjI4rkfhJA0KVBw+Y93nYe/m5zCOQRx7Ahtz1Xs jhVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=64R3bxHMvv5+V0T/PDUTVz1Uk0VIs+H8eHbhqMNvmEQ=; b=CS5wSzU1gJulAh/q9DALxiCRPsUDM74RwLudT5Ey68B2FuuMV1EOJDIVWK/RNHTMgE 1xSrxNEbZcSKJk+hxsQX5rUhSGF5mWZpAk3rZxpL1dg0gWyOgA6/HLANc6+XfLtTPGmr 1CjISslB1pf0s0lj7rPd0if+3TWYNxhNXorR6SKnZ7tZbpN/iYSZJcRRbcWqQkCDXwYP mkhIcZVzYcT+FHwUTlCjzizOd/FDd74a4ry2O+9AeMeM6p3V+8NKYmvDfwy/x4uholW0 krnq3N9ViTuNBJ28mz5tb+mgUe11Nu0N8U9iV5aXd6MNoagJUSXvaYt2GP3qv4twwETW JAow== X-Gm-Message-State: ACgBeo0oLD6ceUDDW4u5f3liUs/SENTgsSakhbx/E/o0rn9FkNkGPdN2 HmizTUjP+LpozPmGQ6CMX+oE2+0eE6ClKYcA X-Google-Smtp-Source: AA6agR6BXiviu/fMhtQYEidUfIBaxKQz8CADJ9PvR9/TZPkp8KxyBxMr5H555oupXKEWVyIiR5O4fg== X-Received: by 2002:a17:90a:7f89:b0:1fa:ad33:7289 with SMTP id m9-20020a17090a7f8900b001faad337289mr8838077pjl.173.1660833136171; Thu, 18 Aug 2022 07:32:16 -0700 (PDT) Received: from vultr.guest ([45.32.72.237]) by smtp.gmail.com with ESMTPSA id h5-20020a63f905000000b003fdc16f5de2sm1379124pgi.15.2022.08.18.07.32.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Aug 2022 07:32:15 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v2 12/12] bpf: Introduce selectable memcg for bpf map Date: Thu, 18 Aug 2022 14:31:18 +0000 Message-Id: <20220818143118.17733-13-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220818143118.17733-1-laoar.shao@gmail.com> References: <20220818143118.17733-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660866809; a=rsa-sha256; cv=none; b=bAX+2AscI7The0DeEZeGYFNyg+YCQZJRCEKO5Sp3H6hmLa+zHoC+9QqXH1tSdHpAuR2ssz M1wSE5BcvTwHwI1NmaWPQM3PKaKc3jCf0jJyJpYYVvAxT+EZyTVwP1lewaJpsjqd5E931h AYcLNFWO3IdlqYbLO41JW00rSVlDjns= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=EX8ou0RT; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf08.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.52 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660866809; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=64R3bxHMvv5+V0T/PDUTVz1Uk0VIs+H8eHbhqMNvmEQ=; b=cVVUfBHWXl2NU8pXA8h4pA0WrVpsk8+Bqpy+tsJyRdRI+WyhxPk4Lc9x4UadawgvcxF+BM kpc5wSZ2t3EdQruT8EWJTq0rtxUmZ3DM0ioXdjz28myUWK5SGqSiyBtKSZ6WYz3NpD7moa uR2/amE6gf1OIjHVCL2eozW0N41SB6Q= X-Stat-Signature: d3thy4wracwk6drjew1ucytrocrz7sdo X-Rspamd-Queue-Id: 9107D160649 X-Rspam-User: Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=EX8ou0RT; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf08.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.52 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com X-Rspamd-Server: rspam11 X-HE-Tag: 1660866809-679766 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A new bpf attr memcg_fd is introduced for map creation. The map creation path in libbpf is changed consequently to set this attr. A new member memcg_fd is introduced into bpf attr of BPF_MAP_CREATE command, which is the fd of an opened cgroup directory. In this cgroup, the memory subsystem must be enabled. The valid memcg_fd must be a postive number, that means it can't be zero(a valid return value of open(2)). Once the kernel get the memory cgroup from this fd, it will set this memcg into bpf map, then all the subsequent memory allocation of this map will be charged to the memcg. The map creation paths in libbpf are also changed consequently. The usage of this new member as follows, struct bpf_map_create_opts map_opts = { .sz = sizeof(map_opts), }; int memcg_fd, map_fd, old_fd; int key, value; memcg_fd = open("/sys/fs/cgroup/memory/bpf", O_DIRECTORY); if (memcg_fd < 0) { perror("memcg dir open"); return -1; } /* 0 is a invalid fd */ if (memcg_fd == 0) { old_fd = memcg_fd; memcg_fd = fcntl(memcg_fd, F_DUPFD_CLOEXEC, 3); close(old_fd); if (memcg_fd < 0) { perror("fcntl"); return -1; } } map_opts.memcg_fd = memcg_fd; map_fd = bpf_map_create(BPF_MAP_TYPE_HASH, "map_for_memcg", sizeof(key), sizeof(value), 1024, &map_opts); if (map_fd <= 0) { close(memcg_fd); perror("map create"); return -1; } Signed-off-by: Yafang Shao --- include/uapi/linux/bpf.h | 1 + kernel/bpf/syscall.c | 45 ++++++++++++++++++++++++++++++++---------- tools/include/uapi/linux/bpf.h | 1 + tools/lib/bpf/bpf.c | 3 ++- tools/lib/bpf/bpf.h | 3 ++- tools/lib/bpf/gen_loader.c | 2 +- tools/lib/bpf/libbpf.c | 2 ++ tools/lib/bpf/skel_internal.h | 2 +- 8 files changed, 45 insertions(+), 14 deletions(-) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 7d1e279..048b692 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1300,6 +1300,7 @@ struct bpf_stack_build_id { * to using 5 hash functions). */ __u64 map_extra; + __u32 memcg_fd; /* selectable memcg */ }; struct { /* anonymous struct used by BPF_MAP_*_ELEM commands */ diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 5f5cade4..fd1f67c 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -294,14 +294,33 @@ static int bpf_map_copy_value(struct bpf_map *map, void *key, void *value, } #ifdef CONFIG_MEMCG_KMEM -static void bpf_map_save_memcg(struct bpf_map *map) +static int bpf_map_save_memcg(struct bpf_map *map, u32 memcg_fd) { - /* Currently if a map is created by a process belonging to the root - * memory cgroup, get_obj_cgroup_from_current() will return NULL. - * So we have to check map->objcg for being NULL each time it's - * being used. - */ - map->objcg = get_obj_cgroup_from_current(); + struct obj_cgroup *objcg; + struct cgroup *cgrp; + + if (memcg_fd) { + cgrp = cgroup_get_from_fd(memcg_fd); + if (IS_ERR(cgrp)) + return -EINVAL; + + objcg = get_obj_cgroup_from_cgroup(cgrp); + if (IS_ERR(objcg)) { + cgroup_put(cgrp); + return PTR_ERR(objcg); + } + cgroup_put(cgrp); + } else { + /* Currently if a map is created by a process belonging to the root + * memory cgroup, get_obj_cgroup_from_current() will return NULL. + * So we have to check map->objcg for being NULL each time it's + * being used. + */ + objcg = get_obj_cgroup_from_current(); + } + + map->objcg = objcg; + return 0; } static void bpf_map_release_memcg(struct bpf_map *map) @@ -311,8 +330,9 @@ static void bpf_map_release_memcg(struct bpf_map *map) } #else -static void bpf_map_save_memcg(struct bpf_map *map) +static int bpf_map_save_memcg(struct bpf_map *map, u32 memcg_fd) { + return 0; } static void bpf_map_release_memcg(struct bpf_map *map) @@ -405,7 +425,12 @@ static u32 bpf_map_flags_retain_permanent(u32 flags) int bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr) { - bpf_map_save_memcg(map); + int err; + + err = bpf_map_save_memcg(map, attr->memcg_fd); + if (err) + return err; + map->map_type = attr->map_type; map->key_size = attr->key_size; map->value_size = attr->value_size; @@ -1091,7 +1116,7 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf, return ret; } -#define BPF_MAP_CREATE_LAST_FIELD map_extra +#define BPF_MAP_CREATE_LAST_FIELD memcg_fd /* called via syscall */ static int map_create(union bpf_attr *attr) { diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index e174ad2..1e9efe7 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1300,6 +1300,7 @@ struct bpf_stack_build_id { * to using 5 hash functions). */ __u64 map_extra; + __u32 memcg_fd; /* selectable memcg */ }; struct { /* anonymous struct used by BPF_MAP_*_ELEM commands */ diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c index 6a96e66..6517553 100644 --- a/tools/lib/bpf/bpf.c +++ b/tools/lib/bpf/bpf.c @@ -171,7 +171,7 @@ int bpf_map_create(enum bpf_map_type map_type, __u32 max_entries, const struct bpf_map_create_opts *opts) { - const size_t attr_sz = offsetofend(union bpf_attr, map_extra); + const size_t attr_sz = offsetofend(union bpf_attr, memcg_fd); union bpf_attr attr; int fd; @@ -199,6 +199,7 @@ int bpf_map_create(enum bpf_map_type map_type, attr.map_extra = OPTS_GET(opts, map_extra, 0); attr.numa_node = OPTS_GET(opts, numa_node, 0); attr.map_ifindex = OPTS_GET(opts, map_ifindex, 0); + attr.memcg_fd = OPTS_GET(opts, memcg_fd, 0); fd = sys_bpf_fd(BPF_MAP_CREATE, &attr, attr_sz); return libbpf_err_errno(fd); diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h index 9c50bea..dd0d929 100644 --- a/tools/lib/bpf/bpf.h +++ b/tools/lib/bpf/bpf.h @@ -51,8 +51,9 @@ struct bpf_map_create_opts { __u32 numa_node; __u32 map_ifindex; + __u32 memcg_fd; }; -#define bpf_map_create_opts__last_field map_ifindex +#define bpf_map_create_opts__last_field memcg_fd LIBBPF_API int bpf_map_create(enum bpf_map_type map_type, const char *map_name, diff --git a/tools/lib/bpf/gen_loader.c b/tools/lib/bpf/gen_loader.c index 23f5c46..f35b014 100644 --- a/tools/lib/bpf/gen_loader.c +++ b/tools/lib/bpf/gen_loader.c @@ -451,7 +451,7 @@ void bpf_gen__map_create(struct bpf_gen *gen, __u32 key_size, __u32 value_size, __u32 max_entries, struct bpf_map_create_opts *map_attr, int map_idx) { - int attr_size = offsetofend(union bpf_attr, map_extra); + int attr_size = offsetofend(union bpf_attr, memcg_fd); bool close_inner_map_fd = false; int map_create_attr, idx; union bpf_attr attr; diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 3f01f5c..ad7aa39 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -511,6 +511,7 @@ struct bpf_map { bool reused; bool autocreate; __u64 map_extra; + __u32 memcg_fd; }; enum extern_type { @@ -4936,6 +4937,7 @@ static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map, b create_attr.map_flags = def->map_flags; create_attr.numa_node = map->numa_node; create_attr.map_extra = map->map_extra; + create_attr.memcg_fd = map->memcg_fd; if (bpf_map__is_struct_ops(map)) create_attr.btf_vmlinux_value_type_id = map->btf_vmlinux_value_type_id; diff --git a/tools/lib/bpf/skel_internal.h b/tools/lib/bpf/skel_internal.h index bd6f450..83403cc 100644 --- a/tools/lib/bpf/skel_internal.h +++ b/tools/lib/bpf/skel_internal.h @@ -222,7 +222,7 @@ static inline int skel_map_create(enum bpf_map_type map_type, __u32 value_size, __u32 max_entries) { - const size_t attr_sz = offsetofend(union bpf_attr, map_extra); + const size_t attr_sz = offsetofend(union bpf_attr, memcg_fd); union bpf_attr attr; memset(&attr, 0, attr_sz);