From patchwork Fri Sep 2 02:29:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12963561 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1200ECAAD3 for ; Fri, 2 Sep 2022 02:30:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 69A9380098; Thu, 1 Sep 2022 22:30:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 64AE78008D; Thu, 1 Sep 2022 22:30:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4767A80098; Thu, 1 Sep 2022 22:30:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3A1BB8008D for ; Thu, 1 Sep 2022 22:30:14 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1662D120EAF for ; Fri, 2 Sep 2022 02:30:14 +0000 (UTC) X-FDA: 79865565948.21.4684BAD Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) by imf14.hostedemail.com (Postfix) with ESMTP id BF1C810002B for ; Fri, 2 Sep 2022 02:30:13 +0000 (UTC) Received: by mail-pg1-f171.google.com with SMTP id x80so816210pgx.0 for ; Thu, 01 Sep 2022 19:30:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=7uI5yxQ2IficxCafBouZLQeDm3f9EVu1uBy3AFKC6gc=; b=OFJXxBSBYOssoMkoywl9EoB3k6QrkEaHd8YL02U+QnpsTC0UA4fodDgpSL5b5LWBLX nLgkHuPVwidqPfsnsj8JHUDLGBuIyEEBczjpRgWekusWdvlQ7O5GLfNXILnfWwkwAPMJ hcJ2ifQutv33yG3kxTre9B7m3UTbVLhx74iQdT6qSkb9YwbhX7EK8Gsn17MRAfi/hNG1 6Svh8N4fpVQKpD4jLnAFBmhHGs6Pbi/lGpjyvsVXTIgrjPlqDpEWOT1O2plNLLZMq3/C Pyiw6ih9zAawcl99mJzJSBR019T2VD/ls6nn2v1UxhwxYmwSSUpvlAAX2eLUKf3JXR6f eZMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=7uI5yxQ2IficxCafBouZLQeDm3f9EVu1uBy3AFKC6gc=; b=UukmcP91bBwoQ7A+py4puNhXarMMTH3dFN+3GzFAUG0TAcGispVh88yhzftQfUy+31 XNijcjx6euZerpvHY0vfZgD0u3oI3ShOe9XkZq7tTBFHLsVtcoKlFjzgN5aa0KsyhbCn nOUlVTCerdJg33947oFsMVylbeKgmygrX9oFvz7aC2x4QPmaoQNmrtr/iQRBe2YlLGUc HOVvF0quTUIcMzaZQC8jAmBbeVVKliWKq7H5xW0RzEycCdJJKlaQ7me0C0UPEaVjIzTk ZTZqst0F3N4NZFlkVS5Hqj308AZ9B17V1cBdClvMm7QhD1hy2DxMlfZ9PvBSS/nRV9oK GKkQ== X-Gm-Message-State: ACgBeo3JK6qxh2swItLXsZE1je1YiiuOL5ayazgDHRwUeg1dRHs2fITG 42J54viUQ2z49CGhWcKiwR0= X-Google-Smtp-Source: AA6agR7KvJ3/nC3lYOZJPIL8xkpq5Mcjsx/aQlho5wzkyPrwzNAvnub08SZ4yh6Sj9fhmkLywjW2qg== X-Received: by 2002:a05:6a02:10e:b0:42a:b42b:5692 with SMTP id bg14-20020a056a02010e00b0042ab42b5692mr28380447pgb.67.1662085812826; Thu, 01 Sep 2022 19:30:12 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:50ea:5400:4ff:fe1f:fbe2]) by smtp.gmail.com with ESMTPSA id j4-20020a170902da8400b0017297a6b39dsm269719plx.265.2022.09.01.19.30.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 19:30:12 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao , Yosry Ahmed Subject: [PATCH bpf-next v3 01/13] cgroup: Update the comment on cgroup_get_from_fd Date: Fri, 2 Sep 2022 02:29:51 +0000 Message-Id: <20220902023003.47124-2-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220902023003.47124-1-laoar.shao@gmail.com> References: <20220902023003.47124-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662085813; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7uI5yxQ2IficxCafBouZLQeDm3f9EVu1uBy3AFKC6gc=; b=nETGgn5wxSS/DXJ0J78Kjk8z/8eya4gRbMEiA7+ccY/dUcM3HSF7duPYoiWUEtumoprDjz jpg/1zaYYKZfuFIOzuGeKJAXXOVM34O8WyAfsnPiVD7bOIFhS2yOCsC2PyQ+agysx/Y0gB g2GdOc4EZLMs1tGD0VdXsgUtS/lTttg= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=OFJXxBSB; spf=pass (imf14.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662085813; a=rsa-sha256; cv=none; b=FjYSTGg2uQ5qI4wrohwMra/viUdYylbsbjntkTjbTxoY5xtY/S+ikYIRug7gFK6uicq1vI dMn37O+LF+9NIe6o2BB+WaqGwlKjXNT2AxZpz4hpbzcKQ6XK/m/m2dC8C4qUYlRCt2TqtJ xE4tyxA3ovRnpIOXbMtLXjWf4DGDlUw= X-Stat-Signature: naf5q179eorrqtr4bo8ea68m3fq8ude9 X-Rspamd-Queue-Id: BF1C810002B Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=OFJXxBSB; spf=pass (imf14.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam12 X-Rspam-User: X-HE-Tag: 1662085813-509964 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: After commit f3a2aebdd6fb ("cgroup: enable cgroup_get_from_file() on cgroup1") we can open a cgroup1 dir as well. So let's update the comment. Cc: Yosry Ahmed Cc: Hao Luo Cc: Tejun Heo Signed-off-by: Yafang Shao --- kernel/cgroup/cgroup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index 5f4502a..b7d2e55 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -6625,7 +6625,7 @@ struct cgroup *cgroup_get_from_path(const char *path) /** * cgroup_get_from_fd - get a cgroup pointer from a fd - * @fd: fd obtained by open(cgroup2_dir) + * @fd: fd obtained by open(cgroup_dir) * * Find the cgroup from a fd which should be obtained * by opening a cgroup directory. Returns a pointer to the From patchwork Fri Sep 2 02:29:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12963562 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12F5BC6FA82 for ; Fri, 2 Sep 2022 02:30:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 711A080099; Thu, 1 Sep 2022 22:30:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6BF608008D; Thu, 1 Sep 2022 22:30:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5607B80099; Thu, 1 Sep 2022 22:30:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 469008008D for ; Thu, 1 Sep 2022 22:30:17 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 15160C0987 for ; Fri, 2 Sep 2022 02:30:17 +0000 (UTC) X-FDA: 79865566074.06.6911B17 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) by imf23.hostedemail.com (Postfix) with ESMTP id BC48914005A for ; Fri, 2 Sep 2022 02:30:15 +0000 (UTC) Received: by mail-pg1-f170.google.com with SMTP id h188so737889pgc.12 for ; Thu, 01 Sep 2022 19:30:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=Ia4obLCjxStcu+ICWYF8jUpLC+DBctMa/v2GHxl/gGU=; b=mGoieldFRW2ZsDHdEN4SGCykyr50uN6imE/RwcMpTgP+UImu3qxKPcjshQFMbcPCH0 uYa4R3RNELwqch3SIr0C74TTS/jL857Xy8RcS9/k21JMyxJJUf2YNaFFVlcjqcqfB+yw /Oe8BEVARTMeYoQ3hPeJvdKL3YSyxw/d0xpdKh1Zngx9QJIJ82ylFj4vVhnLCIQzPjcC DlDpvggQJ9jWbT5zHIXw4sJqpNqrnagdYX8K6LbXA8oO+uLEspmkDZS/yf3+UxDeUQi3 YNWr/PkVUiPiWmBkX+/KeXFXBAEVgHn2Ex8HMJ5uO9B1PmrOuU68MCDIg8ElFlywwvOx Ovkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=Ia4obLCjxStcu+ICWYF8jUpLC+DBctMa/v2GHxl/gGU=; b=XMSjhi7d7UlALtyPJjqRotLDxVxPxheHdZBn3T/aoyIMrH+t9aaWVvCD+FA2QTVkAg eLo4colLvmPnUmWtGQ5bry/AZya1SkKSvpj2QBaJll85e50ojISWFrvyqYac/fYtqJ0d trhNLMMjEiGgB1So8VQ8SWSeawW8QwWV8t7+p5cgd89nUWta7ej00TDZKCfffKvJ8RDU lRYExc2bH2Hx7rLAb5hY38y8u82QJ3MytOEr2JVhaTC3Fj/98lVyE5tKBMg3phAPEbkX gdS0eZayE2yRaR9jHS36O/jl1UgtIMnmt+ry3t5f4clndykxvKAYtFulw4BgaZpyPPLv 55Mg== X-Gm-Message-State: ACgBeo3xcoZkpoc+mwge2KBVi88jSzFYc4IACfQbJHHtn0lg6fLvnc6B 8+Uvw4YCLFULNbKIwWJ8TRg= X-Google-Smtp-Source: AA6agR5zZ7j+yw+Nu++fMWh4vrCtPC8eJ/5esneE7CzVfU+r/WXj/ixS9SxXhktPvUmgl1TQfxfquA== X-Received: by 2002:a63:a516:0:b0:42b:8bb2:7036 with SMTP id n22-20020a63a516000000b0042b8bb27036mr24798037pgf.389.1662085814905; Thu, 01 Sep 2022 19:30:14 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:50ea:5400:4ff:fe1f:fbe2]) by smtp.gmail.com with ESMTPSA id j4-20020a170902da8400b0017297a6b39dsm269719plx.265.2022.09.01.19.30.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 19:30:14 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v3 02/13] bpf: Introduce new helper bpf_map_put_memcg() Date: Fri, 2 Sep 2022 02:29:52 +0000 Message-Id: <20220902023003.47124-3-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220902023003.47124-1-laoar.shao@gmail.com> References: <20220902023003.47124-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662085815; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ia4obLCjxStcu+ICWYF8jUpLC+DBctMa/v2GHxl/gGU=; b=tEu7pqaLGOF0Q0joRLTPOmXKS/VJkjHV5LXehtqlbb+FIQpZRiGBPSLzt3gmorRGY16IF/ UIbNuZvHZ9aF5gyCWL7V/XfZFTaae0g0ZR8W9vRuyAh3DVBxp0Hl3s8Tbs7Px/uuHAKCFI q0g6thq87Na+aBs9dr9cG8FgybGqJM0= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=mGoieldF; spf=pass (imf23.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662085815; a=rsa-sha256; cv=none; b=mPBugVKiiKWRHmFZXw1YlWaPvp4JQtb1wG9IxneROGpHylyKbxJpvVaMqSKydi9YoboqlE Ic5iVhCAUmO7mPV5uD1C10qbj5TU/KWVdgMH5SU+7lNgi7XKhrYVPyIHm1Q3Biw0eAqLp3 UAmg/qQNLvxRhdNrOmQKpmpLmzZVINk= X-Stat-Signature: iidwsrmz8e4h9apqae8dfy711xmcwy9f X-Rspamd-Queue-Id: BC48914005A X-Rspam-User: Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=mGoieldF; spf=pass (imf23.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam01 X-HE-Tag: 1662085815-342362 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace the open-coded mem_cgroup_put() with a new helper bpf_map_put_memcg(). That could make it more clear. Signed-off-by: Yafang Shao --- kernel/bpf/syscall.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 4e9d462..7ce024c 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -441,6 +441,11 @@ static struct mem_cgroup *bpf_map_get_memcg(const struct bpf_map *map) return root_mem_cgroup; } +static void bpf_map_put_memcg(struct mem_cgroup *memcg) +{ + mem_cgroup_put(memcg); +} + void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, int node) { @@ -451,7 +456,7 @@ void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, old_memcg = set_active_memcg(memcg); ptr = kmalloc_node(size, flags | __GFP_ACCOUNT, node); set_active_memcg(old_memcg); - mem_cgroup_put(memcg); + bpf_map_put_memcg(memcg); return ptr; } @@ -465,7 +470,7 @@ void *bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags) old_memcg = set_active_memcg(memcg); ptr = kzalloc(size, flags | __GFP_ACCOUNT); set_active_memcg(old_memcg); - mem_cgroup_put(memcg); + bpf_map_put_memcg(memcg); return ptr; } @@ -480,7 +485,7 @@ void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, old_memcg = set_active_memcg(memcg); ptr = __alloc_percpu_gfp(size, align, flags | __GFP_ACCOUNT); set_active_memcg(old_memcg); - mem_cgroup_put(memcg); + bpf_map_put_memcg(memcg); return ptr; } From patchwork Fri Sep 2 02:29:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12963563 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FD14ECAAD2 for ; Fri, 2 Sep 2022 02:30:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E8A488009A; Thu, 1 Sep 2022 22:30:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E3A6E8008D; Thu, 1 Sep 2022 22:30:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C8C8A8009A; Thu, 1 Sep 2022 22:30:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B7A018008D for ; Thu, 1 Sep 2022 22:30:18 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 89BAD4109E for ; Fri, 2 Sep 2022 02:30:18 +0000 (UTC) X-FDA: 79865566116.06.B1C10CA Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) by imf08.hostedemail.com (Postfix) with ESMTP id 3E558160062 for ; Fri, 2 Sep 2022 02:30:18 +0000 (UTC) Received: by mail-pj1-f45.google.com with SMTP id h13-20020a17090a648d00b001fdb9003787so816165pjj.4 for ; Thu, 01 Sep 2022 19:30:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=4M2wIpbTEjtveSWIAJbVEhxb96NjEAIn1cqGJw9nWyA=; b=jzt0LFNt/yVYD/3NUXzIBoEoI9rvRu241Gab6qFW2E6IBiyGaWJo2ZbTstkJc00kFU 9pZPZRV9AJV3QOLgrCv3w0o+0Zz6uTS264NCkTYPkC1JfBWOvOR9S/q6z3QfpqL91hZc Gw3YnTYgM5ch+cMtv6X6YSKgZFeHqM5lAc0gvuZUP6goEJJhbrHBcgfQHPcp/Hsa0J64 +mB9/MkbcbHohZh88cJ3YPci73xopvwgOehe+rNkUxm4HG3wIVKo9q/fsO4eN9/sJLWH 7EiEn2lzWBAUsRbZFv3jPcIH8tsIo6MVCShqoy1P8lviU8jjl7rXfdx2Su2uGv7ACgRC t22g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=4M2wIpbTEjtveSWIAJbVEhxb96NjEAIn1cqGJw9nWyA=; b=bGL7y5uDxUG/l4QAkvdR/D1LN/OuYgexdDi+1dy+h26r6oYhWa1DJaRXq2lhfBESxG 2baQBpCCYaxVrgTXC/4OjXLZ5v2HqWPSuYT8zj5VR3kDgWuUiXKIZhqylwGXc7p0psE8 WPuCRAha1hhot8U2zfv4ha0UQvM/EEAjXHWJGm7BjbUKXDYLPRf4kZ9iB4Loc4bWQpP/ xUaRSziQ28Bi19EeWObq+kQ0JUt9chC2JPxFB96dNFSHa+uPk2zVuC5dSrL6M1K4S28x jkUvXds/q775H/wN5IUxy7aJWlzA6WFoJmSnmsp3WiN3UOZwCtKy5GWM+/r7HoV7d9Jz 6t+A== X-Gm-Message-State: ACgBeo3twt781CHwuUO+SHfGx+VYkRqoanMhxob7KiWuq+9xIT1dp2sn 3vSqrffoe+0uyKBx922N7z0= X-Google-Smtp-Source: AA6agR7FG/Tgj/X3jzFMLSjqn8xywcVs0S+evyTTodrZ7Ljz0USgvqGYM2KFO6GT0BcGn+RNK9ts3A== X-Received: by 2002:a17:902:db12:b0:16e:eaf9:e98d with SMTP id m18-20020a170902db1200b0016eeaf9e98dmr33174440plx.172.1662085817212; Thu, 01 Sep 2022 19:30:17 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:50ea:5400:4ff:fe1f:fbe2]) by smtp.gmail.com with ESMTPSA id j4-20020a170902da8400b0017297a6b39dsm269719plx.265.2022.09.01.19.30.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 19:30:16 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v3 03/13] bpf: Define bpf_map_{get,put}_memcg for !CONFIG_MEMCG_KMEM Date: Fri, 2 Sep 2022 02:29:53 +0000 Message-Id: <20220902023003.47124-4-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220902023003.47124-1-laoar.shao@gmail.com> References: <20220902023003.47124-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662085818; a=rsa-sha256; cv=none; b=QpvohvsVCQh3kQ5W/q3iK+/N6tgUziPEimoYjuoxMui53HhnBnRUZLz4KskTrZcQmwVQN3 ctKOrOmJbyO5TUV8SRQHLqtCbcKazdIZh8geVUi/PlBGcr7zHfSypTgzhtPrxLzjxHY1PM C19bdC2DXpIvdwxvxnDIXZEzIIIKs5s= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=jzt0LFNt; spf=pass (imf08.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.45 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662085818; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4M2wIpbTEjtveSWIAJbVEhxb96NjEAIn1cqGJw9nWyA=; b=kOhV073HG/b+sb/mXYw658Dxtiy7ww11oBs5w7MLhMEuJ/gicl065yOvjTiEiGICfIkvH2 Wgfq8yLCYGVHA5KJ4bv9Yhl4VS7oWzJ7QFjUS49bCgthVSdm/pBkC1y0mVTO/4swhI7wgg ZL4SR5P6BQPLdQUVPx58dPGbCHn77bs= Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=jzt0LFNt; spf=pass (imf08.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.45 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: k6g9rarrjaa3onmamf4iyzagt1dux5iz X-Rspamd-Queue-Id: 3E558160062 X-Rspamd-Server: rspam04 X-Rspam-User: X-HE-Tag: 1662085818-753989 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We can use this helper when CONFIG_MEMCG_KMEM or CONFIG_MEMCG is not set. It also moves bpf_map_{get,put}_memcg into include/linux/bpf.h, so these two helpers can be used in other source files. Signed-off-by: Yafang Shao --- include/linux/bpf.h | 26 ++++++++++++++++++++++++++ include/linux/memcontrol.h | 10 ++++++++++ kernel/bpf/syscall.c | 13 ------------- 3 files changed, 36 insertions(+), 13 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 9c16749..a50d29c 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -27,6 +27,7 @@ #include #include #include +#include struct bpf_verifier_env; struct bpf_verifier_log; @@ -2594,4 +2595,29 @@ static inline void bpf_cgroup_atype_get(u32 attach_btf_id, int cgroup_atype) {} static inline void bpf_cgroup_atype_put(int cgroup_atype) {} #endif /* CONFIG_BPF_LSM */ +#ifdef CONFIG_MEMCG_KMEM +static inline struct mem_cgroup *bpf_map_get_memcg(const struct bpf_map *map) +{ + if (map->objcg) + return get_mem_cgroup_from_objcg(map->objcg); + + return root_mem_cgroup; +} + +static inline void bpf_map_put_memcg(struct mem_cgroup *memcg) +{ + mem_cgroup_put(memcg); +} + +#else +static inline struct mem_cgroup *bpf_map_get_memcg(const struct bpf_map *map) +{ + return root_memcg(); +} + +static inline void bpf_map_put_memcg(struct mem_cgroup *memcg) +{ +} +#endif + #endif /* _LINUX_BPF_H */ diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 4d31ce5..6040b5c 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -361,6 +361,11 @@ struct mem_cgroup { extern struct mem_cgroup *root_mem_cgroup; +static inline struct mem_cgroup *root_memcg(void) +{ + return root_mem_cgroup; +} + enum page_memcg_data_flags { /* page->memcg_data is a pointer to an objcgs vector */ MEMCG_DATA_OBJCGS = (1UL << 0), @@ -1147,6 +1152,11 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, #define MEM_CGROUP_ID_SHIFT 0 #define MEM_CGROUP_ID_MAX 0 +static inline struct mem_cgroup *root_memcg(void) +{ + return NULL; +} + static inline struct mem_cgroup *folio_memcg(struct folio *folio) { return NULL; diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 7ce024c..eaf1906 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -433,19 +433,6 @@ static void bpf_map_release_memcg(struct bpf_map *map) obj_cgroup_put(map->objcg); } -static struct mem_cgroup *bpf_map_get_memcg(const struct bpf_map *map) -{ - if (map->objcg) - return get_mem_cgroup_from_objcg(map->objcg); - - return root_mem_cgroup; -} - -static void bpf_map_put_memcg(struct mem_cgroup *memcg) -{ - mem_cgroup_put(memcg); -} - void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, int node) { From patchwork Fri Sep 2 02:29:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12963564 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4375DC38145 for ; Fri, 2 Sep 2022 02:30:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D2AD38009B; Thu, 1 Sep 2022 22:30:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CDA758008D; Thu, 1 Sep 2022 22:30:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B7B2F8009B; Thu, 1 Sep 2022 22:30:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A93968008D for ; Thu, 1 Sep 2022 22:30:20 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 86DC2810DC for ; Fri, 2 Sep 2022 02:30:20 +0000 (UTC) X-FDA: 79865566200.29.C21DF39 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by imf06.hostedemail.com (Postfix) with ESMTP id 2F82F180042 for ; Fri, 2 Sep 2022 02:30:20 +0000 (UTC) Received: by mail-pf1-f180.google.com with SMTP id 199so603021pfz.2 for ; Thu, 01 Sep 2022 19:30:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=lQgh1HsLC8Sr1uJR87Ka1cZJSomKx/sCkb/6FvelFCU=; b=GNM1otao9St9RcA3yQ2IOmoal2de7Aanqe1/0mcaal6MCxkxyZmTnCpGD/Cr1ARKSw yVZEPKD3vM/kyoPXf5rqU1fLzWmSHim+pt0OJWmVkiEksLM2022QbUh/BXPbjW8YXsph mhWZQ54ypqiUslBquWro1iXnjY0hHT4KE6fvb661vn6c9dYttYelYz26bQWMzsAjysfu dFACgNLHHgI6EeTssgJAGAx8z9hDLdnvG0lR0JA0X6HhbNmY/qMzZq+bHaQoacm8KA5Q UYGm9M48RoD1SDpE8qT9irswpe4ZvkSgp/ApaWtl/Ozsks59OCMC+wLvx63d0AZdYbvx sJeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=lQgh1HsLC8Sr1uJR87Ka1cZJSomKx/sCkb/6FvelFCU=; b=rADYAXidQMTN72Jq2iXQCprWn3Q0rinwwQctnpsrDkZEkfAnIogcmGRTw1ZNv1U5KD o/YHNVTCRr4qq/eF5E1DmYIoO5yA73hxbKro4KHxVLHKnIq1UL2wiAfwWsHPOL2Yx9Oa o+E4wu+2TzSPv/tYaPnZ6ELtNUDjzFDoVDdKkS0coj1lU+rqp2XAVA7TA8laQJudk2pH 5KhatKrhn6duEe0MIkDre8JUj96azUhwq4FGZ0dGGyxuL/3LH5yUIGrUOyxD+7aRVw7x IHXfmGnzIBc7nhjMA1gfoxmvIyynAu220Ba9bspRmIeiJu2UDbEnptk1ekIGeUjBpWbo /PNw== X-Gm-Message-State: ACgBeo0qXznDBX7bhpvfTt2C69lTsMA3BlzYR0xL+1WGunIfhdyZIIa+ VV+yF6ZOUPUG5GFzo+JVimI= X-Google-Smtp-Source: AA6agR5HU/rgule0aTc2QEkg4SDD8rHlkky3g0ZAFabRaE2x0c1avsI5e9NALko3T9SBxP9fcdqRZw== X-Received: by 2002:a63:fb4a:0:b0:429:8605:6ebf with SMTP id w10-20020a63fb4a000000b0042986056ebfmr28474780pgj.225.1662085819281; Thu, 01 Sep 2022 19:30:19 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:50ea:5400:4ff:fe1f:fbe2]) by smtp.gmail.com with ESMTPSA id j4-20020a170902da8400b0017297a6b39dsm269719plx.265.2022.09.01.19.30.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 19:30:18 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v3 04/13] bpf: Call bpf_map_init_from_attr() immediately after map creation Date: Fri, 2 Sep 2022 02:29:54 +0000 Message-Id: <20220902023003.47124-5-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220902023003.47124-1-laoar.shao@gmail.com> References: <20220902023003.47124-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662085820; a=rsa-sha256; cv=none; b=d/gqiQKJZp/0aPcsh1eN2nZNtcgCMl6EwMwA/Z7RH8/2I1OxCoKlnn+VKa0YSTZ/QXUhAF 2WVlK//BkfUPr5XiEktZU9o8wUPJRMyV8jn42x/dd1F843qkdaD3gNXlyNCOmm/WTyeQU7 BQBLI9xwR665cTMImeHSZp22nG24I8k= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=GNM1otao; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf06.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662085820; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lQgh1HsLC8Sr1uJR87Ka1cZJSomKx/sCkb/6FvelFCU=; b=jjtmNOY4aiuPac4eHxyBTckS76H0x1ob4cm2OdX0Lc/xwmgxgy1XaiMNg5uj2yUNi8c9L7 tfARJOjCtJr/7kGymG9Tbu4Y7ojxS6hE5D47DjCgu/aHKX+IL4q8g7gt2vrM0Tivg+aPlE 1HU5rdrHvpXKC6j0hoMEuCImnvfBmMY= X-Stat-Signature: m7t4p5y464wi6cg9skx5tcuxop9ezqfw X-Rspamd-Queue-Id: 2F82F180042 X-Rspam-User: Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=GNM1otao; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf06.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com X-Rspamd-Server: rspam08 X-HE-Tag: 1662085820-731474 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In order to make all other map related memory allocations been allocated after memcg is saved in the map, we should save the memcg immediately after map creation. But the map is created in bpf_map_area_alloc(), within which we can't get the related bpf_map (except with a pointer casting which may be error prone), so we can do it in bpf_map_init_from_attr(), which is used by all bpf maps. bpf_map_init_from_attr() is executed immediately after bpf_map_area_alloc() for almost all bpf maps except bpf_struct_ops, devmap and hashmap, so this patch changes these three maps. In the future we will change the return type of bpf_map_init_from_attr() from void to int for error cases, so put it immediately after bpf_map_area_alloc() will make it eary to handle the error case. Signed-off-by: Yafang Shao --- kernel/bpf/bpf_struct_ops.c | 2 +- kernel/bpf/devmap.c | 5 ++--- kernel/bpf/hashtab.c | 4 ++-- 3 files changed, 5 insertions(+), 6 deletions(-) diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index 84b2d9d..36f24f8 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -624,6 +624,7 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) st_map->st_ops = st_ops; map = &st_map->map; + bpf_map_init_from_attr(map, attr); st_map->uvalue = bpf_map_area_alloc(vt->size, NUMA_NO_NODE); st_map->links = @@ -637,7 +638,6 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) mutex_init(&st_map->lock); set_vm_flush_reset_perms(st_map->image); - bpf_map_init_from_attr(map, attr); return map; } diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index f9a87dc..20decc7 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -127,9 +127,6 @@ static int dev_map_init_map(struct bpf_dtab *dtab, union bpf_attr *attr) */ attr->map_flags |= BPF_F_RDONLY_PROG; - - bpf_map_init_from_attr(&dtab->map, attr); - if (attr->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { dtab->n_buckets = roundup_pow_of_two(dtab->map.max_entries); @@ -167,6 +164,8 @@ static struct bpf_map *dev_map_alloc(union bpf_attr *attr) if (!dtab) return ERR_PTR(-ENOMEM); + bpf_map_init_from_attr(&dtab->map, attr); + err = dev_map_init_map(dtab, attr); if (err) { bpf_map_area_free(dtab); diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index eb1263f..fc7242c 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -508,10 +508,10 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) if (!htab) return ERR_PTR(-ENOMEM); - lockdep_register_key(&htab->lockdep_key); - bpf_map_init_from_attr(&htab->map, attr); + lockdep_register_key(&htab->lockdep_key); + if (percpu_lru) { /* ensure each CPU's lru list has >=1 elements. * since we are at it, make each lru list has the same From patchwork Fri Sep 2 02:29:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12963565 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73BB2C6FA83 for ; Fri, 2 Sep 2022 02:30:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0F6888009C; Thu, 1 Sep 2022 22:30:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0A4088008D; Thu, 1 Sep 2022 22:30:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E880D8009C; Thu, 1 Sep 2022 22:30:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D93AF8008D for ; Thu, 1 Sep 2022 22:30:22 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B3523AB991 for ; Fri, 2 Sep 2022 02:30:22 +0000 (UTC) X-FDA: 79865566284.19.3434B29 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf16.hostedemail.com (Postfix) with ESMTP id 68D2A18004C for ; Fri, 2 Sep 2022 02:30:22 +0000 (UTC) Received: by mail-pl1-f176.google.com with SMTP id p18so549573plr.8 for ; Thu, 01 Sep 2022 19:30:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=LrcJ3Xr0SYAuTpz8SD3A1h/kSVGH6jotE0l6oNv4Ovg=; b=ZetJgIdL8eRQ2zz4NzHA6ObyiXfbJvwMkPYVo2n1uUyfC9h/UkyEgIdyJxGeuZZPB+ MKebbSXi9Jo981jLRgLU92Es9yPmL1OjzcUu+mOdJ7syv5yrMM1G0xSItA1vl6ePfV0S uwQ262f2pJ6KdqgdZTdZFhRcBE/r6/XT2Hz+NYAJP+569e7Ks9I9dVJyeuygoeSiJhvn BQpVQ1qGiXqubD462c7rS1lMlpO2RYPeJws82y2tYIXUjKNNcrX7uONJQsDm1cJeFGUy AcQYvo6tvkak47JdptSNtojmCaiR5z7i11jdq094azfoLTI3b/CSnDjCNucThPfkhinH uA9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=LrcJ3Xr0SYAuTpz8SD3A1h/kSVGH6jotE0l6oNv4Ovg=; b=afS5YCTi5s9/DUQXQa0cPX9fj41Vp9o4EfodpvRjKYs2Nvxeyn7LE7yGjY+q/hzpDU 0koLp5kc1mpC52ij175FLyKhcL/SPTLIrhD4Wv2YcqFe3AA890X/TFehCQTMlCl8NAUb zWqo+i9fg6CJ/szb0/9hzjdSYaU+EDnvXXCfNiQC7raeg+0sxW/3wRb5o1WJZXa4XKjt sRYZ1DaO80sqJ9T5opjvdCL0z49tqaYtnCtv0rXmikYmzsEO0DYgUh7mX1IdT/fjwoUI P83N4WJPBRHBjYBFVaegnSzNKxZ1gD1LuRqRlZq5xh4zTSyvdo+eAoCACC6LJE+Upng+ 0USg== X-Gm-Message-State: ACgBeo2KPB0weIttFp9xCARqEbK1n7VlBt9sYHIjWn8GSa+0eLlLPvQT 1Rr5uNkrgMd4MwphsYoWrWQ= X-Google-Smtp-Source: AA6agR7hlVlkAm2JGo4tmeE+4PNAOKEAkJuR8AvlN+BkMFHxW3Owwi6CIyhrjA2RUwUBWCf3MEBnyg== X-Received: by 2002:a17:902:d181:b0:174:11d5:b2e0 with SMTP id m1-20020a170902d18100b0017411d5b2e0mr32896156plb.114.1662085821381; Thu, 01 Sep 2022 19:30:21 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:50ea:5400:4ff:fe1f:fbe2]) by smtp.gmail.com with ESMTPSA id j4-20020a170902da8400b0017297a6b39dsm269719plx.265.2022.09.01.19.30.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 19:30:20 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v3 05/13] bpf: Save memcg in bpf_map_init_from_attr() Date: Fri, 2 Sep 2022 02:29:55 +0000 Message-Id: <20220902023003.47124-6-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220902023003.47124-1-laoar.shao@gmail.com> References: <20220902023003.47124-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662085822; a=rsa-sha256; cv=none; b=6Fn3WYVaMJalzM+3SQ6vkjGieowWNx6l7X+eC4u8KaP6YnDcoh2at9cBi0xaCF4+veqWPr Bl+7VHHq+kBopHBn1n71KvT103m/lmKZProH4TyjvcZ8t8HdH4Yx/hkvHoUFA1mt8q1oGN lXx7k5tj/eGZd4/HQ8Et09AREwj5KpY= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ZetJgIdL; spf=pass (imf16.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662085822; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LrcJ3Xr0SYAuTpz8SD3A1h/kSVGH6jotE0l6oNv4Ovg=; b=kHvnSii2RlCZPLjxbwZ8EH/y8fRImIUBIsDj9uYeFxu3IEckKCG+bY/6Q//nAlPY+WXfZD I/6+nnK7NCY+Oktj3HZzultGK8ezGshCoCfA3EKvZz71VXKd8H1DURV3LRiO9UMjPixbXf Xg48YF6N/sJXEiwwvGxeebbSuGtKWCQ= X-Stat-Signature: oxa7a68j5wosyuyqojekargzto88zynw X-Rspam-User: X-Rspamd-Queue-Id: 68D2A18004C X-Rspamd-Server: rspam07 Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ZetJgIdL; spf=pass (imf16.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-HE-Tag: 1662085822-21282 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move bpf_map_save_memcg() into bpf_map_init_from_attr(), then all other map related memory allocation will be allocated after saving the memcg. And then we can get memcg from the map in the followup memory allocation. To pair with this change, bpf_map_release_memcg() is moved into bpf_map_area_free(). A new parameter struct bpf_map is introduced into bpf_map_area_free() for this purpose. Signed-off-by: Yafang Shao --- include/linux/bpf.h | 2 +- kernel/bpf/arraymap.c | 8 +++--- kernel/bpf/bloom_filter.c | 2 +- kernel/bpf/bpf_local_storage.c | 4 +-- kernel/bpf/bpf_struct_ops.c | 6 ++--- kernel/bpf/cpumap.c | 6 ++--- kernel/bpf/devmap.c | 8 +++--- kernel/bpf/hashtab.c | 10 +++---- kernel/bpf/local_storage.c | 2 +- kernel/bpf/lpm_trie.c | 2 +- kernel/bpf/offload.c | 4 +-- kernel/bpf/queue_stack_maps.c | 2 +- kernel/bpf/reuseport_array.c | 2 +- kernel/bpf/ringbuf.c | 8 +++--- kernel/bpf/stackmap.c | 8 +++--- kernel/bpf/syscall.c | 60 ++++++++++++++++++++++-------------------- net/core/sock_map.c | 12 ++++----- net/xdp/xskmap.c | 2 +- 18 files changed, 76 insertions(+), 72 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index a50d29c..729ddf7 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1638,7 +1638,7 @@ struct bpf_prog *bpf_prog_get_type_dev(u32 ufd, enum bpf_prog_type type, void bpf_map_put(struct bpf_map *map); void *bpf_map_area_alloc(u64 size, int numa_node); void *bpf_map_area_mmapable_alloc(u64 size, int numa_node); -void bpf_map_area_free(void *base); +void bpf_map_area_free(void *base, struct bpf_map *map); bool bpf_map_write_active(const struct bpf_map *map); void bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr); int generic_map_lookup_batch(struct bpf_map *map, diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index 6245274..7eebdbe 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -147,7 +147,7 @@ static struct bpf_map *array_map_alloc(union bpf_attr *attr) array->elem_size = elem_size; if (percpu && bpf_array_alloc_percpu(array)) { - bpf_map_area_free(array); + bpf_map_area_free(array, &array->map); return ERR_PTR(-ENOMEM); } @@ -430,9 +430,9 @@ static void array_map_free(struct bpf_map *map) bpf_array_free_percpu(array); if (array->map.map_flags & BPF_F_MMAPABLE) - bpf_map_area_free(array_map_vmalloc_addr(array)); + bpf_map_area_free(array_map_vmalloc_addr(array), map); else - bpf_map_area_free(array); + bpf_map_area_free(array, map); } static void array_map_seq_show_elem(struct bpf_map *map, void *key, @@ -780,7 +780,7 @@ static void fd_array_map_free(struct bpf_map *map) for (i = 0; i < array->map.max_entries; i++) BUG_ON(array->ptrs[i] != NULL); - bpf_map_area_free(array); + bpf_map_area_free(array, map); } static void *fd_array_map_lookup_elem(struct bpf_map *map, void *key) diff --git a/kernel/bpf/bloom_filter.c b/kernel/bpf/bloom_filter.c index b9ea539..e59064d 100644 --- a/kernel/bpf/bloom_filter.c +++ b/kernel/bpf/bloom_filter.c @@ -168,7 +168,7 @@ static void bloom_map_free(struct bpf_map *map) struct bpf_bloom_filter *bloom = container_of(map, struct bpf_bloom_filter, map); - bpf_map_area_free(bloom); + bpf_map_area_free(bloom, map); } static void *bloom_map_lookup_elem(struct bpf_map *map, void *key) diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index 802fc15..7b68d846 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -582,7 +582,7 @@ void bpf_local_storage_map_free(struct bpf_local_storage_map *smap, synchronize_rcu(); kvfree(smap->buckets); - bpf_map_area_free(smap); + bpf_map_area_free(smap, &smap->map); } int bpf_local_storage_map_alloc_check(union bpf_attr *attr) @@ -623,7 +623,7 @@ struct bpf_local_storage_map *bpf_local_storage_map_alloc(union bpf_attr *attr) smap->buckets = kvcalloc(sizeof(*smap->buckets), nbuckets, GFP_USER | __GFP_NOWARN | __GFP_ACCOUNT); if (!smap->buckets) { - bpf_map_area_free(smap); + bpf_map_area_free(smap, &smap->map); return ERR_PTR(-ENOMEM); } diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index 36f24f8..9fb8ad1 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -577,10 +577,10 @@ static void bpf_struct_ops_map_free(struct bpf_map *map) if (st_map->links) bpf_struct_ops_map_put_progs(st_map); - bpf_map_area_free(st_map->links); + bpf_map_area_free(st_map->links, NULL); bpf_jit_free_exec(st_map->image); - bpf_map_area_free(st_map->uvalue); - bpf_map_area_free(st_map); + bpf_map_area_free(st_map->uvalue, NULL); + bpf_map_area_free(st_map, map); } static int bpf_struct_ops_map_alloc_check(union bpf_attr *attr) diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index b5ba34d..7de2ae6 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -118,7 +118,7 @@ static struct bpf_map *cpu_map_alloc(union bpf_attr *attr) return &cmap->map; free_cmap: - bpf_map_area_free(cmap); + bpf_map_area_free(cmap, &cmap->map); return ERR_PTR(err); } @@ -622,8 +622,8 @@ static void cpu_map_free(struct bpf_map *map) /* bq flush and cleanup happens after RCU grace-period */ __cpu_map_entry_replace(cmap, i, NULL); /* call_rcu */ } - bpf_map_area_free(cmap->cpu_map); - bpf_map_area_free(cmap); + bpf_map_area_free(cmap->cpu_map, NULL); + bpf_map_area_free(cmap, map); } /* Elements are kept alive by RCU; either by rcu_read_lock() (from syscall) or diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 20decc7..3268ce7 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -168,7 +168,7 @@ static struct bpf_map *dev_map_alloc(union bpf_attr *attr) err = dev_map_init_map(dtab, attr); if (err) { - bpf_map_area_free(dtab); + bpf_map_area_free(dtab, &dtab->map); return ERR_PTR(err); } @@ -221,7 +221,7 @@ static void dev_map_free(struct bpf_map *map) } } - bpf_map_area_free(dtab->dev_index_head); + bpf_map_area_free(dtab->dev_index_head, NULL); } else { for (i = 0; i < dtab->map.max_entries; i++) { struct bpf_dtab_netdev *dev; @@ -236,10 +236,10 @@ static void dev_map_free(struct bpf_map *map) kfree(dev); } - bpf_map_area_free(dtab->netdev_map); + bpf_map_area_free(dtab->netdev_map, NULL); } - bpf_map_area_free(dtab); + bpf_map_area_free(dtab, &dtab->map); } static int dev_map_get_next_key(struct bpf_map *map, void *key, void *next_key) diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index fc7242c..063989e 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -303,7 +303,7 @@ static void htab_free_elems(struct bpf_htab *htab) cond_resched(); } free_elems: - bpf_map_area_free(htab->elems); + bpf_map_area_free(htab->elems, NULL); } /* The LRU list has a lock (lru_lock). Each htab bucket has a lock @@ -585,10 +585,10 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) free_map_locked: for (i = 0; i < HASHTAB_MAP_LOCK_COUNT; i++) free_percpu(htab->map_locked[i]); - bpf_map_area_free(htab->buckets); + bpf_map_area_free(htab->buckets, NULL); free_htab: lockdep_unregister_key(&htab->lockdep_key); - bpf_map_area_free(htab); + bpf_map_area_free(htab, &htab->map); return ERR_PTR(err); } @@ -1501,11 +1501,11 @@ static void htab_map_free(struct bpf_map *map) bpf_map_free_kptr_off_tab(map); free_percpu(htab->extra_elems); - bpf_map_area_free(htab->buckets); + bpf_map_area_free(htab->buckets, NULL); for (i = 0; i < HASHTAB_MAP_LOCK_COUNT; i++) free_percpu(htab->map_locked[i]); lockdep_unregister_key(&htab->lockdep_key); - bpf_map_area_free(htab); + bpf_map_area_free(htab, map); } static void htab_map_seq_show_elem(struct bpf_map *map, void *key, diff --git a/kernel/bpf/local_storage.c b/kernel/bpf/local_storage.c index 098cf33..c705d66 100644 --- a/kernel/bpf/local_storage.c +++ b/kernel/bpf/local_storage.c @@ -345,7 +345,7 @@ static void cgroup_storage_map_free(struct bpf_map *_map) WARN_ON(!RB_EMPTY_ROOT(&map->root)); WARN_ON(!list_empty(&map->list)); - bpf_map_area_free(map); + bpf_map_area_free(map, _map); } static int cgroup_storage_delete_elem(struct bpf_map *map, void *key) diff --git a/kernel/bpf/lpm_trie.c b/kernel/bpf/lpm_trie.c index d833496..fd99360 100644 --- a/kernel/bpf/lpm_trie.c +++ b/kernel/bpf/lpm_trie.c @@ -609,7 +609,7 @@ static void trie_free(struct bpf_map *map) } out: - bpf_map_area_free(trie); + bpf_map_area_free(trie, map); } static int trie_get_next_key(struct bpf_map *map, void *_key, void *_next_key) diff --git a/kernel/bpf/offload.c b/kernel/bpf/offload.c index 13e4efc..c9941a9 100644 --- a/kernel/bpf/offload.c +++ b/kernel/bpf/offload.c @@ -404,7 +404,7 @@ struct bpf_map *bpf_map_offload_map_alloc(union bpf_attr *attr) err_unlock: up_write(&bpf_devs_lock); rtnl_unlock(); - bpf_map_area_free(offmap); + bpf_map_area_free(offmap, &offmap->map); return ERR_PTR(err); } @@ -428,7 +428,7 @@ void bpf_map_offload_map_free(struct bpf_map *map) up_write(&bpf_devs_lock); rtnl_unlock(); - bpf_map_area_free(offmap); + bpf_map_area_free(offmap, map); } int bpf_map_offload_lookup_elem(struct bpf_map *map, void *key, void *value) diff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c index 8a5e060..f2ec0c4 100644 --- a/kernel/bpf/queue_stack_maps.c +++ b/kernel/bpf/queue_stack_maps.c @@ -92,7 +92,7 @@ static void queue_stack_map_free(struct bpf_map *map) { struct bpf_queue_stack *qs = bpf_queue_stack(map); - bpf_map_area_free(qs); + bpf_map_area_free(qs, map); } static int __queue_map_get(struct bpf_map *map, void *value, bool delete) diff --git a/kernel/bpf/reuseport_array.c b/kernel/bpf/reuseport_array.c index 85fa9db..3ac9276 100644 --- a/kernel/bpf/reuseport_array.c +++ b/kernel/bpf/reuseport_array.c @@ -143,7 +143,7 @@ static void reuseport_array_free(struct bpf_map *map) * Once reaching here, all sk->sk_user_data is not * referencing this "array". "array" can be freed now. */ - bpf_map_area_free(array); + bpf_map_area_free(array, map); } static struct bpf_map *reuseport_array_alloc(union bpf_attr *attr) diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index b483aea..74dd8dc 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -116,7 +116,7 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) err_free_pages: for (i = 0; i < nr_pages; i++) __free_page(pages[i]); - bpf_map_area_free(pages); + bpf_map_area_free(pages, NULL); return NULL; } @@ -172,7 +172,7 @@ static struct bpf_map *ringbuf_map_alloc(union bpf_attr *attr) rb_map->rb = bpf_ringbuf_alloc(attr->max_entries, rb_map->map.numa_node); if (!rb_map->rb) { - bpf_map_area_free(rb_map); + bpf_map_area_free(rb_map, &rb_map->map); return ERR_PTR(-ENOMEM); } @@ -190,7 +190,7 @@ static void bpf_ringbuf_free(struct bpf_ringbuf *rb) vunmap(rb); for (i = 0; i < nr_pages; i++) __free_page(pages[i]); - bpf_map_area_free(pages); + bpf_map_area_free(pages, NULL); } static void ringbuf_map_free(struct bpf_map *map) @@ -199,7 +199,7 @@ static void ringbuf_map_free(struct bpf_map *map) rb_map = container_of(map, struct bpf_ringbuf_map, map); bpf_ringbuf_free(rb_map->rb); - bpf_map_area_free(rb_map); + bpf_map_area_free(rb_map, map); } static void *ringbuf_map_lookup_elem(struct bpf_map *map, void *key) diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index 1adbe67..042b7d2 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -62,7 +62,7 @@ static int prealloc_elems_and_freelist(struct bpf_stack_map *smap) return 0; free_elems: - bpf_map_area_free(smap->elems); + bpf_map_area_free(smap->elems, NULL); return err; } @@ -120,7 +120,7 @@ static struct bpf_map *stack_map_alloc(union bpf_attr *attr) put_buffers: put_callchain_buffers(); free_smap: - bpf_map_area_free(smap); + bpf_map_area_free(smap, &smap->map); return ERR_PTR(err); } @@ -648,9 +648,9 @@ static void stack_map_free(struct bpf_map *map) { struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map); - bpf_map_area_free(smap->elems); + bpf_map_area_free(smap->elems, NULL); pcpu_freelist_destroy(&smap->freelist); - bpf_map_area_free(smap); + bpf_map_area_free(smap, map); put_callchain_buffers(); } diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index eaf1906..68a6f59 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -293,6 +293,34 @@ static int bpf_map_copy_value(struct bpf_map *map, void *key, void *value, return err; } +#ifdef CONFIG_MEMCG_KMEM +static void bpf_map_save_memcg(struct bpf_map *map) +{ + /* Currently if a map is created by a process belonging to the root + * memory cgroup, get_obj_cgroup_from_current() will return NULL. + * So we have to check map->objcg for being NULL each time it's + * being used. + */ + map->objcg = get_obj_cgroup_from_current(); +} + +static void bpf_map_release_memcg(struct bpf_map *map) +{ + if (map->objcg) + obj_cgroup_put(map->objcg); +} + +#else +static void bpf_map_save_memcg(struct bpf_map *map) +{ +} + +static void bpf_map_release_memcg(struct bpf_map *map) +{ +} + +#endif + /* Please, do not use this function outside from the map creation path * (e.g. in map update path) without taking care of setting the active * memory cgroup (see at bpf_map_kmalloc_node() for example). @@ -344,8 +372,10 @@ void *bpf_map_area_mmapable_alloc(u64 size, int numa_node) return __bpf_map_area_alloc(size, numa_node, true); } -void bpf_map_area_free(void *area) +void bpf_map_area_free(void *area, struct bpf_map *map) { + if (map) + bpf_map_release_memcg(map); kvfree(area); } @@ -363,6 +393,7 @@ static u32 bpf_map_flags_retain_permanent(u32 flags) void bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr) { + bpf_map_save_memcg(map); map->map_type = attr->map_type; map->key_size = attr->key_size; map->value_size = attr->value_size; @@ -417,22 +448,6 @@ void bpf_map_free_id(struct bpf_map *map, bool do_idr_lock) } #ifdef CONFIG_MEMCG_KMEM -static void bpf_map_save_memcg(struct bpf_map *map) -{ - /* Currently if a map is created by a process belonging to the root - * memory cgroup, get_obj_cgroup_from_current() will return NULL. - * So we have to check map->objcg for being NULL each time it's - * being used. - */ - map->objcg = get_obj_cgroup_from_current(); -} - -static void bpf_map_release_memcg(struct bpf_map *map) -{ - if (map->objcg) - obj_cgroup_put(map->objcg); -} - void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, int node) { @@ -477,14 +492,6 @@ void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, return ptr; } -#else -static void bpf_map_save_memcg(struct bpf_map *map) -{ -} - -static void bpf_map_release_memcg(struct bpf_map *map) -{ -} #endif static int bpf_map_kptr_off_cmp(const void *a, const void *b) @@ -605,7 +612,6 @@ static void bpf_map_free_deferred(struct work_struct *work) security_bpf_map_free(map); kfree(map->off_arr); - bpf_map_release_memcg(map); /* implementation dependent freeing, map_free callback also does * bpf_map_free_kptr_off_tab, if needed. */ @@ -1154,8 +1160,6 @@ static int map_create(union bpf_attr *attr) if (err) goto free_map_sec; - bpf_map_save_memcg(map); - err = bpf_map_new_fd(map, f_flags); if (err < 0) { /* failed to allocate fd. diff --git a/net/core/sock_map.c b/net/core/sock_map.c index a660bae..8da9fd4 100644 --- a/net/core/sock_map.c +++ b/net/core/sock_map.c @@ -52,7 +52,7 @@ static struct bpf_map *sock_map_alloc(union bpf_attr *attr) sizeof(struct sock *), stab->map.numa_node); if (!stab->sks) { - bpf_map_area_free(stab); + bpf_map_area_free(stab, &stab->map); return ERR_PTR(-ENOMEM); } @@ -360,8 +360,8 @@ static void sock_map_free(struct bpf_map *map) /* wait for psock readers accessing its map link */ synchronize_rcu(); - bpf_map_area_free(stab->sks); - bpf_map_area_free(stab); + bpf_map_area_free(stab->sks, NULL); + bpf_map_area_free(stab, map); } static void sock_map_release_progs(struct bpf_map *map) @@ -1115,7 +1115,7 @@ static struct bpf_map *sock_hash_alloc(union bpf_attr *attr) return &htab->map; free_htab: - bpf_map_area_free(htab); + bpf_map_area_free(htab, &htab->map); return ERR_PTR(err); } @@ -1167,8 +1167,8 @@ static void sock_hash_free(struct bpf_map *map) /* wait for psock readers accessing its map link */ synchronize_rcu(); - bpf_map_area_free(htab->buckets); - bpf_map_area_free(htab); + bpf_map_area_free(htab->buckets, NULL); + bpf_map_area_free(htab, map); } static void *sock_hash_lookup_sys(struct bpf_map *map, void *key) diff --git a/net/xdp/xskmap.c b/net/xdp/xskmap.c index acc8e52..5abb87e 100644 --- a/net/xdp/xskmap.c +++ b/net/xdp/xskmap.c @@ -90,7 +90,7 @@ static void xsk_map_free(struct bpf_map *map) struct xsk_map *m = container_of(map, struct xsk_map, map); synchronize_net(); - bpf_map_area_free(m); + bpf_map_area_free(m, map); } static int xsk_map_get_next_key(struct bpf_map *map, void *key, void *next_key) From patchwork Fri Sep 2 02:29:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12963566 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BD43ECAAD2 for ; Fri, 2 Sep 2022 02:30:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2076E8009D; Thu, 1 Sep 2022 22:30:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1B6F58008D; Thu, 1 Sep 2022 22:30:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F25E78009D; Thu, 1 Sep 2022 22:30:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E2B1E8008D for ; Thu, 1 Sep 2022 22:30:24 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B3102C07DA for ; Fri, 2 Sep 2022 02:30:24 +0000 (UTC) X-FDA: 79865566368.06.B6904FD Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) by imf04.hostedemail.com (Postfix) with ESMTP id 6913340054 for ; Fri, 2 Sep 2022 02:30:24 +0000 (UTC) Received: by mail-pg1-f178.google.com with SMTP id r69so781141pgr.2 for ; Thu, 01 Sep 2022 19:30:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=d52EByDRkZy6FSBYa32Z9k09ZdChhn+PMU2CBRjQSD0=; b=jQaj1rFYPYqud+kx+l6MgcNA+2B0GanmlTdOaPZoR69PjiXwTATAWLSeAUlpOhVy9i eCeIFonJykQk4PVpdFMcHkVAHywxyy5FMhCfSHQGigLTwoY6DKbLDmC3FHFoqggNBSr3 aZQHxLzxgO/eBCPiFfFG4IbfDAcKY1ukxNrGQAz98ZXYnpZjLNeCnjm72k/asJROPvkY ABTAdaUC0tyV9W0Sx5tKUzIJ/i18osTElJ5Jq37Vyfyqapu2AzaswEz0aUMrNRVjljUQ aG2tB92mKTTYGL3TCz5ACchreGLu0BHPgZjjoHwHhT8CvaCMVrE39J6onJORtr1bq3Xl EQ7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=d52EByDRkZy6FSBYa32Z9k09ZdChhn+PMU2CBRjQSD0=; b=O10I+guETtwyLg4HS8PXluS+QNJ19iEHYAdSCtBDMwDTWHXMwKDI71+PXlFpnXQO+x fSOLiSquX4BBYa6AImqIG5/ucfM7E1sxzHPWufREequl4Y00kU2pMyojnemPsI+uJ9wU fFMrFkQcVcrKqKalUC7MqHom9ICX0QY0/yC4fiCmGYtgnNMyG+WavxbF94c76DluIdNu D4u1cY7179Gj9Luz+kL8YjKYZP2u4TZi1EiRvTaxLPrKkJtuj+ZoINoH2Fr0tg1nCDOz XbEdrVHGYRvT6GbRWicAJc54VTjYUtkqTUOf9x1G3L4Tun9j++tM1co4Me3TYFuIGqV+ 2Kzg== X-Gm-Message-State: ACgBeo2MmCmT36Uw2yo7BsNvRCq5TRmeGjQ8Nyjx07TZdDGiw21q7IuT FSX/l3IB4suNbRgiWxwkHps= X-Google-Smtp-Source: AA6agR6mG+dcQVuRFfaLKaQkPzMWMkXyIGqJIXOsblH9hJwE5q+optrmrPsZc2vJWFn2zAZnxDAPUw== X-Received: by 2002:a05:6a02:104:b0:430:93ec:776f with SMTP id bg4-20020a056a02010400b0043093ec776fmr3328929pgb.544.1662085823442; Thu, 01 Sep 2022 19:30:23 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:50ea:5400:4ff:fe1f:fbe2]) by smtp.gmail.com with ESMTPSA id j4-20020a170902da8400b0017297a6b39dsm269719plx.265.2022.09.01.19.30.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 19:30:22 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v3 06/13] bpf: Use scoped-based charge in bpf_map_area_alloc Date: Fri, 2 Sep 2022 02:29:56 +0000 Message-Id: <20220902023003.47124-7-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220902023003.47124-1-laoar.shao@gmail.com> References: <20220902023003.47124-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662085824; a=rsa-sha256; cv=none; b=dlu6o0TeEKp/E7RpObIf19e5DaOQazWt/OvdDa6ZPgVZJEYqSidMbe9Xxbf1/xsQGKXQQr Nxx0JbiYY1Vo3BOYhYOf9i6xJk6alqW5Smo9CWWT5zkU8cM0NNfd7KdE89r70JvHQ9/hnx b8fJOCOBHCDeZUgsNyzwqw06RkwlMnM= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=jQaj1rFY; spf=pass (imf04.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.215.178 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662085824; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=d52EByDRkZy6FSBYa32Z9k09ZdChhn+PMU2CBRjQSD0=; b=J97fH/JQ6qMrvDXLfRyXWmiQ8DfNzgBp5ZKxeYzjCcyWSkNi5PDKdQpon2Zz7OypBgFRBw 1ZXgcCLQqw+2uv1OaGqc8n6M7FO82ZEllKVuSXsP62wJdwpcKdRoFZTNWK6jG2tZxxiOAr l9TZA+laNsIOQWqZw7L51CN6nKNPnBQ= X-Stat-Signature: mo6beu594fzcs85dboddbfdhd4pbyaxb X-Rspam-User: X-Rspamd-Queue-Id: 6913340054 X-Rspamd-Server: rspam07 Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=jQaj1rFY; spf=pass (imf04.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.215.178 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-HE-Tag: 1662085824-992452 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently bpf_map_area_alloc() is used to allocate a container of struct bpf_map or members in this container. To distinguish the map creation and the other case, a new parameter struct bpf_map is added into bpf_map_area_alloc(). Then for the non-map-creation case, we could get the memcg from the map instead of using the current memcg. Signed-off-by: Yafang Shao --- include/linux/bpf.h | 2 +- kernel/bpf/arraymap.c | 2 +- kernel/bpf/bloom_filter.c | 2 +- kernel/bpf/bpf_local_storage.c | 2 +- kernel/bpf/bpf_struct_ops.c | 6 +++--- kernel/bpf/cpumap.c | 5 +++-- kernel/bpf/devmap.c | 13 ++++++++----- kernel/bpf/hashtab.c | 8 +++++--- kernel/bpf/local_storage.c | 2 +- kernel/bpf/lpm_trie.c | 2 +- kernel/bpf/offload.c | 2 +- kernel/bpf/queue_stack_maps.c | 2 +- kernel/bpf/reuseport_array.c | 2 +- kernel/bpf/ringbuf.c | 15 +++++++++------ kernel/bpf/stackmap.c | 5 +++-- kernel/bpf/syscall.c | 16 ++++++++++++++-- net/core/sock_map.c | 10 ++++++---- net/xdp/xskmap.c | 2 +- 18 files changed, 61 insertions(+), 37 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 729ddf7..e8ac29f 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1636,7 +1636,7 @@ struct bpf_prog *bpf_prog_get_type_dev(u32 ufd, enum bpf_prog_type type, struct bpf_map * __must_check bpf_map_inc_not_zero(struct bpf_map *map); void bpf_map_put_with_uref(struct bpf_map *map); void bpf_map_put(struct bpf_map *map); -void *bpf_map_area_alloc(u64 size, int numa_node); +void *bpf_map_area_alloc(u64 size, int numa_node, struct bpf_map *map); void *bpf_map_area_mmapable_alloc(u64 size, int numa_node); void bpf_map_area_free(void *base, struct bpf_map *map); bool bpf_map_write_active(const struct bpf_map *map); diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index 7eebdbe..114fbda 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -135,7 +135,7 @@ static struct bpf_map *array_map_alloc(union bpf_attr *attr) array = data + PAGE_ALIGN(sizeof(struct bpf_array)) - offsetof(struct bpf_array, value); } else { - array = bpf_map_area_alloc(array_size, numa_node); + array = bpf_map_area_alloc(array_size, numa_node, NULL); } if (!array) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/bloom_filter.c b/kernel/bpf/bloom_filter.c index e59064d..6691f79 100644 --- a/kernel/bpf/bloom_filter.c +++ b/kernel/bpf/bloom_filter.c @@ -142,7 +142,7 @@ static struct bpf_map *bloom_map_alloc(union bpf_attr *attr) } bitset_bytes = roundup(bitset_bytes, sizeof(unsigned long)); - bloom = bpf_map_area_alloc(sizeof(*bloom) + bitset_bytes, numa_node); + bloom = bpf_map_area_alloc(sizeof(*bloom) + bitset_bytes, numa_node, NULL); if (!bloom) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index 7b68d846..44498d7d 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -610,7 +610,7 @@ struct bpf_local_storage_map *bpf_local_storage_map_alloc(union bpf_attr *attr) unsigned int i; u32 nbuckets; - smap = bpf_map_area_alloc(sizeof(*smap), NUMA_NO_NODE); + smap = bpf_map_area_alloc(sizeof(*smap), NUMA_NO_NODE, NULL); if (!smap) return ERR_PTR(-ENOMEM); bpf_map_init_from_attr(&smap->map, attr); diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index 9fb8ad1..37ba5c0 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -618,7 +618,7 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) */ (vt->size - sizeof(struct bpf_struct_ops_value)); - st_map = bpf_map_area_alloc(st_map_size, NUMA_NO_NODE); + st_map = bpf_map_area_alloc(st_map_size, NUMA_NO_NODE, NULL); if (!st_map) return ERR_PTR(-ENOMEM); @@ -626,10 +626,10 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) map = &st_map->map; bpf_map_init_from_attr(map, attr); - st_map->uvalue = bpf_map_area_alloc(vt->size, NUMA_NO_NODE); + st_map->uvalue = bpf_map_area_alloc(vt->size, NUMA_NO_NODE, map); st_map->links = bpf_map_area_alloc(btf_type_vlen(t) * sizeof(struct bpf_links *), - NUMA_NO_NODE); + NUMA_NO_NODE, map); st_map->image = bpf_jit_alloc_exec(PAGE_SIZE); if (!st_map->uvalue || !st_map->links || !st_map->image) { bpf_struct_ops_map_free(map); diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index 7de2ae6..b593157 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -97,7 +97,7 @@ static struct bpf_map *cpu_map_alloc(union bpf_attr *attr) attr->map_flags & ~BPF_F_NUMA_NODE) return ERR_PTR(-EINVAL); - cmap = bpf_map_area_alloc(sizeof(*cmap), NUMA_NO_NODE); + cmap = bpf_map_area_alloc(sizeof(*cmap), NUMA_NO_NODE, NULL); if (!cmap) return ERR_PTR(-ENOMEM); @@ -112,7 +112,8 @@ static struct bpf_map *cpu_map_alloc(union bpf_attr *attr) /* Alloc array for possible remote "destination" CPUs */ cmap->cpu_map = bpf_map_area_alloc(cmap->map.max_entries * sizeof(struct bpf_cpu_map_entry *), - cmap->map.numa_node); + cmap->map.numa_node, + &cmap->map); if (!cmap->cpu_map) goto free_cmap; diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 3268ce7..807a4cd 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -89,12 +89,13 @@ struct bpf_dtab { static LIST_HEAD(dev_map_list); static struct hlist_head *dev_map_create_hash(unsigned int entries, - int numa_node) + int numa_node, + struct bpf_map *map) { int i; struct hlist_head *hash; - hash = bpf_map_area_alloc((u64) entries * sizeof(*hash), numa_node); + hash = bpf_map_area_alloc((u64) entries * sizeof(*hash), numa_node, map); if (hash != NULL) for (i = 0; i < entries; i++) INIT_HLIST_HEAD(&hash[i]); @@ -136,7 +137,8 @@ static int dev_map_init_map(struct bpf_dtab *dtab, union bpf_attr *attr) if (attr->map_type == BPF_MAP_TYPE_DEVMAP_HASH) { dtab->dev_index_head = dev_map_create_hash(dtab->n_buckets, - dtab->map.numa_node); + dtab->map.numa_node, + &dtab->map); if (!dtab->dev_index_head) return -ENOMEM; @@ -144,7 +146,8 @@ static int dev_map_init_map(struct bpf_dtab *dtab, union bpf_attr *attr) } else { dtab->netdev_map = bpf_map_area_alloc((u64) dtab->map.max_entries * sizeof(struct bpf_dtab_netdev *), - dtab->map.numa_node); + dtab->map.numa_node, + &dtab->map); if (!dtab->netdev_map) return -ENOMEM; } @@ -160,7 +163,7 @@ static struct bpf_map *dev_map_alloc(union bpf_attr *attr) if (!capable(CAP_NET_ADMIN)) return ERR_PTR(-EPERM); - dtab = bpf_map_area_alloc(sizeof(*dtab), NUMA_NO_NODE); + dtab = bpf_map_area_alloc(sizeof(*dtab), NUMA_NO_NODE, NULL); if (!dtab) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 063989e..975bcc1 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -341,7 +341,8 @@ static int prealloc_init(struct bpf_htab *htab) num_entries += num_possible_cpus(); htab->elems = bpf_map_area_alloc((u64)htab->elem_size * num_entries, - htab->map.numa_node); + htab->map.numa_node, + &htab->map); if (!htab->elems) return -ENOMEM; @@ -504,7 +505,7 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) struct bpf_htab *htab; int err, i; - htab = bpf_map_area_alloc(sizeof(*htab), NUMA_NO_NODE); + htab = bpf_map_area_alloc(sizeof(*htab), NUMA_NO_NODE, NULL); if (!htab) return ERR_PTR(-ENOMEM); @@ -543,7 +544,8 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) err = -ENOMEM; htab->buckets = bpf_map_area_alloc(htab->n_buckets * sizeof(struct bucket), - htab->map.numa_node); + htab->map.numa_node, + &htab->map); if (!htab->buckets) goto free_htab; diff --git a/kernel/bpf/local_storage.c b/kernel/bpf/local_storage.c index c705d66..fcc7ece 100644 --- a/kernel/bpf/local_storage.c +++ b/kernel/bpf/local_storage.c @@ -313,7 +313,7 @@ static struct bpf_map *cgroup_storage_map_alloc(union bpf_attr *attr) /* max_entries is not used and enforced to be 0 */ return ERR_PTR(-EINVAL); - map = bpf_map_area_alloc(sizeof(struct bpf_cgroup_storage_map), numa_node); + map = bpf_map_area_alloc(sizeof(struct bpf_cgroup_storage_map), numa_node, NULL); if (!map) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/lpm_trie.c b/kernel/bpf/lpm_trie.c index fd99360..3d329ae 100644 --- a/kernel/bpf/lpm_trie.c +++ b/kernel/bpf/lpm_trie.c @@ -558,7 +558,7 @@ static struct bpf_map *trie_alloc(union bpf_attr *attr) attr->value_size > LPM_VAL_SIZE_MAX) return ERR_PTR(-EINVAL); - trie = bpf_map_area_alloc(sizeof(*trie), NUMA_NO_NODE); + trie = bpf_map_area_alloc(sizeof(*trie), NUMA_NO_NODE, NULL); if (!trie) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/offload.c b/kernel/bpf/offload.c index c9941a9..87c59da 100644 --- a/kernel/bpf/offload.c +++ b/kernel/bpf/offload.c @@ -372,7 +372,7 @@ struct bpf_map *bpf_map_offload_map_alloc(union bpf_attr *attr) attr->map_type != BPF_MAP_TYPE_HASH) return ERR_PTR(-EINVAL); - offmap = bpf_map_area_alloc(sizeof(*offmap), NUMA_NO_NODE); + offmap = bpf_map_area_alloc(sizeof(*offmap), NUMA_NO_NODE, NULL); if (!offmap) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c index f2ec0c4..bf57e45 100644 --- a/kernel/bpf/queue_stack_maps.c +++ b/kernel/bpf/queue_stack_maps.c @@ -74,7 +74,7 @@ static struct bpf_map *queue_stack_map_alloc(union bpf_attr *attr) size = (u64) attr->max_entries + 1; queue_size = sizeof(*qs) + size * attr->value_size; - qs = bpf_map_area_alloc(queue_size, numa_node); + qs = bpf_map_area_alloc(queue_size, numa_node, NULL); if (!qs) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/reuseport_array.c b/kernel/bpf/reuseport_array.c index 3ac9276..447f664 100644 --- a/kernel/bpf/reuseport_array.c +++ b/kernel/bpf/reuseport_array.c @@ -155,7 +155,7 @@ static struct bpf_map *reuseport_array_alloc(union bpf_attr *attr) return ERR_PTR(-EPERM); /* allocate all map elements and zero-initialize them */ - array = bpf_map_area_alloc(struct_size(array, ptrs, attr->max_entries), numa_node); + array = bpf_map_area_alloc(struct_size(array, ptrs, attr->max_entries), numa_node, NULL); if (!array) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index 74dd8dc..5eb7820 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -59,7 +59,8 @@ struct bpf_ringbuf_hdr { u32 pg_off; }; -static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) +static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node, + struct bpf_map *map) { const gfp_t flags = GFP_KERNEL_ACCOUNT | __GFP_RETRY_MAYFAIL | __GFP_NOWARN | __GFP_ZERO; @@ -89,7 +90,7 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) * user-space implementations significantly. */ array_size = (nr_meta_pages + 2 * nr_data_pages) * sizeof(*pages); - pages = bpf_map_area_alloc(array_size, numa_node); + pages = bpf_map_area_alloc(array_size, numa_node, map); if (!pages) return NULL; @@ -127,11 +128,12 @@ static void bpf_ringbuf_notify(struct irq_work *work) wake_up_all(&rb->waitq); } -static struct bpf_ringbuf *bpf_ringbuf_alloc(size_t data_sz, int numa_node) +static struct bpf_ringbuf *bpf_ringbuf_alloc(size_t data_sz, int numa_node, + struct bpf_map *map) { struct bpf_ringbuf *rb; - rb = bpf_ringbuf_area_alloc(data_sz, numa_node); + rb = bpf_ringbuf_area_alloc(data_sz, numa_node, map); if (!rb) return NULL; @@ -164,13 +166,14 @@ static struct bpf_map *ringbuf_map_alloc(union bpf_attr *attr) return ERR_PTR(-E2BIG); #endif - rb_map = bpf_map_area_alloc(sizeof(*rb_map), NUMA_NO_NODE); + rb_map = bpf_map_area_alloc(sizeof(*rb_map), NUMA_NO_NODE, NULL); if (!rb_map) return ERR_PTR(-ENOMEM); bpf_map_init_from_attr(&rb_map->map, attr); - rb_map->rb = bpf_ringbuf_alloc(attr->max_entries, rb_map->map.numa_node); + rb_map->rb = bpf_ringbuf_alloc(attr->max_entries, rb_map->map.numa_node, + &rb_map->map); if (!rb_map->rb) { bpf_map_area_free(rb_map, &rb_map->map); return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index 042b7d2..9440fab 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -49,7 +49,8 @@ static int prealloc_elems_and_freelist(struct bpf_stack_map *smap) int err; smap->elems = bpf_map_area_alloc(elem_size * smap->map.max_entries, - smap->map.numa_node); + smap->map.numa_node, + &smap->map); if (!smap->elems) return -ENOMEM; @@ -100,7 +101,7 @@ static struct bpf_map *stack_map_alloc(union bpf_attr *attr) return ERR_PTR(-E2BIG); cost = n_buckets * sizeof(struct stack_map_bucket *) + sizeof(*smap); - smap = bpf_map_area_alloc(cost, bpf_map_attr_numa_node(attr)); + smap = bpf_map_area_alloc(cost, bpf_map_attr_numa_node(attr), NULL); if (!smap) return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 68a6f59..eefe590 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -362,9 +362,21 @@ static void *__bpf_map_area_alloc(u64 size, int numa_node, bool mmapable) flags, numa_node, __builtin_return_address(0)); } -void *bpf_map_area_alloc(u64 size, int numa_node) +void *bpf_map_area_alloc(u64 size, int numa_node, struct bpf_map *map) { - return __bpf_map_area_alloc(size, numa_node, false); + struct mem_cgroup *memcg, *old_memcg; + void *ptr; + + if (!map) + return __bpf_map_area_alloc(size, numa_node, false); + + memcg = bpf_map_get_memcg(map); + old_memcg = set_active_memcg(memcg); + ptr = __bpf_map_area_alloc(size, numa_node, false); + set_active_memcg(old_memcg); + bpf_map_put_memcg(memcg); + + return ptr; } void *bpf_map_area_mmapable_alloc(u64 size, int numa_node) diff --git a/net/core/sock_map.c b/net/core/sock_map.c index 8da9fd4..25a5ac4 100644 --- a/net/core/sock_map.c +++ b/net/core/sock_map.c @@ -41,7 +41,7 @@ static struct bpf_map *sock_map_alloc(union bpf_attr *attr) attr->map_flags & ~SOCK_CREATE_FLAG_MASK) return ERR_PTR(-EINVAL); - stab = bpf_map_area_alloc(sizeof(*stab), NUMA_NO_NODE); + stab = bpf_map_area_alloc(sizeof(*stab), NUMA_NO_NODE, NULL); if (!stab) return ERR_PTR(-ENOMEM); @@ -50,7 +50,8 @@ static struct bpf_map *sock_map_alloc(union bpf_attr *attr) stab->sks = bpf_map_area_alloc((u64) stab->map.max_entries * sizeof(struct sock *), - stab->map.numa_node); + stab->map.numa_node, + &stab->map); if (!stab->sks) { bpf_map_area_free(stab, &stab->map); return ERR_PTR(-ENOMEM); @@ -1085,7 +1086,7 @@ static struct bpf_map *sock_hash_alloc(union bpf_attr *attr) if (attr->key_size > MAX_BPF_STACK) return ERR_PTR(-E2BIG); - htab = bpf_map_area_alloc(sizeof(*htab), NUMA_NO_NODE); + htab = bpf_map_area_alloc(sizeof(*htab), NUMA_NO_NODE, NULL); if (!htab) return ERR_PTR(-ENOMEM); @@ -1102,7 +1103,8 @@ static struct bpf_map *sock_hash_alloc(union bpf_attr *attr) htab->buckets = bpf_map_area_alloc(htab->buckets_num * sizeof(struct bpf_shtab_bucket), - htab->map.numa_node); + htab->map.numa_node, + &htab->map); if (!htab->buckets) { err = -ENOMEM; goto free_htab; diff --git a/net/xdp/xskmap.c b/net/xdp/xskmap.c index 5abb87e..beb11fd 100644 --- a/net/xdp/xskmap.c +++ b/net/xdp/xskmap.c @@ -75,7 +75,7 @@ static struct bpf_map *xsk_map_alloc(union bpf_attr *attr) numa_node = bpf_map_attr_numa_node(attr); size = struct_size(m, xsk_map, attr->max_entries); - m = bpf_map_area_alloc(size, numa_node); + m = bpf_map_area_alloc(size, numa_node, NULL); if (!m) return ERR_PTR(-ENOMEM); From patchwork Fri Sep 2 02:29:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12963567 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F104C38145 for ; Fri, 2 Sep 2022 02:30:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1E2DF8009E; Thu, 1 Sep 2022 22:30:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 190FF8008D; Thu, 1 Sep 2022 22:30:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 00AE48009E; Thu, 1 Sep 2022 22:30:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D8FDF8008D for ; Thu, 1 Sep 2022 22:30:26 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id B60C61405D3 for ; Fri, 2 Sep 2022 02:30:26 +0000 (UTC) X-FDA: 79865566452.24.60497A1 Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) by imf08.hostedemail.com (Postfix) with ESMTP id 701F1160061 for ; Fri, 2 Sep 2022 02:30:26 +0000 (UTC) Received: by mail-pg1-f181.google.com with SMTP id x80so816543pgx.0 for ; Thu, 01 Sep 2022 19:30:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=E8MAYkdVIQP0FJEM6EBF4l8Xg/qgqUjpGYemv3sFZKw=; b=WS87BarBUJUmdJvwHVOUJY8VhLnc3qqplexcN9euuSS2P7/7zhZnOc+Mdfx1nLfPp4 9WwWX1Zu+Wrt6tRcQXJGP+9X5wVNcqCKy7Ytx+0yvkRORBzJMZPRSfGfU5GKm/YFHg22 CbIxqWhkwiFvp6k3naiQouLi/puXIuV6j5aVe07KNixUP6a85g9RtfkPff9rzfmEO4Td iGUdUahKFP8/Nv+/ndoJhg+QwuPa20S42nMTGRzV3onP3zPd8NxcsLe87WY1UXyZtYHB 03hF7uKZlzxujTSM4oSCEJFzU7k8OW1wQlzMsrpPnHhpeL0IifVBg2WuiV3EgidomGr0 D6bQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=E8MAYkdVIQP0FJEM6EBF4l8Xg/qgqUjpGYemv3sFZKw=; b=eDObbLWKsn1SOWKw/ojky2uWn8/Jmh6RKo+M9/+6tfiC0Q2tTHHvHH3LwEEIK1IYj2 QvzfZw3t7nBis5lmkLU43ggS/DSR78SJBtlG8lHOKMiTnbmsH2ov0ap/JOwVRn4dM/rZ 3481LzaK6vdSET80TgyIyHfCGTI0tBLsaFDJJISU8PZbm8HKOi1fpwuLq4BIhvncBGHZ iK3A7OC1tBz8pRrVQ9RmQuhCtrBdWQ9Jfy0gxcOkyRHB/ENnYqjqk5etoiu/q/iNYnOI Wvpb/if0EXwDAFRDil4Mu6r8CtRZrbS2rfXI3knB8/YWvGD/bLOUxHGkbhnzyxQ/c6Q1 9aUQ== X-Gm-Message-State: ACgBeo08JERtnuiD0NngXXQO3XKB/vTIlGlbq5/zRehsqm2hWZiRiRWF +rVh85pbTgJt4zpc9PQ2Dew= X-Google-Smtp-Source: AA6agR7LqQuyAM3qmmIWpEDGymm/gkFBBR7fNB6mvH0YDlAlzs+wsP1hmzMC4vbU/wwv1JDDBSFZJA== X-Received: by 2002:a63:e25:0:b0:41c:30f7:c39c with SMTP id d37-20020a630e25000000b0041c30f7c39cmr28488997pgl.147.1662085825466; Thu, 01 Sep 2022 19:30:25 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:50ea:5400:4ff:fe1f:fbe2]) by smtp.gmail.com with ESMTPSA id j4-20020a170902da8400b0017297a6b39dsm269719plx.265.2022.09.01.19.30.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 19:30:24 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v3 07/13] bpf: Introduce new helpers bpf_ringbuf_pages_{alloc,free} Date: Fri, 2 Sep 2022 02:29:57 +0000 Message-Id: <20220902023003.47124-8-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220902023003.47124-1-laoar.shao@gmail.com> References: <20220902023003.47124-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662085826; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=E8MAYkdVIQP0FJEM6EBF4l8Xg/qgqUjpGYemv3sFZKw=; b=WC4dGc6tZiPeNGvOITroutsiHtv0CIeUjfCUw5jFdjhm0vueB6xYcy+EKBfnPxfGZlwZZK jA+dPVKSjKzMHD56OCI1hWTppToVEx0fWZUlTKdG2PWMb6Kvpq+lhdhcd+WYOc0adFeCIE 42eLfcpQFTCZZsw2TcgFgnLVcSyMtA4= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=WS87BarB; spf=pass (imf08.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.215.181 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662085826; a=rsa-sha256; cv=none; b=hDYvJt1x5yS/Es5JQwdN81CRI5Y0vHfE8NNlq9WqjAethHlDjf+Vs/L2lDarCdcMI1W+LJ ed0iaRj//7L5zphldJl1ueAB/KR08bS3Ow99T565n8nUwGOuPv6w+LDlBu9u1rBAzgxxWV gW3k2VUL+6DudUpYNAEU2QMqs/vaA8U= X-Stat-Signature: dhyma5um56y6joo9556ukgn6bni34wfm X-Rspamd-Queue-Id: 701F1160061 Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=WS87BarB; spf=pass (imf08.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.215.181 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam05 X-Rspam-User: X-HE-Tag: 1662085826-648081 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allocate pages related memory into the new helper bpf_ringbuf_pages_alloc(), then it can be handled as a single unit. Suggested-by: Andrii Nakryiko Signed-off-by: Yafang Shao Acked-by: Andrii Nakryiko --- kernel/bpf/ringbuf.c | 80 ++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 56 insertions(+), 24 deletions(-) diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index 5eb7820..1e7284c 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -59,6 +59,57 @@ struct bpf_ringbuf_hdr { u32 pg_off; }; +static void bpf_ringbuf_pages_free(struct page **pages, int nr_pages) +{ + int i; + + for (i = 0; i < nr_pages; i++) + __free_page(pages[i]); + bpf_map_area_free(pages, NULL); +} + +static struct page **bpf_ringbuf_pages_alloc(struct bpf_map *map, + int nr_meta_pages, + int nr_data_pages, + int numa_node, + const gfp_t flags) +{ + int nr_pages = nr_meta_pages + nr_data_pages; + struct mem_cgroup *memcg, *old_memcg; + struct page **pages, *page; + int array_size; + int i; + + memcg = bpf_map_get_memcg(map); + old_memcg = set_active_memcg(memcg); + array_size = (nr_meta_pages + 2 * nr_data_pages) * sizeof(*pages); + pages = bpf_map_area_alloc(array_size, numa_node, NULL); + if (!pages) + goto err; + + for (i = 0; i < nr_pages; i++) { + page = alloc_pages_node(numa_node, flags, 0); + if (!page) { + nr_pages = i; + goto err_free_pages; + } + pages[i] = page; + if (i >= nr_meta_pages) + pages[nr_data_pages + i] = page; + } + set_active_memcg(old_memcg); + bpf_map_put_memcg(memcg); + + return pages; + +err_free_pages: + bpf_ringbuf_pages_free(pages, nr_pages); +err: + set_active_memcg(old_memcg); + bpf_map_put_memcg(memcg); + return NULL; +} + static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node, struct bpf_map *map) { @@ -67,10 +118,8 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node, int nr_meta_pages = RINGBUF_PGOFF + RINGBUF_POS_PAGES; int nr_data_pages = data_sz >> PAGE_SHIFT; int nr_pages = nr_meta_pages + nr_data_pages; - struct page **pages, *page; struct bpf_ringbuf *rb; - size_t array_size; - int i; + struct page **pages; /* Each data page is mapped twice to allow "virtual" * continuous read of samples wrapping around the end of ring @@ -89,22 +138,11 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node, * when mmap()'ed in user-space, simplifying both kernel and * user-space implementations significantly. */ - array_size = (nr_meta_pages + 2 * nr_data_pages) * sizeof(*pages); - pages = bpf_map_area_alloc(array_size, numa_node, map); + pages = bpf_ringbuf_pages_alloc(map, nr_meta_pages, nr_data_pages, + numa_node, flags); if (!pages) return NULL; - for (i = 0; i < nr_pages; i++) { - page = alloc_pages_node(numa_node, flags, 0); - if (!page) { - nr_pages = i; - goto err_free_pages; - } - pages[i] = page; - if (i >= nr_meta_pages) - pages[nr_data_pages + i] = page; - } - rb = vmap(pages, nr_meta_pages + 2 * nr_data_pages, VM_MAP | VM_USERMAP, PAGE_KERNEL); if (rb) { @@ -114,10 +152,6 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node, return rb; } -err_free_pages: - for (i = 0; i < nr_pages; i++) - __free_page(pages[i]); - bpf_map_area_free(pages, NULL); return NULL; } @@ -188,12 +222,10 @@ static void bpf_ringbuf_free(struct bpf_ringbuf *rb) * to unmap rb itself with vunmap() below */ struct page **pages = rb->pages; - int i, nr_pages = rb->nr_pages; + int nr_pages = rb->nr_pages; vunmap(rb); - for (i = 0; i < nr_pages; i++) - __free_page(pages[i]); - bpf_map_area_free(pages, NULL); + bpf_ringbuf_pages_free(pages, nr_pages); } static void ringbuf_map_free(struct bpf_map *map) From patchwork Fri Sep 2 02:29:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12963568 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B46EDECAAD2 for ; Fri, 2 Sep 2022 02:30:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4EDFC8009F; Thu, 1 Sep 2022 22:30:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 49DD48008D; Thu, 1 Sep 2022 22:30:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2F0B58009F; Thu, 1 Sep 2022 22:30:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1EA008008D for ; Thu, 1 Sep 2022 22:30:29 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id DFAA01A1167 for ; Fri, 2 Sep 2022 02:30:28 +0000 (UTC) X-FDA: 79865566536.17.93993AD Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf18.hostedemail.com (Postfix) with ESMTP id 996B11C0049 for ; Fri, 2 Sep 2022 02:30:28 +0000 (UTC) Received: by mail-pl1-f175.google.com with SMTP id u22so507425plq.12 for ; Thu, 01 Sep 2022 19:30:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=V0gc6AUiAQY/WQln3OGyPk91MKpBrNPoI3+sFkmiGew=; b=YdKJPsj3wxmrtdhJ63cf//SS9jJ7JPgMY5Ex1l+fOx9UNzapxQ7i8fyfCKfg4DC/fk SKCqoIu3Lb00g2SEoCnPl+BmSgfvaKisdl966nvF6kLOv46V4GjP7QAko0FRxIpQUgDP +BAUY24JOpf1j9T/XcBIZ1nzFMUebVEKtDxZ3QHbKE/m/bDB7+h36BTsKpp/lHfXQVv6 GmxW8iDrtqqVi0VcXAO6aBbWMwwAb6hXr2tJE3pBNIxaczYssgaQaXmHSMnehmD0T9In MbelyenA6iu3l7pz8fEFlMFkPExLSWN/gjke+3if8/R7WdY2vWMLVfeC38NKGyCWOiVt Vhjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=V0gc6AUiAQY/WQln3OGyPk91MKpBrNPoI3+sFkmiGew=; b=X2491jjeVKvu7bAiEtU/GExDR4QWJZ4BAcATjL01XNodDhQo6a6PoS9WzGqFrlbDEK /ET/+RAVsAY7DAvAujz12NtX6N+I8vYIJYSJp3DF2/hVxCVspzkztO2HmHyfz4wY0hzp yhG/5B454gAgMMHvWm+PFqqGT0FQOz/27dXVrdq39m0HaCgpy8XGsa/fqbtb+UiWzfFB N7eTfja7CsUA347hHR9/IeAh9HnNgKLlK8Ok3P6lm7F/fW15aAW041lTLipaVdr3Wn6m qbjL2OSQzWqhsHRiuJjqlMdAW6upJOXEVsxKbw/OVBdtuRKJViXKsm/xgCBxIZ8CgdGA WAZQ== X-Gm-Message-State: ACgBeo3T2KNA8jzGUspSn2i6e92/ar49jVXdDqEseRXmjTcgSczGcl4a dBhn3xZaoF6U6NPDYWIZBsU= X-Google-Smtp-Source: AA6agR5uktgD0liwSY6UrfEhRG+mLXx9clJMktiplgxkJLtJtcoS4ulnUGKjv6sSBbLKMm+PhmeuJQ== X-Received: by 2002:a17:902:8e88:b0:172:d1f8:efcb with SMTP id bg8-20020a1709028e8800b00172d1f8efcbmr32528601plb.27.1662085827706; Thu, 01 Sep 2022 19:30:27 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:50ea:5400:4ff:fe1f:fbe2]) by smtp.gmail.com with ESMTPSA id j4-20020a170902da8400b0017297a6b39dsm269719plx.265.2022.09.01.19.30.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 19:30:26 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v3 08/13] bpf: Use bpf_map_kzalloc in arraymap Date: Fri, 2 Sep 2022 02:29:58 +0000 Message-Id: <20220902023003.47124-9-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220902023003.47124-1-laoar.shao@gmail.com> References: <20220902023003.47124-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662085828; a=rsa-sha256; cv=none; b=c+cSmfjtAE2UK2MZhK902p4miWqM/JoyTQL3DteDFvqRWumKYOBZuLgustNKaZF9TIVK36 8u0rJNmgtF4hzzNZ0LPyVPkp/UveAQCFaqY8a4Y7yr16iCYfbsoPpijlB6pSAdUCLBnctf b64VDwR9wlrKfTyyjXqfNsXnID3yzcc= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=YdKJPsj3; spf=pass (imf18.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662085828; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V0gc6AUiAQY/WQln3OGyPk91MKpBrNPoI3+sFkmiGew=; b=AjFz9IR6ipkZft2mJi6wGNcK7fMmvXp59YLxJPOdk9tzPxbqydji1khsjL+VwecRTn5Ea+ L9y2JlE42r2blGvPxKrY9SOfaf+wm8Fz1tf09jE+XH/mNiMOSk0yjxtqVClSOYgi8rsbjY lWJhGM3LgtNSDGodU5eG5sn3q/V/RZ8= X-Stat-Signature: itddzmrnq9zqujxoxc5iipg4z3mqor16 X-Rspam-User: X-Rspamd-Queue-Id: 996B11C0049 X-Rspamd-Server: rspam07 Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=YdKJPsj3; spf=pass (imf18.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-HE-Tag: 1662085828-83888 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allocates memory after map creation, then we can use the generic helper bpf_map_kzalloc() instead of the open-coded kzalloc(). Signed-off-by: Yafang Shao --- kernel/bpf/arraymap.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index 114fbda..f953acc 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -1096,20 +1096,20 @@ static struct bpf_map *prog_array_map_alloc(union bpf_attr *attr) struct bpf_array_aux *aux; struct bpf_map *map; - aux = kzalloc(sizeof(*aux), GFP_KERNEL_ACCOUNT); - if (!aux) + map = array_map_alloc(attr); + if (IS_ERR(map)) return ERR_PTR(-ENOMEM); + aux = bpf_map_kzalloc(map, sizeof(*aux), GFP_KERNEL); + if (!aux) { + array_map_free(map); + return ERR_PTR(-ENOMEM); + } + INIT_WORK(&aux->work, prog_array_map_clear_deferred); INIT_LIST_HEAD(&aux->poke_progs); mutex_init(&aux->poke_mutex); - map = array_map_alloc(attr); - if (IS_ERR(map)) { - kfree(aux); - return map; - } - container_of(map, struct bpf_array, map)->aux = aux; aux->map = map; From patchwork Fri Sep 2 02:29:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12963569 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 195D3ECAAD2 for ; Fri, 2 Sep 2022 02:30:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A6BE0800A0; Thu, 1 Sep 2022 22:30:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A1ABF8008D; Thu, 1 Sep 2022 22:30:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8BC0D800A0; Thu, 1 Sep 2022 22:30:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7C7DC8008D for ; Thu, 1 Sep 2022 22:30:31 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4B449C07DA for ; Fri, 2 Sep 2022 02:30:31 +0000 (UTC) X-FDA: 79865566662.04.731C392 Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) by imf01.hostedemail.com (Postfix) with ESMTP id D5B134004D for ; Fri, 2 Sep 2022 02:30:30 +0000 (UTC) Received: by mail-pg1-f182.google.com with SMTP id q63so751901pga.9 for ; Thu, 01 Sep 2022 19:30:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=Y3yvFjkfqipQf2Go42LgC412HcIZU4AC9eg6yB/qVpc=; b=CypXU69T4bEXY+gyJWc7edwtpEaxlWD1Ugkcbwbvka6ieQ5OsoQOHYfhs+YODiohDf pNAR5QbYm6Zshluujo9IVXCkqVCjbXi44Xe0kptUT2lh7dy4LacuWeHyDLh9IvlhSymK taagPSh3KOQuxrfcWbeZNborlIBHsmdJBlsL2r0gmjjig0PrP0KsOpnrKnG/RcqbgF57 kVlCz2rIVCZ5qsHx5yTbDgRpZa4+y45I6Adih3Pyju1d2Cj+o9e1zA1y324ULU+zZpRt 4LrQju+8fICS/me/SAomVEpSeA7fGxqZ/gSEB+v/Y4U0cWI3zfWxEtAdiLZ5RKbbhEEr 8Jhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=Y3yvFjkfqipQf2Go42LgC412HcIZU4AC9eg6yB/qVpc=; b=KK7FS8LCcasWbFE1r9C6N1v5gnyJwTm0aiPA4p28gL7W7EzDRW1wcQtIeLpY83T4BS Uk+R3vy5OrM44PF9R/V/v5nc1KB8PtlaLIJdTbGZpBXcPbWRBhxN8UKSkto13BnQKoyR 2kW+eVf9ABVGiKiSaYXgkB9RDrXCo2r7YpVMIdVwoncTympm4ito5E56QR1XBtFl7k4B Nuhbq3AQ29trtfJfg9PVafebAEjdVLJ2i5XDuRH5Fdl0+5pIajXclTtkkyxMwPPQpBgb JQYuuzj/HfSl/qlMIAgO1S7HaMi4sfIQXdraNskAz70RoKdC5GXkAF4Tm574sB4Lgi9F 7XBQ== X-Gm-Message-State: ACgBeo0ZbHSl3O4GAsQhGSrykLHqBnAlqgtFImxr/Zpi4y2hca5tlJ7E Iif3KS8YWcxHnYsYcI4iDwQ= X-Google-Smtp-Source: AA6agR4wYf+PukhsNAMGFdVQd31/KYbkBHPDW2ZE/BYHzB0NpNVJUfvxe0otbZctReSshoToCfvzmA== X-Received: by 2002:a63:485a:0:b0:41d:ed37:d937 with SMTP id x26-20020a63485a000000b0041ded37d937mr29497744pgk.336.1662085829772; Thu, 01 Sep 2022 19:30:29 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:50ea:5400:4ff:fe1f:fbe2]) by smtp.gmail.com with ESMTPSA id j4-20020a170902da8400b0017297a6b39dsm269719plx.265.2022.09.01.19.30.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 19:30:28 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v3 09/13] bpf: Use bpf_map_kvcalloc in bpf_local_storage Date: Fri, 2 Sep 2022 02:29:59 +0000 Message-Id: <20220902023003.47124-10-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220902023003.47124-1-laoar.shao@gmail.com> References: <20220902023003.47124-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662085830; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Y3yvFjkfqipQf2Go42LgC412HcIZU4AC9eg6yB/qVpc=; b=O3wtkbN9akKOWXWdSvT8jh9d+xle9chEwEsnVfFNqu5Lpf93fMBIrju8y9E4SHzyP2HCzZ pmjyPdRgyvNCjQ3hWhb2Olb2Au37H5SvQoobqWADYEIz8YLaC4YYL9uzAiLuQj2rFNdyEh t24uaJFlRv6VmogKFadMFn1o4MeDMFE= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=CypXU69T; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.215.182 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662085830; a=rsa-sha256; cv=none; b=64OJzbUq9SLMkunPNajs1m4NZl8mXYYZ8DsT4ub/b3H0kz1TzQcpdqcvNDXXr8ff+U3WLL EK0/E5Ptwfch+4AkiZZrfQH5uidVvZICS9WszWPqTFJI4XcSngfxqJekEwev2QArdCYwqy od/FUwpULZUO4oko5LIHD+lCB1JsSw4= Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=CypXU69T; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.215.182 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com X-Rspamd-Queue-Id: D5B134004D X-Stat-Signature: 3epoq7aqiy9g8mq4mjux4rcq353kiuzu X-Rspam-User: X-Rspamd-Server: rspam02 X-HE-Tag: 1662085830-623484 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce new helper bpf_map_kvcalloc() for this memory allocation. Signed-off-by: Yafang Shao --- include/linux/bpf.h | 8 ++++++++ kernel/bpf/bpf_local_storage.c | 4 ++-- kernel/bpf/syscall.c | 15 +++++++++++++++ 3 files changed, 25 insertions(+), 2 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index e8ac29f..52d8df0 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1657,6 +1657,8 @@ int generic_map_delete_batch(struct bpf_map *map, void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, int node); void *bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags); +void *bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size, + gfp_t flags); void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, size_t align, gfp_t flags); #else @@ -1673,6 +1675,12 @@ void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, return kzalloc(size, flags); } +static inline void * +bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size, gfp_t flags) +{ + return kvcalloc(n, size, flags); +} + static inline void __percpu * bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, size_t align, gfp_t flags) diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index 44498d7d..8a24828 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -620,8 +620,8 @@ struct bpf_local_storage_map *bpf_local_storage_map_alloc(union bpf_attr *attr) nbuckets = max_t(u32, 2, nbuckets); smap->bucket_log = ilog2(nbuckets); - smap->buckets = kvcalloc(sizeof(*smap->buckets), nbuckets, - GFP_USER | __GFP_NOWARN | __GFP_ACCOUNT); + smap->buckets = bpf_map_kvcalloc(&smap->map, sizeof(*smap->buckets), + nbuckets, GFP_USER | __GFP_NOWARN); if (!smap->buckets) { bpf_map_area_free(smap, &smap->map); return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index eefe590..034accd 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -489,6 +489,21 @@ void *bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags) return ptr; } +void *bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size, + gfp_t flags) +{ + struct mem_cgroup *memcg, *old_memcg; + void *ptr; + + memcg = bpf_map_get_memcg(map); + old_memcg = set_active_memcg(memcg); + ptr = kvcalloc(n, size, flags | __GFP_ACCOUNT); + set_active_memcg(old_memcg); + bpf_map_put_memcg(memcg); + + return ptr; +} + void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, size_t align, gfp_t flags) { From patchwork Fri Sep 2 02:30:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12963570 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16758C6FA82 for ; Fri, 2 Sep 2022 02:30:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9F65F800A1; Thu, 1 Sep 2022 22:30:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9A6148008D; Thu, 1 Sep 2022 22:30:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7FA66800A1; Thu, 1 Sep 2022 22:30:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 6A0BE8008D for ; Thu, 1 Sep 2022 22:30:33 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 377F61A1167 for ; Fri, 2 Sep 2022 02:30:33 +0000 (UTC) X-FDA: 79865566746.14.2B611C2 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) by imf03.hostedemail.com (Postfix) with ESMTP id E57EA2004D for ; Fri, 2 Sep 2022 02:30:32 +0000 (UTC) Received: by mail-pg1-f173.google.com with SMTP id 78so731409pgb.13 for ; Thu, 01 Sep 2022 19:30:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=oTzSf85iWJHNEVVdQJQawTYzDnCEuMbWuvl4YMwFJHo=; b=HpH5nbR7f38apaxWi0KaD3SeNAKzaoXicey2rg/miPioqCgeCWVOK43hCtIftBuo1N QbqGHEF6XUCu0BCMIRlFwsthhpD00GqnpDf3e3iIVAx6t7zqW0H7lguDqeHmf9F9kjxZ S6VeDMuodddj2nFJ9QdZANE0ynOkPTs8CSWtXQtVPUlS/WeUrrRH8JcbisME3QJuXhjN PBriHpMPWSoNwoECmcNJcBgaMUza/8shHez4l6SJ+A6QdqVQeyaccN2u0IE0CWhjJnBn +56WGiPw4jZ5f7N+AU/OOQCp50Ok1xUJm48Hr0097jDlSOabNoWqfp7K4Ct7NsN5+B+u K5ZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=oTzSf85iWJHNEVVdQJQawTYzDnCEuMbWuvl4YMwFJHo=; b=o1BqWuSj764FxuRiSNGuqPtE07/J+bJqE2IgysoX8UXateOWAr69vj1aUCZcMjBGkR g1OEA6ekECucZLZGhixvMc1PaTTOq7R8iKPrvaXAOEI4X4YFG6kCg9pE296imJ4qG0Jn 7o6I5j+DdAnUug8EA5ON3UDFrrP8Ul9VV0nyXioo3fKcSo9H412qIFBlFm/0ZWav5w1n KG9EnIoHqC0Q9az2JUfLwzQW4n4d9TXxBy0WgLshuJC/3m7CI1J10eMPdEGeavKgASsg ocrdeB2V0k3/BL0d5WITlDxwpna15FvCIu2Qrge4HAoQEJMYmYpbY4JUAOPdZmRlcR13 9EbQ== X-Gm-Message-State: ACgBeo3LPTRuEvxcNo5IBOPfW4/uc7n1aEcNSDRkHVzICQo/ZfuHp854 EYnzJV4m/XLkS7n/csCjNxA= X-Google-Smtp-Source: AA6agR7Ygx377TAYolYTUSgnk0C2dXozijEaoUr+0SdI+Cp29mIWbr48/yrAJDnYVJ/ugfGgDTLCgQ== X-Received: by 2002:a65:6047:0:b0:42b:313e:d331 with SMTP id a7-20020a656047000000b0042b313ed331mr29312059pgp.179.1662085832022; Thu, 01 Sep 2022 19:30:32 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:50ea:5400:4ff:fe1f:fbe2]) by smtp.gmail.com with ESMTPSA id j4-20020a170902da8400b0017297a6b39dsm269719plx.265.2022.09.01.19.30.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 19:30:31 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v3 10/13] mm, memcg: Add new helper get_obj_cgroup_from_cgroup Date: Fri, 2 Sep 2022 02:30:00 +0000 Message-Id: <20220902023003.47124-11-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220902023003.47124-1-laoar.shao@gmail.com> References: <20220902023003.47124-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662085832; a=rsa-sha256; cv=none; b=pKnlmaoqYfKLqm68Z0urKhynz1lvxsPxcj3cdbm2WstPS/beL8zmL1jn+RYN4uJXsdu6vh fbJl0jJTmp8zs0k2rOPhl3k/5jFUCJhTHI5yLpOJzXGqejSFa/+A50GWpA4L/CJZfBa2F1 mKTnmRbLfdOcvGO0IgeXLix7OqorDM8= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=HpH5nbR7; spf=pass (imf03.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.215.173 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662085832; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oTzSf85iWJHNEVVdQJQawTYzDnCEuMbWuvl4YMwFJHo=; b=8K0O6/JzTn4BH4BAJeYPtfZvXpIYIbwiCuIgOtiVOVxLU0tMn6vndpzl1njT7H/2KlDAAM PMc76iZZFTeKrqdozeMbWOHYO/IWj/gNLwswef7TeQBv4S6rgWAm/iOsJJgjvZI6uS/ySZ Sf4cJ4l0lmtUbK8Uz+ew85wU6WR65fI= X-Rspam-User: X-Stat-Signature: cfteaqkka4fa9e6o9q6bjzqgwr81d8u6 X-Rspamd-Queue-Id: E57EA2004D X-Rspamd-Server: rspam06 Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=HpH5nbR7; spf=pass (imf03.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.215.173 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-HE-Tag: 1662085832-669633 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We want to open a cgroup directory and pass the fd into kernel, and then in the kernel we can charge the allocated memory into the open cgroup if it has valid memory subsystem. In the bpf subsystem, the opened cgroup will be store as a struct obj_cgroup pointer, so a new helper get_obj_cgroup_from_cgroup() is introduced. Signed-off-by: Yafang Shao --- include/linux/memcontrol.h | 1 + mm/memcontrol.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 6040b5c..7a7f252 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1734,6 +1734,7 @@ static inline void set_shrinker_bit(struct mem_cgroup *memcg, int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order); void __memcg_kmem_uncharge_page(struct page *page, int order); +struct obj_cgroup *get_obj_cgroup_from_cgroup(struct cgroup *cgrp); struct obj_cgroup *get_obj_cgroup_from_current(void); struct obj_cgroup *get_obj_cgroup_from_page(struct page *page); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index b69979c..4e3b51e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2940,6 +2940,7 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) { struct obj_cgroup *objcg = NULL; + WARN_ON_ONCE(!rcu_read_lock_held()); for (; memcg != root_mem_cgroup; memcg = parent_mem_cgroup(memcg)) { objcg = rcu_dereference(memcg->objcg); if (objcg && obj_cgroup_tryget(objcg)) @@ -2949,6 +2950,53 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) return objcg; } +static struct obj_cgroup *get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) +{ + struct obj_cgroup *objcg; + + if (memcg_kmem_bypass()) + return NULL; + + rcu_read_lock(); + objcg = __get_obj_cgroup_from_memcg(memcg); + rcu_read_unlock(); + return objcg; +} + +/** + * get_obj_cgroup_from_cgroup: Obtain a reference on given cgroup's objcg. + * @cgrp: cgroup from which objcg should be extracted. It can't be NULL. + * The memory subsystem of this cgroup must be enabled, otherwise + * -EINVAL will be returned. + * On success, the objcg will be returned. + * On failure, -EINVAL will be returned. + */ +struct obj_cgroup *get_obj_cgroup_from_cgroup(struct cgroup *cgrp) +{ + struct cgroup_subsys_state *css; + struct mem_cgroup *memcg; + struct obj_cgroup *objcg; + + rcu_read_lock(); + css = rcu_dereference(cgrp->subsys[memory_cgrp_id]); + if (!css || !css_tryget(css)) { + rcu_read_unlock(); + return ERR_PTR(-EINVAL); + } + rcu_read_unlock(); + + memcg = mem_cgroup_from_css(css); + if (!memcg) { + css_put(css); + return ERR_PTR(-EINVAL); + } + + objcg = get_obj_cgroup_from_memcg(memcg); + css_put(css); + + return objcg; +} + __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) { struct obj_cgroup *objcg = NULL; From patchwork Fri Sep 2 02:30:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12963571 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20A43C6FA83 for ; Fri, 2 Sep 2022 02:30:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AE33E800A2; Thu, 1 Sep 2022 22:30:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A6B848008D; Thu, 1 Sep 2022 22:30:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 86DF7800A2; Thu, 1 Sep 2022 22:30:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7506B8008D for ; Thu, 1 Sep 2022 22:30:35 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 5880E1405D3 for ; Fri, 2 Sep 2022 02:30:35 +0000 (UTC) X-FDA: 79865566830.12.9D417B0 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by imf14.hostedemail.com (Postfix) with ESMTP id 2421B10002B for ; Fri, 2 Sep 2022 02:30:34 +0000 (UTC) Received: by mail-pj1-f44.google.com with SMTP id q3so722117pjg.3 for ; Thu, 01 Sep 2022 19:30:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=eAg0PZWl4zJUTIoBr/3Gv3urgLIcQ4VDCzv1IFZNeuY=; b=Gy+cv6XxxiYO3hr3Muw+hBZMy0efagnoQa+lkQO0z4mGYIW5CA1XBEpuraxfeAkfxK G8jIecWEoNFtvzfvGj7MUoaQaJt6vAiOKaeeaE0MPYK18/zDhN7V4FvEnghuG12f7i8k Wygblna21VTeaQ1/2AbJRacHSL9o97+r1sF9tnGeMEcckZfF2mRzApylow2eQjSffdB/ hmzgT9VOAVh2e1R9frgjc8IuTH7PdBnGqPgwVc5a7Y0hMwIth/hwYmEVonEu1nG6Pnij zqNPwiEXu0QWL90ZfnVnEaOkufmAIS5nPaMLhxm/P9a6FaAONxHMU+n2OSgyMVpdr9Sa 6ayQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=eAg0PZWl4zJUTIoBr/3Gv3urgLIcQ4VDCzv1IFZNeuY=; b=BqECOMosXRjpnSuqsURU/r6JYvLIhgrFrdyggzSkEjf3T5oW3PRtIrcn6Fvd1Z5kOm czZxGqOiujQgE0f0Mi4Q3T1ORZKiPV0DJX9iIYSc7L9dMXWEGbEoSUkQmZYhC7/BXoD2 m2QdNWF+L8h5i0xTkNemqJWlbSb8aaYpkaqs3zsMLFukbO9IIsjTQHgF5k7a85gyq+ov YVWJXhpv5dVmRAMnUNRbJxdYrplkBiCqZ/SKPRBDd57eCUiLEefiUX/8So/NoMCkpc9Q sEzQcGAKUGjwQSWextE9s8XH6HEkHS00K7eU4A7pkPUAVmKHKzjAxarxMxu66PQtOj0k GZdQ== X-Gm-Message-State: ACgBeo13YTKb5Vm+ptCvvUQRTyRD0E8+g6rRgUPv7Sx2CFQscpIa/3W0 puMPK/Z4O8roKWtQK5xyqpI= X-Google-Smtp-Source: AA6agR4DsBfz/UK1tZcnfi/DCBBeE+JJsSZfcGsqqa8xiOuQfkFBNrN6XShqRSgVYStdq2UPWrbWLQ== X-Received: by 2002:a17:903:2284:b0:174:8681:3f6e with SMTP id b4-20020a170903228400b0017486813f6emr25252605plh.5.1662085834206; Thu, 01 Sep 2022 19:30:34 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:50ea:5400:4ff:fe1f:fbe2]) by smtp.gmail.com with ESMTPSA id j4-20020a170902da8400b0017297a6b39dsm269719plx.265.2022.09.01.19.30.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 19:30:33 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v3 11/13] mm, memcg: Add new helper task_under_memcg_hierarchy Date: Fri, 2 Sep 2022 02:30:01 +0000 Message-Id: <20220902023003.47124-12-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220902023003.47124-1-laoar.shao@gmail.com> References: <20220902023003.47124-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662085835; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eAg0PZWl4zJUTIoBr/3Gv3urgLIcQ4VDCzv1IFZNeuY=; b=Ec7D0aqBEJMxAAPcynr+6N0/B35oNU0Hfdp+proLxnrA7M7p0r1VZwRZ15pFnAxCZG4+Qq 9IIQrq/zLAFJCzm5Vg+Zw5xTb+0tSLvRo0ASYN8vi+L9bw+wzDY9s3IR5oEWM8J0UH/ZVV UCWWSLBbl2VvOqqMq4Q3Y3LxO1bw27E= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Gy+cv6Xx; spf=pass (imf14.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662085835; a=rsa-sha256; cv=none; b=lUqXIRJOEyWIh5GPoLBvY1hc13ZCJdQm+9y0zbsa+CYc7D72doa0ib7bxi9chcBWTlZVg1 bQAws8e0GEQlfoerfj/Pvs0lGwJQuXq1mv9AhE7U7J+zHiB19yGTY+ZIm9KkYcwazHZs+E VFg5Ji4Y9bBoj8mUBnLzssECsAVuhZI= X-Stat-Signature: zprsfw61zcycfnx1hyphtro6b5fj6pj1 X-Rspamd-Queue-Id: 2421B10002B X-Rspam-User: Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Gy+cv6Xx; spf=pass (imf14.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam01 X-HE-Tag: 1662085834-428051 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce a new helper to check if a task belongs to a specific memcg. It is similar to mm_match_cgroup() except that the new helper is checked against a task rather than a mm struct. So with this new helper we can check a task directly. Signed-off-by: Yafang Shao --- include/linux/memcontrol.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 7a7f252..3b8a8dd 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -887,6 +887,20 @@ static inline bool mem_cgroup_is_descendant(struct mem_cgroup *memcg, return cgroup_is_descendant(memcg->css.cgroup, root->css.cgroup); } +static inline bool task_under_memcg_hierarchy(struct task_struct *p, + struct mem_cgroup *memcg) +{ + struct mem_cgroup *task_memcg; + bool match = false; + + rcu_read_lock(); + task_memcg = mem_cgroup_from_task(p); + if (task_memcg) + match = mem_cgroup_is_descendant(task_memcg, memcg); + rcu_read_unlock(); + return match; +} + static inline bool mm_match_cgroup(struct mm_struct *mm, struct mem_cgroup *memcg) { From patchwork Fri Sep 2 02:30:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12963572 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40591C6FA82 for ; Fri, 2 Sep 2022 02:30:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C8F33800A3; Thu, 1 Sep 2022 22:30:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C3E1F8008D; Thu, 1 Sep 2022 22:30:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A90E0800A3; Thu, 1 Sep 2022 22:30:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9A2BD8008D for ; Thu, 1 Sep 2022 22:30:37 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 71E3BA118A for ; Fri, 2 Sep 2022 02:30:37 +0000 (UTC) X-FDA: 79865566914.15.656FB5C Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by imf31.hostedemail.com (Postfix) with ESMTP id 2D10420060 for ; Fri, 2 Sep 2022 02:30:37 +0000 (UTC) Received: by mail-pf1-f173.google.com with SMTP id q15so562403pfn.11 for ; Thu, 01 Sep 2022 19:30:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=id7LbOxlALrRnZ+Ut5/GrJgo6c5SmgE89mpDoPOy7aI=; b=b/FhSzr48XdH8ITEzsmU7pO0Iea4baMm/IeFrWlaRiC7Uid/oALkZydAJq39A2Lzng 4Vu/2/Mk2X8aTl2eKYhu8QyakFkSW+R0lUl+EAIG26UKKOGRkSiNb+reBU1x4q10cLPc 1b67Ch8msWNBnRzmMPGGcOSIRET43rQVgSsVds6oadnMn4nRZoMAjlNvp65IZyozr3WM lwPjX+z3B8tF12T/qJS8yozoWS7xB2F561q8E8024pRzhnDPBcNmJTy1EiiyoWawvvqG ZDYZ4N8aW5RqnjiCI4wa9q0KIiznttD0/Ho8TuJ187TVBI5V6WIEICfBYYwr/W/nBl9/ 8DOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=id7LbOxlALrRnZ+Ut5/GrJgo6c5SmgE89mpDoPOy7aI=; b=6Z+F2MsmC9/B4zaYh6zb4VOE+9Fn4WVULYKXv4KmOvtpVKcid5npueUcEB2zqrNWmY 4z3uJ0VtXxKxtmsxdmwXHpWvdj8DN6GAjV3YO4cZ/SQCmwF7B+49aSnLdasAndFvgGnO qGk9yBPH3wcqck881ucyNVale9m3IZi9p5PDqJhCAqhGpglV0Ixzs7O4TkuNjqQmQI9/ rW8N+mT24YyZdvmwh5zBKUG8snNc18mJ44VucHiq/9upTWDSw3TxFphT80Z4AUT7/rR3 ZWCkVNpKoB0f7xbTLAaEHkLV+pW/OfI0MFM0oM4wXu32FisusR4u+zlsB0endqwjRa78 NhmA== X-Gm-Message-State: ACgBeo1JIlgJHKUrwajBe3eydWqdq3wBnJGQQuozBxFbaVcoRLaXFA2g JqDR09x8H4epAEohFZ6+0Zw= X-Google-Smtp-Source: AA6agR7VXJjYao0ws/HPQpOTTWWHuyvB83zybImvWsY/2TThBtrRoJE1wX6a8BX530eRmwDhsfEjLw== X-Received: by 2002:a65:450d:0:b0:42b:9117:ba5b with SMTP id n13-20020a65450d000000b0042b9117ba5bmr25581878pgq.418.1662085836290; Thu, 01 Sep 2022 19:30:36 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:50ea:5400:4ff:fe1f:fbe2]) by smtp.gmail.com with ESMTPSA id j4-20020a170902da8400b0017297a6b39dsm269719plx.265.2022.09.01.19.30.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 19:30:35 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v3 12/13] bpf: Add return value for bpf_map_init_from_attr Date: Fri, 2 Sep 2022 02:30:02 +0000 Message-Id: <20220902023003.47124-13-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220902023003.47124-1-laoar.shao@gmail.com> References: <20220902023003.47124-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662085837; a=rsa-sha256; cv=none; b=296Par8oOhcbqKYeyPcof015HPlcJ8W57+A8q0nXZOwTGZJ8UxUGRfMyaMxt7zVpoc8u2G caJ3QExCmmyTUaO2cBje/3hzDKOpSVyXKqqA6mB3wif/aZROcmWmiy8rJZs5hf0PDMPlQr ja21JNRt48yi3an+SM3r9607P8KyiIA= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="b/FhSzr4"; spf=pass (imf31.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.210.173 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662085837; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=id7LbOxlALrRnZ+Ut5/GrJgo6c5SmgE89mpDoPOy7aI=; b=T00h6rV7CurRjvq3NRTPV5dwIOHf6Wm2aL/ZFZdlhfi8r2dUiiiQVCuVE/ejU/qZKyAt+g bYQautSitJdYVQ1cdzMp9wduDYyDn3P2PVe09hY+cqkeJBC4JkJgeOJLJ1vAAWpxIlXXRL PpbWWs6oS3sRiTgq+NARD0iF6rqQTDE= Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="b/FhSzr4"; spf=pass (imf31.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.210.173 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: wxgu1sbq369tcsqpppxhqwjmbty6oe54 X-Rspamd-Queue-Id: 2D10420060 X-Rspamd-Server: rspam04 X-Rspam-User: X-HE-Tag: 1662085837-688854 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add return value for bpf_map_init_from_attr() to indicate whether it init successfully. This is a preparation of the followup patch. Signed-off-by: Yafang Shao --- include/linux/bpf.h | 2 +- kernel/bpf/arraymap.c | 8 +++++++- kernel/bpf/bloom_filter.c | 7 ++++++- kernel/bpf/bpf_local_storage.c | 7 ++++++- kernel/bpf/bpf_struct_ops.c | 7 ++++++- kernel/bpf/cpumap.c | 6 +++++- kernel/bpf/devmap.c | 6 +++++- kernel/bpf/hashtab.c | 6 +++++- kernel/bpf/local_storage.c | 7 ++++++- kernel/bpf/lpm_trie.c | 8 +++++++- kernel/bpf/offload.c | 6 +++++- kernel/bpf/queue_stack_maps.c | 7 ++++++- kernel/bpf/reuseport_array.c | 7 ++++++- kernel/bpf/ringbuf.c | 7 ++++++- kernel/bpf/syscall.c | 4 +++- net/core/sock_map.c | 8 +++++++- net/xdp/xskmap.c | 8 +++++++- 17 files changed, 94 insertions(+), 17 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 52d8df0..2dd520e 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1640,7 +1640,7 @@ struct bpf_prog *bpf_prog_get_type_dev(u32 ufd, enum bpf_prog_type type, void *bpf_map_area_mmapable_alloc(u64 size, int numa_node); void bpf_map_area_free(void *base, struct bpf_map *map); bool bpf_map_write_active(const struct bpf_map *map); -void bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr); +int bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr); int generic_map_lookup_batch(struct bpf_map *map, const union bpf_attr *attr, union bpf_attr __user *uattr); diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index f953acc..9f08694 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -85,6 +85,7 @@ static struct bpf_map *array_map_alloc(union bpf_attr *attr) bool bypass_spec_v1 = bpf_bypass_spec_v1(); u64 array_size, mask64; struct bpf_array *array; + int err; elem_size = round_up(attr->value_size, 8); @@ -143,7 +144,12 @@ static struct bpf_map *array_map_alloc(union bpf_attr *attr) array->map.bypass_spec_v1 = bypass_spec_v1; /* copy mandatory map attributes */ - bpf_map_init_from_attr(&array->map, attr); + err = bpf_map_init_from_attr(&array->map, attr); + if (err) { + bpf_map_area_free(array, NULL); + return ERR_PTR(err); + } + array->elem_size = elem_size; if (percpu && bpf_array_alloc_percpu(array)) { diff --git a/kernel/bpf/bloom_filter.c b/kernel/bpf/bloom_filter.c index 6691f79..be64227 100644 --- a/kernel/bpf/bloom_filter.c +++ b/kernel/bpf/bloom_filter.c @@ -93,6 +93,7 @@ static struct bpf_map *bloom_map_alloc(union bpf_attr *attr) u32 bitset_bytes, bitset_mask, nr_hash_funcs, nr_bits; int numa_node = bpf_map_attr_numa_node(attr); struct bpf_bloom_filter *bloom; + int err; if (!bpf_capable()) return ERR_PTR(-EPERM); @@ -147,7 +148,11 @@ static struct bpf_map *bloom_map_alloc(union bpf_attr *attr) if (!bloom) return ERR_PTR(-ENOMEM); - bpf_map_init_from_attr(&bloom->map, attr); + err = bpf_map_init_from_attr(&bloom->map, attr); + if (err) { + bpf_map_area_free(bloom, NULL); + return ERR_PTR(err); + } bloom->nr_hash_funcs = nr_hash_funcs; bloom->bitset_mask = bitset_mask; diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index 8a24828..ab7fd6b 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -609,11 +609,16 @@ struct bpf_local_storage_map *bpf_local_storage_map_alloc(union bpf_attr *attr) struct bpf_local_storage_map *smap; unsigned int i; u32 nbuckets; + int err; smap = bpf_map_area_alloc(sizeof(*smap), NUMA_NO_NODE, NULL); if (!smap) return ERR_PTR(-ENOMEM); - bpf_map_init_from_attr(&smap->map, attr); + err = bpf_map_init_from_attr(&smap->map, attr); + if (err) { + bpf_map_area_free(&smap->map, NULL); + return ERR_PTR(err); + } nbuckets = roundup_pow_of_two(num_possible_cpus()); /* Use at least 2 buckets, select_bucket() is undefined behavior with 1 bucket */ diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index 37ba5c0..7cfbabc 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -598,6 +598,7 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) struct bpf_struct_ops_map *st_map; const struct btf_type *t, *vt; struct bpf_map *map; + int err; if (!bpf_capable()) return ERR_PTR(-EPERM); @@ -624,7 +625,11 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) st_map->st_ops = st_ops; map = &st_map->map; - bpf_map_init_from_attr(map, attr); + err = bpf_map_init_from_attr(map, attr); + if (err) { + bpf_map_area_free(st_map, NULL); + return ERR_PTR(err); + } st_map->uvalue = bpf_map_area_alloc(vt->size, NUMA_NO_NODE, map); st_map->links = diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index b593157..e672e62 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -101,7 +101,11 @@ static struct bpf_map *cpu_map_alloc(union bpf_attr *attr) if (!cmap) return ERR_PTR(-ENOMEM); - bpf_map_init_from_attr(&cmap->map, attr); + err = bpf_map_init_from_attr(&cmap->map, attr); + if (err) { + bpf_map_area_free(cmap, NULL); + return ERR_PTR(err); + } /* Pre-limit array size based on NR_CPUS, not final CPU check */ if (cmap->map.max_entries > NR_CPUS) { diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 807a4cd..10f038d 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -167,7 +167,11 @@ static struct bpf_map *dev_map_alloc(union bpf_attr *attr) if (!dtab) return ERR_PTR(-ENOMEM); - bpf_map_init_from_attr(&dtab->map, attr); + err = bpf_map_init_from_attr(&dtab->map, attr); + if (err) { + bpf_map_area_free(dtab, NULL); + return ERR_PTR(err); + } err = dev_map_init_map(dtab, attr); if (err) { diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 975bcc1..705ffdd 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -509,7 +509,11 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) if (!htab) return ERR_PTR(-ENOMEM); - bpf_map_init_from_attr(&htab->map, attr); + err = bpf_map_init_from_attr(&htab->map, attr); + if (err) { + bpf_map_area_free(htab, NULL); + return ERR_PTR(err); + } lockdep_register_key(&htab->lockdep_key); diff --git a/kernel/bpf/local_storage.c b/kernel/bpf/local_storage.c index fcc7ece..1901195 100644 --- a/kernel/bpf/local_storage.c +++ b/kernel/bpf/local_storage.c @@ -287,6 +287,7 @@ static struct bpf_map *cgroup_storage_map_alloc(union bpf_attr *attr) __u32 max_value_size = BPF_LOCAL_STORAGE_MAX_VALUE_SIZE; int numa_node = bpf_map_attr_numa_node(attr); struct bpf_cgroup_storage_map *map; + int err; /* percpu is bound by PCPU_MIN_UNIT_SIZE, non-percu * is the same as other local storages. @@ -318,7 +319,11 @@ static struct bpf_map *cgroup_storage_map_alloc(union bpf_attr *attr) return ERR_PTR(-ENOMEM); /* copy mandatory map attributes */ - bpf_map_init_from_attr(&map->map, attr); + err = bpf_map_init_from_attr(&map->map, attr); + if (err) { + bpf_map_area_free(map, NULL); + return ERR_PTR(err); + } spin_lock_init(&map->lock); map->root = RB_ROOT; diff --git a/kernel/bpf/lpm_trie.c b/kernel/bpf/lpm_trie.c index 3d329ae..38d7b00 100644 --- a/kernel/bpf/lpm_trie.c +++ b/kernel/bpf/lpm_trie.c @@ -543,6 +543,7 @@ static int trie_delete_elem(struct bpf_map *map, void *_key) static struct bpf_map *trie_alloc(union bpf_attr *attr) { struct lpm_trie *trie; + int err; if (!bpf_capable()) return ERR_PTR(-EPERM); @@ -563,7 +564,12 @@ static struct bpf_map *trie_alloc(union bpf_attr *attr) return ERR_PTR(-ENOMEM); /* copy mandatory map attributes */ - bpf_map_init_from_attr(&trie->map, attr); + err = bpf_map_init_from_attr(&trie->map, attr); + if (err) { + bpf_map_area_free(trie, NULL); + return ERR_PTR(err); + } + trie->data_size = attr->key_size - offsetof(struct bpf_lpm_trie_key, data); trie->max_prefixlen = trie->data_size * 8; diff --git a/kernel/bpf/offload.c b/kernel/bpf/offload.c index 87c59da..dba7ed2 100644 --- a/kernel/bpf/offload.c +++ b/kernel/bpf/offload.c @@ -376,7 +376,11 @@ struct bpf_map *bpf_map_offload_map_alloc(union bpf_attr *attr) if (!offmap) return ERR_PTR(-ENOMEM); - bpf_map_init_from_attr(&offmap->map, attr); + err = bpf_map_init_from_attr(&offmap->map, attr); + if (err) { + bpf_map_area_free(offmap, NULL); + return ERR_PTR(err); + } rtnl_lock(); down_write(&bpf_devs_lock); diff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c index bf57e45..f231897 100644 --- a/kernel/bpf/queue_stack_maps.c +++ b/kernel/bpf/queue_stack_maps.c @@ -70,6 +70,7 @@ static struct bpf_map *queue_stack_map_alloc(union bpf_attr *attr) int numa_node = bpf_map_attr_numa_node(attr); struct bpf_queue_stack *qs; u64 size, queue_size; + int err; size = (u64) attr->max_entries + 1; queue_size = sizeof(*qs) + size * attr->value_size; @@ -78,7 +79,11 @@ static struct bpf_map *queue_stack_map_alloc(union bpf_attr *attr) if (!qs) return ERR_PTR(-ENOMEM); - bpf_map_init_from_attr(&qs->map, attr); + err = bpf_map_init_from_attr(&qs->map, attr); + if (err) { + bpf_map_area_free(qs, NULL); + return ERR_PTR(err); + } qs->size = size; diff --git a/kernel/bpf/reuseport_array.c b/kernel/bpf/reuseport_array.c index 447f664..f9604f3 100644 --- a/kernel/bpf/reuseport_array.c +++ b/kernel/bpf/reuseport_array.c @@ -150,6 +150,7 @@ static struct bpf_map *reuseport_array_alloc(union bpf_attr *attr) { int numa_node = bpf_map_attr_numa_node(attr); struct reuseport_array *array; + int err; if (!bpf_capable()) return ERR_PTR(-EPERM); @@ -160,7 +161,11 @@ static struct bpf_map *reuseport_array_alloc(union bpf_attr *attr) return ERR_PTR(-ENOMEM); /* copy mandatory map attributes */ - bpf_map_init_from_attr(&array->map, attr); + err = bpf_map_init_from_attr(&array->map, attr); + if (err) { + bpf_map_area_free(array, NULL); + return ERR_PTR(err); + } return &array->map; } diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index 1e7284c..3edefd3 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -185,6 +185,7 @@ static struct bpf_ringbuf *bpf_ringbuf_alloc(size_t data_sz, int numa_node, static struct bpf_map *ringbuf_map_alloc(union bpf_attr *attr) { struct bpf_ringbuf_map *rb_map; + int err; if (attr->map_flags & ~RINGBUF_CREATE_FLAG_MASK) return ERR_PTR(-EINVAL); @@ -204,7 +205,11 @@ static struct bpf_map *ringbuf_map_alloc(union bpf_attr *attr) if (!rb_map) return ERR_PTR(-ENOMEM); - bpf_map_init_from_attr(&rb_map->map, attr); + err = bpf_map_init_from_attr(&rb_map->map, attr); + if (err) { + bpf_map_area_free(rb_map, NULL); + return ERR_PTR(err); + } rb_map->rb = bpf_ringbuf_alloc(attr->max_entries, rb_map->map.numa_node, &rb_map->map); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 034accd..f710495 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -403,7 +403,7 @@ static u32 bpf_map_flags_retain_permanent(u32 flags) return flags & ~(BPF_F_RDONLY | BPF_F_WRONLY); } -void bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr) +int bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr) { bpf_map_save_memcg(map); map->map_type = attr->map_type; @@ -413,6 +413,8 @@ void bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr) map->map_flags = bpf_map_flags_retain_permanent(attr->map_flags); map->numa_node = bpf_map_attr_numa_node(attr); map->map_extra = attr->map_extra; + + return 0; } static int bpf_map_alloc_id(struct bpf_map *map) diff --git a/net/core/sock_map.c b/net/core/sock_map.c index 25a5ac4..67c502d 100644 --- a/net/core/sock_map.c +++ b/net/core/sock_map.c @@ -31,6 +31,7 @@ static int sock_map_prog_update(struct bpf_map *map, struct bpf_prog *prog, static struct bpf_map *sock_map_alloc(union bpf_attr *attr) { struct bpf_stab *stab; + int err; if (!capable(CAP_NET_ADMIN)) return ERR_PTR(-EPERM); @@ -45,7 +46,12 @@ static struct bpf_map *sock_map_alloc(union bpf_attr *attr) if (!stab) return ERR_PTR(-ENOMEM); - bpf_map_init_from_attr(&stab->map, attr); + err = bpf_map_init_from_attr(&stab->map, attr); + if (err) { + bpf_map_area_free(stab, NULL); + return ERR_PTR(err); + } + raw_spin_lock_init(&stab->lock); stab->sks = bpf_map_area_alloc((u64) stab->map.max_entries * diff --git a/net/xdp/xskmap.c b/net/xdp/xskmap.c index beb11fd..8e7a5a6 100644 --- a/net/xdp/xskmap.c +++ b/net/xdp/xskmap.c @@ -63,6 +63,7 @@ static struct bpf_map *xsk_map_alloc(union bpf_attr *attr) struct xsk_map *m; int numa_node; u64 size; + int err; if (!capable(CAP_NET_ADMIN)) return ERR_PTR(-EPERM); @@ -79,7 +80,12 @@ static struct bpf_map *xsk_map_alloc(union bpf_attr *attr) if (!m) return ERR_PTR(-ENOMEM); - bpf_map_init_from_attr(&m->map, attr); + err = bpf_map_init_from_attr(&m->map, attr); + if (err) { + bpf_map_area_free(m, NULL); + return ERR_PTR(err); + } + spin_lock_init(&m->lock); return &m->map; From patchwork Fri Sep 2 02:30:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12963573 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B64EC6FA85 for ; Fri, 2 Sep 2022 02:30:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C440B800A4; Thu, 1 Sep 2022 22:30:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BC8DA8008D; Thu, 1 Sep 2022 22:30:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9F445800A4; Thu, 1 Sep 2022 22:30:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8FF1A8008D for ; Thu, 1 Sep 2022 22:30:39 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 6A311141106 for ; Fri, 2 Sep 2022 02:30:39 +0000 (UTC) X-FDA: 79865566998.17.93B87A3 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) by imf16.hostedemail.com (Postfix) with ESMTP id 28CC118004F for ; Fri, 2 Sep 2022 02:30:38 +0000 (UTC) Received: by mail-pg1-f173.google.com with SMTP id 202so752266pgc.8 for ; Thu, 01 Sep 2022 19:30:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=LbURm8rqcOZV+rHHC9kc9kgsqQi9dVlL3oSCkwKpjVM=; b=VNb++7b2qt+LPgyZo5BPHw2GfJzb+MvkQEc6QA+/NiK8AE7Hg+ftvUL+ETOuo0W3Ej +rD8w4PJwMrJ1OiYNDt6QrnnBmguqiQ6LjLHQKSkYfEGDY06Bz/q4kKkC4ITYlWFYxz6 +aH5cQv+GhAg3bnpb6Ie3cmiYwXa63zlzrVjBWVgNxAC3DRdCCPWgEhilDYnUuDKbLFc TOkhPwnPwuDaD+Szz7l1rZp0VOmBJ6CjdNMaaTCnnLTSpX6zuprBHGbmpJgE/h+u+jbK o3zFSUQmag8TcC2baZcmOMFDnXDF6xVr9RDtJSI0ENQYBSGq4LIgIsvC03YrVhiVMn+B 97ZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=LbURm8rqcOZV+rHHC9kc9kgsqQi9dVlL3oSCkwKpjVM=; b=IojXdfaMdrZJ3GF9C79V3OJMTUm94Xin2IKFKiimGGxEp5ppkY4+bnB1IKg+9Pyd9G WRs/w+S2hXwfYrJQamU+oJlqT/wf8QLxansGdlydUKTJQMBEOVmQY0IENWTd/D28gZiI 2xuBCVQBhGrA1MXcq/dBhuVNpLF9kelxyjiS3ZQhgFCiR0Ms9YihRL1QQppEycs2gwXt VciROojnHc0VdVOqOpuj9GA+US9ChINKycQeXYMvMk2R1xFPXl1CBcu4Cbs3eQTKLZJe ZzdAw1tiUqOl2PnZHdvtZX8BsCR9cj3ykBdYmA/vGualSvg+LKghpGrkhdbUD1x4xPQ1 LhHw== X-Gm-Message-State: ACgBeo2uuSM5QbNQiF+nLVPxLD/RbOaNfPO6Rj3+sQr7uIRf54W1GLoz nzFu9nFZS5QephqIue1BE0Q= X-Google-Smtp-Source: AA6agR6ktbAX7ONXPPPVIABchY62CUO2VrWR2JdJtRAEEYTq4TduYWLRwi/JwGe+KUXb/FheV0LpXQ== X-Received: by 2002:a63:d745:0:b0:429:d263:615d with SMTP id w5-20020a63d745000000b00429d263615dmr29199167pgi.579.1662085838232; Thu, 01 Sep 2022 19:30:38 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:50ea:5400:4ff:fe1f:fbe2]) by smtp.gmail.com with ESMTPSA id j4-20020a170902da8400b0017297a6b39dsm269719plx.265.2022.09.01.19.30.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Sep 2022 19:30:37 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, tj@kernel.org, lizefan.x@bytedance.com Cc: cgroups@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Yafang Shao Subject: [PATCH bpf-next v3 13/13] bpf: Introduce selectable memcg for bpf map Date: Fri, 2 Sep 2022 02:30:03 +0000 Message-Id: <20220902023003.47124-14-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220902023003.47124-1-laoar.shao@gmail.com> References: <20220902023003.47124-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662085839; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LbURm8rqcOZV+rHHC9kc9kgsqQi9dVlL3oSCkwKpjVM=; b=hmmxmbsSAYJktufjdaGU5OLsgUF8vPzoBrvTehF8qWRfP35z061ghtmlFkkCQIUJnJXGiB x9xZVodCUGn64jSbec1wPs9QSuSmnYOx/3y1p8Ih+RvSc2BR18F8z/+4wC6hR9E7co6c2T SrcQaLa/4msLYj7kjoq/D/qwazKiJfY= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=VNb++7b2; spf=pass (imf16.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.215.173 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662085839; a=rsa-sha256; cv=none; b=jxmFMWWev0YLzZxxPfzZJ0/LlAF7jo+8TiwZwHqr1v+joenQfKg2uYIaxY3BMIZw6OXhm4 pHNdG5JObKSiXqJ1c75ZXxrYPrZepZQAS3+sm/AzFq9rczp9oA8ca0vaFx1o/Zn1YWaVwN LNuAPxGsMJ+FwsR6+YhPUKGqoXm+YwY= X-Stat-Signature: kttbcxripuos4ositb3a1ma9xnzbkdrc X-Rspamd-Queue-Id: 28CC118004F Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=VNb++7b2; spf=pass (imf16.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.215.173 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam12 X-Rspam-User: X-HE-Tag: 1662085838-882225 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A new member memcg_fd is introduced into bpf attr of BPF_MAP_CREATE command, which is the fd of an opened cgroup directory. In this cgroup, the memory subsystem must be enabled. The valid memcg_fd must be a postive number, that means it can't be zero(a valid return value of open(2)). Once the kernel get the memory cgroup from this fd, it will set this memcg into bpf map, then all the subsequent memory allocation of this map will be charged to the memcg. The map creation paths in libbpf are also changed consequently. Currently we only allow to select its ancestors to avoid breaking the memcg hierarchy further. For example, we can select its parent, other ancestors, or the root memcg. Possible use cases of the selectable memcg as follows, - Select the root memcg as bpf-map's memcg Then bpf-map's memory won't be throttled by current memcg limit. - Put current memcg under a fixed memcg dir and select the fixed memcg as bpf-map's memcg The hierarchy as follows, Parent-memcg (A fixed dir, i.e. /sys/fs/cgroup/memory/bpf) \ Current-memcg (Container dir, i.e. /sys/fs/cgroup/memory/bpf/foo) At the map creation time, the bpf-map's memory will be charged into the parent directly without charging into current memcg, and thus current memcg's usage will be consistent among different generations. To limit bpf-map's memory usage, we can set the limit in the parent memcg. Below is an example on how to use this new API, struct bpf_map_create_opts map_opts = { .sz = sizeof(map_opts), }; int memcg_fd, map_fd, old_fd; int key, value; memcg_fd = open("/sys/fs/cgroup/memory/bpf", O_DIRECTORY); if (memcg_fd < 0) { perror("memcg dir open"); return -1; } /* 0 is a invalid fd */ if (memcg_fd == 0) { old_fd = memcg_fd; memcg_fd = fcntl(memcg_fd, F_DUPFD_CLOEXEC, 3); close(old_fd); if (memcg_fd < 0) { perror("fcntl"); return -1; } } map_opts.memcg_fd = memcg_fd; map_fd = bpf_map_create(BPF_MAP_TYPE_HASH, "map_for_memcg", sizeof(key), sizeof(value), 1024, &map_opts); if (map_fd <= 0) { close(memcg_fd); perror("map create"); return -1; } Signed-off-by: Yafang Shao --- include/uapi/linux/bpf.h | 1 + kernel/bpf/syscall.c | 49 +++++++++++++++++++++++++++++++++--------- tools/include/uapi/linux/bpf.h | 1 + tools/lib/bpf/bpf.c | 3 ++- tools/lib/bpf/bpf.h | 3 ++- tools/lib/bpf/gen_loader.c | 2 +- tools/lib/bpf/libbpf.c | 2 ++ tools/lib/bpf/skel_internal.h | 2 +- 8 files changed, 49 insertions(+), 14 deletions(-) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 962960a..9121c4f 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1319,6 +1319,7 @@ struct bpf_stack_build_id { * to using 5 hash functions). */ __u64 map_extra; + __u32 memcg_fd; /* selectable memcg */ }; struct { /* anonymous struct used by BPF_MAP_*_ELEM commands */ diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index f710495..1b1af68 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -294,14 +294,37 @@ static int bpf_map_copy_value(struct bpf_map *map, void *key, void *value, } #ifdef CONFIG_MEMCG_KMEM -static void bpf_map_save_memcg(struct bpf_map *map) +static int bpf_map_save_memcg(struct bpf_map *map, u32 memcg_fd) { - /* Currently if a map is created by a process belonging to the root - * memory cgroup, get_obj_cgroup_from_current() will return NULL. - * So we have to check map->objcg for being NULL each time it's - * being used. - */ - map->objcg = get_obj_cgroup_from_current(); + struct obj_cgroup *objcg; + struct cgroup *cgrp; + + if (memcg_fd) { + cgrp = cgroup_get_from_fd(memcg_fd); + if (IS_ERR(cgrp)) + return -EINVAL; + + objcg = get_obj_cgroup_from_cgroup(cgrp); + cgroup_put(cgrp); + if (IS_ERR(objcg)) + return PTR_ERR(objcg); + + /* Currently we only allow to select its ancestors. */ + if (objcg && !task_under_memcg_hierarchy(current, objcg->memcg)) { + obj_cgroup_put(objcg); + return -EINVAL; + } + } else { + /* Currently if a map is created by a process belonging to the root + * memory cgroup, get_obj_cgroup_from_current() will return NULL. + * So we have to check map->objcg for being NULL each time it's + * being used. + */ + objcg = get_obj_cgroup_from_current(); + } + + map->objcg = objcg; + return 0; } static void bpf_map_release_memcg(struct bpf_map *map) @@ -311,8 +334,9 @@ static void bpf_map_release_memcg(struct bpf_map *map) } #else -static void bpf_map_save_memcg(struct bpf_map *map) +static int bpf_map_save_memcg(struct bpf_map *map, u32 memcg_fd) { + return 0; } static void bpf_map_release_memcg(struct bpf_map *map) @@ -405,7 +429,12 @@ static u32 bpf_map_flags_retain_permanent(u32 flags) int bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr) { - bpf_map_save_memcg(map); + int err; + + err = bpf_map_save_memcg(map, attr->memcg_fd); + if (err) + return err; + map->map_type = attr->map_type; map->key_size = attr->key_size; map->value_size = attr->value_size; @@ -1091,7 +1120,7 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf, return ret; } -#define BPF_MAP_CREATE_LAST_FIELD map_extra +#define BPF_MAP_CREATE_LAST_FIELD memcg_fd /* called via syscall */ static int map_create(union bpf_attr *attr) { diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index f4ba82a..fc19366 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1319,6 +1319,7 @@ struct bpf_stack_build_id { * to using 5 hash functions). */ __u64 map_extra; + __u32 memcg_fd; /* selectable memcg */ }; struct { /* anonymous struct used by BPF_MAP_*_ELEM commands */ diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c index 1d49a03..b475b28 100644 --- a/tools/lib/bpf/bpf.c +++ b/tools/lib/bpf/bpf.c @@ -169,7 +169,7 @@ int bpf_map_create(enum bpf_map_type map_type, __u32 max_entries, const struct bpf_map_create_opts *opts) { - const size_t attr_sz = offsetofend(union bpf_attr, map_extra); + const size_t attr_sz = offsetofend(union bpf_attr, memcg_fd); union bpf_attr attr; int fd; @@ -197,6 +197,7 @@ int bpf_map_create(enum bpf_map_type map_type, attr.map_extra = OPTS_GET(opts, map_extra, 0); attr.numa_node = OPTS_GET(opts, numa_node, 0); attr.map_ifindex = OPTS_GET(opts, map_ifindex, 0); + attr.memcg_fd = OPTS_GET(opts, memcg_fd, 0); fd = sys_bpf_fd(BPF_MAP_CREATE, &attr, attr_sz); return libbpf_err_errno(fd); diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h index 9c50bea..dd0d929 100644 --- a/tools/lib/bpf/bpf.h +++ b/tools/lib/bpf/bpf.h @@ -51,8 +51,9 @@ struct bpf_map_create_opts { __u32 numa_node; __u32 map_ifindex; + __u32 memcg_fd; }; -#define bpf_map_create_opts__last_field map_ifindex +#define bpf_map_create_opts__last_field memcg_fd LIBBPF_API int bpf_map_create(enum bpf_map_type map_type, const char *map_name, diff --git a/tools/lib/bpf/gen_loader.c b/tools/lib/bpf/gen_loader.c index 23f5c46..f35b014 100644 --- a/tools/lib/bpf/gen_loader.c +++ b/tools/lib/bpf/gen_loader.c @@ -451,7 +451,7 @@ void bpf_gen__map_create(struct bpf_gen *gen, __u32 key_size, __u32 value_size, __u32 max_entries, struct bpf_map_create_opts *map_attr, int map_idx) { - int attr_size = offsetofend(union bpf_attr, map_extra); + int attr_size = offsetofend(union bpf_attr, memcg_fd); bool close_inner_map_fd = false; int map_create_attr, idx; union bpf_attr attr; diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 3ad1392..ce04d93 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -512,6 +512,7 @@ struct bpf_map { bool reused; bool autocreate; __u64 map_extra; + __u32 memcg_fd; }; enum extern_type { @@ -4948,6 +4949,7 @@ static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map, b create_attr.map_flags = def->map_flags; create_attr.numa_node = map->numa_node; create_attr.map_extra = map->map_extra; + create_attr.memcg_fd = map->memcg_fd; if (bpf_map__is_struct_ops(map)) create_attr.btf_vmlinux_value_type_id = map->btf_vmlinux_value_type_id; diff --git a/tools/lib/bpf/skel_internal.h b/tools/lib/bpf/skel_internal.h index 1e82ab0..8760747 100644 --- a/tools/lib/bpf/skel_internal.h +++ b/tools/lib/bpf/skel_internal.h @@ -222,7 +222,7 @@ static inline int skel_map_create(enum bpf_map_type map_type, __u32 value_size, __u32 max_entries) { - const size_t attr_sz = offsetofend(union bpf_attr, map_extra); + const size_t attr_sz = offsetofend(union bpf_attr, memcg_fd); union bpf_attr attr; memset(&attr, 0, attr_sz);