From patchwork Sun Jun 19 15:50:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 12886759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 582F2C43334 for ; Sun, 19 Jun 2022 15:50:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ED0F78D000C; Sun, 19 Jun 2022 11:50:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E58738D0006; Sun, 19 Jun 2022 11:50:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF8A58D000C; Sun, 19 Jun 2022 11:50:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BA1758D0006 for ; Sun, 19 Jun 2022 11:50:58 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 9267D60967 for ; Sun, 19 Jun 2022 15:50:58 +0000 (UTC) X-FDA: 79595423796.23.08772DE Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf28.hostedemail.com (Postfix) with ESMTP id 2EB46C0009 for ; Sun, 19 Jun 2022 15:50:58 +0000 (UTC) Received: by mail-pl1-f171.google.com with SMTP id h1so7731739plf.11 for ; Sun, 19 Jun 2022 08:50:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8axVjIKJzoKX7/vYAqMF/TYbFLVxAIKs18CKF3m+oFs=; b=PuPpsRhrY7YHspdQWWpU1xoiN04UvGnMB4vC75wdnvBWiS+Y1ytnUMJqu3Ho9YMv7j yUVPddPMbPZG1XjtCjIyGWJoikOcQ0GatJvErGEg6QrKCQj3dvmPOZSUtiXTU+dFvoSp YcIX9PwU4tdeb+QuUi7/vdBpJBPcFXrepnvYHjd/drtSfgthuwT/jXJ9dp48KqVK1jGQ L78qryWdc++QJK17LRHW7muc+RPiMWMgaJ4tfBg+wB226QbRz/Vy32swaf+QXSEiOP6S DXPQMqrQ0zysXkG87o3+oTm4V7pbmv2EYd17jJv4SCOll+2gRmAW4MAkIbBXsxPkuf++ IiPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8axVjIKJzoKX7/vYAqMF/TYbFLVxAIKs18CKF3m+oFs=; b=dCdhTR6jXFsjOiZG8c1U8s7sCGoMKFyehuzGUwV89JeyWL8hQl63BW5i09EgCGJiPT ep+MpAQCGv2C6R0IKwN06Bw1IQX/QYwf1OzNNuFTn1/5b3AeqSagD7F6zSQvN5iV5pzl 4ghYt4I2eh1wI7ui7uVCTmV+4zOUuqSQRggBDZBwJAEHaPv0Dfk51S0YxJl8/5bAqK/2 w0g2GVMvVfHrsBXdlbdVHHSUlHJFGQlQ8MMY1eeYhc8U58OQiknCnZwgB2iSNKerzx0q zjlH4SNdNr6HNdlXPxhSsghutqBRSieAej4YR6PcT+ThNThcnBLIEpHu7+UkK9mUx5Ne N2Ag== X-Gm-Message-State: AJIora8pl4AFrvhHtY+HAHvjyFFyDJ2VScThQzo7VzqNlUPKnxOE+kFZ PTLZJNbncr+vOCLBDF3wgIY= X-Google-Smtp-Source: AGRyM1tWFLmDZ1xZ4c7PpIxT4CG5PqHtIf/jPAzTTqfH8RQEghLLoAKdBEQZ9RMRcWjmMK8GqJOEtA== X-Received: by 2002:a17:902:7686:b0:168:de55:8c45 with SMTP id m6-20020a170902768600b00168de558c45mr19518921pll.129.1655653857305; Sun, 19 Jun 2022 08:50:57 -0700 (PDT) Received: from vultr.guest ([2001:19f0:6001:2b24:5400:4ff:fe09:b144]) by smtp.gmail.com with ESMTPSA id z10-20020a1709027e8a00b001690a7df347sm6381761pla.96.2022.06.19.08.50.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Jun 2022 08:50:56 -0700 (PDT) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, quentin@isovalent.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, songmuchun@bytedance.com, akpm@linux-foundation.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 07/10] mm: Add helper to recharge percpu address Date: Sun, 19 Jun 2022 15:50:29 +0000 Message-Id: <20220619155032.32515-8-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220619155032.32515-1-laoar.shao@gmail.com> References: <20220619155032.32515-1-laoar.shao@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655653858; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8axVjIKJzoKX7/vYAqMF/TYbFLVxAIKs18CKF3m+oFs=; b=AuEHW+lG4lrHlvO83QPDmCJjyyEvQBDs4vmmyd0kdvoJHTQSZeC/ccPPZCd2+N0Z0C5dsR d8ES+jDCtIX2rFJCcZVQcBVx3Q9MGqPZzE/FitCuf++ibQrpHbrctrH+Vk1Nb4scxs3VyG Hd/drPzyJorYDX70cwYhDoXKchk/Nn0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655653858; a=rsa-sha256; cv=none; b=QGntzomlUBCEdisi8gyYLGDiXHtwoz7X/SniF/++1IfQFtcFGZXislKcBnhMkJW//seCxh ICxcdlGV8A1b/fU20YMTOHU/C9yV8mCiGMd4C1FsYic1pNsVTUNT6m0LwQYgX2A0+O1zbX IdoE0V7tHAdsROvg9+D5thwV2wTv3gU= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=PuPpsRhr; spf=pass (imf28.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: c74w5bn9uj3btcu4uk4zmaazdupxy8kj Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=PuPpsRhr; spf=pass (imf28.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Queue-Id: 2EB46C0009 X-Rspamd-Server: rspam02 X-Rspam-User: X-HE-Tag: 1655653858-895542 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch introduces a helper to recharge the corresponding pages of a given percpu address. It is similar with how to recharge a kmalloc'ed address. Signed-off-by: Yafang Shao --- include/linux/percpu.h | 1 + mm/percpu.c | 98 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 99 insertions(+) diff --git a/include/linux/percpu.h b/include/linux/percpu.h index f1ec5ad1351c..e88429410179 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -128,6 +128,7 @@ extern void __init setup_per_cpu_areas(void); extern void __percpu *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp) __alloc_size(1); extern void __percpu *__alloc_percpu(size_t size, size_t align) __alloc_size(1); extern void free_percpu(void __percpu *__pdata); +bool recharge_percpu(void __percpu *__pdata, int step); extern phys_addr_t per_cpu_ptr_to_phys(void *addr); #define alloc_percpu_gfp(type, gfp) \ diff --git a/mm/percpu.c b/mm/percpu.c index 3633eeefaa0d..fd81f4d79f2f 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -2310,6 +2310,104 @@ void free_percpu(void __percpu *ptr) } EXPORT_SYMBOL_GPL(free_percpu); +#ifdef CONFIG_MEMCG_KMEM +bool recharge_percpu(void __percpu *ptr, int step) +{ + int bit_off, off, bits, size, end; + struct obj_cgroup *objcg_old; + struct obj_cgroup *objcg_new; + struct pcpu_chunk *chunk; + unsigned long flags; + void *addr; + + WARN_ON(!in_task()); + + if (!ptr) + return true; + + addr = __pcpu_ptr_to_addr(ptr); + spin_lock_irqsave(&pcpu_lock, flags); + chunk = pcpu_chunk_addr_search(addr); + off = addr - chunk->base_addr; + objcg_old = chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT]; + if (!objcg_old && step != MEMCG_KMEM_POST_CHARGE) { + spin_unlock_irqrestore(&pcpu_lock, flags); + return true; + } + + bit_off = off / PCPU_MIN_ALLOC_SIZE; + /* find end index */ + end = find_next_bit(chunk->bound_map, pcpu_chunk_map_bits(chunk), + bit_off + 1); + bits = end - bit_off; + size = bits * PCPU_MIN_ALLOC_SIZE; + + switch (step) { + case MEMCG_KMEM_PRE_CHARGE: + objcg_new = get_obj_cgroup_from_current(); + WARN_ON(!objcg_new); + if (obj_cgroup_charge(objcg_new, GFP_KERNEL, + size * num_possible_cpus())) { + obj_cgroup_put(objcg_new); + spin_unlock_irqrestore(&pcpu_lock, flags); + return false; + } + break; + case MEMCG_KMEM_UNCHARGE: + obj_cgroup_uncharge(objcg_old, size * num_possible_cpus()); + rcu_read_lock(); + mod_memcg_state(obj_cgroup_memcg(objcg_old), MEMCG_PERCPU_B, + -(size * num_possible_cpus())); + rcu_read_unlock(); + chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = NULL; + obj_cgroup_put(objcg_old); + break; + case MEMCG_KMEM_POST_CHARGE: + rcu_read_lock(); + chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = obj_cgroup_from_current(); + mod_memcg_state(mem_cgroup_from_task(current), MEMCG_PERCPU_B, + (size * num_possible_cpus())); + rcu_read_unlock(); + break; + case MEMCG_KMEM_CHARGE_ERR: + /* + * In case fail to charge to the new one in the pre charge state, + * for example, we have pre-charged one memcg successfully but fail + * to pre-charge the second memcg, then we should uncharge the first + * memcg. + */ + objcg_new = obj_cgroup_from_current(); + obj_cgroup_uncharge(objcg_new, size * num_possible_cpus()); + obj_cgroup_put(objcg_new); + rcu_read_lock(); + mod_memcg_state(obj_cgroup_memcg(objcg_new), MEMCG_PERCPU_B, + -(size * num_possible_cpus())); + rcu_read_unlock(); + + break; + } + + spin_unlock_irqrestore(&pcpu_lock, flags); + + return true; +} +EXPORT_SYMBOL(recharge_percpu); + +#else /* CONFIG_MEMCG_KMEM */ + +bool charge_percpu(void __percpu *ptr, bool charge) +{ + return true; +} +EXPORT_SYMBOL(charge_percpu); + +void uncharge_percpu(void __percpu *ptr) +{ +} +EXPORT_SYMBOL(uncharge_percpu); + +#endif /* CONFIG_MEMCG_KMEM */ + bool __is_kernel_percpu_address(unsigned long addr, unsigned long *can_addr) { #ifdef CONFIG_SMP