From patchwork Mon Dec 12 00:37:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13070696 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 174CBC4332F for ; Mon, 12 Dec 2022 00:37:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7898A8E0007; Sun, 11 Dec 2022 19:37:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 73A248E0002; Sun, 11 Dec 2022 19:37:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5DBA48E0007; Sun, 11 Dec 2022 19:37:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4B9AA8E0002 for ; Sun, 11 Dec 2022 19:37:54 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 19E1F1A06BE for ; Mon, 12 Dec 2022 00:37:54 +0000 (UTC) X-FDA: 80231791668.04.97E19AF Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf17.hostedemail.com (Postfix) with ESMTP id 7C4C940006 for ; Mon, 12 Dec 2022 00:37:52 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=blDmjwOG; spf=pass (imf17.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670805472; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/FpiiW8tlCQuBLdcxwgK57oWmDCwhuuS+u14RJYhlmk=; b=upu5sSnKi3BPvbcXLzDe6prUtTugZyy/qlG0SMPmC+WuFoQTOFQtqQaCKEJxIj0/nF6LrP kllw8KdtsGntAkrFr1xCZkqF87Nt9A/VFKFVwG1cd/01H1NGkuPPqVnOJna1qmfGgTVijc wS2qRzFPBhZTBjuTVZRLhO4eqrrAcZI= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=blDmjwOG; spf=pass (imf17.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670805472; a=rsa-sha256; cv=none; b=w8lDRqoCl3yaVRGP/sDb9M3+C8BhlQN2L4Q+fUB3hArM/EXzlOgf+Vf+Sryr3MEFtho5td Itwn7zD0zgq8xKnIy9YcVIFF4byh56gctzoJFbeZOnvRIxiOkegotolTIYbyoS6YKU0Mr6 nyyVkx1Zv7XURELGPrG++3jpi1A/7ok= Received: by mail-pl1-f174.google.com with SMTP id 4so4460066plj.3 for ; Sun, 11 Dec 2022 16:37:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/FpiiW8tlCQuBLdcxwgK57oWmDCwhuuS+u14RJYhlmk=; b=blDmjwOG5yBpOKGDu4OqOJfK6YDC2tIgsaA++wSa0spxRtLKgIr7tNI1/nRT1GOLtp kmICcOuS0tuAzHuhsh3L4tSLzOkTgZDXK8KlHTw1Zulv2p2rFKaK5DBXCtOzTWLLXLXM nFdtzntL+7qlYziM7IZ6Z9Eu20pf9537iCQMLsJ0nKLZ5Z4nBKTo3jKnKyTXIkZWEMWK uToyEoyYinHFpycXiVUdruUq96vWChVYpvMHF5dpYLdnhpth42d/gxX097KBpmpTJCFt EHawh4kk5/jSb54tQVSkfcXVjAF6va6Pb8fF0SIuMb5dMRpI7dzXrhcTgE9Jtr+vFpZy 8VbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/FpiiW8tlCQuBLdcxwgK57oWmDCwhuuS+u14RJYhlmk=; b=HW5bzaMqMaCRNKYn2Q+mjPNov5hlKUn8A0h8o8vPoxS52QYb52XmoE9XFzf3dUi8YU yiUi9EO71bZqFaiAnAkQhp6ZpYeupj0vKanlNy+D/pCGvINIdeTmLZURIpAdoT/z5KA7 kdotjKMEpbfFEVJ3gS3EYNFi8tHUbg8Djox9hCHrm35uWJqd32ovIUPI1SLBi/9HaYts snwzfsxxbMvpgQP6jvHYsCzvwVZUaYm2RSWNg2tBYKbS6rndBDngrhGcETvB2BHnTWe9 mMoW8KY0O2p4QLrxADCP2ow8nsVJGGN+Xu5vzp0qmnoYW0PPrzd3boG3q3nrd84PwBLb 348Q== X-Gm-Message-State: ANoB5pkJLxIySQH+dWC9sWEavikvZOfoasUujn+Xt2EYmMuqTIpNrqWp EesIeqOGjWl8wA+XzD6RAsk= X-Google-Smtp-Source: AA0mqf6sGfx5c19Rrfw5tyzmw9AMZXV+iZ/hNtnw3UdNnuW5oRiZL8CBPsG6dFQMVoMUC2lhjL16RA== X-Received: by 2002:a17:902:b58c:b0:189:184d:f0ff with SMTP id a12-20020a170902b58c00b00189184df0ffmr13651625pls.11.1670805471482; Sun, 11 Dec 2022 16:37:51 -0800 (PST) Received: from vultr.guest ([2001:19f0:7002:8c7:5400:4ff:fe3d:656a]) by smtp.gmail.com with ESMTPSA id w9-20020a170902e88900b00177fb862a87sm4895960plg.20.2022.12.11.16.37.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 11 Dec 2022 16:37:50 -0800 (PST) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 3/9] mm: percpu: Account active vm for percpu Date: Mon, 12 Dec 2022 00:37:05 +0000 Message-Id: <20221212003711.24977-4-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20221212003711.24977-1-laoar.shao@gmail.com> References: <20221212003711.24977-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 7C4C940006 X-Stat-Signature: qgeh9na5afozwacaz1u534id4hupe4qr X-Rspam-User: X-HE-Tag: 1670805472-475186 X-HE-Meta: U2FsdGVkX19ImJ8OSX7KNCam6h/5+MqcA3i/kQsVDVOgscFT1vpSchd/hVKkDK0lN3MUnEJedi1BOiVzdvttzzHSYlIABUWegOl5Ds58eV/aJdlHIAB8HesVVbqw1MFDE/Fr3RZ7r5jCKyZQSJNIrahBBITFojF+fmnmuNJtqlTY/yufquN4AxQdWKZTEpiIR32BxpKtPOP7m7OtnFH2N0WozgMVAny8vgA+ChLlyL05A+coUo08S3DeHTzZBdgdrgRAjQr7iHfgguEfCKxVjbD2VfEEsaReKWc3Z5I3UfVarKW8vL+sAgkebjLlgaQvEMklldQPCyituibrvv5zMXaAIO5agBuae423vcNRwiQ7KXEPsM86yyt6UD2Kz0cM07Itm3hUqu2BZL4hWGE/n/NryQgUC4WM2hFtGwEZxMyuo1rDnwqZyMRrbrfXeHXrO6HbI4k0FWdv94z5XHPC5NH6kCYObsMGvq/Vxg24tb54fM/4haHG8uF4zq72YLc0p7Y9OquXvVNl3I4zUYuL23kivt3tKtIxuklGYDrX9sTEdOSIWlZdK3x3GmA8avxGeI22FOAYn8oFqMpEkczcLDaoqVg+L/ppjs7ERpXA4HycFmfCbps0M2a/yMsh9N7XjQcHR2jZj1eo/ZlZYySeqwpBAMwbZgE1bJ9UCkw+XgWgKjfLSjDMbryiftWSP54yWV+AXDs6V9PlLov0SaFE2+NM3oVjwHUXLcMIUdTa8QikY4lLIzDlRMBJau60THNHsbVqZIUHQpJPVjQ/FSsTMKiPueTaFbIxNdf1pLtM9yFxhAzHiEELIdU+B98pBN+kjEi8N6IskAHk4L3bIXMLR0oQXppBrUtVIMYpzJKxZwSUoiUpS0l1F6yjkry7WBkDn+I/yIqm3poiJ4XLov5w+PEk18S/Y9BWue94uE4gnfyLypXX2Wj33woS5mNjj1avjN8n3mNqLpZ8gxoPnGm RcWBRzmd YnLAJylc81hw2p3D42lGoORTzV2QqrJ+LvaWga3d6Fs0yJJN9qUiXunnwK65waVdA9TDdLZsb4y8DVG6kGDsuwfc5alIynGOhhW1Ll6sMaBH0VNwL6XIVSJKDthOzDqQbWYuyi0oPygNmDRjFNwpzIigeShD3g0t9LmG4qTd/4sAhtPhSrFXWbptzX4WqnknvL8ibMY7bALO2/4iY0b8FyJP8WuYYinXZGGBpeWBPdAgsjFM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Account percpu allocation when active vm item is set. The percpu memory is accounted at percpu alloc and unaccount at percpu free. To record which part of percpu chunk is enabled with active vm, we have to allocate extra memory for this percpu chunk to record the active vm information. This extra memory will be freed when this percpu chunk is freed. Signed-off-by: Yafang Shao --- mm/percpu-internal.h | 3 +++ mm/percpu.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index 70b1ea23f4d2..f56e236a2cf3 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -63,6 +63,9 @@ struct pcpu_chunk { int nr_pages; /* # of pages served by this chunk */ int nr_populated; /* # of populated pages */ int nr_empty_pop_pages; /* # of empty populated pages */ +#ifdef CONFIG_ACTIVE_VM + int *active_vm; /* vector of activem vm items */ +#endif unsigned long populated[]; /* populated bitmap */ }; diff --git a/mm/percpu.c b/mm/percpu.c index 27697b2429c2..05858981ed4a 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -98,6 +98,7 @@ #include #include "percpu-internal.h" +#include "active_vm.h" /* * The slots are sorted by the size of the biggest continuous free area. @@ -1398,6 +1399,9 @@ static struct pcpu_chunk * __init pcpu_alloc_first_chunk(unsigned long tmp_addr, #ifdef CONFIG_MEMCG_KMEM /* first chunk is free to use */ chunk->obj_cgroups = NULL; +#endif +#ifdef CONFIG_ACTIVE_VM + chunk->active_vm = NULL; #endif pcpu_init_md_blocks(chunk); @@ -1476,6 +1480,14 @@ static struct pcpu_chunk *pcpu_alloc_chunk(gfp_t gfp) } #endif +#ifdef CONFIG_ACTIVE_VM + if (active_vm_enabled()) { + chunk->active_vm = pcpu_mem_zalloc(pcpu_chunk_map_bits(chunk) * + sizeof(int), gfp); + if (!chunk->active_vm) + goto active_vm_fail; + } +#endif pcpu_init_md_blocks(chunk); /* init metadata */ @@ -1483,6 +1495,12 @@ static struct pcpu_chunk *pcpu_alloc_chunk(gfp_t gfp) return chunk; +#ifdef CONFIG_ACTIVE_VM +active_vm_fail: +#ifdef CONFIG_MEMCG_KMEM + pcpu_mem_free(chunk->obj_cgroups); +#endif +#endif #ifdef CONFIG_MEMCG_KMEM objcg_fail: pcpu_mem_free(chunk->md_blocks); @@ -1501,6 +1519,9 @@ static void pcpu_free_chunk(struct pcpu_chunk *chunk) { if (!chunk) return; +#ifdef CONFIG_ACTIVE_VM + pcpu_mem_free(chunk->active_vm); +#endif #ifdef CONFIG_MEMCG_KMEM pcpu_mem_free(chunk->obj_cgroups); #endif @@ -1890,6 +1911,17 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, pcpu_memcg_post_alloc_hook(objcg, chunk, off, size); +#ifdef CONFIG_ACTIVE_VM + if (active_vm_enabled() && chunk->active_vm && (gfp & __GFP_ACCOUNT)) { + int item = active_vm_item(); + + if (item > 0) { + chunk->active_vm[off >> PCPU_MIN_ALLOC_SHIFT] = item; + active_vm_item_add(item, size); + } + } +#endif + return ptr; fail_unlock: @@ -2283,6 +2315,17 @@ void free_percpu(void __percpu *ptr) pcpu_memcg_free_hook(chunk, off, size); +#ifdef CONFIG_ACTIVE_VM + if (active_vm_enabled() && chunk->active_vm) { + int item = chunk->active_vm[off >> PCPU_MIN_ALLOC_SHIFT]; + + if (item > 0) { + active_vm_item_sub(item, size); + chunk->active_vm[off >> PCPU_MIN_ALLOC_SHIFT] = 0; + } + } +#endif + /* * If there are more than one fully free chunks, wake up grim reaper. * If the chunk is isolated, it may be in the process of being