From patchwork Thu Apr 8 03:57:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 12189919 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F61EC43460 for ; Thu, 8 Apr 2021 03:58:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1FAF861139 for ; Thu, 8 Apr 2021 03:58:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1FAF861139 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 843726B0082; Wed, 7 Apr 2021 23:58:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7CC0E6B0085; Wed, 7 Apr 2021 23:58:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5FCE06B0087; Wed, 7 Apr 2021 23:58:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0190.hostedemail.com [216.40.44.190]) by kanga.kvack.org (Postfix) with ESMTP id 34A226B0082 for ; Wed, 7 Apr 2021 23:58:21 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id E90FA8786 for ; Thu, 8 Apr 2021 03:58:20 +0000 (UTC) X-FDA: 78007842360.14.2EAEB8D Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf14.hostedemail.com (Postfix) with ESMTP id 99C8BC0007C1 for ; Thu, 8 Apr 2021 03:58:15 +0000 (UTC) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 1383uNox008464 for ; Wed, 7 Apr 2021 20:58:19 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=twOL+x7dWTU1or/wZU/ar4/+kx+XkJUUrC30nFb+VAw=; b=K4AILJ+yroqCe2BRfrojA6CaefIjSggFHuY5SJl1J3tuSkifoFCNuWhyauwwfs+vRDlI 1oeN8n0k5xTkRh7jCuLrHLqq/KnQkyWaqSo6/dXauLG6bUURgLRX7rAGiPeSnr0onQwd jV9lRf5SKwNFU8arrRlbWtUjCU5UHYyVZtc= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com with ESMTP id 37sg04kmye-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 07 Apr 2021 20:58:19 -0700 Received: from intmgw002.48.prn1.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:21d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Wed, 7 Apr 2021 20:57:47 -0700 Received: by devvm3388.prn0.facebook.com (Postfix, from userid 111017) id 58BCE602B00C; Wed, 7 Apr 2021 20:57:41 -0700 (PDT) From: Roman Gushchin To: Dennis Zhou CC: Tejun Heo , Christoph Lameter , Andrew Morton , Vlastimil Babka , , , Roman Gushchin Subject: [PATCH v3 1/6] percpu: fix a comment about the chunks ordering Date: Wed, 7 Apr 2021 20:57:31 -0700 Message-ID: <20210408035736.883861-2-guro@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210408035736.883861-1-guro@fb.com> References: <20210408035736.883861-1-guro@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: 4lgQu-3HH84eIPXbZRe520mwLNqaBOan X-Proofpoint-GUID: 4lgQu-3HH84eIPXbZRe520mwLNqaBOan X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.761 definitions=2021-04-08_01:2021-04-07,2021-04-08 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 adultscore=0 mlxscore=0 suspectscore=0 priorityscore=1501 bulkscore=0 lowpriorityscore=0 clxscore=1015 mlxlogscore=819 phishscore=0 malwarescore=0 spamscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104060000 definitions=main-2104080023 X-FB-Internal: deliver X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 99C8BC0007C1 X-Stat-Signature: 1si4brep6bb6o775rzy6a8x4keht3m8q Received-SPF: none (fb.com>: No applicable sender policy available) receiver=imf14; identity=mailfrom; envelope-from=""; helo=mx0a-00082601.pphosted.com; client-ip=67.231.145.42 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617854295-332154 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since the commit 3e54097beb22 ("percpu: manage chunks based on contig_bits instead of free_bytes") chunks are sorted based on the size of the biggest continuous free area instead of the total number of free bytes. Update the corresponding comment to reflect this. Signed-off-by: Roman Gushchin --- mm/percpu.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/percpu.c b/mm/percpu.c index 6596a0a4286e..2f27123bb489 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -99,7 +99,10 @@ #include "percpu-internal.h" -/* the slots are sorted by free bytes left, 1-31 bytes share the same slot */ +/* + * The slots are sorted by the size of the biggest continuous free area. + * 1-31 bytes share the same slot. + */ #define PCPU_SLOT_BASE_SHIFT 5 /* chunks in slots below this are subject to being sidelined on failed alloc */ #define PCPU_SLOT_FAIL_THRESHOLD 3 From patchwork Thu Apr 8 03:57:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 12189907 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57919C43460 for ; Thu, 8 Apr 2021 03:57:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D1E1D610FA for ; Thu, 8 Apr 2021 03:57:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D1E1D610FA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5D3478E0001; Wed, 7 Apr 2021 23:57:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 55AFB6B0082; Wed, 7 Apr 2021 23:57:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 338768D0001; Wed, 7 Apr 2021 23:57:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0059.hostedemail.com [216.40.44.59]) by kanga.kvack.org (Postfix) with ESMTP id 1107B6B0080 for ; Wed, 7 Apr 2021 23:57:51 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id C9792181D5520 for ; Thu, 8 Apr 2021 03:57:50 +0000 (UTC) X-FDA: 78007841100.08.A22E4FD Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by imf12.hostedemail.com (Postfix) with ESMTP id 0A204D2 for ; Thu, 8 Apr 2021 03:57:46 +0000 (UTC) Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.16.0.43/8.16.0.43) with SMTP id 1383r9U1017880 for ; Wed, 7 Apr 2021 20:57:49 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=HMmvNQkcS7gbusULGAPo0RXdrobA76PG+Ea0H7xaLXk=; b=rMewr7bnmR7IXwtstXqzjNQzmVfzeiVydOhBLo/PpgU15WibXSSoNidQ7F6UFFy3XT5C AXzsL6YQd228s29mTgWMAZerq1yjU65LEGiHfjFPgVECCxHzA7GCGW/nKJeunS9hwuTf WVT4uB5H3nqc1ZwIXhyPymE7INABn3PNAVc= Received: from mail.thefacebook.com ([163.114.132.120]) by m0001303.ppops.net with ESMTP id 37spqus30s-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 07 Apr 2021 20:57:49 -0700 Received: from intmgw002.46.prn1.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:21d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Wed, 7 Apr 2021 20:57:47 -0700 Received: by devvm3388.prn0.facebook.com (Postfix, from userid 111017) id 5D3F2602B00E; Wed, 7 Apr 2021 20:57:41 -0700 (PDT) From: Roman Gushchin To: Dennis Zhou CC: Tejun Heo , Christoph Lameter , Andrew Morton , Vlastimil Babka , , , Roman Gushchin Subject: [PATCH v3 2/6] percpu: split __pcpu_balance_workfn() Date: Wed, 7 Apr 2021 20:57:32 -0700 Message-ID: <20210408035736.883861-3-guro@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210408035736.883861-1-guro@fb.com> References: <20210408035736.883861-1-guro@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: X_MQc2IWl7mjOTZKG5I7RMk2xKVDCa7J X-Proofpoint-GUID: X_MQc2IWl7mjOTZKG5I7RMk2xKVDCa7J X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.761 definitions=2021-04-08_01:2021-04-07,2021-04-08 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 phishscore=0 lowpriorityscore=0 suspectscore=0 bulkscore=0 mlxlogscore=999 spamscore=0 adultscore=0 priorityscore=1501 clxscore=1015 mlxscore=0 malwarescore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104060000 definitions=main-2104080023 X-FB-Internal: deliver X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 0A204D2 X-Stat-Signature: hst3ot7xiuc7poyqrwj6e5fwhkprykfx Received-SPF: none (fb.com>: No applicable sender policy available) receiver=imf12; identity=mailfrom; envelope-from=""; helo=mx0a-00082601.pphosted.com; client-ip=67.231.153.30 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617854266-987349 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: __pcpu_balance_workfn() became fairly big and hard to follow, but in fact it consists of two fully independent parts, responsible for the destruction of excessive free chunks and population of necessarily amount of free pages. In order to simplify the code and prepare for adding of a new functionality, split it in two functions: 1) pcpu_balance_free, 2) pcpu_balance_populated. Move the taking/releasing of the pcpu_alloc_mutex to an upper level to keep the current synchronization in place. Signed-off-by: Roman Gushchin Reviewed-by: Dennis Zhou --- mm/percpu.c | 46 +++++++++++++++++++++++++++++----------------- 1 file changed, 29 insertions(+), 17 deletions(-) diff --git a/mm/percpu.c b/mm/percpu.c index 2f27123bb489..7e31e1b8725f 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1933,31 +1933,22 @@ void __percpu *__alloc_reserved_percpu(size_t size, size_t align) } /** - * __pcpu_balance_workfn - manage the amount of free chunks and populated pages + * pcpu_balance_free - manage the amount of free chunks * @type: chunk type * - * Reclaim all fully free chunks except for the first one. This is also - * responsible for maintaining the pool of empty populated pages. However, - * it is possible that this is called when physical memory is scarce causing - * OOM killer to be triggered. We should avoid doing so until an actual - * allocation causes the failure as it is possible that requests can be - * serviced from already backed regions. + * Reclaim all fully free chunks except for the first one. */ -static void __pcpu_balance_workfn(enum pcpu_chunk_type type) +static void pcpu_balance_free(enum pcpu_chunk_type type) { - /* gfp flags passed to underlying allocators */ - const gfp_t gfp = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN; LIST_HEAD(to_free); struct list_head *pcpu_slot = pcpu_chunk_list(type); struct list_head *free_head = &pcpu_slot[pcpu_nr_slots - 1]; struct pcpu_chunk *chunk, *next; - int slot, nr_to_pop, ret; /* * There's no reason to keep around multiple unused chunks and VM * areas can be scarce. Destroy all free chunks except for one. */ - mutex_lock(&pcpu_alloc_mutex); spin_lock_irq(&pcpu_lock); list_for_each_entry_safe(chunk, next, free_head, list) { @@ -1985,6 +1976,25 @@ static void __pcpu_balance_workfn(enum pcpu_chunk_type type) pcpu_destroy_chunk(chunk); cond_resched(); } +} + +/** + * pcpu_balance_populated - manage the amount of populated pages + * @type: chunk type + * + * Maintain a certain amount of populated pages to satisfy atomic allocations. + * It is possible that this is called when physical memory is scarce causing + * OOM killer to be triggered. We should avoid doing so until an actual + * allocation causes the failure as it is possible that requests can be + * serviced from already backed regions. + */ +static void pcpu_balance_populated(enum pcpu_chunk_type type) +{ + /* gfp flags passed to underlying allocators */ + const gfp_t gfp = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN; + struct list_head *pcpu_slot = pcpu_chunk_list(type); + struct pcpu_chunk *chunk; + int slot, nr_to_pop, ret; /* * Ensure there are certain number of free populated pages for @@ -2054,22 +2064,24 @@ static void __pcpu_balance_workfn(enum pcpu_chunk_type type) goto retry_pop; } } - - mutex_unlock(&pcpu_alloc_mutex); } /** * pcpu_balance_workfn - manage the amount of free chunks and populated pages * @work: unused * - * Call __pcpu_balance_workfn() for each chunk type. + * Call pcpu_balance_free() and pcpu_balance_populated() for each chunk type. */ static void pcpu_balance_workfn(struct work_struct *work) { enum pcpu_chunk_type type; - for (type = 0; type < PCPU_NR_CHUNK_TYPES; type++) - __pcpu_balance_workfn(type); + for (type = 0; type < PCPU_NR_CHUNK_TYPES; type++) { + mutex_lock(&pcpu_alloc_mutex); + pcpu_balance_free(type); + pcpu_balance_populated(type); + mutex_unlock(&pcpu_alloc_mutex); + } } /** From patchwork Thu Apr 8 03:57:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 12189917 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85ACCC433B4 for ; Thu, 8 Apr 2021 03:58:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 34A4D611BD for ; Thu, 8 Apr 2021 03:58:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 34A4D611BD Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A18578D0003; Wed, 7 Apr 2021 23:58:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C91A6B0085; Wed, 7 Apr 2021 23:58:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 81BA88D0003; Wed, 7 Apr 2021 23:58:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0175.hostedemail.com [216.40.44.175]) by kanga.kvack.org (Postfix) with ESMTP id 6229D6B0082 for ; Wed, 7 Apr 2021 23:58:20 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 19A398786 for ; Thu, 8 Apr 2021 03:58:20 +0000 (UTC) X-FDA: 78007842360.23.0684265 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by imf02.hostedemail.com (Postfix) with ESMTP id 8461440002CE for ; Thu, 8 Apr 2021 03:58:10 +0000 (UTC) Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 1383u7cq004510 for ; Wed, 7 Apr 2021 20:58:19 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=4vZaaBFsaOB4F8XygoHlUQdcYtLDpiQQyW830Fvwijw=; b=KWiObgVqOxfkfQkH1qAh7b5fYEAlSnSYcTTE8iRyF0B4mgOIvljh3uRP+QOTqdRLIxT5 njVsxBaUwod7byvxvrYBuiV0HAgdtsncXakWZ7Fp/zSYFsrheB3eaZxJPoiIs0Rw1Q1L Idt34p0xB0X/ptTKxOCPc2D858YX19KrNBw= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com with ESMTP id 37sh2nk8df-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 07 Apr 2021 20:58:19 -0700 Received: from intmgw001.05.prn6.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:21d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Wed, 7 Apr 2021 20:57:47 -0700 Received: by devvm3388.prn0.facebook.com (Postfix, from userid 111017) id 61F75602B010; Wed, 7 Apr 2021 20:57:41 -0700 (PDT) From: Roman Gushchin To: Dennis Zhou CC: Tejun Heo , Christoph Lameter , Andrew Morton , Vlastimil Babka , , , Roman Gushchin Subject: [PATCH v3 3/6] percpu: make pcpu_nr_empty_pop_pages per chunk type Date: Wed, 7 Apr 2021 20:57:33 -0700 Message-ID: <20210408035736.883861-4-guro@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210408035736.883861-1-guro@fb.com> References: <20210408035736.883861-1-guro@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: Zlm7OkgGVyBMGDErxSoYMJOVa2zTyO6q X-Proofpoint-GUID: Zlm7OkgGVyBMGDErxSoYMJOVa2zTyO6q X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.761 definitions=2021-04-08_01:2021-04-07,2021-04-08 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 impostorscore=0 bulkscore=0 spamscore=0 priorityscore=1501 mlxscore=0 mlxlogscore=999 phishscore=0 adultscore=0 suspectscore=0 lowpriorityscore=0 clxscore=1015 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104060000 definitions=main-2104080023 X-FB-Internal: deliver X-Stat-Signature: un393nd3dg7bjyu57juh9xwyiurhco9s X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 8461440002CE Received-SPF: none (fb.com>: No applicable sender policy available) receiver=imf02; identity=mailfrom; envelope-from=""; helo=mx0b-00082601.pphosted.com; client-ip=67.231.153.30 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617854290-609694 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: nr_empty_pop_pages is used to guarantee that there are some free populated pages to satisfy atomic allocations. Accounted and non-accounted allocations are using separate sets of chunks, so both need to have a surplus of empty pages. This commit makes pcpu_nr_empty_pop_pages and the corresponding logic per chunk type. Signed-off-by: Roman Gushchin --- mm/percpu-internal.h | 2 +- mm/percpu-stats.c | 9 +++++++-- mm/percpu.c | 14 +++++++------- 3 files changed, 15 insertions(+), 10 deletions(-) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index 18b768ac7dca..095d7eaa0db4 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -87,7 +87,7 @@ extern spinlock_t pcpu_lock; extern struct list_head *pcpu_chunk_lists; extern int pcpu_nr_slots; -extern int pcpu_nr_empty_pop_pages; +extern int pcpu_nr_empty_pop_pages[]; extern struct pcpu_chunk *pcpu_first_chunk; extern struct pcpu_chunk *pcpu_reserved_chunk; diff --git a/mm/percpu-stats.c b/mm/percpu-stats.c index c8400a2adbc2..f6026dbcdf6b 100644 --- a/mm/percpu-stats.c +++ b/mm/percpu-stats.c @@ -145,6 +145,7 @@ static int percpu_stats_show(struct seq_file *m, void *v) int slot, max_nr_alloc; int *buffer; enum pcpu_chunk_type type; + int nr_empty_pop_pages; alloc_buffer: spin_lock_irq(&pcpu_lock); @@ -165,7 +166,11 @@ static int percpu_stats_show(struct seq_file *m, void *v) goto alloc_buffer; } -#define PL(X) \ + nr_empty_pop_pages = 0; + for (type = 0; type < PCPU_NR_CHUNK_TYPES; type++) + nr_empty_pop_pages += pcpu_nr_empty_pop_pages[type]; + +#define PL(X) \ seq_printf(m, " %-20s: %12lld\n", #X, (long long int)pcpu_stats_ai.X) seq_printf(m, @@ -196,7 +201,7 @@ static int percpu_stats_show(struct seq_file *m, void *v) PU(nr_max_chunks); PU(min_alloc_size); PU(max_alloc_size); - P("empty_pop_pages", pcpu_nr_empty_pop_pages); + P("empty_pop_pages", nr_empty_pop_pages); seq_putc(m, '\n'); #undef PU diff --git a/mm/percpu.c b/mm/percpu.c index 7e31e1b8725f..61339b3d9337 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -176,10 +176,10 @@ struct list_head *pcpu_chunk_lists __ro_after_init; /* chunk list slots */ static LIST_HEAD(pcpu_map_extend_chunks); /* - * The number of empty populated pages, protected by pcpu_lock. The - * reserved chunk doesn't contribute to the count. + * The number of empty populated pages by chunk type, protected by pcpu_lock. + * The reserved chunk doesn't contribute to the count. */ -int pcpu_nr_empty_pop_pages; +int pcpu_nr_empty_pop_pages[PCPU_NR_CHUNK_TYPES]; /* * The number of populated pages in use by the allocator, protected by @@ -559,7 +559,7 @@ static inline void pcpu_update_empty_pages(struct pcpu_chunk *chunk, int nr) { chunk->nr_empty_pop_pages += nr; if (chunk != pcpu_reserved_chunk) - pcpu_nr_empty_pop_pages += nr; + pcpu_nr_empty_pop_pages[pcpu_chunk_type(chunk)] += nr; } /* @@ -1835,7 +1835,7 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, mutex_unlock(&pcpu_alloc_mutex); } - if (pcpu_nr_empty_pop_pages < PCPU_EMPTY_POP_PAGES_LOW) + if (pcpu_nr_empty_pop_pages[type] < PCPU_EMPTY_POP_PAGES_LOW) pcpu_schedule_balance_work(); /* clear the areas and return address relative to base address */ @@ -2013,7 +2013,7 @@ static void pcpu_balance_populated(enum pcpu_chunk_type type) pcpu_atomic_alloc_failed = false; } else { nr_to_pop = clamp(PCPU_EMPTY_POP_PAGES_HIGH - - pcpu_nr_empty_pop_pages, + pcpu_nr_empty_pop_pages[type], 0, PCPU_EMPTY_POP_PAGES_HIGH); } @@ -2595,7 +2595,7 @@ void __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai, /* link the first chunk in */ pcpu_first_chunk = chunk; - pcpu_nr_empty_pop_pages = pcpu_first_chunk->nr_empty_pop_pages; + pcpu_nr_empty_pop_pages[PCPU_CHUNK_ROOT] = pcpu_first_chunk->nr_empty_pop_pages; pcpu_chunk_relocate(pcpu_first_chunk, -1); /* include all regions of the first chunk */ From patchwork Thu Apr 8 03:57:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 12189909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79252C43461 for ; Thu, 8 Apr 2021 03:57:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 035E361132 for ; Thu, 8 Apr 2021 03:57:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 035E361132 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8270C6B0081; Wed, 7 Apr 2021 23:57:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7A4968E0002; Wed, 7 Apr 2021 23:57:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3FC858D0003; Wed, 7 Apr 2021 23:57:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0007.hostedemail.com [216.40.44.7]) by kanga.kvack.org (Postfix) with ESMTP id 1D4766B0081 for ; Wed, 7 Apr 2021 23:57:51 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id C6E84181D551B for ; Thu, 8 Apr 2021 03:57:50 +0000 (UTC) X-FDA: 78007841100.26.E8743AC Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf05.hostedemail.com (Postfix) with ESMTP id A3083E000104 for ; Thu, 8 Apr 2021 03:57:49 +0000 (UTC) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 1383tnlJ022033 for ; Wed, 7 Apr 2021 20:57:48 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=qAEXwgccNCJwApJKnF3tbkjWwzvQ4ovmNeHysDx48R0=; b=TfTrvIPmSEVVOhwoeNCdym6WDTkE78AKcIQ9bQBQjWD4WQOH6CdiVBg1mYQN9lnbrex3 /+c47xj9y/GdviCPlzd7pjQrQMIfYTRHiT2xYuXp0b/zVRobSo6uxjYDdCIUneBCstpg jFiXI8R4ljHGEIsl8E5/OM/7IlNcCbMW/Mk= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com with ESMTP id 37smjwhwnc-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 07 Apr 2021 20:57:48 -0700 Received: from intmgw002.46.prn1.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:11d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Wed, 7 Apr 2021 20:57:47 -0700 Received: by devvm3388.prn0.facebook.com (Postfix, from userid 111017) id 67A4E602B012; Wed, 7 Apr 2021 20:57:41 -0700 (PDT) From: Roman Gushchin To: Dennis Zhou CC: Tejun Heo , Christoph Lameter , Andrew Morton , Vlastimil Babka , , , Roman Gushchin Subject: [PATCH v3 4/6] percpu: generalize pcpu_balance_populated() Date: Wed, 7 Apr 2021 20:57:34 -0700 Message-ID: <20210408035736.883861-5-guro@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210408035736.883861-1-guro@fb.com> References: <20210408035736.883861-1-guro@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: GFHEjN2cySM2Ia08pC_lfTknEp74SUIE X-Proofpoint-ORIG-GUID: GFHEjN2cySM2Ia08pC_lfTknEp74SUIE X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.761 definitions=2021-04-08_01:2021-04-07,2021-04-08 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 impostorscore=0 bulkscore=0 clxscore=1015 adultscore=0 malwarescore=0 priorityscore=1501 mlxscore=0 spamscore=0 phishscore=0 mlxlogscore=999 suspectscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104060000 definitions=main-2104080023 X-FB-Internal: deliver X-Rspamd-Queue-Id: A3083E000104 X-Stat-Signature: c8edxeox893xymx8cgjskchoqwh8j5j4 X-Rspamd-Server: rspam02 Received-SPF: none (fb.com>: No applicable sender policy available) receiver=imf05; identity=mailfrom; envelope-from=""; helo=mx0a-00082601.pphosted.com; client-ip=67.231.145.42 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617854269-826198 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To prepare for the depopulation of percpu chunks, split out the populating part of the pcpu_balance_populated() into the new pcpu_grow_populated() (with an intention to add pcpu_shrink_populated() in the next commit). The goal of pcpu_balance_populated() is to determine whether there is a shortage or an excessive amount of empty percpu pages and call into the corresponding function. pcpu_grow_populated() takes a desired number of pages as an argument (nr_to_pop). If it creates a new chunk, nr_to_pop should be updated to reflect that the new chunk could be created already populated. Otherwise an infinite loop might appear. Signed-off-by: Roman Gushchin --- mm/percpu.c | 63 +++++++++++++++++++++++++++++++++-------------------- 1 file changed, 39 insertions(+), 24 deletions(-) diff --git a/mm/percpu.c b/mm/percpu.c index 61339b3d9337..e20119668c42 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1979,7 +1979,7 @@ static void pcpu_balance_free(enum pcpu_chunk_type type) } /** - * pcpu_balance_populated - manage the amount of populated pages + * pcpu_grow_populated - populate chunk(s) to satisfy atomic allocations * @type: chunk type * * Maintain a certain amount of populated pages to satisfy atomic allocations. @@ -1988,35 +1988,15 @@ static void pcpu_balance_free(enum pcpu_chunk_type type) * allocation causes the failure as it is possible that requests can be * serviced from already backed regions. */ -static void pcpu_balance_populated(enum pcpu_chunk_type type) +static void pcpu_grow_populated(enum pcpu_chunk_type type, int nr_to_pop) { /* gfp flags passed to underlying allocators */ const gfp_t gfp = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN; struct list_head *pcpu_slot = pcpu_chunk_list(type); struct pcpu_chunk *chunk; - int slot, nr_to_pop, ret; + int slot, ret; - /* - * Ensure there are certain number of free populated pages for - * atomic allocs. Fill up from the most packed so that atomic - * allocs don't increase fragmentation. If atomic allocation - * failed previously, always populate the maximum amount. This - * should prevent atomic allocs larger than PAGE_SIZE from keeping - * failing indefinitely; however, large atomic allocs are not - * something we support properly and can be highly unreliable and - * inefficient. - */ retry_pop: - if (pcpu_atomic_alloc_failed) { - nr_to_pop = PCPU_EMPTY_POP_PAGES_HIGH; - /* best effort anyway, don't worry about synchronization */ - pcpu_atomic_alloc_failed = false; - } else { - nr_to_pop = clamp(PCPU_EMPTY_POP_PAGES_HIGH - - pcpu_nr_empty_pop_pages[type], - 0, PCPU_EMPTY_POP_PAGES_HIGH); - } - for (slot = pcpu_size_to_slot(PAGE_SIZE); slot < pcpu_nr_slots; slot++) { unsigned int nr_unpop = 0, rs, re; @@ -2060,12 +2040,47 @@ static void pcpu_balance_populated(enum pcpu_chunk_type type) if (chunk) { spin_lock_irq(&pcpu_lock); pcpu_chunk_relocate(chunk, -1); + nr_to_pop = max_t(int, 0, nr_to_pop - chunk->nr_populated); spin_unlock_irq(&pcpu_lock); - goto retry_pop; + if (nr_to_pop) + goto retry_pop; } } } +/** + * pcpu_balance_populated - manage the amount of populated pages + * @type: chunk type + * + * Populate or depopulate chunks to maintain a certain amount + * of free pages to satisfy atomic allocations, but not waste + * large amounts of memory. + */ +static void pcpu_balance_populated(enum pcpu_chunk_type type) +{ + int nr_to_pop; + + /* + * Ensure there are certain number of free populated pages for + * atomic allocs. Fill up from the most packed so that atomic + * allocs don't increase fragmentation. If atomic allocation + * failed previously, always populate the maximum amount. This + * should prevent atomic allocs larger than PAGE_SIZE from keeping + * failing indefinitely; however, large atomic allocs are not + * something we support properly and can be highly unreliable and + * inefficient. + */ + if (pcpu_atomic_alloc_failed) { + nr_to_pop = PCPU_EMPTY_POP_PAGES_HIGH; + /* best effort anyway, don't worry about synchronization */ + pcpu_atomic_alloc_failed = false; + pcpu_grow_populated(type, nr_to_pop); + } else if (pcpu_nr_empty_pop_pages[type] < PCPU_EMPTY_POP_PAGES_HIGH) { + nr_to_pop = PCPU_EMPTY_POP_PAGES_HIGH - pcpu_nr_empty_pop_pages[type]; + pcpu_grow_populated(type, nr_to_pop); + } +} + /** * pcpu_balance_workfn - manage the amount of free chunks and populated pages * @work: unused From patchwork Thu Apr 8 03:57:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 12189913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C091C43460 for ; Thu, 8 Apr 2021 03:57:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C96F5610FA for ; Thu, 8 Apr 2021 03:57:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C96F5610FA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E7BFE6B0080; Wed, 7 Apr 2021 23:57:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DE7FA8D0001; Wed, 7 Apr 2021 23:57:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B01906B0082; Wed, 7 Apr 2021 23:57:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0003.hostedemail.com [216.40.44.3]) by kanga.kvack.org (Postfix) with ESMTP id 7AFD28E0003 for ; Wed, 7 Apr 2021 23:57:51 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 36B215DF5 for ; Thu, 8 Apr 2021 03:57:51 +0000 (UTC) X-FDA: 78007841142.18.28871F2 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf22.hostedemail.com (Postfix) with ESMTP id 98F51C0001EE for ; Thu, 8 Apr 2021 03:57:48 +0000 (UTC) Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 1383qcZL010009 for ; Wed, 7 Apr 2021 20:57:49 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=nfieNUNbNb1D09215aq0Md0ODibFOEFLCBSQhP0UblY=; b=Ei+F+GPAXOR6U7V8sokiEiHmPHl3xizv7f+HIBM04VR4nK3TayJLBK9RopBf9i3a5UDo Kq0KHkMK+f26xQi1M9w4ZL2apqcBTVTgbILwGMd985t/fA6sgCty2ER6aRK9sXmkXU02 qDTzAdVkfAUsLfK4Vfr/CDU5ZVfrC3tBJ+M= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com with ESMTP id 37sf0qm5cq-9 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 07 Apr 2021 20:57:49 -0700 Received: from intmgw002.46.prn1.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:11d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Wed, 7 Apr 2021 20:57:47 -0700 Received: by devvm3388.prn0.facebook.com (Postfix, from userid 111017) id 6D269602B014; Wed, 7 Apr 2021 20:57:41 -0700 (PDT) From: Roman Gushchin To: Dennis Zhou CC: Tejun Heo , Christoph Lameter , Andrew Morton , Vlastimil Babka , , , Roman Gushchin Subject: [PATCH v3 5/6] percpu: factor out pcpu_check_chunk_hint() Date: Wed, 7 Apr 2021 20:57:35 -0700 Message-ID: <20210408035736.883861-6-guro@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210408035736.883861-1-guro@fb.com> References: <20210408035736.883861-1-guro@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: DHrS6iLFNTzUkjMd-hG8o-O2oBUnPggq X-Proofpoint-ORIG-GUID: DHrS6iLFNTzUkjMd-hG8o-O2oBUnPggq X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.761 definitions=2021-04-08_01:2021-04-07,2021-04-08 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 suspectscore=0 adultscore=0 phishscore=0 impostorscore=0 mlxlogscore=999 clxscore=1015 priorityscore=1501 lowpriorityscore=0 malwarescore=0 spamscore=0 bulkscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104060000 definitions=main-2104080023 X-FB-Internal: deliver X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 98F51C0001EE X-Stat-Signature: haz14un7iefdgf95sqo7gcwz7fbx7p3g Received-SPF: none (fb.com>: No applicable sender policy available) receiver=imf22; identity=mailfrom; envelope-from=""; helo=mx0a-00082601.pphosted.com; client-ip=67.231.145.42 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617854268-265034 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Factor out the pcpu_check_chunk_hint() helper, which will be useful in the future. The new function checks if the allocation can likely fit the given chunk. Signed-off-by: Roman Gushchin --- mm/percpu.c | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/mm/percpu.c b/mm/percpu.c index e20119668c42..357fd6994278 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -306,6 +306,26 @@ static unsigned long pcpu_block_off_to_off(int index, int off) return index * PCPU_BITMAP_BLOCK_BITS + off; } +/** + * pcpu_check_chunk_hint - check that allocation can fit a chunk + * @chunk_md: chunk's block + * @bits: size of request in allocation units + * @align: alignment of area (max PAGE_SIZE) + * + * Check to see if the allocation can fit in the chunk's contig hint. + * This is an optimization to prevent scanning by assuming if it + * cannot fit in the global hint, there is memory pressure and creating + * a new chunk would happen soon. + */ +static bool pcpu_check_chunk_hint(struct pcpu_block_md *chunk_md, int bits, + size_t align) +{ + int bit_off = ALIGN(chunk_md->contig_hint_start, align) - + chunk_md->contig_hint_start; + + return bit_off + bits <= chunk_md->contig_hint; +} + /* * pcpu_next_hint - determine which hint to use * @block: block of interest @@ -1065,15 +1085,7 @@ static int pcpu_find_block_fit(struct pcpu_chunk *chunk, int alloc_bits, struct pcpu_block_md *chunk_md = &chunk->chunk_md; int bit_off, bits, next_off; - /* - * Check to see if the allocation can fit in the chunk's contig hint. - * This is an optimization to prevent scanning by assuming if it - * cannot fit in the global hint, there is memory pressure and creating - * a new chunk would happen soon. - */ - bit_off = ALIGN(chunk_md->contig_hint_start, align) - - chunk_md->contig_hint_start; - if (bit_off + alloc_bits > chunk_md->contig_hint) + if (!pcpu_check_chunk_hint(chunk_md, alloc_bits, align)) return -1; bit_off = pcpu_next_hint(chunk_md, alloc_bits); From patchwork Thu Apr 8 03:57:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 12189915 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A01CEC433B4 for ; Thu, 8 Apr 2021 03:58:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4B4B46113C for ; Thu, 8 Apr 2021 03:58:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4B4B46113C Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D74238D0001; Wed, 7 Apr 2021 23:58:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D23126B0085; Wed, 7 Apr 2021 23:58:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BC4F58D0001; Wed, 7 Apr 2021 23:58:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0066.hostedemail.com [216.40.44.66]) by kanga.kvack.org (Postfix) with ESMTP id 9C0716B0082 for ; Wed, 7 Apr 2021 23:58:11 -0400 (EDT) Received: from smtpin34.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 536EC8248047 for ; Thu, 8 Apr 2021 03:58:11 +0000 (UTC) X-FDA: 78007841982.34.D88C120 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf09.hostedemail.com (Postfix) with ESMTP id BA737600010D for ; Thu, 8 Apr 2021 03:58:08 +0000 (UTC) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 1383umT9008593 for ; Wed, 7 Apr 2021 20:58:09 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=ST/7cmPLqMeg4QVviVPk0E9T4usB55zMBAxd/+0O/f8=; b=FbX7NSNMf5CH8PkUCozWLlmkQ7P/X/O9fvmcWkywf9xc2BDOTCsGWAO2rLY4uLZmHzwI sdR5amD8u6HbpN8tbcqEmZsvOACvxPf9jKttGB3AoQMlToM1ExtMkgaUnuuaOHRk17H/ 7Mrvp07zBjvlH5NlTpiGDDJTY/7ixIsyKU8= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com with ESMTP id 37sg04kmx6-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 07 Apr 2021 20:58:09 -0700 Received: from intmgw001.46.prn1.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:11d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Wed, 7 Apr 2021 20:57:51 -0700 Received: by devvm3388.prn0.facebook.com (Postfix, from userid 111017) id 72783602B016; Wed, 7 Apr 2021 20:57:41 -0700 (PDT) From: Roman Gushchin To: Dennis Zhou CC: Tejun Heo , Christoph Lameter , Andrew Morton , Vlastimil Babka , , , Roman Gushchin Subject: [PATCH v3 6/6] percpu: implement partial chunk depopulation Date: Wed, 7 Apr 2021 20:57:36 -0700 Message-ID: <20210408035736.883861-7-guro@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210408035736.883861-1-guro@fb.com> References: <20210408035736.883861-1-guro@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: GF2iBRwO2h_B3ldxuep0yFBl6nfDJSoD X-Proofpoint-GUID: GF2iBRwO2h_B3ldxuep0yFBl6nfDJSoD X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.761 definitions=2021-04-08_01:2021-04-07,2021-04-08 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 adultscore=0 mlxscore=0 suspectscore=0 priorityscore=1501 bulkscore=0 lowpriorityscore=0 clxscore=1015 mlxlogscore=999 phishscore=0 malwarescore=0 spamscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104060000 definitions=main-2104080023 X-FB-Internal: deliver X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: BA737600010D X-Stat-Signature: ju7ca4fpfubg14qjsxq88bcoktbwtehs Received-SPF: none (fb.com>: No applicable sender policy available) receiver=imf09; identity=mailfrom; envelope-from=""; helo=mx0a-00082601.pphosted.com; client-ip=67.231.145.42 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617854288-962122 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch implements partial depopulation of percpu chunks. As now, a chunk can be depopulated only as a part of the final destruction, if there are no more outstanding allocations. However to minimize a memory waste it might be useful to depopulate a partially filed chunk, if a small number of outstanding allocations prevents the chunk from being fully reclaimed. This patch implements the following depopulation process: it scans over the chunk pages, looks for a range of empty and populated pages and performs the depopulation. To avoid races with new allocations, the chunk is previously isolated. After the depopulation the chunk is sidelined to a special list or freed. New allocations can't be served using a sidelined chunk. The chunk can be moved back to a corresponding slot if there are not enough chunks with empty populated pages. The depopulation is scheduled on the free path. Is the chunk: 1) has more than 1/4 of total pages free and populated 2) the system has enough free percpu pages aside of this chunk 3) isn't the reserved chunk 4) isn't the first chunk 5) isn't entirely free it's a good target for depopulation. If it's already depopulated but got free populated pages, it's a good target too. The chunk is moved to a special pcpu_depopulate_list, chunk->isolate flag is set and the async balancing is scheduled. The async balancing moves pcpu_depopulate_list to a local list (because pcpu_depopulate_list can be changed when pcpu_lock is releases), and then tries to depopulate each chunk. The depopulation is performed in the reverse direction to keep populated pages close to the beginning, if the global number of empty pages is reached. Depopulated chunks are sidelined to prevent further allocations. Skipped and fully empty chunks are returned to the corresponding slot. On the allocation path, if there are no suitable chunks found, the list of sidelined chunks in scanned prior to creating a new chunk. If there is a good sidelined chunk, it's placed back to the slot and the scanning is restarted. Many thanks to Dennis Zhou for his great ideas and a very constructive discussion which led to many improvements in this patchset! Signed-off-by: Roman Gushchin --- mm/percpu-internal.h | 2 + mm/percpu.c | 158 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 158 insertions(+), 2 deletions(-) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index 095d7eaa0db4..8e432663c41e 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -67,6 +67,8 @@ struct pcpu_chunk { void *data; /* chunk data */ bool immutable; /* no [de]population allowed */ + bool isolated; /* isolated from chunk slot lists */ + bool depopulated; /* sidelined after depopulation */ int start_offset; /* the overlap with the previous region to have a page aligned base_addr */ diff --git a/mm/percpu.c b/mm/percpu.c index 357fd6994278..5bb294e394b3 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -181,6 +181,19 @@ static LIST_HEAD(pcpu_map_extend_chunks); */ int pcpu_nr_empty_pop_pages[PCPU_NR_CHUNK_TYPES]; +/* + * List of chunks with a lot of free pages. Used to depopulate them + * asynchronously. + */ +static struct list_head pcpu_depopulate_list[PCPU_NR_CHUNK_TYPES]; + +/* + * List of previously depopulated chunks. They are not usually used for new + * allocations, but can be returned back to service if a need arises. + */ +static struct list_head pcpu_sideline_list[PCPU_NR_CHUNK_TYPES]; + + /* * The number of populated pages in use by the allocator, protected by * pcpu_lock. This number is kept per a unit per chunk (i.e. when a page gets @@ -562,6 +575,12 @@ static void pcpu_chunk_relocate(struct pcpu_chunk *chunk, int oslot) { int nslot = pcpu_chunk_slot(chunk); + /* + * Keep isolated and depopulated chunks on a sideline. + */ + if (chunk->isolated || chunk->depopulated) + return; + if (oslot != nslot) __pcpu_chunk_move(chunk, nslot, oslot < nslot); } @@ -1790,6 +1809,19 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, } } + /* search through sidelined depopulated chunks */ + list_for_each_entry(chunk, &pcpu_sideline_list[type], list) { + /* + * If the allocation can fit the chunk, place the chunk back + * into corresponding slot and restart the scanning. + */ + if (pcpu_check_chunk_hint(&chunk->chunk_md, bits, bit_align)) { + chunk->depopulated = false; + pcpu_chunk_relocate(chunk, -1); + goto restart; + } + } + spin_unlock_irqrestore(&pcpu_lock, flags); /* @@ -2060,6 +2092,106 @@ static void pcpu_grow_populated(enum pcpu_chunk_type type, int nr_to_pop) } } +/** + * pcpu_shrink_populated - scan chunks and release unused pages to the system + * @type: chunk type + * + * Scan over chunks in the depopulate list, try to release unused populated + * pages to the system. Depopulated chunks are sidelined to prevent further + * allocations without a need. Skipped and fully free chunks are returned + * to corresponding slots. Stop depopulating if the number of empty populated + * pages reaches the threshold. Each chunk is scanned in the reverse order to + * keep populated pages close to the beginning of the chunk. + */ +static void pcpu_shrink_populated(enum pcpu_chunk_type type) +{ + struct pcpu_block_md *block; + struct pcpu_chunk *chunk, *tmp; + LIST_HEAD(to_depopulate); + bool depopulated; + int i, end; + + spin_lock_irq(&pcpu_lock); + + list_splice_init(&pcpu_depopulate_list[type], &to_depopulate); + + list_for_each_entry_safe(chunk, tmp, &to_depopulate, list) { + WARN_ON(chunk->immutable); + depopulated = false; + + /* + * Scan chunk's pages in the reverse order to keep populated + * pages close to the beginning of the chunk. + */ + for (i = chunk->nr_pages - 1, end = -1; i >= 0; i--) { + /* + * If the chunk has no empty pages or + * we're short on empty pages in general, + * just put the chunk back into the original slot. + */ + if (!chunk->nr_empty_pop_pages || + pcpu_nr_empty_pop_pages[type] <= + PCPU_EMPTY_POP_PAGES_HIGH) + break; + + /* + * If the page is empty and populated, start or + * extend the (i, end) range. If i == 0, decrease + * i and perform the depopulation to cover the last + * (first) page in the chunk. + */ + block = chunk->md_blocks + i; + if (block->contig_hint == PCPU_BITMAP_BLOCK_BITS && + test_bit(i, chunk->populated)) { + if (end == -1) + end = i; + if (i > 0) + continue; + i--; + } + + /* + * Otherwise check if there is an active range, + * and if yes, depopulate it. + */ + if (end == -1) + continue; + + depopulated = true; + + spin_unlock_irq(&pcpu_lock); + pcpu_depopulate_chunk(chunk, i + 1, end + 1); + cond_resched(); + spin_lock_irq(&pcpu_lock); + + pcpu_chunk_depopulated(chunk, i + 1, end + 1); + + /* + * Reset the range and continue. + */ + end = -1; + } + + chunk->isolated = false; + if (chunk->free_bytes == pcpu_unit_size || !depopulated) { + /* + * If the chunk is empty or hasn't been depopulated, + * return it to the original slot. + */ + pcpu_chunk_relocate(chunk, -1); + } else { + /* + * Otherwise put the chunk to the list of depopulated + * chunks. + */ + chunk->depopulated = true; + list_move(&chunk->list, &pcpu_sideline_list[type]); + } + } + + spin_unlock_irq(&pcpu_lock); +} + /** * pcpu_balance_populated - manage the amount of populated pages * @type: chunk type @@ -2090,6 +2222,8 @@ static void pcpu_balance_populated(enum pcpu_chunk_type type) } else if (pcpu_nr_empty_pop_pages[type] < PCPU_EMPTY_POP_PAGES_HIGH) { nr_to_pop = PCPU_EMPTY_POP_PAGES_HIGH - pcpu_nr_empty_pop_pages[type]; pcpu_grow_populated(type, nr_to_pop); + } else if (!list_empty(&pcpu_depopulate_list[type])) { + pcpu_shrink_populated(type); } } @@ -2147,7 +2281,13 @@ void free_percpu(void __percpu *ptr) pcpu_memcg_free_hook(chunk, off, size); - /* if there are more than one fully free chunks, wake up grim reaper */ + /* + * If there are more than one fully free chunks, wake up grim reaper. + * Otherwise if at least 1/4 of its pages are empty and there is no + * system-wide shortage of empty pages aside from this chunk, isolate + * the chunk and schedule an async depopulation. If the chunk was + * depopulated previously and got free pages, depopulate it too. + */ if (chunk->free_bytes == pcpu_unit_size) { struct pcpu_chunk *pos; @@ -2156,6 +2296,16 @@ void free_percpu(void __percpu *ptr) need_balance = true; break; } + } else if (chunk != pcpu_first_chunk && chunk != pcpu_reserved_chunk && + !chunk->isolated && + (pcpu_nr_empty_pop_pages[pcpu_chunk_type(chunk)] > + PCPU_EMPTY_POP_PAGES_HIGH + chunk->nr_empty_pop_pages) && + ((chunk->depopulated && chunk->nr_empty_pop_pages) || + (chunk->nr_empty_pop_pages >= chunk->nr_pages / 4))) { + list_move(&chunk->list, &pcpu_depopulate_list[pcpu_chunk_type(chunk)]); + chunk->isolated = true; + chunk->depopulated = false; + need_balance = true; } trace_percpu_free_percpu(chunk->base_addr, off, ptr); @@ -2583,10 +2733,14 @@ void __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai, pcpu_nr_slots * sizeof(pcpu_chunk_lists[0]) * PCPU_NR_CHUNK_TYPES); - for (type = 0; type < PCPU_NR_CHUNK_TYPES; type++) + for (type = 0; type < PCPU_NR_CHUNK_TYPES; type++) { for (i = 0; i < pcpu_nr_slots; i++) INIT_LIST_HEAD(&pcpu_chunk_list(type)[i]); + INIT_LIST_HEAD(&pcpu_depopulate_list[type]); + INIT_LIST_HEAD(&pcpu_sideline_list[type]); + } + /* * The end of the static region needs to be aligned with the * minimum allocation size as this offsets the reserved and