From patchwork Mon Apr 10 08:46:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "zhaoyang.huang" X-Patchwork-Id: 13206195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0802EC76196 for ; Mon, 10 Apr 2023 08:48:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 52A3F280008; Mon, 10 Apr 2023 04:48:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D9DA280002; Mon, 10 Apr 2023 04:48:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A1D2280008; Mon, 10 Apr 2023 04:48:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2A190280002 for ; Mon, 10 Apr 2023 04:48:50 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E469D16027E for ; Mon, 10 Apr 2023 08:48:49 +0000 (UTC) X-FDA: 80664855978.26.96541FC Received: from SHSQR01.spreadtrum.com (unknown [222.66.158.135]) by imf30.hostedemail.com (Postfix) with ESMTP id 06B3B80004 for ; Mon, 10 Apr 2023 08:48:46 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf30.hostedemail.com: domain of zhaoyang.huang@unisoc.com designates 222.66.158.135 as permitted sender) smtp.mailfrom=zhaoyang.huang@unisoc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681116528; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references; bh=RpsPoJZeo3673tgyArEa9y/kXzGIl+uRGEifWWA+y/g=; b=cY0jwkhPziupoVsDLXL3W7bMqUZIAa3AezEpOmAWIPq0yYNA/eSe2qplcqK0stbr6eCEsw a7roULKcibZyvk4apMk8g3Qb9ww4cYRi1vmYJ9K4ucvWb2nSBadKeNFM69e3xaEq3EBrSs F+7paw7puli+12bGRmt+6fKxTlHhAWw= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf30.hostedemail.com: domain of zhaoyang.huang@unisoc.com designates 222.66.158.135 as permitted sender) smtp.mailfrom=zhaoyang.huang@unisoc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681116528; a=rsa-sha256; cv=none; b=UI2OZEQjQKMIYmPxPu1pZEft4XV8HBLVgbrD6ysANXKWns5D2pCu9wR1DLOsG4F2U/iNVc q9rcqwuBEm0pm8urRlhMWaUqHFwaH0ovJHZJOl8H/c+vOp8gzEnhFRVFbDeQDf4jIs+WsZ OS6+AwZzG+UCeoLNWb3cjH5KBuIyTko= Received: from SHSend.spreadtrum.com (bjmbx01.spreadtrum.com [10.0.64.7]) by SHSQR01.spreadtrum.com with ESMTP id 33A8krjN036709; Mon, 10 Apr 2023 16:46:53 +0800 (+08) (envelope-from zhaoyang.huang@unisoc.com) Received: from bj03382pcu.spreadtrum.com (10.0.74.65) by BJMBX01.spreadtrum.com (10.0.64.7) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Mon, 10 Apr 2023 16:46:50 +0800 From: "zhaoyang.huang" To: Andrew Morton , Minchan Kim , Joonsoo Kim , , , Zhaoyang Huang , Subject: [RFC PATCHv2] mm: introduce defer free for cma Date: Mon, 10 Apr 2023 16:46:35 +0800 Message-ID: <1681116395-18633-1-git-send-email-zhaoyang.huang@unisoc.com> X-Mailer: git-send-email 1.9.1 MIME-Version: 1.0 X-Originating-IP: [10.0.74.65] X-ClientProxiedBy: SHCAS03.spreadtrum.com (10.0.1.207) To BJMBX01.spreadtrum.com (10.0.64.7) X-MAIL: SHSQR01.spreadtrum.com 33A8krjN036709 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 06B3B80004 X-Rspam-User: X-Stat-Signature: f3kwqn5oegekh6pw1j9mihokihru3f9i X-HE-Tag: 1681116526-296487 X-HE-Meta: U2FsdGVkX1/YaIhrK7jeY4OokIIadxSiR4gEKYbiHXG/RlXuh9vu2uv8M8s3pYq7IjWM7uu6sERUlhiPjwRq3rkRxMN9tzEwEESExwWTbzuBG6JJNJ+kDVC612+HV5lNARhGP2MM4gbczSuWzfj+cPIj8/dpTB8u8NtU6Nr/vOfzmMpNSfPCdBm8EzAd3eTHzntqvFjRfwbQp9rzAOU7nwcBBLQItuL1oxu41obzo+/ernh9Zy/47tUKRNCYC3CKgZVo3drok5euAkZ3feeRu0+3fPrvoECVuSItUyrbhvxuW78NzQD4qvMmhqibkj7ijHgbs6mHm50VXY088SlUHmLSNXHSBpoUQEzD+7i2oI/u4ChiPWs6agVnYek1615ESEzFt+iJ71L3MIkmdqP2MyUX4hf49gnk2CXf8q5HnFwd8hP8S5R8NZ2BWgcOvE6eWE9SIuaHy0oa3fNztwfXDi6XOSOmsR7vlAAWwWwTKoSchsEvow7JTiUnU1xM/IjIsWJQz0r1urvXJHU2JSoOZENYbUNTd++v+FjN/E7CWHPY7BjA3rBQONIsxMaKVTwy6T151xuPbJaDN9jVWPFzCuVRvWcRxLmAvEmh3+xSGzl902qw/jIKr0Fj242QHO0e/a50Cc5EpQdLktahqj9szPXJEoeGAWdey52aUVfe2/5+9Rvsdp8OTeK/K+Bbj0fCqJdaVU9EXbulHe8sQTAstNgIQx6vKfNIC33XlxNqR7PdSf0TfZ5nQZILT526PIkfnKK6yihP7ddNEafA+5C9GjLXSUMAF6Bmh2l9n0MMZYBxbjX23HLGdUZH3nTkhTJTjie7hFuD9DPzF7rwMCl0F2muOm9rAYNiEXt4+epqC7trLR7bZ/2X+HpIrbbhfQfxq4scH9oySQSV2x3klQslw12EdQ8DCfuurvvEVLNV1CDSDhFhcHZRxny/VoMCNJOI7JfA9upC+bz8eyKGqwP +IsKPVJL LrnabAgKft/UdF9ejg8mJvErEsPKRGIefMU6ZnnGrYQKpbTOtL5t99BJX8/DP22Ive27aL8MsnjaDWsg/uEeAFMOrQpuO6bGfp8E25MF9dsFI++qkY02oOz7dSEVh9OpVQ4CnvzYefWkG0Ym2f8Tc3GFYB4LKdjrLmxIgEe55VhpHuHMjDkTmprU9ihe5ueKYbxc/8bnuaB+hjw0RqFdf96PStCExQcXC/u7hhYRVpcORL1u2H45EG/S/0MK7faDr+sJ7ygW2YbzzGZMmlZaNs1pD3UTO5Q9k6yXZF08ej5C+hJb9VjlI2djsKe3U/OpHo0Bt X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Zhaoyang Huang Continues page blocks are expensive for the system. Introducing defer free mechanism to buffer some which make the allocation easier. The shrinker will ensure the page block can be reclaimed when there is memory pressure. Signed-off-by: Zhaoyang Huang --- v2: fix build warning and regist shrinker --- --- mm/cma.c | 151 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- mm/cma.h | 11 +++++ 2 files changed, 160 insertions(+), 2 deletions(-) diff --git a/mm/cma.c b/mm/cma.c index 4a978e0..6d2fd24 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -39,6 +39,10 @@ unsigned cma_area_count; static DEFINE_MUTEX(cma_mutex); +static unsigned long cma_defer_free_count(struct shrinker *shrinker, + struct shrink_control *sc); +static unsigned long cma_defer_free_scan(struct shrinker *shrinker, + struct shrink_control *sc); phys_addr_t cma_get_base(const struct cma *cma) { return PFN_PHYS(cma->base_pfn); @@ -153,6 +157,20 @@ static int __init cma_init_reserved_areas(void) } core_initcall(cma_init_reserved_areas); +static unsigned long cma_free_get(struct cma *cma) +{ + unsigned long used; + unsigned long val; + + spin_lock_irq(&cma->lock); + /* pages counter is smaller than sizeof(int) */ + used = bitmap_weight(cma->bitmap, (int)cma_bitmap_maxno(cma)); + val = cma->count - ((u64)used << cma->order_per_bit); + spin_unlock_irq(&cma->lock); + + return val; +} + void __init cma_reserve_pages_on_error(struct cma *cma) { cma->reserve_pages_on_error = true; @@ -212,6 +230,13 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, cma_area_count++; totalcma_pages += (size / PAGE_SIZE); + cma->batch = cma->count >> 1; + cma->shrinker.count_objects = cma_defer_free_count; + cma->shrinker.scan_objects = cma_defer_free_scan; + cma->shrinker.seeks = DEFAULT_SEEKS; + cma->shrinker.batch = 0; + + register_shrinker(&cma->shrinker, "cma-shrinker"); return 0; } @@ -411,6 +436,46 @@ static void cma_debug_show_areas(struct cma *cma) static inline void cma_debug_show_areas(struct cma *cma) { } #endif +static int cma_defer_area_fetch(struct cma *cma, unsigned long pfn, + unsigned long count) +{ + struct cma_defer_free_area *area; + unsigned long new_pfn; + int ret = -1; + + if (!atomic64_read(&cma->defer_count)) + return ret; + if (count <= atomic64_read(&cma->defer_count)) { + spin_lock_irq(&cma->lock); + list_for_each_entry(area, &cma->defer_free, list) { + /*area found for given pfn and count*/ + if (pfn >= area->pfn && count <= area->count) { + list_del(&area->list); + /*set bits for allocated pfn*/ + bitmap_set(cma->bitmap, pfn - cma->base_pfn, count); + kfree(area); + atomic64_sub(count, &cma->defer_count); + /*release the rest pfn to cma*/ + if (!list_empty(&cma->defer_free) && (pfn == area->pfn)) { + new_pfn = pfn + count; + cma_release(cma, pfn_to_page(new_pfn), area->count - count); + } + ret = 0; + spin_unlock_irq(&cma->lock); + return ret; + } + } + } + /*no area found, release all to buddy*/ + list_for_each_entry(area, &cma->defer_free, list) { + list_del(&area->list); + free_contig_range(area->pfn, area->count); + cma_clear_bitmap(cma, area->pfn, area->count); + kfree(area); + } + spin_unlock_irq(&cma->lock); + return ret; +} /** * cma_alloc() - allocate pages from contiguous area * @cma: Contiguous memory region for which the allocation is performed. @@ -469,9 +534,11 @@ struct page *cma_alloc(struct cma *cma, unsigned long count, spin_unlock_irq(&cma->lock); pfn = cma->base_pfn + (bitmap_no << cma->order_per_bit); + mutex_lock(&cma_mutex); - ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA, - GFP_KERNEL | (no_warn ? __GFP_NOWARN : 0)); + /*search defer area first*/ + ret = cma_defer_area_fetch(cma, pfn, count) ? alloc_contig_range(pfn, pfn + count, MIGRATE_CMA, + GFP_KERNEL | (no_warn ? __GFP_NOWARN : 0)) : 0; mutex_unlock(&cma_mutex); if (ret == 0) { page = pfn_to_page(pfn); @@ -556,6 +623,8 @@ bool cma_release(struct cma *cma, const struct page *pages, unsigned long count) { unsigned long pfn; + unsigned long flags; + struct cma_defer_free_area *defer_area; if (!cma_pages_valid(cma, pages, count)) return false; @@ -566,6 +635,19 @@ bool cma_release(struct cma *cma, const struct page *pages, VM_BUG_ON(pfn + count > cma->base_pfn + cma->count); + if (atomic64_read(&cma->defer_count) < cma->batch) { + defer_area = kmalloc(sizeof(struct cma_defer_free_area), GFP_KERNEL); + if (defer_area) { + defer_area->pfn = pfn; + defer_area->count = count; + spin_lock_irqsave(&cma->lock, flags); + list_add(&defer_area->list, &cma->defer_free); + atomic64_add(count, &cma->defer_count); + spin_unlock_irqrestore(&cma->lock, flags); + cma_clear_bitmap(cma, pfn, count); + return true; + } + } free_contig_range(pfn, count); cma_clear_bitmap(cma, pfn, count); trace_cma_release(cma->name, pfn, pages, count); @@ -586,3 +668,68 @@ int cma_for_each_area(int (*it)(struct cma *cma, void *data), void *data) return 0; } + +static unsigned long cma_defer_free_count(struct shrinker *shrinker, + struct shrink_control *sc) +{ + struct cma *cma = container_of(shrinker, struct cma, shrinker); + unsigned long val; + + val = atomic64_read(&cma->defer_count); + return val; +} + +static unsigned long cma_defer_free_scan(struct shrinker *shrinker, + struct shrink_control *sc) +{ + struct cma *cma = container_of(shrinker, struct cma, shrinker); + unsigned long to_scan; + struct cma_defer_free_area *area; + unsigned long new_pfn; + unsigned long defer_count; + + if (sc->nr_to_scan < cma->batch) + return 0; + + to_scan = cma->batch - sc->nr_to_scan; + defer_count = atomic64_read(&cma->defer_count); + spin_lock_irq(&cma->lock); + + /*large to_scan, free all node*/ + if (to_scan >= defer_count) { + list_for_each_entry(area, &cma->defer_free, list) { + list_del(&area->list); + free_contig_range(area->pfn, area->count); + cma_clear_bitmap(cma, area->pfn, area->count); + kfree(area); + } + atomic64_set(&cma->defer_count, 0); + return defer_count; + } + /*iterate all defer_area*/ + list_for_each_entry(area, &cma->defer_free, list) { + if (to_scan <= area->count) { + list_del(&area->list); + free_contig_range(area->pfn, area->count); + cma_clear_bitmap(cma, area->pfn, area->count); + kfree(area); + atomic64_sub(to_scan, &cma->defer_count); + /*release the rest pfn to cma*/ + if (!list_empty(&cma->defer_free)) { + new_pfn = area->pfn + to_scan; + cma_release(cma, pfn_to_page(new_pfn), area->count - to_scan); + } + break; + } + else { + list_del(&area->list); + free_contig_range(area->pfn, area->count); + cma_clear_bitmap(cma, area->pfn, area->count); + kfree(area); + to_scan = to_scan - atomic64_read(&cma->defer_count); + continue; + } + } + spin_unlock_irq(&cma->lock); + return 0; +} diff --git a/mm/cma.h b/mm/cma.h index 88a0595..e1e3e2f 100644 --- a/mm/cma.h +++ b/mm/cma.h @@ -4,6 +4,7 @@ #include #include +#include struct cma_kobject { struct kobject kobj; @@ -31,6 +32,16 @@ struct cma { struct cma_kobject *cma_kobj; #endif bool reserve_pages_on_error; + struct list_head defer_free; + atomic64_t defer_count; + unsigned long batch; + struct shrinker shrinker; +}; + +struct cma_defer_free_area { + unsigned long pfn; + unsigned long count; + struct list_head list; }; extern struct cma cma_areas[MAX_CMA_AREAS];