From patchwork Tue Dec 15 19:36:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 11975663 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 749A5C4361B for ; Tue, 15 Dec 2020 19:36:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E5FDA22B2D for ; Tue, 15 Dec 2020 19:36:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E5FDA22B2D Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 45C6E6B0036; Tue, 15 Dec 2020 14:36:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 40C936B005D; Tue, 15 Dec 2020 14:36:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2FB386B0068; Tue, 15 Dec 2020 14:36:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0236.hostedemail.com [216.40.44.236]) by kanga.kvack.org (Postfix) with ESMTP id 173536B0036 for ; Tue, 15 Dec 2020 14:36:23 -0500 (EST) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id BFA458249980 for ; Tue, 15 Dec 2020 19:36:22 +0000 (UTC) X-FDA: 77596523004.07.paste85_291674c27426 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id AE1F91803F9B0 for ; Tue, 15 Dec 2020 19:36:22 +0000 (UTC) X-HE-Tag: paste85_291674c27426 X-Filterd-Recvd-Size: 9338 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Tue, 15 Dec 2020 19:36:21 +0000 (UTC) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 0BFJVQmT015563 for ; Tue, 15 Dec 2020 11:36:20 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : mime-version : content-transfer-encoding : content-type; s=facebook; bh=XiyLS6boex6NduO3Y3HlXsJcjGSnFa1JrSnocOQadMg=; b=UgOwBbqxibA3GU5iadoMx1tYi3LO/XUT4tVj5ZOKe6XVk9NxlroegagZFsBrMfYzhhAW 4xdYhf9yJOePsPc4+wwhCV7M+SKP6ly189HFhGM8UISbmVycDipdpVy/wcwxk5k3UqoE e/sO6MoYQhR9+na3fJ0d6kLh8XWtHfUrnXw= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com with ESMTP id 35desucsxa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 15 Dec 2020 11:36:20 -0800 Received: from intmgw001.41.prn1.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:21d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1979.3; Tue, 15 Dec 2020 11:36:19 -0800 Received: by devvm3388.prn0.facebook.com (Postfix, from userid 111017) id AC6701FE954D; Tue, 15 Dec 2020 11:36:16 -0800 (PST) From: Roman Gushchin To: Andrew Morton , Mike Rapoport , CC: Joonsoo Kim , Rik van Riel , Michal Hocko , , , Roman Gushchin Subject: [PATCH] mm: cma: allocate cma areas bottom-up Date: Tue, 15 Dec 2020 11:36:15 -0800 Message-ID: <20201215193615.1867115-1-guro@fb.com> X-Mailer: git-send-email 2.24.1 MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2020-12-15_12:2020-12-15,2020-12-15 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 spamscore=0 mlxlogscore=999 phishscore=0 priorityscore=1501 impostorscore=0 malwarescore=0 adultscore=0 lowpriorityscore=0 suspectscore=0 mlxscore=0 bulkscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2012150130 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently cma areas without a fixed base address are allocated close to the end of the node. This placement is sub-optimal because of how the compaction works: it effectively moves pages into the cma area. In particular, it often brings in hot executable pages, even if there is a plenty of free memory on the machine. This results in more cma allocation failures. Instead let's place cma areas close to the beginning of a node. Cma first tries to start with highmem_start, so we shouldn't mess up with DMA32. In this case the compaction will help to free cma areas, resulting in better cma allocation success rates. Signed-off-by: Roman Gushchin --- include/linux/memblock.h | 5 +++-- mm/cma.c | 4 ++-- mm/memblock.c | 26 +++++++++++++++----------- 3 files changed, 20 insertions(+), 15 deletions(-) diff --git a/include/linux/memblock.h b/include/linux/memblock.h index 9c5cc95c7cee..698188066450 100644 --- a/include/linux/memblock.h +++ b/include/linux/memblock.h @@ -384,8 +384,9 @@ static inline int memblock_get_region_node(const struct memblock_region *r) phys_addr_t memblock_phys_alloc_range(phys_addr_t size, phys_addr_t align, phys_addr_t start, phys_addr_t end); phys_addr_t memblock_alloc_range_nid(phys_addr_t size, - phys_addr_t align, phys_addr_t start, - phys_addr_t end, int nid, bool exact_nid); + phys_addr_t align, phys_addr_t start, + phys_addr_t end, int nid, bool exact_nid, + bool bottom_up); phys_addr_t memblock_phys_alloc_try_nid(phys_addr_t size, phys_addr_t align, int nid); static inline phys_addr_t memblock_phys_alloc(phys_addr_t size, diff --git a/mm/cma.c b/mm/cma.c index 20c4f6f40037..1b42be6d059b 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -332,13 +332,13 @@ int __init cma_declare_contiguous_nid(phys_addr_t base, */ if (base < highmem_start && limit > highmem_start) { addr = memblock_alloc_range_nid(size, alignment, - highmem_start, limit, nid, true); + highmem_start, limit, nid, true, true); limit = highmem_start; } if (!addr) { addr = memblock_alloc_range_nid(size, alignment, base, - limit, nid, true); + limit, nid, true, true); if (!addr) { ret = -ENOMEM; goto err; diff --git a/mm/memblock.c b/mm/memblock.c index b8b7be0561c4..c334b401fe16 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -272,6 +272,7 @@ __memblock_find_range_top_down(phys_addr_t start, phys_addr_t end, * %MEMBLOCK_ALLOC_ACCESSIBLE * @nid: nid of the free area to find, %NUMA_NO_NODE for any node * @flags: pick from blocks based on memory attributes + * @bottom_up: force bottom-up allocation * * Find @size free area aligned to @align in the specified range and node. * @@ -289,7 +290,8 @@ __memblock_find_range_top_down(phys_addr_t start, phys_addr_t end, static phys_addr_t __init_memblock memblock_find_in_range_node(phys_addr_t size, phys_addr_t align, phys_addr_t start, phys_addr_t end, int nid, - enum memblock_flags flags) + enum memblock_flags flags, + bool bottom_up) { phys_addr_t kernel_end, ret; @@ -305,9 +307,10 @@ static phys_addr_t __init_memblock memblock_find_in_range_node(phys_addr_t size, /* * try bottom-up allocation only when bottom-up mode - * is set and @end is above the kernel image. + * is set and @end is above the kernel image or + * the bottom-up mode is enforced. */ - if (memblock_bottom_up() && end > kernel_end) { + if ((memblock_bottom_up() && end > kernel_end) || bottom_up) { phys_addr_t bottom_up_start; /* make sure we will allocate above the kernel */ @@ -359,7 +362,7 @@ phys_addr_t __init_memblock memblock_find_in_range(phys_addr_t start, again: ret = memblock_find_in_range_node(size, align, start, end, - NUMA_NO_NODE, flags); + NUMA_NO_NODE, flags, false); if (!ret && (flags & MEMBLOCK_MIRROR)) { pr_warn("Could not allocate %pap bytes of mirrored memory\n", @@ -1331,6 +1334,7 @@ __next_mem_pfn_range_in_zone(u64 *idx, struct zone *zone, * @end: the upper bound of the memory region to allocate (phys address) * @nid: nid of the free area to find, %NUMA_NO_NODE for any node * @exact_nid: control the allocation fall back to other nodes + * @bottom_up: force bottom-up allocation * * The allocation is performed from memory region limited by * memblock.current_limit if @end == %MEMBLOCK_ALLOC_ACCESSIBLE. @@ -1351,7 +1355,7 @@ __next_mem_pfn_range_in_zone(u64 *idx, struct zone *zone, phys_addr_t __init memblock_alloc_range_nid(phys_addr_t size, phys_addr_t align, phys_addr_t start, phys_addr_t end, int nid, - bool exact_nid) + bool exact_nid, bool bottom_up) { enum memblock_flags flags = choose_memblock_flags(); phys_addr_t found; @@ -1367,14 +1371,14 @@ phys_addr_t __init memblock_alloc_range_nid(phys_addr_t size, again: found = memblock_find_in_range_node(size, align, start, end, nid, - flags); + flags, bottom_up); if (found && !memblock_reserve(found, size)) goto done; if (nid != NUMA_NO_NODE && !exact_nid) { found = memblock_find_in_range_node(size, align, start, end, NUMA_NO_NODE, - flags); + flags, bottom_up); if (found && !memblock_reserve(found, size)) goto done; } @@ -1423,7 +1427,7 @@ phys_addr_t __init memblock_phys_alloc_range(phys_addr_t size, __func__, (u64)size, (u64)align, &start, &end, (void *)_RET_IP_); return memblock_alloc_range_nid(size, align, start, end, NUMA_NO_NODE, - false); + false, false); } /** @@ -1442,7 +1446,7 @@ phys_addr_t __init memblock_phys_alloc_range(phys_addr_t size, phys_addr_t __init memblock_phys_alloc_try_nid(phys_addr_t size, phys_addr_t align, int nid) { return memblock_alloc_range_nid(size, align, 0, - MEMBLOCK_ALLOC_ACCESSIBLE, nid, false); + MEMBLOCK_ALLOC_ACCESSIBLE, nid, false, false); } /** @@ -1484,12 +1488,12 @@ static void * __init memblock_alloc_internal( max_addr = memblock.current_limit; alloc = memblock_alloc_range_nid(size, align, min_addr, max_addr, nid, - exact_nid); + exact_nid, false); /* retry allocation without lower limit */ if (!alloc && min_addr) alloc = memblock_alloc_range_nid(size, align, 0, max_addr, nid, - exact_nid); + exact_nid, false); if (!alloc) return NULL;