From patchwork Tue Sep 13 19:55:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62187C6FA82 for ; Tue, 13 Sep 2022 19:59:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 013556B0099; Tue, 13 Sep 2022 15:59:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F2CE46B009A; Tue, 13 Sep 2022 15:59:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA79B8D0001; Tue, 13 Sep 2022 15:59:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CB7106B0099 for ; Tue, 13 Sep 2022 15:59:01 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id AE107A0BD7 for ; Tue, 13 Sep 2022 19:59:01 +0000 (UTC) X-FDA: 79908125682.11.095F15F Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) by imf22.hostedemail.com (Postfix) with ESMTP id 716C1C00C2 for ; Tue, 13 Sep 2022 19:59:01 +0000 (UTC) Received: by mail-qt1-f177.google.com with SMTP id s18so6629362qtx.6 for ; Tue, 13 Sep 2022 12:59:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=fHU5ZUx5v+GumCekHv98dt5HtboTaZeeu0Gp9AlVmvA=; b=ZkQM2Xi16fac9Kxsd6JfpJUyj9dFTpL2a9fMxPQYb5tsP1AGTq9LrS3GQ23PdeUUAq KiAJoUd0eFmdtlIe31jI3xWYF3lNu2z6iPVbkhYktSoNmKAD7mOQ11g6lhzdx8PeDKqG lPt1LWxw7/FsEvYMWeWC/86MaywQjplVNZrIXdmUyLe5Nr4lwjgNum55ZdLf4luz/yKw pKJIZBfhsG2I2a6u/v5b8+uct94R+RrovFIPHm9V4i65FXFauBq/I0QO/UcUPxNQEMMR iJaQjUMSz8SAuIngfV/EGkXgYLEYcOdGnoAVuWBboXNtLXNpt0Nqru8cUMiF0ZW7269J 6y+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=fHU5ZUx5v+GumCekHv98dt5HtboTaZeeu0Gp9AlVmvA=; b=5QQnZiVqaF84cL4Jksipdzk8BbCl7y+1r42iMM/KhEltjsgD2Ak2EjJTW8qEIImlnX 7qwgv3GpJ6BhquyDrGVhxgjxj/uNby88fxQJFIcBD/gUHJDgn3bumfrxiyEPddtS1DIc ShI1RFq2D+ISASvylgF9Cg2Pipf970sMlacyCSmTdkIQVSipPRdpaA0fQUle0SNsHMlJ HON3u+St/KSqCFlezitHUQhio6Tz6l2chfrOTyG4CauvSP8DcUABN0hJfoLpg8xIJ2+f k72IJ/Q+NvqmIK1UArEHpZBdLxZ+OhebCkxg2X6whStcfvJgCr1Z2gl6ILaK3iF4Gl+e 6RtQ== X-Gm-Message-State: ACgBeo3Rzn+5n0VO/y6d6YdsYaJ49pZxjn/2+TxN/eikSD3apWZGTbhr 9tEgamf9L4sdaaGseV/4s3w= X-Google-Smtp-Source: AA6agR7yxxGK0L7lb4UY6ElTAIAR0o3jzl9zqXW4dIwQkZBtshIlrdqdJVz3avbMktN2rc9zUTYXIg== X-Received: by 2002:ac8:570a:0:b0:343:73af:7cd8 with SMTP id 10-20020ac8570a000000b0034373af7cd8mr30203309qtw.16.1663099140678; Tue, 13 Sep 2022 12:59:00 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:59:00 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 18/21] mm/cma: support CMA in Designated Movable Blocks Date: Tue, 13 Sep 2022 12:55:05 -0700 Message-Id: <20220913195508.3511038-19-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099141; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fHU5ZUx5v+GumCekHv98dt5HtboTaZeeu0Gp9AlVmvA=; b=qz/yftUNlPoBpKWgTsGAAvlbknBIar/ih0Ihg96e//vvr0h7grv3mW1qyW1yroMu/hOTEh IgvpYt+KjZYcHXbQSDh4EnmZUWcxpr6reSCSFapY4h7vToLJ87M4syW5W6f684Yt9ZhJEM 1CE/J6jCIFQZNw3zYe2TBt268AosANc= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ZkQM2Xi1; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of opendmb@gmail.com designates 209.85.160.177 as permitted sender) smtp.mailfrom=opendmb@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099141; a=rsa-sha256; cv=none; b=iYaBytCU0l460WS2b64fg/WfknDuGKWjsDka98L8F0NrppLRAb9hC8hd77zvLTO93qxRMH W7P69UYoSttCytgY0ScCFfNm4x4569oFz4opSuvV4x3hm9vpBNevoOe3ySl+W/Fzfq/rcs TSOU/ntBSd9XcyYHnbqAwhy41GPk5jo= Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ZkQM2Xi1; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of opendmb@gmail.com designates 209.85.160.177 as permitted sender) smtp.mailfrom=opendmb@gmail.com X-Stat-Signature: r1a4xikucnzyrozmwa5i8kmnhadbc5ch X-Rspamd-Queue-Id: 716C1C00C2 X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1663099141-308227 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This commit allows for different page allocator handling for CMA areas that are within Designated Movable Blocks. Specifically, the pageblocks are allowed to remain migratetype MIGRATE_MOVABLE to allow more aggressive utilization by the page allocator. This also means that the page allocator should not consider these pages as part of the nr_free_cma metric it uses for managing MIGRATE_CMA type pageblocks. This leads to the decision to remove these areas from the CmaTotal metrics after initialization to avoid confusion. Signed-off-by: Doug Berger Reported-by: kernel test robot Reported-by: kernel test robot Reported-by: kernel test robot --- include/linux/cma.h | 13 ++++++--- mm/cma.c | 55 +++++++++++++++++++++++++----------- mm/page_alloc.c | 69 +++++++++++++++++++++++++++++---------------- 3 files changed, 92 insertions(+), 45 deletions(-) diff --git a/include/linux/cma.h b/include/linux/cma.h index 63873b93deaa..ffbb8ea2c5f8 100644 --- a/include/linux/cma.h +++ b/include/linux/cma.h @@ -31,11 +31,13 @@ extern phys_addr_t cma_get_base(const struct cma *cma); extern unsigned long cma_get_size(const struct cma *cma); extern const char *cma_get_name(const struct cma *cma); -extern int __init cma_declare_contiguous_nid(phys_addr_t base, +extern int __init __cma_declare_contiguous_nid(phys_addr_t base, phys_addr_t size, phys_addr_t limit, phys_addr_t alignment, unsigned int order_per_bit, bool fixed, const char *name, struct cma **res_cma, - int nid); + int nid, bool in_dmb); +#define cma_declare_contiguous_nid(b, s, l, a, o, f, n, r_c, nid) \ + __cma_declare_contiguous_nid(b, s, l, a, o, f, n, r_c, nid, false) static inline int __init cma_declare_contiguous(phys_addr_t base, phys_addr_t size, phys_addr_t limit, phys_addr_t alignment, unsigned int order_per_bit, @@ -44,10 +46,13 @@ static inline int __init cma_declare_contiguous(phys_addr_t base, return cma_declare_contiguous_nid(base, size, limit, alignment, order_per_bit, fixed, name, res_cma, NUMA_NO_NODE); } -extern int cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, +extern int __cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, unsigned int order_per_bit, const char *name, - struct cma **res_cma); + struct cma **res_cma, + bool in_dmb); +#define cma_init_reserved_mem(base, size, order, name, res_cma) \ + __cma_init_reserved_mem(base, size, order, name, res_cma, 0) extern struct page *cma_alloc(struct cma *cma, unsigned long count, unsigned int align, bool no_warn); extern bool cma_pages_valid(struct cma *cma, const struct page *pages, unsigned long count); diff --git a/mm/cma.c b/mm/cma.c index 6208a3e1cd9d..4f33cd54db9e 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -33,6 +33,7 @@ #include #include #include +#include #include "cma.h" @@ -98,6 +99,10 @@ static void __init cma_activate_area(struct cma *cma) { unsigned long base_pfn = cma->base_pfn, pfn; struct zone *zone; + int is_dmb = dmb_intersects(base_pfn, base_pfn + cma->count); + + if (is_dmb == DMB_MIXED) + goto out_error; cma->bitmap = bitmap_zalloc(cma_bitmap_maxno(cma), GFP_KERNEL); if (!cma->bitmap) @@ -116,13 +121,17 @@ static void __init cma_activate_area(struct cma *cma) goto not_in_zone; } - for (pfn = base_pfn; pfn < base_pfn + cma->count; - pfn += pageblock_nr_pages) { - struct page *page = pfn_to_page(pfn); + if (is_dmb == DMB_INTERSECTS) { + totalcma_pages -= cma->count; + } else { + for (pfn = base_pfn; pfn < base_pfn + cma->count; + pfn += pageblock_nr_pages) { + struct page *page = pfn_to_page(pfn); - set_pageblock_migratetype(page, MIGRATE_CMA); - init_reserved_pageblock(page); - page_zone(page)->cma_pages += pageblock_nr_pages; + set_pageblock_migratetype(page, MIGRATE_CMA); + init_reserved_pageblock(page); + page_zone(page)->cma_pages += pageblock_nr_pages; + } } spin_lock_init(&cma->lock); @@ -141,7 +150,8 @@ static void __init cma_activate_area(struct cma *cma) if (!cma->reserve_pages_on_error) { for (pfn = base_pfn; pfn < base_pfn + cma->count; pfn += pageblock_nr_pages) - init_reserved_pageblock(pfn_to_page(pfn)); + if (!dmb_intersects(pfn, pfn + pageblock_nr_pages)) + init_reserved_pageblock(pfn_to_page(pfn)); } totalcma_pages -= cma->count; cma->count = 0; @@ -166,7 +176,7 @@ void __init cma_reserve_pages_on_error(struct cma *cma) } /** - * cma_init_reserved_mem() - create custom contiguous area from reserved memory + * __cma_init_reserved_mem() - create custom contiguous area in reserved memory * @base: Base address of the reserved area * @size: Size of the reserved area (in bytes), * @order_per_bit: Order of pages represented by one bit on bitmap. @@ -174,15 +184,18 @@ void __init cma_reserve_pages_on_error(struct cma *cma) * the area will be set to "cmaN", where N is a running counter of * used areas. * @res_cma: Pointer to store the created cma region. + * @in_dmb: Designate the reserved memory as a Designated Movable Block. * * This function creates custom contiguous area from already reserved memory. */ -int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, - unsigned int order_per_bit, - const char *name, - struct cma **res_cma) +int __init __cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, + unsigned int order_per_bit, + const char *name, + struct cma **res_cma, + bool in_dmb) { struct cma *cma; + int err; /* Sanity checks */ if (cma_area_count == ARRAY_SIZE(cma_areas)) { @@ -201,6 +214,14 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, if (!IS_ALIGNED(base | size, CMA_MIN_ALIGNMENT_BYTES)) return -EINVAL; + if (in_dmb) { + err = dmb_reserve(base, size, NULL); + if (err) { + pr_err("Cannot reserve DMB for CMA!\n"); + return err; + } + } + /* * Each reserved area must be initialised later, when more kernel * subsystems (like slab allocator) are available. @@ -223,7 +244,7 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, } /** - * cma_declare_contiguous_nid() - reserve custom contiguous area + * __cma_declare_contiguous_nid() - reserve custom contiguous area * @base: Base address of the reserved area optional, use 0 for any * @size: Size of the reserved area (in bytes), * @limit: End address of the reserved memory (optional, 0 for any). @@ -233,6 +254,7 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, * @name: The name of the area. See function cma_init_reserved_mem() * @res_cma: Pointer to store the created cma region. * @nid: nid of the free area to find, %NUMA_NO_NODE for any node + * @in_dmb: Designate the reserved memory as a Designated Movable Block. * * This function reserves memory from early allocator. It should be * called by arch specific code once the early allocator (memblock or bootmem) @@ -242,11 +264,11 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, * If @fixed is true, reserve contiguous area at exactly @base. If false, * reserve in range from @base to @limit. */ -int __init cma_declare_contiguous_nid(phys_addr_t base, +int __init __cma_declare_contiguous_nid(phys_addr_t base, phys_addr_t size, phys_addr_t limit, phys_addr_t alignment, unsigned int order_per_bit, bool fixed, const char *name, struct cma **res_cma, - int nid) + int nid, bool in_dmb) { phys_addr_t memblock_end = memblock_end_of_DRAM(); phys_addr_t highmem_start; @@ -374,7 +396,8 @@ int __init cma_declare_contiguous_nid(phys_addr_t base, base = addr; } - ret = cma_init_reserved_mem(base, size, order_per_bit, name, res_cma); + ret = __cma_init_reserved_mem(base, size, order_per_bit, name, res_cma, + in_dmb); if (ret) goto free_mem; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e38dd1b32771..09d00c178bc8 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -9233,29 +9233,8 @@ int __alloc_contig_migrate_range(struct compact_control *cc, return 0; } -/** - * alloc_contig_range() -- tries to allocate given range of pages - * @start: start PFN to allocate - * @end: one-past-the-last PFN to allocate - * @migratetype: migratetype of the underlying pageblocks (either - * #MIGRATE_MOVABLE or #MIGRATE_CMA). All pageblocks - * in range must have the same migratetype and it must - * be either of the two. - * @gfp_mask: GFP mask to use during compaction - * - * The PFN range does not have to be pageblock aligned. The PFN range must - * belong to a single zone. - * - * The first thing this routine does is attempt to MIGRATE_ISOLATE all - * pageblocks in the range. Once isolated, the pageblocks should not - * be modified by others. - * - * Return: zero on success or negative error code. On success all - * pages which PFN is in [start, end) are allocated for the caller and - * need to be freed with free_contig_range(). - */ -int alloc_contig_range(unsigned long start, unsigned long end, - unsigned migratetype, gfp_t gfp_mask) +int _alloc_contig_range(unsigned long start, unsigned long end, + unsigned int migratetype, gfp_t gfp_mask) { unsigned long outer_start, outer_end; int order; @@ -9379,6 +9358,46 @@ int alloc_contig_range(unsigned long start, unsigned long end, undo_isolate_page_range(start, end, migratetype); return ret; } + +/** + * alloc_contig_range() -- tries to allocate given range of pages + * @start: start PFN to allocate + * @end: one-past-the-last PFN to allocate + * @migratetype: migratetype of the underlying pageblocks (either + * #MIGRATE_MOVABLE or #MIGRATE_CMA). All pageblocks + * in range must have the same migratetype and it must + * be either of the two. + * @gfp_mask: GFP mask to use during compaction + * + * The PFN range does not have to be pageblock aligned. The PFN range must + * belong to a single zone. + * + * The first thing this routine does is attempt to MIGRATE_ISOLATE all + * pageblocks in the range. Once isolated, the pageblocks should not + * be modified by others. + * + * Return: zero on success or negative error code. On success all + * pages which PFN is in [start, end) are allocated for the caller and + * need to be freed with free_contig_range(). + */ +int alloc_contig_range(unsigned long start, unsigned long end, + unsigned int migratetype, gfp_t gfp_mask) +{ + switch (dmb_intersects(start, end)) { + case DMB_DISJOINT: + break; + case DMB_INTERSECTS: + if (migratetype == MIGRATE_CMA) + migratetype = MIGRATE_MOVABLE; + else + return -EBUSY; + break; + default: + return -EBUSY; + } + + return _alloc_contig_range(start, end, migratetype, gfp_mask); +} EXPORT_SYMBOL(alloc_contig_range); static int __alloc_contig_pages(unsigned long start_pfn, @@ -9386,8 +9405,8 @@ static int __alloc_contig_pages(unsigned long start_pfn, { unsigned long end_pfn = start_pfn + nr_pages; - return alloc_contig_range(start_pfn, end_pfn, MIGRATE_MOVABLE, - gfp_mask); + return _alloc_contig_range(start_pfn, end_pfn, MIGRATE_MOVABLE, + gfp_mask); } static bool pfn_range_valid_contig(struct zone *z, unsigned long start_pfn,