From patchwork Tue Sep 13 19:54:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975309 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99BE3C6FA8A for ; Tue, 13 Sep 2022 19:57:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 236FD6B0074; Tue, 13 Sep 2022 15:57:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1E7AB6B0075; Tue, 13 Sep 2022 15:57:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0119B8D0001; Tue, 13 Sep 2022 15:57:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E67866B0074 for ; Tue, 13 Sep 2022 15:57:58 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C2A1AA09EF for ; Tue, 13 Sep 2022 19:57:58 +0000 (UTC) X-FDA: 79908123036.18.603B6A5 Received: from mail-qt1-f174.google.com (mail-qt1-f174.google.com [209.85.160.174]) by imf02.hostedemail.com (Postfix) with ESMTP id 7557C800A6 for ; Tue, 13 Sep 2022 19:57:58 +0000 (UTC) Received: by mail-qt1-f174.google.com with SMTP id r20so8904689qtn.12 for ; Tue, 13 Sep 2022 12:57:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=rWqwJImtlBlKgE2e4HCMCjZ57hDnUIYZvYnGoYO3Ko0=; b=d2+jGtD0TKLMdHImIPIOFRc/CCsLBd1zTd7Z32acL/4ee7Q///DRfLAeWKVby8PAWE fgb6z8hXf1k5PM+mERKCcjYlhJUc7YcVmkzbKybhsAlNUjEqr+U7/BKoMqeCMB/HNIHI g9CIZx3cpM7r7QtrN3niww/63rwDZ4ABYIOm7CcpFGEgc/xw03gOtoM8VQgFq7CdtHOy PnM3xHgjGzGq4Cl6QAxH02OPTmRMVzFrDgXO8l6Rjf0jEf15sxaNGCsaRPE9K0g1rGX0 Q9q2x8LOpIavofodITDiPKTwnIXo7FGcbxZ+dhF4xLlkpGof9K3nuM7AOF386w+Nu9xd TF4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=rWqwJImtlBlKgE2e4HCMCjZ57hDnUIYZvYnGoYO3Ko0=; b=E4q8uETcyRhl/rxjIYl8LbDEqMOr9/tyYAaulowHpdAkShI1Iovy8qQEgA1wfPoXbN asEzTLKHjPHehMww01MZ0yKPmRyoCRoM/rBMAcyIiavuNcoqw1mdLwDR52FODH5bOTp4 u2xisgClksS72W8YukDFYsvgjMsa6cBGWj4XqiyiTKP5skx41gx0B11DBMdOxBKEiDPR kZxZRkS/qIsJr4sjbA3JZzwmWZrkX8BySlps9378wXPLu2LyaZNi05KzMjXifcPEwvUm +WJ901DWHLnLJfKej67A9fwx+44uGaISX3q4R0L95vbD6XNJSoal9PHUzdhkwsUzVqQ3 Q1Dw== X-Gm-Message-State: ACgBeo2hLkHcUeWOSob2NWFXlTRIOKps2TLf8Zw0slBS1vjqnAnr7hel yeMYF/pS05KHpYDQ7dEhr/w= X-Google-Smtp-Source: AA6agR5i53agCunM1nP5BNWdgMMXAxgKBBgHP+YifFsQRT8ONLZddir0hPCAFH311ymBKWli946Oxg== X-Received: by 2002:a05:622a:40e:b0:343:7769:5895 with SMTP id n14-20020a05622a040e00b0034377695895mr30562094qtx.467.1663099078015; Tue, 13 Sep 2022 12:57:58 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.57.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:57:57 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 01/21] mm/page_isolation: protect cma from isolate_single_pageblock Date: Tue, 13 Sep 2022 12:54:48 -0700 Message-Id: <20220913195508.3511038-2-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099078; a=rsa-sha256; cv=none; b=nWqPTrwfZrsxmqCBrcGd5BQscpHpT9PzyMxyfBEFjqnJo3VrOp00iACeF+LTngtIoGxr3t ELcKMCM1j4r/fBGTrIN3eA5ZtcTPsmY2IYRyYjAEuIMChyde64/bz1IVNYrb/xrG11T0ed tDwR/EZbKmvTXDweYTDlJyvZ3iXaCVE= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=d2+jGtD0; spf=pass (imf02.hostedemail.com: domain of opendmb@gmail.com designates 209.85.160.174 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099078; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rWqwJImtlBlKgE2e4HCMCjZ57hDnUIYZvYnGoYO3Ko0=; b=3fnYma6ufhw5ZOZVPZ9C2j/oBkUsnvwzXRBhil0pRU41Y7I4btWpbAuDT1Vv4gcKln5/X7 C496dZhBX0siZb3QD2L2FUKXZkrAnckcvR5aomyqISGDj05x2vM7Mmg4vJmw7V5ncxYSbB dhoz77qPjAaPK1kN5FfOBFogRuWfua8= X-Rspam-User: Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=d2+jGtD0; spf=pass (imf02.hostedemail.com: domain of opendmb@gmail.com designates 209.85.160.174 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam02 X-Stat-Signature: st568sre5m4o7wfqaxw1atfif5pn1yhw X-Rspamd-Queue-Id: 7557C800A6 X-HE-Tag: 1663099078-229508 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The function set_migratetype_isolate() has special handling for pageblocks of MIGRATE_CMA type that protects them from being isolated for MIGRATE_MOVABLE requests. Since isolate_single_pageblock() doesn't receive the migratetype argument of start_isolate_page_range() it used the migratetype of the pageblock instead of the requested migratetype which defeats this MIGRATE_CMA check. This allows an attempt to create a gigantic page within a CMA region to change the migratetype of the first and last pageblocks from MIGRATE_CMA to MIGRATE_MOVABLE when they are restored after failure, which corrupts the CMA region. The calls to (un)set_migratetype_isolate() for the first and last pageblocks of the start_isolate_page_range() are moved back into that function to allow access to its migratetype argument and make it easier to see how all of the pageblocks in the range are isolated. Fixes: b2c9e2fbba32 ("mm: make alloc_contig_range work at pageblock granularity") Signed-off-by: Doug Berger Reported-by: kernel test robot --- mm/page_isolation.c | 75 +++++++++++++++++++++------------------------ 1 file changed, 35 insertions(+), 40 deletions(-) diff --git a/mm/page_isolation.c b/mm/page_isolation.c index 9d73dc38e3d7..8e16aa22cb61 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -286,8 +286,6 @@ __first_valid_page(unsigned long pfn, unsigned long nr_pages) * @flags: isolation flags * @gfp_flags: GFP flags used for migrating pages * @isolate_before: isolate the pageblock before the boundary_pfn - * @skip_isolation: the flag to skip the pageblock isolation in second - * isolate_single_pageblock() * * Free and in-use pages can be as big as MAX_ORDER-1 and contain more than one * pageblock. When not all pageblocks within a page are isolated at the same @@ -302,9 +300,8 @@ __first_valid_page(unsigned long pfn, unsigned long nr_pages) * the in-use page then splitting the free page. */ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, - gfp_t gfp_flags, bool isolate_before, bool skip_isolation) + gfp_t gfp_flags, bool isolate_before) { - unsigned char saved_mt; unsigned long start_pfn; unsigned long isolate_pageblock; unsigned long pfn; @@ -328,18 +325,6 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, start_pfn = max(ALIGN_DOWN(isolate_pageblock, MAX_ORDER_NR_PAGES), zone->zone_start_pfn); - saved_mt = get_pageblock_migratetype(pfn_to_page(isolate_pageblock)); - - if (skip_isolation) - VM_BUG_ON(!is_migrate_isolate(saved_mt)); - else { - ret = set_migratetype_isolate(pfn_to_page(isolate_pageblock), saved_mt, flags, - isolate_pageblock, isolate_pageblock + pageblock_nr_pages); - - if (ret) - return ret; - } - /* * Bail out early when the to-be-isolated pageblock does not form * a free or in-use page across boundary_pfn: @@ -428,7 +413,7 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, ret = set_migratetype_isolate(page, page_mt, flags, head_pfn, head_pfn + nr_pages); if (ret) - goto failed; + return ret; } ret = __alloc_contig_migrate_range(&cc, head_pfn, @@ -443,7 +428,7 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, unset_migratetype_isolate(page, page_mt); if (ret) - goto failed; + return -EBUSY; /* * reset pfn to the head of the free page, so * that the free page handling code above can split @@ -459,24 +444,19 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, while (!PageBuddy(pfn_to_page(outer_pfn))) { /* stop if we cannot find the free page */ if (++order >= MAX_ORDER) - goto failed; + return -EBUSY; outer_pfn &= ~0UL << order; } pfn = outer_pfn; continue; } else #endif - goto failed; + return -EBUSY; } pfn++; } return 0; -failed: - /* restore the original migratetype */ - if (!skip_isolation) - unset_migratetype_isolate(pfn_to_page(isolate_pageblock), saved_mt); - return -EBUSY; } /** @@ -534,21 +514,30 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, unsigned long isolate_start = ALIGN_DOWN(start_pfn, pageblock_nr_pages); unsigned long isolate_end = ALIGN(end_pfn, pageblock_nr_pages); int ret; - bool skip_isolation = false; /* isolate [isolate_start, isolate_start + pageblock_nr_pages) pageblock */ - ret = isolate_single_pageblock(isolate_start, flags, gfp_flags, false, skip_isolation); + ret = set_migratetype_isolate(pfn_to_page(isolate_start), migratetype, + flags, isolate_start, isolate_start + pageblock_nr_pages); if (ret) return ret; - - if (isolate_start == isolate_end - pageblock_nr_pages) - skip_isolation = true; + ret = isolate_single_pageblock(isolate_start, flags, gfp_flags, false); + if (ret) + goto unset_start_block; /* isolate [isolate_end - pageblock_nr_pages, isolate_end) pageblock */ - ret = isolate_single_pageblock(isolate_end, flags, gfp_flags, true, skip_isolation); + pfn = isolate_end - pageblock_nr_pages; + if (isolate_start != pfn) { + ret = set_migratetype_isolate(pfn_to_page(pfn), migratetype, + flags, pfn, pfn + pageblock_nr_pages); + if (ret) + goto unset_start_block; + } + ret = isolate_single_pageblock(isolate_end, flags, gfp_flags, true); if (ret) { - unset_migratetype_isolate(pfn_to_page(isolate_start), migratetype); - return ret; + if (isolate_start != pfn) + goto unset_end_block; + else + goto unset_start_block; } /* skip isolated pageblocks at the beginning and end */ @@ -557,15 +546,21 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, pfn += pageblock_nr_pages) { page = __first_valid_page(pfn, pageblock_nr_pages); if (page && set_migratetype_isolate(page, migratetype, flags, - start_pfn, end_pfn)) { - undo_isolate_page_range(isolate_start, pfn, migratetype); - unset_migratetype_isolate( - pfn_to_page(isolate_end - pageblock_nr_pages), - migratetype); - return -EBUSY; - } + start_pfn, end_pfn)) + goto unset_isolated_blocks; } return 0; + +unset_isolated_blocks: + ret = -EBUSY; + undo_isolate_page_range(isolate_start + pageblock_nr_pages, pfn, + migratetype); +unset_end_block: + unset_migratetype_isolate(pfn_to_page(isolate_end - pageblock_nr_pages), + migratetype); +unset_start_block: + unset_migratetype_isolate(pfn_to_page(isolate_start), migratetype); + return ret; } /* From patchwork Tue Sep 13 19:54:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975310 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89E3EC6FA8A for ; Tue, 13 Sep 2022 19:58:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 062436B0075; Tue, 13 Sep 2022 15:58:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 011A96B0078; Tue, 13 Sep 2022 15:58:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DCDC38D0001; Tue, 13 Sep 2022 15:58:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C68D76B0075 for ; Tue, 13 Sep 2022 15:58:02 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A08681A0B8B for ; Tue, 13 Sep 2022 19:58:02 +0000 (UTC) X-FDA: 79908123204.02.326B5CF Received: from mail-qt1-f171.google.com (mail-qt1-f171.google.com [209.85.160.171]) by imf23.hostedemail.com (Postfix) with ESMTP id 6343A1400B5 for ; Tue, 13 Sep 2022 19:58:02 +0000 (UTC) Received: by mail-qt1-f171.google.com with SMTP id g12so9221976qts.1 for ; Tue, 13 Sep 2022 12:58:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=Ae0x/cYtHCzUmIbnlOleUipnU3BBydliG8CbSfbHHfI=; b=lQy156yj9m33v2+lsCppN/1SSiL1+HSKUeGu4OKDvKzsWQP9uaxSZeRF6KYff4/ZX1 T8ar9K8mrC7poQnpcOlicqZBPrnyrX4lQmy/efXceAo8c/r+O7+kebNQPiRqxM6b8RHk nqDdqIjYqMl2UkmBaLyC3jNoxqdBf+LTtzsYTc4L2U/V/TdvsthKIhCb1FgggCffFcVL 4UCRPgfEkwehGvOaibbczRrkl3etaXkViPMT1ZepP3U6LDqZx2jP70X41REaFOYLplOf cOzr55DR2HMCT6kc/P98YRx0lPP2AD9Ny8hyMOuXORVZJrRjvmaKZrJZ1BXySGhbtpI9 HDig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=Ae0x/cYtHCzUmIbnlOleUipnU3BBydliG8CbSfbHHfI=; b=bzKzWy8H14Zk8DITKAGF/blnSh7JuRkj5HDTvZdwxB8ovuPr6eZ2pT8jmQqe/yH0g6 xIMXoaqeHyQmfZBGYqtGddIXHhmvX1Bf5Qo+lZdEX9G2udcKRlKH3rT4CyDhov5oGDPj K5wvHVvK+8Nv86898t/CWyxQcccb6Fmnj+w+EdBjHJoc3g4r1opZym1OKwAw0HbUGcGg bVQqrrEGIBftCtjgA7MMQYgibpiS54kQVZe/wTRJLMKg8xanloNdQZkVeBI4SzDqRx8n iokpvn6KiP3neAcYQ1koEwfV3aTF/ad7Yg+OzvsKhf4waiFPSUoeFJkbh0s6x4dwrvVy JOnA== X-Gm-Message-State: ACgBeo2xj6xv6veS9k+cK7pbds2QxqKwbPZiyKW/M34fnhUJqoGBVls2 bx4fX1+s7yeC4p+yZ/wjlPQ= X-Google-Smtp-Source: AA6agR5RQ3Vz73r+e9pp4sPmIM/iMa/zHS6QVY1LvvDFqMY99XQUTUB/nmtdr1ce7F9gfpRjTDrD5Q== X-Received: by 2002:ac8:5790:0:b0:35b:fd92:6fb8 with SMTP id v16-20020ac85790000000b0035bfd926fb8mr4051143qta.53.1663099081690; Tue, 13 Sep 2022 12:58:01 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.57.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:01 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 02/21] mm/hugetlb: correct max_huge_pages accounting on demote Date: Tue, 13 Sep 2022 12:54:49 -0700 Message-Id: <20220913195508.3511038-3-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099082; a=rsa-sha256; cv=none; b=EOPkkuvICGLy8Z6WO9mx/GxVZCM/sh0nSIgtxzUafrOn/cA1a/oeu/ZTy4Ps2vKakSbsCY oAAVrxKXYnQUGdYHG0f8L1LjcoddXV4CSx4p3dJZp1+o+ad5Ygof7aJ3RNo7FnwoDPKstU WNgPRCIpiRRTAmgUN/8rhbuanCMOJlk= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=lQy156yj; spf=pass (imf23.hostedemail.com: domain of opendmb@gmail.com designates 209.85.160.171 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099082; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ae0x/cYtHCzUmIbnlOleUipnU3BBydliG8CbSfbHHfI=; b=d+iuKDTbhn0I0Ht14g0arcDE1U2fimKi09B7VXXSkjHkPt+V/3s7dtE1xi45GqoqUd8gkJ acAWZ6fptg2PKYUwTKQIbUO+Nv2u3bogb6canmS7ktnV9NAi9tbdzIlJxuMd7PnMwpMzLE ZvtQGATEHKDnhetfiTZUebe5FAkxRXQ= X-Rspam-User: Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=lQy156yj; spf=pass (imf23.hostedemail.com: domain of opendmb@gmail.com designates 209.85.160.171 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam02 X-Stat-Signature: t87rctg6h8sqwr6qbjwocabewowudqcc X-Rspamd-Queue-Id: 6343A1400B5 X-HE-Tag: 1663099082-425423 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When demoting a hugepage to a smaller order, the number of pages added to the target hstate will be the size of the large page divided by the size of the smaller page. Fixes: 8531fc6f52f5 ("hugetlb: add hugetlb demote page support") Signed-off-by: Doug Berger --- mm/hugetlb.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index e070b8593b37..79949893ac12 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3472,7 +3472,8 @@ static int demote_free_huge_page(struct hstate *h, struct page *page) * based on pool changes for the demoted page. */ h->max_huge_pages--; - target_hstate->max_huge_pages += pages_per_huge_page(h); + target_hstate->max_huge_pages += pages_per_huge_page(h) / + pages_per_huge_page(target_hstate); return rc; } From patchwork Tue Sep 13 19:54:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975311 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 673D8C6FA82 for ; Tue, 13 Sep 2022 19:58:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E6BBB6B007B; Tue, 13 Sep 2022 15:58:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E1BB08D0001; Tue, 13 Sep 2022 15:58:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE3166B007E; Tue, 13 Sep 2022 15:58:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BEBAD6B007B for ; Tue, 13 Sep 2022 15:58:06 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 9867FA09EF for ; Tue, 13 Sep 2022 19:58:06 +0000 (UTC) X-FDA: 79908123372.25.2998B18 Received: from mail-qv1-f44.google.com (mail-qv1-f44.google.com [209.85.219.44]) by imf01.hostedemail.com (Postfix) with ESMTP id 58A534009D for ; Tue, 13 Sep 2022 19:58:06 +0000 (UTC) Received: by mail-qv1-f44.google.com with SMTP id ml1so10115837qvb.1 for ; Tue, 13 Sep 2022 12:58:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=dgAC9wdSWdNnQd7c6lUo6ch6gbgfZ8Ik+m13B5omrFE=; b=pgU7qB1cfn3bbb6aqaPKVsP0QULztBXM6ZTdmlLajCShk8VBE78cxzG8Cv0iNse9pr raJ8GgLPcDyEkXzvMHViIvbgw2EPxCCCxmBWVydWXsbvHPXhV9LnTIQcxrhx+uvf4Xru 5WYEg1qQVSMXl/Ik2LXwDiiTB+uN5XnQ7jp32HYymQ2HZJZADmguB3tnteKHk9qIukRc jpn8LdBlaPcxTkVAmOKzkV+NLvIVifwROYLBZ5KLKak/eoCCJdA+KloOJgYdmZJeYMVG xPc5vPtISXEXXxdKiktdhoBVSWbwV6cguyuU1u/JwyBG7hVEPpa11URawNPjZH6cK9Db YImw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=dgAC9wdSWdNnQd7c6lUo6ch6gbgfZ8Ik+m13B5omrFE=; b=s9KXVGQ2dSZ7drNJhocvRMTzWZxqBhqEv2MB+jtk6c3er/6Mb9fKe2H6rfkgajvEt2 aqKUlW18op/LaramVBLNx9i7CyVewBIL0mU/j2SamJQ/0xgk9zdA6k5woWT5DJp1ifzU pXSWU6APSDuqzTbe80F3mowDb+rBElPFGROKRtbhJlXSCeXy+4umACYdPIT4mtIvEfYX msv5r3kSDGtU724m1ABkqO6Uzlf0tuh3ZpazdHRHmG6GYzCCXkUtmE+drRui42z++pN9 fCPYEVZYtI4UucHPJohg4pkc1dcO2CPHNFd3PJ9f5HYQYWtQP0vUTBF/rq+hKzqScAV1 dEdA== X-Gm-Message-State: ACgBeo1XRI8QCAmBJZqGouaUUMIv/qNHUkHfNnibDqzs483S/hnsg4Ml wBjiYR8IDCXBiaDfjLVQTJI= X-Google-Smtp-Source: AA6agR5N0IDxlcIGSy6m2BrDrpoJ44hk3cGJOE0+/g5noyEETZjDxRdlg2yegkHVtpvIIdmASzximw== X-Received: by 2002:a0c:a901:0:b0:4aa:a283:ef4a with SMTP id y1-20020a0ca901000000b004aaa283ef4amr28465376qva.53.1663099085481; Tue, 13 Sep 2022 12:58:05 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:04 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 03/21] mm/hugetlb: correct demote page offset logic Date: Tue, 13 Sep 2022 12:54:50 -0700 Message-Id: <20220913195508.3511038-4-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=pgU7qB1c; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of opendmb@gmail.com designates 209.85.219.44 as permitted sender) smtp.mailfrom=opendmb@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099086; a=rsa-sha256; cv=none; b=LAT37yN9RDGn+7Y8f9Zx6SUCl7ynyDfeRdJ3aVOxeHRqNGtjWdo2oN1lRsVtghH4pOgEXI kqaM24vEi2Oe022RBk+GkEy+1VfRpkPV25FwgInyiI3gKmHnvmTAl+xo6S0A/NRaN42V1O qib5Zr4cYHH7VTcjBHWxCnJJ1K1CBoc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099086; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dgAC9wdSWdNnQd7c6lUo6ch6gbgfZ8Ik+m13B5omrFE=; b=t5UqZuoNjsMu7qF1ZO9msg6QPDhRN4KAj2OrhkqQip/RtNWiIy5TqLfnMmTlMkk1Vbhw/j ziIg4Cjx3BClb2HhekPdNiiajfyOf7zwIjQSeJhVZ/DI87DCcwNbtOLm76T90eZr+F7/yf jBS8Kn9AEpze97QJjrbESMP7dPpD2Ck= X-Rspam-User: Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=pgU7qB1c; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of opendmb@gmail.com designates 209.85.219.44 as permitted sender) smtp.mailfrom=opendmb@gmail.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 58A534009D X-Stat-Signature: xdese6toqen3qmr6ptit9hmb3xhn7y1f X-HE-Tag: 1663099086-482691 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With gigantic pages it may not be true that struct page structures are contiguous across the entire gigantic page. The mem_map_offset function is used here in place of direct pointer arithmetic to correct for this. Fixes: 8531fc6f52f5 ("hugetlb: add hugetlb demote page support") Signed-off-by: Doug Berger Acked-by: Muchun Song --- mm/hugetlb.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 79949893ac12..a1d51a1f0404 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3420,6 +3420,7 @@ static int demote_free_huge_page(struct hstate *h, struct page *page) { int i, nid = page_to_nid(page); struct hstate *target_hstate; + struct page *subpage; int rc = 0; target_hstate = size_to_hstate(PAGE_SIZE << h->demote_order); @@ -3453,15 +3454,16 @@ static int demote_free_huge_page(struct hstate *h, struct page *page) mutex_lock(&target_hstate->resize_lock); for (i = 0; i < pages_per_huge_page(h); i += pages_per_huge_page(target_hstate)) { + subpage = mem_map_offset(page, i); if (hstate_is_gigantic(target_hstate)) - prep_compound_gigantic_page_for_demote(page + i, + prep_compound_gigantic_page_for_demote(subpage, target_hstate->order); else - prep_compound_page(page + i, target_hstate->order); - set_page_private(page + i, 0); - set_page_refcounted(page + i); - prep_new_huge_page(target_hstate, page + i, nid); - put_page(page + i); + prep_compound_page(subpage, target_hstate->order); + set_page_private(subpage, 0); + set_page_refcounted(subpage); + prep_new_huge_page(target_hstate, subpage, nid); + put_page(subpage); } mutex_unlock(&target_hstate->resize_lock); From patchwork Tue Sep 13 19:54:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975312 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DCBCDC6FA82 for ; Tue, 13 Sep 2022 19:58:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8027E6B007E; Tue, 13 Sep 2022 15:58:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7B24B6B0080; Tue, 13 Sep 2022 15:58:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 62C6F6B0081; Tue, 13 Sep 2022 15:58:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 516AB6B007E for ; Tue, 13 Sep 2022 15:58:10 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2F02CA027F for ; Tue, 13 Sep 2022 19:58:10 +0000 (UTC) X-FDA: 79908123540.02.7E1AB1A Received: from mail-qk1-f177.google.com (mail-qk1-f177.google.com [209.85.222.177]) by imf07.hostedemail.com (Postfix) with ESMTP id E142F40089 for ; Tue, 13 Sep 2022 19:58:09 +0000 (UTC) Received: by mail-qk1-f177.google.com with SMTP id h28so9342516qka.0 for ; Tue, 13 Sep 2022 12:58:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=yHgeZ0+Ocv6GrFoaruOXhrYrRwLmKzzR+EPUoXoCFLk=; b=MO7TGhjFAweThFGVYteWrV+nJ7j5y3LZRa4C7ACnJ1qWyIG54wf5kg5kO3xvw/FqDQ s4wKsy9yFNOEBO191kejMvdr64z7Q9PD3aPBFXDJT69Mpap/Qm7BagM/tmOC2sbVk5Ek f646vwKxfSqOitl6VKXJ3CuYfAma7EN7IglgyNYbVTkgDe6A8Ejr2VK5Z6ft+DXrx+l4 fQRhw7+DxWu3uXxtthnmrlFhvdM2zwLPoqSspHMHQaTzl30h2Y5TuDS9T3a1oeXGiuia JD/Wvi1/kbP+I5YXUEfkPCFyR+Q73ACUQMRNqUffeivDuZhRpUvnXKzfC7RG3PK+ZTef LsyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=yHgeZ0+Ocv6GrFoaruOXhrYrRwLmKzzR+EPUoXoCFLk=; b=VNiPp38XRMEm23d+EHFRzcvTYsSNWP4dxPAPKMUPxNsAsUV0IhMoZPLas4p1U1IVRY 3WmBykX0uKVGmoOs8vvJ2QFywOzEWNG0C6pFd7Whb25PaVdOKpo/9QBhQNCxOPxZNppY 6Abg0z0zP2KOXUtedoRvHJSjOgyCxUqikXfNi3nxk1XwolxkmeT0ZIRcL4b7hcwmK1Tf 8M7EjMGmOFD6XI50jnHCawOqaluSREBWpebybV+B3mNp10v1mZGAqzl/Iy6Wt6CN+5vp S9t691kG82OebH6bPzDSHmezMLtKhatdNBuIb9nAGemFhP9PC+m3TMU98I9Qwc5hcEMA Lu4w== X-Gm-Message-State: ACgBeo3F7T4BGT7cpIWSl+Bz0u4daRMyjGqJtePMcbKDM6ayjJ4n8VuR upFq7RFuTS8Igi8JMfjBINI= X-Google-Smtp-Source: AA6agR5MEfLRKDjJcxpoJCjM2jEAW8H06DRSio04kdGXrEcQ2RvULscAsN8q5RMRJ4UP1SunQTdfRQ== X-Received: by 2002:a05:620a:488f:b0:6ce:634e:8963 with SMTP id ea15-20020a05620a488f00b006ce634e8963mr3462100qkb.16.1663099089149; Tue, 13 Sep 2022 12:58:09 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:08 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 04/21] mm/hugetlb: refactor alloc_and_dissolve_huge_page Date: Tue, 13 Sep 2022 12:54:51 -0700 Message-Id: <20220913195508.3511038-5-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099089; a=rsa-sha256; cv=none; b=AXSjXmw1FqR4moHh5QREhQOhemOnjCC/h8jf83/BR6Wt/MkNZoNwo4Y3ej/tR7nhB2dBcx 3HggbSZtzZdDxjTnYd7Kjq1Kx/ekNBMqqRJ3VvkbcXT2IHGJ3gEEj05J+TQcQodd0T5g66 QmT6vYAey9bObbAW9gjAaWO5iR5M8Lo= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=MO7TGhjF; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of opendmb@gmail.com designates 209.85.222.177 as permitted sender) smtp.mailfrom=opendmb@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099089; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yHgeZ0+Ocv6GrFoaruOXhrYrRwLmKzzR+EPUoXoCFLk=; b=Tb6BK8dhWE407KrCeDWDVPE6NWkCm+9d/fJrmkASg2/HWeU1ZACy9jQmWLTgn4N1v/icwz RDUGizEQ5zeI6nYXfi7rQh7JwJmapIRaAG5E2j1pV2oz2LuiC7iCrbuEuc/Kd9xOb12n+t nwmuTclRevlHkobTGcjVj3x7CiWwvk0= Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=MO7TGhjF; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of opendmb@gmail.com designates 209.85.222.177 as permitted sender) smtp.mailfrom=opendmb@gmail.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: E142F40089 X-Stat-Signature: 7hr387p3cmy98q844kbk5bkrub51f18w X-Rspam-User: X-HE-Tag: 1663099089-872306 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The alloc_replacement_page() and replace_hugepage() functions are created from code in the alloc_and_dissolve_huge_page() function to allow their reuse by the next commit. Signed-off-by: Doug Berger --- mm/hugetlb.c | 84 +++++++++++++++++++++++++++++++--------------------- 1 file changed, 51 insertions(+), 33 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a1d51a1f0404..f232a37df4b6 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2709,32 +2709,22 @@ void restore_reserve_on_error(struct hstate *h, struct vm_area_struct *vma, } /* - * alloc_and_dissolve_huge_page - Allocate a new page and dissolve the old one - * @h: struct hstate old page belongs to - * @old_page: Old page to dissolve - * @list: List to isolate the page in case we need to - * Returns 0 on success, otherwise negated error. + * Before dissolving the page, we need to allocate a new one for the + * pool to remain stable. Here, we allocate the page and 'prep' it + * by doing everything but actually updating counters and adding to + * the pool. This simplifies and let us do most of the processing + * under the lock. */ -static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, - struct list_head *list) +static struct page *alloc_replacement_page(struct hstate *h, int nid) { gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE; - int nid = page_to_nid(old_page); bool alloc_retry = false; struct page *new_page; - int ret = 0; - /* - * Before dissolving the page, we need to allocate a new one for the - * pool to remain stable. Here, we allocate the page and 'prep' it - * by doing everything but actually updating counters and adding to - * the pool. This simplifies and let us do most of the processing - * under the lock. - */ alloc_retry: new_page = alloc_buddy_huge_page(h, gfp_mask, nid, NULL, NULL); if (!new_page) - return -ENOMEM; + return ERR_PTR(-ENOMEM); /* * If all goes well, this page will be directly added to the free * list in the pool. For this the ref count needs to be zero. @@ -2748,7 +2738,7 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, SetHPageTemporary(new_page); if (!put_page_testzero(new_page)) { if (alloc_retry) - return -EBUSY; + return ERR_PTR(-EBUSY); alloc_retry = true; goto alloc_retry; @@ -2757,6 +2747,48 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, __prep_new_huge_page(h, new_page); + return new_page; +} + +static void replace_hugepage(struct hstate *h, int nid, struct page *old_page, + struct page *new_page) +{ + lockdep_assert_held(&hugetlb_lock); + /* + * Ok, old_page is still a genuine free hugepage. Remove it from + * the freelist and decrease the counters. These will be + * incremented again when calling __prep_account_new_huge_page() + * and enqueue_huge_page() for new_page. The counters will remain + * stable since this happens under the lock. + */ + remove_hugetlb_page(h, old_page, false); + + /* + * Ref count on new page is already zero as it was dropped + * earlier. It can be directly added to the pool free list. + */ + __prep_account_new_huge_page(h, nid); + enqueue_huge_page(h, new_page); +} + +/* + * alloc_and_dissolve_huge_page - Allocate a new page and dissolve the old one + * @h: struct hstate old page belongs to + * @old_page: Old page to dissolve + * @list: List to isolate the page in case we need to + * Returns 0 on success, otherwise negated error. + */ +static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, + struct list_head *list) +{ + int nid = page_to_nid(old_page); + struct page *new_page; + int ret = 0; + + new_page = alloc_replacement_page(h, nid); + if (IS_ERR(new_page)) + return PTR_ERR(new_page); + retry: spin_lock_irq(&hugetlb_lock); if (!PageHuge(old_page)) { @@ -2783,21 +2815,7 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, cond_resched(); goto retry; } else { - /* - * Ok, old_page is still a genuine free hugepage. Remove it from - * the freelist and decrease the counters. These will be - * incremented again when calling __prep_account_new_huge_page() - * and enqueue_huge_page() for new_page. The counters will remain - * stable since this happens under the lock. - */ - remove_hugetlb_page(h, old_page, false); - - /* - * Ref count on new page is already zero as it was dropped - * earlier. It can be directly added to the pool free list. - */ - __prep_account_new_huge_page(h, nid); - enqueue_huge_page(h, new_page); + replace_hugepage(h, nid, old_page, new_page); /* * Pages have been replaced, we can safely free the old one. From patchwork Tue Sep 13 19:54:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975313 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6DB6ECAAD8 for ; Tue, 13 Sep 2022 19:58:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 51DFE6B0080; Tue, 13 Sep 2022 15:58:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F45E6B0081; Tue, 13 Sep 2022 15:58:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 346476B0082; Tue, 13 Sep 2022 15:58:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2629C6B0080 for ; Tue, 13 Sep 2022 15:58:14 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 042CF1C6486 for ; Tue, 13 Sep 2022 19:58:13 +0000 (UTC) X-FDA: 79908123666.30.DE81FD6 Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42]) by imf28.hostedemail.com (Postfix) with ESMTP id B921FC00C1 for ; Tue, 13 Sep 2022 19:58:13 +0000 (UTC) Received: by mail-qv1-f42.google.com with SMTP id v15so10086365qvi.11 for ; Tue, 13 Sep 2022 12:58:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=ivekPqFwDTKnplQubqOcOnwSj5SwqgEqKMLIx8G0mTA=; b=kvUC28iyEDcMoby9xrnJHhocivoBZj9gqY+ui6eXHvBbq0tXS2dzYAAmqBdXCNT1wy Q0fTtOM1kwBq1neZJJDtv8OVb2E4WTXaookEyrn9UWfnw0gFlPMPSsLKdxzfJXEQRdps TBY/dxDfKDXG5nzytFERu/dU7OZ7FBxZvIzoh4dkW/QYs2YW41IfiKdkkwlFW8nnTbi2 RA0eP5SekM1KC2ny0+Tc+vzS7nujN6k/2bx9b8Rj/IE1ke6REKO17fe0+no3v3V+6DPQ hTcXXufvBklDdoPK3SevxPEu7k9onHuyK1D6iBkExepS0BnC5h2ewbJv8V7xNvAFJRmS S5FA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=ivekPqFwDTKnplQubqOcOnwSj5SwqgEqKMLIx8G0mTA=; b=lqXVCYH+YcLp+Mjg5k9GEtbZsGXqR9OmqW2fwoTERSJf/mY1GJMDtAbAV8kvDxczG5 GPgapFSfalCClkS4+mIWNmHkCM2yl844FXvrp+L8S0Nv3H/CHp3omZfyYPMGOug2eIja LJzI+as1JGl36kVZurcgXj2By6zK0D+JY26SUfp5LDGIEGRSqOVYoRxPS2Z1+nuTNWzs PZb02LE6sWsoWMrnOmbqu5c+ltnidNS/JMsBgdHtpyUVxswFLOoec8Of9+ujLL5Usl5Z e3Qj4OeVx/s6dH/AEiKetf22eq1rzAEj2BMw9edw5u+8jTPQmAKsDK4XV2yvqYncV5UY 4wtQ== X-Gm-Message-State: ACgBeo34QG02sPC6JsT9Nc4KJQZicWyT37oYYSgl0A7fnqvyEw01cuz8 fsnp7LoJzmS0GuY0L1PdmMU= X-Google-Smtp-Source: AA6agR5Uplm7zgaGZn1afr+syf2bXEr/tRhasRhXystutC9gt9duBJzLy1nlXABisQhehw9c0OtbIg== X-Received: by 2002:a05:6214:1c09:b0:4ac:9160:7484 with SMTP id u9-20020a0562141c0900b004ac91607484mr17181443qvc.13.1663099092959; Tue, 13 Sep 2022 12:58:12 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:12 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 05/21] mm/hugetlb: allow migrated hugepage to dissolve when freed Date: Tue, 13 Sep 2022 12:54:52 -0700 Message-Id: <20220913195508.3511038-6-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099093; a=rsa-sha256; cv=none; b=v8l4ExIkOjhUKLu/jZn46Yc7PWNMjwCyc6OioVo8gwy6aHVWzFqTe5BNwlFgbH2GlHC6ST 6rozTl4IxHB5o2zkuazv7AFnPWNmB/UBF2xt5AhbLn6cnvdrjszV/64IlNA2y+MIo9nK0u rPM2Deutcjz5tTbmI866MhdptnLAnMM= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=kvUC28iy; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of opendmb@gmail.com designates 209.85.219.42 as permitted sender) smtp.mailfrom=opendmb@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099093; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ivekPqFwDTKnplQubqOcOnwSj5SwqgEqKMLIx8G0mTA=; b=KgdAan3iZVC8QnFl9r89ctOUC9Bm/zjr4a3kVsWTjtdi/M6z9kLfYo46Lz4IyMRA+kptqa WomDkr6/+W46p2rV2mdRzrstrQTBbS8hF62zFaqHSxCl3L76DlJYqRnJtd8Wh8fwYgHXME 07HTkf64wsYU8Jjwp1VVDEJgKdnZPUM= Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=kvUC28iy; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of opendmb@gmail.com designates 209.85.219.42 as permitted sender) smtp.mailfrom=opendmb@gmail.com X-Rspam-User: X-Rspamd-Server: rspam01 X-Stat-Signature: bnin4mrcm7zwctnq9wnhga7it1hwy6qe X-Rspamd-Queue-Id: B921FC00C1 X-HE-Tag: 1663099093-254426 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There is no isolation mechanism for hugepages so a hugepage that is migrated is returned to its hugepage freelist. This creates problems for alloc_contig_range() because migrated hugepages can be allocated as migrate targets for subsequent hugepage migration attempts. Even if the migration succeeds the alloc_contig_range() attempt will fail because test_pages_isolated() will find the now free hugepages haven't been dissolved. A subsequent attempt by alloc_contig_range() is necessary for the isolate_migratepages_range() function to find the freed hugepage and dissolve it (assuming it has not been reallocated). A workqueue is introduced to perform the equivalent functionality of alloc_and_dissolve_huge_page() for a migrated hugepage when it is freed so that the pages can be released to the isolated page lists of the buddy allocator allowing the alloc_contig_range() attempt to succeed. The HPG_dissolve hugepage flag is introduced to allow tagging migratable hugepages that should be dissolved when freed. Signed-off-by: Doug Berger --- include/linux/hugetlb.h | 5 +++ mm/hugetlb.c | 72 ++++++++++++++++++++++++++++++++++++++--- mm/migrate.c | 1 + mm/page_alloc.c | 1 + 4 files changed, 75 insertions(+), 4 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 3ec981a0d8b3..0e6e21805e51 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -222,6 +222,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, bool is_hugetlb_entry_migration(pte_t pte); void hugetlb_unshare_all_pmds(struct vm_area_struct *vma); +void sync_hugetlb_dissolve(void); #else /* !CONFIG_HUGETLB_PAGE */ @@ -430,6 +431,8 @@ static inline vm_fault_t hugetlb_fault(struct mm_struct *mm, static inline void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) { } +static inline void sync_hugetlb_dissolve(void) { } + #endif /* !CONFIG_HUGETLB_PAGE */ /* * hugepages at page global directory. If arch support @@ -574,6 +577,7 @@ enum hugetlb_page_flags { HPG_freed, HPG_vmemmap_optimized, HPG_raw_hwp_unreliable, + HPG_dissolve, __NR_HPAGEFLAGS, }; @@ -621,6 +625,7 @@ HPAGEFLAG(Temporary, temporary) HPAGEFLAG(Freed, freed) HPAGEFLAG(VmemmapOptimized, vmemmap_optimized) HPAGEFLAG(RawHwpUnreliable, raw_hwp_unreliable) +HPAGEFLAG(Dissolve, dissolve) #ifdef CONFIG_HUGETLB_PAGE diff --git a/mm/hugetlb.c b/mm/hugetlb.c index f232a37df4b6..da80889e1436 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1582,6 +1582,10 @@ static void __update_and_free_page(struct hstate *h, struct page *page) } } +static LLIST_HEAD(hpage_dissolvelist); +static void dissolve_hpage_workfn(struct work_struct *work); +static DECLARE_WORK(dissolve_hpage_work, dissolve_hpage_workfn); + /* * As update_and_free_page() can be called under any context, so we cannot * use GFP_KERNEL to allocate vmemmap pages. However, we can defer the @@ -1628,6 +1632,8 @@ static inline void flush_free_hpage_work(struct hstate *h) { if (hugetlb_vmemmap_optimizable(h)) flush_work(&free_hpage_work); + if (!hstate_is_gigantic(h)) + flush_work(&dissolve_hpage_work); } static void update_and_free_page(struct hstate *h, struct page *page, @@ -1679,7 +1685,7 @@ void free_huge_page(struct page *page) struct hstate *h = page_hstate(page); int nid = page_to_nid(page); struct hugepage_subpool *spool = hugetlb_page_subpool(page); - bool restore_reserve; + bool restore_reserve, dissolve; unsigned long flags; VM_BUG_ON_PAGE(page_count(page), page); @@ -1691,6 +1697,8 @@ void free_huge_page(struct page *page) page->mapping = NULL; restore_reserve = HPageRestoreReserve(page); ClearHPageRestoreReserve(page); + dissolve = HPageDissolve(page); + ClearHPageDissolve(page); /* * If HPageRestoreReserve was set on page, page allocation consumed a @@ -1729,6 +1737,11 @@ void free_huge_page(struct page *page) remove_hugetlb_page(h, page, true); spin_unlock_irqrestore(&hugetlb_lock, flags); update_and_free_page(h, page, true); + } else if (dissolve) { + spin_unlock_irqrestore(&hugetlb_lock, flags); + if (llist_add((struct llist_node *)&page->mapping, + &hpage_dissolvelist)) + schedule_work(&dissolve_hpage_work); } else { arch_clear_hugepage_flags(page); enqueue_huge_page(h, page); @@ -2771,6 +2784,49 @@ static void replace_hugepage(struct hstate *h, int nid, struct page *old_page, enqueue_huge_page(h, new_page); } +static void dissolve_hpage_workfn(struct work_struct *work) +{ + struct llist_node *node; + + node = llist_del_all(&hpage_dissolvelist); + + while (node) { + struct page *oldpage, *newpage; + struct hstate *h; + int nid; + + oldpage = container_of((struct address_space **)node, + struct page, mapping); + node = node->next; + oldpage->mapping = NULL; + + h = page_hstate(oldpage); + nid = page_to_nid(oldpage); + + newpage = alloc_replacement_page(h, nid); + + spin_lock_irq(&hugetlb_lock); + /* finish freeing oldpage */ + arch_clear_hugepage_flags(oldpage); + enqueue_huge_page(h, oldpage); + if (IS_ERR(newpage)) { + /* cannot dissolve so just leave free */ + spin_unlock_irq(&hugetlb_lock); + goto next; + } + + replace_hugepage(h, nid, oldpage, newpage); + + /* + * Pages have been replaced, we can safely free the old one. + */ + spin_unlock_irq(&hugetlb_lock); + __update_and_free_page(h, oldpage); +next: + cond_resched(); + } +} + /* * alloc_and_dissolve_huge_page - Allocate a new page and dissolve the old one * @h: struct hstate old page belongs to @@ -2803,6 +2859,7 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, */ spin_unlock_irq(&hugetlb_lock); ret = isolate_hugetlb(old_page, list); + SetHPageDissolve(old_page); spin_lock_irq(&hugetlb_lock); goto free_new; } else if (!HPageFreed(old_page)) { @@ -2864,14 +2921,21 @@ int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list) if (hstate_is_gigantic(h)) return -ENOMEM; - if (page_count(head) && !isolate_hugetlb(head, list)) + if (page_count(head) && !isolate_hugetlb(head, list)) { + SetHPageDissolve(head); ret = 0; - else if (!page_count(head)) + } else if (!page_count(head)) { ret = alloc_and_dissolve_huge_page(h, head, list); - + } return ret; } +void sync_hugetlb_dissolve(void) +{ + flush_work(&free_hpage_work); + flush_work(&dissolve_hpage_work); +} + struct page *alloc_huge_page(struct vm_area_struct *vma, unsigned long addr, int avoid_reserve) { diff --git a/mm/migrate.c b/mm/migrate.c index 6a1597c92261..b6c6123e614c 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -141,6 +141,7 @@ void putback_movable_pages(struct list_head *l) list_for_each_entry_safe(page, page2, l, lru) { if (unlikely(PageHuge(page))) { + ClearHPageDissolve(page); putback_active_hugepage(page); continue; } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e5486d47406e..6bf76bbc0308 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -9235,6 +9235,7 @@ int alloc_contig_range(unsigned long start, unsigned long end, if (ret && ret != -EBUSY) goto done; ret = 0; + sync_hugetlb_dissolve(); /* * Pages from [start, end) are within a pageblock_nr_pages From patchwork Tue Sep 13 19:54:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975314 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63753C6FA82 for ; Tue, 13 Sep 2022 19:58:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F2FA66B0081; Tue, 13 Sep 2022 15:58:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB71C6B0082; Tue, 13 Sep 2022 15:58:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D584C8D0001; Tue, 13 Sep 2022 15:58:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C661E6B0081 for ; Tue, 13 Sep 2022 15:58:17 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id A96C11401D4 for ; Tue, 13 Sep 2022 19:58:17 +0000 (UTC) X-FDA: 79908123834.06.4FE241F Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) by imf06.hostedemail.com (Postfix) with ESMTP id 782F418008B for ; Tue, 13 Sep 2022 19:58:17 +0000 (UTC) Received: by mail-qk1-f180.google.com with SMTP id f13so5069451qkk.6 for ; Tue, 13 Sep 2022 12:58:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=5478I3WyUfDJcI1zhJnzCnFwBV1ia1XR+sLHu3XJI6Q=; b=ngWpsEYVUBqyRCK+w+5Ame50HZp1DYM8gstxDexc5gHtIO2fEu4wX/+7Noqq7EWBX5 SX4JOXX5B1E8Qf1s/pvzk9Bz2iJJBgv4Hk3Ull+tKjmtldfM+s3z+vbM1B3di5w6gxjz dcVCWO2NZ50adOCK8bPYxw3tCqWOsDx7M7cTAjTnYf/V4tSaPDhi8fqtVljCEiP5o1JE EBkXOKZZNuxNzz6s5U3pOqHYezTLOaL4mQLClkKUJN8hT0raCVH32ZKrxQcROVGentqi 38kFA0gMphAG/6qqKzQ2VCttdNXcqVd0gmzjLhB0hmlMfPBCKPcE2Rytw17lBrFXHH3/ rqCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=5478I3WyUfDJcI1zhJnzCnFwBV1ia1XR+sLHu3XJI6Q=; b=vGDuEGP6cxHWWxquPSbypBLZc6YTtchKGqzbAEiMaDKhhuTpNN/ayK/XhMZkJJelp5 C3N9ts4nTZ6J+fQqJ9DWqEFI+fXFSWvwJHZx/IfBR7LoRiaHq1Ru5Z2RpG3eU3jlUXWx ovXqjApQilsd1lWGP6LQyG80TkanvXBMIK/tXhF89jGzHSPHlscC0PmUro0Q3th8hmcF Z5QRPDVzhviAvuOlqj99RPZxNS3hiSV4JFNH2ancCSDZdEa7qJ1fYsz6F0i1qzq9Cpf1 NDsulMIShzbbQ9u2g5yojSxIHXKQNPhf0GkrVJr7/Y9jgLIWBj5HW3jbj9exFFvQLNut JQ4Q== X-Gm-Message-State: ACgBeo02T8VX2C7CJh5sfwPHh/fTVz4HPUw3sWacF18ub1ED4QGPyWCG qlqNKj16toAEQVNSstpLZ2I= X-Google-Smtp-Source: AA6agR5yg7LQIQNFFYJ0sDmMy6v2CLDWuXatdLv4m+yPVXKNj8MYR/nHEcTFAbCbbhbRcvfawH5dww== X-Received: by 2002:a05:620a:4496:b0:6bb:8db4:6b49 with SMTP id x22-20020a05620a449600b006bb8db46b49mr24594751qkp.703.1663099096778; Tue, 13 Sep 2022 12:58:16 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:16 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 06/21] mm/hugetlb: add hugepage isolation support Date: Tue, 13 Sep 2022 12:54:53 -0700 Message-Id: <20220913195508.3511038-7-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099097; a=rsa-sha256; cv=none; b=AQqXZvBYqY9lMeziz442TGwMvV2EdklmzZ4B3a6+6ptSJ2vuGfkbS+39E3H8JPRnid0Eit KnmXP8GkjFmNBpdGfjsV/39CMmiewMGKBWh8zDqBCFMsqUYl9YkFY94qMTeijmH9KkMp9w r313vJJ7ByPTUr3b7pQJG4UM7VVgst4= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ngWpsEYV; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf06.hostedemail.com: domain of opendmb@gmail.com designates 209.85.222.180 as permitted sender) smtp.mailfrom=opendmb@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099097; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5478I3WyUfDJcI1zhJnzCnFwBV1ia1XR+sLHu3XJI6Q=; b=FWG04cdfVILDQbZS1Rb+uKzMBL40Dq8Uyz6njh0gyuIpNCI/X9v9WfrEvrYHmbcMMzMvZA j2vSDcp+ZfeS8CrRS+VokAwrumSzlMRjrFcS9I2B/znp2kkgQ6jY79kvG7cdWZTQArI6kQ vKF75Kd76HGYNpSxOjeqrCACJoEKMxc= X-Rspam-User: Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ngWpsEYV; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf06.hostedemail.com: domain of opendmb@gmail.com designates 209.85.222.180 as permitted sender) smtp.mailfrom=opendmb@gmail.com X-Rspamd-Server: rspam11 X-Stat-Signature: shma7tg86g8dt4gdbfjgtifp839qesge X-Rspamd-Queue-Id: 782F418008B X-HE-Tag: 1663099097-504124 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a range of pageblocks is isolated there is at most one hugepage that has only tail pages overlapping that range (i.e. a hugepage that overlaps the beginning of the range). However, that hugepage is the first migration target for an alloc_contig_range() attempt so it already receives special attention. Checking whether the pageblock containing the head of a hugepage is isolated is an inexpensive way to avoid hugepage allocations from isolated pageblocks which makes alloc_contig_range() more efficient. Signed-off-by: Doug Berger --- mm/hugetlb.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index da80889e1436..2f354423f50f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -33,6 +33,7 @@ #include #include #include +#include #include #include @@ -1135,6 +1136,10 @@ static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) if (PageHWPoison(page)) continue; + /* Check head pageblock isolation */ + if (is_migrate_isolate_page(page)) + continue; + list_move(&page->lru, &h->hugepage_activelist); set_page_refcounted(page); ClearHPageFreed(page); From patchwork Tue Sep 13 19:54:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975315 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31C69C54EE9 for ; Tue, 13 Sep 2022 19:58:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BD3C56B0083; Tue, 13 Sep 2022 15:58:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B83196B0085; Tue, 13 Sep 2022 15:58:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B1F18D0001; Tue, 13 Sep 2022 15:58:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8C98F6B0083 for ; Tue, 13 Sep 2022 15:58:21 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6D798A66C8 for ; Tue, 13 Sep 2022 19:58:21 +0000 (UTC) X-FDA: 79908123960.24.948673B Received: from mail-qv1-f50.google.com (mail-qv1-f50.google.com [209.85.219.50]) by imf23.hostedemail.com (Postfix) with ESMTP id 1C2691400B5 for ; Tue, 13 Sep 2022 19:58:20 +0000 (UTC) Received: by mail-qv1-f50.google.com with SMTP id s13so10080186qvq.10 for ; Tue, 13 Sep 2022 12:58:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=hB/8CHaP3epa66ZDI7eIxcYHinx9lai7E+SzdR8JK34=; b=DH5wOpdpiHaFFXY8TAT3JRecX64XiPNxiR/CcG1LrVu07Readvn0PWViKgBmDA60Pb UBITk9aWlYro2ikNI4WmLJPOCXwQOqtprdUB0CzoJZ72kknTQx9o/RitCRSffQDxQ5Q6 VTZyKVoIHd28mrFZHIQl9dz1uLmu560sdtEcQWDtP1q9dt0hIGQCuqCTjE9fttuhjONt 0HIT34gX8eS66xaVSIttBxYAN/YQSf5UtfsLj6aZ6r0xFgT3C7geS9uBf8xzXOro8Ch1 QuB8O5yFa+P3URJ5XpTXammfwNsUv1UpGQMU/vjO4KV44uIZqk+ZBhqofvaHP1BLFOQP ilEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=hB/8CHaP3epa66ZDI7eIxcYHinx9lai7E+SzdR8JK34=; b=wd47I37gojnsKsogfl1ScK5sVUse+XfgxrBQPR0rTgMDBxlSknIj041mfy4kxyGc5c 0HWkzWbNRsQ/iFsmYiYNFFVKulXBsAnDJqhIdDuyahfs0hWGQ5kQF0gXJIf5ZXzkqRld 7XS7rln6yaofC1TwF4R755ar3MnWi1ixJJPlCLjuGaME9ZTeYPBPCdpvgAarX7lZdhyT QFD453lIc8pjK/nFVP8Ktd144v3jqZztVW/3GeX6lf96+D/WTr9CU0bk2pu4bdcpPtiI uVOkEiHPy6W3zI1XpZ1qmEFPZ0zbMNH0V42H6TJeU6M3AudloariqR52wsFNphLx824F 66fg== X-Gm-Message-State: ACgBeo2Fog26OF3nbHYJZxDB7KRldK+y4TeF5dBugAn5bRpgQY3LgUVX 15A0v6duGI0sAEjyzF6Mlik= X-Google-Smtp-Source: AA6agR4VzptNv0IdYnihlFGaNyhbElSt3SCLpuN9uI3qoIO24r77G7BDUS/ieSa0RDNa+EBnGiO84g== X-Received: by 2002:a0c:c98a:0:b0:4ac:9f4b:efed with SMTP id b10-20020a0cc98a000000b004ac9f4befedmr14045129qvk.35.1663099100395; Tue, 13 Sep 2022 12:58:20 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:19 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 07/21] lib/show_mem.c: display MovableOnly Date: Tue, 13 Sep 2022 12:54:54 -0700 Message-Id: <20220913195508.3511038-8-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=DH5wOpdp; spf=pass (imf23.hostedemail.com: domain of opendmb@gmail.com designates 209.85.219.50 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099101; a=rsa-sha256; cv=none; b=17R68LQ+p2upqsusnvqBt+aoLoiZ0TLsS9C2jNa7vHZBBZoGn4fXgycLFHk7HftDhZUH8D QqOp+HnxPQgT2AZkXxHQJx8qVix0SeTMBoDsxpUbL6lXvmxRR13Lbf7ep/ydB93PkhlTpN jvWfpThefqnTpcBxM5WX24feJRnKvjE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099101; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hB/8CHaP3epa66ZDI7eIxcYHinx9lai7E+SzdR8JK34=; b=xQYb7wtY4rJsBmSGV50z3/mK2IeY1jCw21hVtQzLLZaNBT8XhuQb8ImXweMn34NQW/Hbul ayL1AQs+PDNJxQjpW3eA+SjOOMzxeQ61P1e+Vwf/cZ5VdeBU+t96sityYYRrNEUI+VRO3h 29rxtPr5H5CEK8UxQUVhKa9C9dW2EEw= Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=DH5wOpdp; spf=pass (imf23.hostedemail.com: domain of opendmb@gmail.com designates 209.85.219.50 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam12 X-Stat-Signature: ar1udrnky87iqn6h4jdyidyh7jz1dhex X-Rspamd-Queue-Id: 1C2691400B5 X-Rspam-User: X-HE-Tag: 1663099100-405339 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The comment for commit c78e93630d15 ("mm: do not walk all of system memory during show_mem") indicates it "also corrects the reporting of HighMem as HighMem/MovableOnly as ZONE_MOVABLE has similar problems to HighMem with respect to lowmem/highmem exhaustion." Presuming the similar problems are with regard to the general exclusion of kernel allocations from either zone, I believe it makes sense to include all ZONE_MOVABLE memory even on systems without HighMem. To the extent that this was the intent of the original commit I have included a "Fixes" tag, but it seems unnecessary to submit to linux-stable. Fixes: c78e93630d15 ("mm: do not walk all of system memory during show_mem") Signed-off-by: Doug Berger --- lib/show_mem.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/show_mem.c b/lib/show_mem.c index 1c26c14ffbb9..337c870a5e59 100644 --- a/lib/show_mem.c +++ b/lib/show_mem.c @@ -27,7 +27,7 @@ void show_mem(unsigned int filter, nodemask_t *nodemask) total += zone->present_pages; reserved += zone->present_pages - zone_managed_pages(zone); - if (is_highmem_idx(zoneid)) + if (zoneid == ZONE_MOVABLE || is_highmem_idx(zoneid)) highmem += zone->present_pages; } } From patchwork Tue Sep 13 19:54:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975316 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9796EC54EE9 for ; Tue, 13 Sep 2022 19:58:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 347C76B0085; Tue, 13 Sep 2022 15:58:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F79B6B0087; Tue, 13 Sep 2022 15:58:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 173528D0001; Tue, 13 Sep 2022 15:58:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0689C6B0085 for ; Tue, 13 Sep 2022 15:58:25 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id DBDB9120A3E for ; Tue, 13 Sep 2022 19:58:24 +0000 (UTC) X-FDA: 79908124128.11.7FDB8C6 Received: from mail-qv1-f50.google.com (mail-qv1-f50.google.com [209.85.219.50]) by imf23.hostedemail.com (Postfix) with ESMTP id 858811400B5 for ; Tue, 13 Sep 2022 19:58:24 +0000 (UTC) Received: by mail-qv1-f50.google.com with SMTP id s13so10080313qvq.10 for ; Tue, 13 Sep 2022 12:58:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=6/+RylwAUS1L6kGkdbzNevLkV7SCUovxqo9fjOaS6JU=; b=IIvzQBml/wGri/ZBWsuig4iC5yLnXMtk1VHesQ1eh/9HzasiMgV8YxfdpWHpGIoYep M/INFOPjeTVC3OEHHCjy2E1Hs8kTN4G53N8xRA/QBMd1ulJjTypxrYcp9Vasl1Q/eOJB qHAj6LLFWCEJm7FVRIifJLn+g0xAhtpqiGJt/0Lazj8LpFDzEnyQtx4bqcugERvtsdZG jeSnKtUCKGP4SXlnb3HB7CkPQRbT8a6KgjjUz3Q32zw4SQOfC5UckDKNJQGxML3W3hgv kauff9Q/Ua47CAvmWSZZgrFSzvsXxKLS/PIpqLn25rigG7/KetjxEqCiT7a4LgLAH6NO U6Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=6/+RylwAUS1L6kGkdbzNevLkV7SCUovxqo9fjOaS6JU=; b=0AiwN3rVWvPf8cLkUihfnvjUF78hXezu4+m15/Zo0cc+xcfGio3/YsqeKx5tc+YZD4 D9pP+jgMV2rDusl7rcl4prtoduI7FJCwi3Rlhl56R4aCEpPL5mnuKiss+61H4eV6PiHE x3CgW7fcS590BYvkB4DyY/J8GUzrkFc54IV5oMZsG+fl9rwm3yt4S8oHNf9BJq0TBYgK MB9HSAISFsENS11Zs+M9GzH/RCDJehmzi0Pz/OfWxR1glpqJle50uNAu2IFeTmIMw5U+ 6PwNQ/NwXTvJsWeG2XM95a//TFIxzVa8aH+AZ5NkDsYdnmAPVIWkutpTueka1OnZjCPY peIw== X-Gm-Message-State: ACgBeo0AsNFjwdUVy2OMTU06aGCbqEAuCINocxxFGVIMzqFcB6dVRD70 aJTj+S2EJ7xaOhzKuYkW50k= X-Google-Smtp-Source: AA6agR7HhxHAKHSAet4vBy7WKGiBHiOPyi3Gu3lpC/Y1Ix8I/hDUvAWKeNeCYEL5N+Wcelu80g5S7w== X-Received: by 2002:a0c:d987:0:b0:4a5:192c:faff with SMTP id y7-20020a0cd987000000b004a5192cfaffmr29341373qvj.106.1663099104077; Tue, 13 Sep 2022 12:58:24 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:23 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 08/21] mm/vmstat: show start_pfn when zone spans pages Date: Tue, 13 Sep 2022 12:54:55 -0700 Message-Id: <20220913195508.3511038-9-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=IIvzQBml; spf=pass (imf23.hostedemail.com: domain of opendmb@gmail.com designates 209.85.219.50 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099104; a=rsa-sha256; cv=none; b=t4CNNRSu9LSxS2L611abyr6ALVByrmyGVm1I1oTaP8xKZ/MQbTDsAK+DqRI5/ucWAvr8LK hirchtZEcpg8ENeiFPjm8m9PCNQJl1WTXPm+zyUQL8gzFAaNagEAFYBckgVnqurM+rZg7n FQL0M73A88rZJ4ehaiF618rYsDrXVh4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099104; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6/+RylwAUS1L6kGkdbzNevLkV7SCUovxqo9fjOaS6JU=; b=wia1tfjojNyD1TQTFIprqkfn9zVmPAZVkq1K4m0dl0doEcXBXxkx42A1/AftFLq7Xd+hNJ PASINJsntg26Y7KfuRCBOo/2SyzQqQ8tMBiaPI57gRDlRVzTpnOUVdK2z91/aciKU84nya 4os9ZRhlpsbVb2f5D57uQZ7AZdOU2bc= Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=IIvzQBml; spf=pass (imf23.hostedemail.com: domain of opendmb@gmail.com designates 209.85.219.50 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam12 X-Stat-Signature: rgziuttgio9b7r5jmu1z3eimfqysyswa X-Rspamd-Queue-Id: 858811400B5 X-Rspam-User: X-HE-Tag: 1663099104-739665 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A zone that overlaps with another zone may span a range of pages that are not present. In this case, displaying the start_pfn of the zone allows the zone page range to be identified. Signed-off-by: Doug Berger --- mm/vmstat.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/vmstat.c b/mm/vmstat.c index 90af9a8572f5..e2f19f2b7615 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1717,6 +1717,11 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat, /* If unpopulated, no other information is useful */ if (!populated_zone(zone)) { + /* Show start_pfn for empty overlapped zones */ + if (zone->spanned_pages) + seq_printf(m, + "\n start_pfn: %lu", + zone->zone_start_pfn); seq_putc(m, '\n'); return; } From patchwork Tue Sep 13 19:54:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7AD83C6FA82 for ; Tue, 13 Sep 2022 19:58:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 14CC16B0087; Tue, 13 Sep 2022 15:58:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0FD566B0088; Tue, 13 Sep 2022 15:58:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EECEA8D0001; Tue, 13 Sep 2022 15:58:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id DA4526B0087 for ; Tue, 13 Sep 2022 15:58:28 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C194F1401D4 for ; Tue, 13 Sep 2022 19:58:28 +0000 (UTC) X-FDA: 79908124296.30.66E7888 Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) by imf01.hostedemail.com (Postfix) with ESMTP id 6CEC940092 for ; Tue, 13 Sep 2022 19:58:28 +0000 (UTC) Received: by mail-qk1-f179.google.com with SMTP id k12so9303218qkj.8 for ; Tue, 13 Sep 2022 12:58:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=kefl6D8eKpN1GxKRqpPd+2PbW8yMD7NaxrgEOy2EJKU=; b=byR/eekYsW9DhH0NqLev2fS+tpkFMN+hxgB8j3wmXvKTocIMCm6zZesYvYgQ87wjhP q+4AfKw8TXxcj2+vnpVcn5ECPh4qGk7Jtzj+zNbyb6YCFLIiwPOzRUIUuAqm/vme1CGD HXFFRmWffOsc2lXoXpa+V0KHw2MwGlrFjaT+dLfIFP4Da4Lspw/3GTIWmZkSLzazc/cQ hUYdezyBusY92UGvKeXjT7Rg0vsemlXyaK6ln+reN6yeiO3uNu7F3nGVR2B+OsucHch6 d5MKRdQDkKjHro2ts06Z/prjBlf3W1AQZuN3IrXE7aLh74QN4KD+aUTWaAkLm41rgl56 mOWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=kefl6D8eKpN1GxKRqpPd+2PbW8yMD7NaxrgEOy2EJKU=; b=XWBs9jrsikDxrlA+06g0Gqs157xFM+NWDLEWbbkNpeB59TOwco/NcCExxPiAswN4cm 2vp1yQJ8sBKghhZRaCjXzHOfeA7jujxgCWaLQwSPIsPH4JraSmydAimgOt5eZtHhDtGn NhjZhDKLoAfGISGPo/eubHe+zoYmk+fwXOV2NUft7FKis+wjqf8w4xfUi8Za5skSq5SR xdrls4R9WsvrzBChSg449npGgnU3RQi4LYMh1Pa4ZSWEBvyPNUjMe+Va1V569gfl7ENu XIQV9zvXBhG4c65d9u4T+hWTXN/Nrdf+iCLtHD6GeZb45kP6gxwGJ1wCnnOdGuYwjKZM aKiw== X-Gm-Message-State: ACgBeo3N0NWqpO4ysk4h1QXQHjp+55aVz7hMwTFguju0qyMQqLwRmOdp Txctu0K9AFBJCFblfhbr0Uk= X-Google-Smtp-Source: AA6agR4kCtYZpwUOe+kdoRL5XUeDY9Udr6YA+L+RrDdkM+ZkGSBIa8BGCY7yWmaW8ey8FgVIise+hw== X-Received: by 2002:a37:916:0:b0:6ce:5f1c:d5d9 with SMTP id 22-20020a370916000000b006ce5f1cd5d9mr5678519qkj.737.1663099107720; Tue, 13 Sep 2022 12:58:27 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:27 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 09/21] mm/page_alloc: calculate node_spanned_pages from pfns Date: Tue, 13 Sep 2022 12:54:56 -0700 Message-Id: <20220913195508.3511038-10-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="byR/eekY"; spf=pass (imf01.hostedemail.com: domain of opendmb@gmail.com designates 209.85.222.179 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099108; a=rsa-sha256; cv=none; b=RxTBS5EyAG3tAQxLNO58O5F+fFHom96wS6etOvcbTZvKVzoSywnELGeAP137bGdnfSMmM1 vdczO6LXyPBs5iaEoKfUGk8BnCWiSPiBRHPAIM64PGhRTvZ/3mDbLskBghB6OCeWE/NVh4 XVRKw6vOwY1zxPIuAo3gmxSEiYI/1/I= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099108; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kefl6D8eKpN1GxKRqpPd+2PbW8yMD7NaxrgEOy2EJKU=; b=YOGregR6ub1D6yX2PW5sbQ8aVPdb/Yxev4JEAbCxL0BW9+PBVu5H7KLwPdJsWirKfNa45y mcBQ1MRWJrruogUSaT2CpxMbIFly4TR2arZ07VYrherSSE9fVRegPKhOQQVX4o5xZaazPz 9xM7vO4v4wWOjxSx2ydwgoHGm8vKkXI= Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="byR/eekY"; spf=pass (imf01.hostedemail.com: domain of opendmb@gmail.com designates 209.85.222.179 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam12 X-Stat-Signature: 55kuzadwu8dg17y9m61i4eupahzh843f X-Rspamd-Queue-Id: 6CEC940092 X-Rspam-User: X-HE-Tag: 1663099108-874912 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since the start and end pfns of the node are passed as arguments to calculate_node_totalpages() they might as well be used to specify the node_spanned_pages value for the node rather than accumulating the spans of member zones. This prevents the need for additional adjustments if zones are allowed to overlap. Signed-off-by: Doug Berger --- mm/page_alloc.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 6bf76bbc0308..b6074961fb59 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7452,7 +7452,7 @@ static void __init calculate_node_totalpages(struct pglist_data *pgdat, unsigned long node_start_pfn, unsigned long node_end_pfn) { - unsigned long realtotalpages = 0, totalpages = 0; + unsigned long realtotalpages = 0; enum zone_type i; for (i = 0; i < MAX_NR_ZONES; i++) { @@ -7483,11 +7483,10 @@ static void __init calculate_node_totalpages(struct pglist_data *pgdat, zone->present_early_pages = real_size; #endif - totalpages += size; realtotalpages += real_size; } - pgdat->node_spanned_pages = totalpages; + pgdat->node_spanned_pages = node_end_pfn - node_start_pfn; pgdat->node_present_pages = realtotalpages; pr_debug("On node %d totalpages: %lu\n", pgdat->node_id, realtotalpages); } From patchwork Tue Sep 13 19:54:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975318 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C475ECAAD8 for ; Tue, 13 Sep 2022 19:58:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A4CFA6B0088; Tue, 13 Sep 2022 15:58:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9FD1F6B0089; Tue, 13 Sep 2022 15:58:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 876BA8D0001; Tue, 13 Sep 2022 15:58:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 77D516B0088 for ; Tue, 13 Sep 2022 15:58:32 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 56C1A1C6486 for ; Tue, 13 Sep 2022 19:58:32 +0000 (UTC) X-FDA: 79908124464.30.B1D19F7 Received: from mail-qt1-f169.google.com (mail-qt1-f169.google.com [209.85.160.169]) by imf25.hostedemail.com (Postfix) with ESMTP id 0CC84A00BD for ; Tue, 13 Sep 2022 19:58:31 +0000 (UTC) Received: by mail-qt1-f169.google.com with SMTP id g12so9223173qts.1 for ; Tue, 13 Sep 2022 12:58:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=bjPZ9urefAsJcftBbAgQB1diXayOj809VCbylW0WU+4=; b=ElB8vGfwnTgMrOHPV8jOPMjX9e4dyOnpsA/daCy84nADITgBEDs0AWwUlYd7LmzJ4R Fd+wihlwr+eHGiOYJevIadmukrrhPQQQbV14He/EgVL8UId/ZjEQspu78E84lCBkqAk3 9AGpDbnJvnF63g83iAedqf+9zINjZ62a64qbSfO2T1vB7Ckgl7brAPRPzZAIUuLk9YsF YnnaDAUhOF2TFJBs1+hdGFIi2/6/xETSCDXaO1b95XuqZwbhceSd0bqWp4uVAOWzs4lk zAxkAToGZXGzD1zml0W1YllS9/kHr+k5y9Z8yZENt9c1+33raLf8akoDdgYfGMOxfC4A ITFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=bjPZ9urefAsJcftBbAgQB1diXayOj809VCbylW0WU+4=; b=gTa+SNwQQIvxnp43Q3haRpfAQYaej/Nz+l/Tpx8NkISU/Y8gF1eAgK2EdYK8HNu7rl IYDTjx5h0yudrmZCFIQ9x/tN2L41RznfmMCLLULm/xMM3ngy0IsnHhgW5oxbgKWnFPto eyPlhY02wkNxEe7HHafK1GsTeOQIItCbXhc7Awo3rwFrd5ELdHQ6elfpAi3IcO0WXlIY f7RYR+vVoLkzUz9922edHDPQg5GpgyC6mWZ7Q6gsNyL9BAoNj90EqWug2h4vFYyjp8Up OwMlQvbCLtP/Lq4u2WTNla06ZJzEkKz2MHhHYXGKZBs1g4lxfBLyY24Gh0nYwxCeBlP5 JrbA== X-Gm-Message-State: ACgBeo0rsj96ghpvvPmcvGBgdUKsPe2ZVfoPvpmzybxaLST1p+62DhvJ eoSPbUf/nWfOaUvq2oUsKAJAKbUN9pw= X-Google-Smtp-Source: AA6agR4w6aZMPi6cpzcCwIclFPfEwGVo8/KdfU9B4JBAlrDz19GDQ32yg1HkC6aVwcDa1gVOoM50cA== X-Received: by 2002:a05:622a:174b:b0:343:1fc:14d8 with SMTP id l11-20020a05622a174b00b0034301fc14d8mr29317121qtk.579.1663099111308; Tue, 13 Sep 2022 12:58:31 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:30 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 10/21] mm/page_alloc.c: allow oversized movablecore Date: Tue, 13 Sep 2022 12:54:57 -0700 Message-Id: <20220913195508.3511038-11-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099112; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bjPZ9urefAsJcftBbAgQB1diXayOj809VCbylW0WU+4=; b=fnKqKJGbdgp2Meten+bKOKiJJwJBCZ4tC2rc7LlflEBW958HqyG+m2n7IeSTrnGppY2R3x 9Wb+wAVnmmNzU08NsTMBw+tOYt/RLPA2oOAgyJYDiouqTapF3Xfq2h8tCTVK1V+vGHZ4NA SEY3Igep1jp0JZNcHdIIM7PrbQJyrvY= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ElB8vGfw; spf=pass (imf25.hostedemail.com: domain of opendmb@gmail.com designates 209.85.160.169 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099112; a=rsa-sha256; cv=none; b=sad7pg+wO96fv2INbADjfIzQ5Q4UcBZpgTOzfpjDWgnvSmjRfU2RPzK0WKiHV9JremyYwh HeVR07a69gJuKKs8Rr1c+6a4RCexTI2Ig077DLOAC2ECNZChPakRAH0JH/lNJJp91gPeep A4Assyzbig3Ri8bvmYjOQ/8F68bJmsA= X-Rspam-User: X-Stat-Signature: rhuu7ub1e4m34qn68nzxdfotkdodszg4 X-Rspamd-Queue-Id: 0CC84A00BD Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ElB8vGfw; spf=pass (imf25.hostedemail.com: domain of opendmb@gmail.com designates 209.85.160.169 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam08 X-HE-Tag: 1663099111-590146 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that the error in computation of corepages has been corrected by commit 9fd745d450e7 ("mm: fix overflow in find_zone_movable_pfns_for_nodes()"), oversized specifications of movablecore will result in a zero value for required_kernelcore if it is not also specified. It is unintuitive for such a request to lead to no ZONE_MOVABLE memory when the kernel parameters are clearly requesting some. The current behavior when requesting an oversized kernelcore is to classify all of the pages in movable_zone as kernelcore. The new behavior when requesting an oversized movablecore (when not also specifying kernelcore) is to similarly classify all of the pages in movable_zone as movablecore. Signed-off-by: Doug Berger --- mm/page_alloc.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index b6074961fb59..ad38a81203e5 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -8041,13 +8041,13 @@ static void __init find_zone_movable_pfns_for_nodes(void) corepages = totalpages - required_movablecore; required_kernelcore = max(required_kernelcore, corepages); + } else if (!required_kernelcore) { + /* If kernelcore was not specified, there is no ZONE_MOVABLE */ + goto out; } - /* - * If kernelcore was not specified or kernelcore size is larger - * than totalpages, there is no ZONE_MOVABLE. - */ - if (!required_kernelcore || required_kernelcore >= totalpages) + /* If kernelcore size exceeds totalpages, there is no ZONE_MOVABLE */ + if (required_kernelcore >= totalpages) goto out; /* usable_startpfn is the lowest possible pfn ZONE_MOVABLE can be at */ From patchwork Tue Sep 13 19:54:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B35D9ECAAD8 for ; Tue, 13 Sep 2022 19:58:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 544BE6B008A; Tue, 13 Sep 2022 15:58:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F41F8D0001; Tue, 13 Sep 2022 15:58:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 394AC6B008C; Tue, 13 Sep 2022 15:58:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 291D86B0089 for ; Tue, 13 Sep 2022 15:58:36 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 018A540754 for ; Tue, 13 Sep 2022 19:58:35 +0000 (UTC) X-FDA: 79908124632.14.FE74262 Received: from mail-qv1-f43.google.com (mail-qv1-f43.google.com [209.85.219.43]) by imf11.hostedemail.com (Postfix) with ESMTP id A3D2E4008F for ; Tue, 13 Sep 2022 19:58:35 +0000 (UTC) Received: by mail-qv1-f43.google.com with SMTP id c6so10090979qvn.6 for ; Tue, 13 Sep 2022 12:58:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=KeLoRQJB3eAOdyXC4NTd1TP4on3QCykrOlScFxq8ZPE=; b=FeFlqfsiEQU3HSeYb1n4gkpg1qmnB5WeDR3FybX0zPrw0OvjL/48yUYTNjXdAaMlAU JdyQzl2XCc8s8RP5OO/H5J7S/0xDcSLDPZ/g9EIPbcsw9LhFgl8pcNCL7ia4xS0HbBWk UrWVqsTOfQjwUdrXVkxZ/eY8sVsnX0gm6/fSQE07CBzEH1IsIWTpZVsPIgNRukD5QnzD c6mLxWXf85eq1Q9U3EFaUBdh/wEyMg/Ynk1wbDDmvjWml5kn5SQbnOB+ZusyeRT2gzZB 30p18F8NjILejz6FdnbFmeCsFxJLxim3pDKbO19ZAHgQZI8N1Ooww3p9oQH3KqqVXxOk 7Eog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=KeLoRQJB3eAOdyXC4NTd1TP4on3QCykrOlScFxq8ZPE=; b=Jj6mUUR9OE39t0LGLjFdcH2gykS71YhxKl51TwGA7PVzpF8C9G6YRqIhlo2SyEmJVQ ciKBznXeIRgyXlBuuiMu9d4T/HpF8ZrvKPWuxm8xJ/JQK0Sg2JKkziN9Gz3qzuJf4Cqc IS4Yebk8voJYn7R6z1/kn10CUdAMbG5McVCCGSyInTQ6F4wNbDPBm5OJlYhTRmvhKles hynCwanqwJYcnem+1MxqeCT2W4opt/8xmQP5MRVp3QKNihtKfNoxl9jOJ03UQniFhZOR 50ON09jCAr9a64eRxqXDz3O5GFNHQgMOypwo9K5rvigFzemuU8ieEpn2D9pHRd0XdO5x qJ0w== X-Gm-Message-State: ACgBeo0P2J+THNgDYEcr1JZAd4+q+1K4QPsjYZBIVlRrVTuPHXwktlHB 82ZWeA7+fWpgBo0i4neCpH4= X-Google-Smtp-Source: AA6agR6LdySz/okFv68gwh3HYvltSAZalCbl/B/p5X5bZPgANvOoXngUoiwr94jsBmavX1SN+2thWQ== X-Received: by 2002:a05:6214:301a:b0:4ac:a4ec:b8b1 with SMTP id ke26-20020a056214301a00b004aca4ecb8b1mr13793217qvb.122.1663099114897; Tue, 13 Sep 2022 12:58:34 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:34 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 11/21] mm/page_alloc: introduce init_reserved_pageblock() Date: Tue, 13 Sep 2022 12:54:58 -0700 Message-Id: <20220913195508.3511038-12-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099115; a=rsa-sha256; cv=none; b=CmxouZPwMJ8kiuIgrF5UbXl7QY9tanlv9tnorcQWqSB3WjDieIoVFaP+qztE4+6g0xMF9g xn7W2OKeT9c7B3wOWXx6s0Ga4O+kEzolGbxN7Ya38lzHKaXst9K3MXp9P6jBMdxwL+FYrs qGalCshjrZt9UzS6/dlyfT8WnTYs6fE= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=FeFlqfsi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf11.hostedemail.com: domain of opendmb@gmail.com designates 209.85.219.43 as permitted sender) smtp.mailfrom=opendmb@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099115; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KeLoRQJB3eAOdyXC4NTd1TP4on3QCykrOlScFxq8ZPE=; b=t+jrTfaCnSC9fxM2amZQL5LZQrdYeHQ4NPRQw2b1oKCXt48CZg5c+o94gLsm/5o9Hw6eAr e7dF4SeErZ94GREDBg5d6Q5ou5hJmD64eVXqoJNJnTc22WVsoei97Kwom/FTCqRYtmkIGY 72OVsL7S2nIIsKfRa+oqdwn1E3xRvSc= Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=FeFlqfsi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf11.hostedemail.com: domain of opendmb@gmail.com designates 209.85.219.43 as permitted sender) smtp.mailfrom=opendmb@gmail.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: A3D2E4008F X-Stat-Signature: koyq5iwrpsk76x3yjreq8wqsfs5paxgy X-Rspam-User: X-HE-Tag: 1663099115-580735 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Most of the implementation of init_cma_reserved_pageblock() is common to the initialization of any reserved pageblock for use by the page allocator. This commit breaks that functionality out into the new common function init_reserved_pageblock() for use by code other than CMA. The CMA specific code is relocated from page_alloc to the point where init_cma_reserved_pageblock() was invoked and the new function is used there instead. The error path is also updated to use the function to operate on pageblocks rather than pages. Signed-off-by: Doug Berger --- include/linux/gfp.h | 5 +---- mm/cma.c | 15 +++++++++++---- mm/page_alloc.c | 8 ++------ 3 files changed, 14 insertions(+), 14 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index f314be58fa77..71ed687be406 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -367,9 +367,6 @@ extern struct page *alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask, #endif void free_contig_range(unsigned long pfn, unsigned long nr_pages); -#ifdef CONFIG_CMA -/* CMA stuff */ -extern void init_cma_reserved_pageblock(struct page *page); -#endif +extern void init_reserved_pageblock(struct page *page); #endif /* __LINUX_GFP_H */ diff --git a/mm/cma.c b/mm/cma.c index 4a978e09547a..6208a3e1cd9d 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -31,6 +31,7 @@ #include #include #include +#include #include #include "cma.h" @@ -116,8 +117,13 @@ static void __init cma_activate_area(struct cma *cma) } for (pfn = base_pfn; pfn < base_pfn + cma->count; - pfn += pageblock_nr_pages) - init_cma_reserved_pageblock(pfn_to_page(pfn)); + pfn += pageblock_nr_pages) { + struct page *page = pfn_to_page(pfn); + + set_pageblock_migratetype(page, MIGRATE_CMA); + init_reserved_pageblock(page); + page_zone(page)->cma_pages += pageblock_nr_pages; + } spin_lock_init(&cma->lock); @@ -133,8 +139,9 @@ static void __init cma_activate_area(struct cma *cma) out_error: /* Expose all pages to the buddy, they are useless for CMA. */ if (!cma->reserve_pages_on_error) { - for (pfn = base_pfn; pfn < base_pfn + cma->count; pfn++) - free_reserved_page(pfn_to_page(pfn)); + for (pfn = base_pfn; pfn < base_pfn + cma->count; + pfn += pageblock_nr_pages) + init_reserved_pageblock(pfn_to_page(pfn)); } totalcma_pages -= cma->count; cma->count = 0; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ad38a81203e5..1682d8815efa 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2302,9 +2302,8 @@ void __init page_alloc_init_late(void) set_zone_contiguous(zone); } -#ifdef CONFIG_CMA -/* Free whole pageblock and set its migration type to MIGRATE_CMA. */ -void __init init_cma_reserved_pageblock(struct page *page) +/* Free whole pageblock */ +void __init init_reserved_pageblock(struct page *page) { unsigned i = pageblock_nr_pages; struct page *p = page; @@ -2314,14 +2313,11 @@ void __init init_cma_reserved_pageblock(struct page *page) set_page_count(p, 0); } while (++p, --i); - set_pageblock_migratetype(page, MIGRATE_CMA); set_page_refcounted(page); __free_pages(page, pageblock_order); adjust_managed_page_count(page, pageblock_nr_pages); - page_zone(page)->cma_pages += pageblock_nr_pages; } -#endif /* * The order of subdivision here is critical for the IO subsystem. From patchwork Tue Sep 13 19:54:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975320 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B622ECAAD8 for ; Tue, 13 Sep 2022 19:58:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3BDA06B0092; Tue, 13 Sep 2022 15:58:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 36C9A6B008C; Tue, 13 Sep 2022 15:58:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 20E826B0092; Tue, 13 Sep 2022 15:58:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0F07F6B0089 for ; Tue, 13 Sep 2022 15:58:40 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E442716042F for ; Tue, 13 Sep 2022 19:58:39 +0000 (UTC) X-FDA: 79908124758.12.A9990DD Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) by imf30.hostedemail.com (Postfix) with ESMTP id 960F8800C8 for ; Tue, 13 Sep 2022 19:58:39 +0000 (UTC) Received: by mail-qt1-f177.google.com with SMTP id c11so9603060qtw.8 for ; Tue, 13 Sep 2022 12:58:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=5WWAAzNcGRX09yf3QcgLeiV4ytIgw0QIAz2zMov/ZDY=; b=Yz1mtV6YiuvArbfHbQBXNKT9eLELbvn7bnniGDLD1BumkiBsr2ZDlVa36+Wy8wKT5o ryYhGAJx49vUh11TKsorymepYWGSO76xJbhP0rWVGH3ntJ/cZBgl0HG4OoxTwsRCKi5P r8fZHkmi9dhCwHqvtScDgQwe7gg6nWmJBlbT6hYOH0ew7iRaIAjSwvoSk6SKm+tRfQu+ rKZsLcxmkz1cCAli2W9VyR6gyYWGuJOVoZMSgPsY0YmHve5v7DOrmmIyu3LMUZg6pUfw lIU/zKc3QMqDUaQMXnCFOtpC/YhN4nYgSjeSX0Y2GgcVNB6wvWWTn+zhjMxvZzDNv7zs 69cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=5WWAAzNcGRX09yf3QcgLeiV4ytIgw0QIAz2zMov/ZDY=; b=wjKGozw4aukSZQz/ZT/BRNZ2/JRRBuSosjalUscv1SgYxUKOt9oQYdtG0uBhvB7HrV dbsH1B5jJAEPr9ZDmTShR/G6LFcEkfttbxht5JU6taj8oTltw4Pxa795olQnTqn1GDCp STOXblbxakELmS4n/en6haQVfCNQDPuysNLawrPg9Pj6dQw2KILRJwTCyBzvQbUfhjFY t+1KgKjFmu31peBxT5/RHTrHNQecqj0yQWyP2nH+hg6DWwHyt2n3J6IPSSabmKuXN1w+ qPU591z93DqeKBf3O4EqxO1Ce9M6Wuhkn3XqFVsTyyN/F5pE2zOubGzVgvNdj3pNhWgY bUCw== X-Gm-Message-State: ACgBeo17aimG8H+aFiWfcfh9Ih2m10paMy3BgtU1XyP48BLQcRC5/87R V7Lx65XtpqV4rDS3ub5vi8Q= X-Google-Smtp-Source: AA6agR6JFHHtW1Pd7kJf3d4lhxO33YWLvSfzGCmyAGQLZ1dEkgSuVHnr1X0agG5L8d/cKj0D7xYuZA== X-Received: by 2002:a05:622a:1998:b0:343:6452:dbd9 with SMTP id u24-20020a05622a199800b003436452dbd9mr30254647qtc.423.1663099118855; Tue, 13 Sep 2022 12:58:38 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:38 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 12/21] memblock: introduce MEMBLOCK_MOVABLE flag Date: Tue, 13 Sep 2022 12:54:59 -0700 Message-Id: <20220913195508.3511038-13-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099119; a=rsa-sha256; cv=none; b=ddyKt8H1Klh06Ihcb3SSg+huOfgBGdmVJGHbr0YB845cNilvjBb7hlSc1e25AVCVlsnC7I awhXxwCttg+zzwVGN+v6y3gD+NPEXY12eF4ceV9iVnsx9Qi9CgUHQ2R8KKVp75LIGeuwye 1617qcXb3Ty9AcbebpHKeDAxqaG8W8Q= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Yz1mtV6Y; spf=pass (imf30.hostedemail.com: domain of opendmb@gmail.com designates 209.85.160.177 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099119; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5WWAAzNcGRX09yf3QcgLeiV4ytIgw0QIAz2zMov/ZDY=; b=QFDGIMWsGecm9TeztZmEfsJPx8a0Rk7djidPUuyHggOkdTb444qePsKv3r3xa8NElm0D7E N6K3x0UPjtT1hObzrnqqW/nVysXHqedcM4k0dACTeCcuhTMWtdDsx/enVQoGOw8JCTZV4s Ks71biuYWLfKjvTW8jsK+QnMm1JgLxg= X-Rspam-User: Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Yz1mtV6Y; spf=pass (imf30.hostedemail.com: domain of opendmb@gmail.com designates 209.85.160.177 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam02 X-Stat-Signature: rs7akzgt4fseca3wqax3ft5wakweidxu X-Rspamd-Queue-Id: 960F8800C8 X-HE-Tag: 1663099119-116793 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The MEMBLOCK_MOVABLE flag is introduced to designate a memblock as only supporting movable allocations by the page allocator. Signed-off-by: Doug Berger --- include/linux/memblock.h | 8 ++++++++ mm/memblock.c | 24 ++++++++++++++++++++++++ 2 files changed, 32 insertions(+) diff --git a/include/linux/memblock.h b/include/linux/memblock.h index 50ad19662a32..8eb3ca32dfa7 100644 --- a/include/linux/memblock.h +++ b/include/linux/memblock.h @@ -47,6 +47,7 @@ enum memblock_flags { MEMBLOCK_MIRROR = 0x2, /* mirrored region */ MEMBLOCK_NOMAP = 0x4, /* don't add to kernel direct mapping */ MEMBLOCK_DRIVER_MANAGED = 0x8, /* always detected via a driver */ + MEMBLOCK_MOVABLE = 0x10, /* designated movable block */ }; /** @@ -125,6 +126,8 @@ int memblock_clear_hotplug(phys_addr_t base, phys_addr_t size); int memblock_mark_mirror(phys_addr_t base, phys_addr_t size); int memblock_mark_nomap(phys_addr_t base, phys_addr_t size); int memblock_clear_nomap(phys_addr_t base, phys_addr_t size); +int memblock_mark_movable(phys_addr_t base, phys_addr_t size); +int memblock_clear_movable(phys_addr_t base, phys_addr_t size); void memblock_free_all(void); void memblock_free(void *ptr, size_t size); @@ -265,6 +268,11 @@ static inline bool memblock_is_driver_managed(struct memblock_region *m) return m->flags & MEMBLOCK_DRIVER_MANAGED; } +static inline bool memblock_is_movable(struct memblock_region *m) +{ + return m->flags & MEMBLOCK_MOVABLE; +} + int memblock_search_pfn_nid(unsigned long pfn, unsigned long *start_pfn, unsigned long *end_pfn); void __next_mem_pfn_range(int *idx, int nid, unsigned long *out_start_pfn, diff --git a/mm/memblock.c b/mm/memblock.c index b5d3026979fc..5d6a210d98ec 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -979,6 +979,30 @@ int __init_memblock memblock_clear_nomap(phys_addr_t base, phys_addr_t size) return memblock_setclr_flag(base, size, 0, MEMBLOCK_NOMAP); } +/** + * memblock_mark_movable - Mark designated movable block with MEMBLOCK_MOVABLE. + * @base: the base phys addr of the region + * @size: the size of the region + * + * Return: 0 on success, -errno on failure. + */ +int __init_memblock memblock_mark_movable(phys_addr_t base, phys_addr_t size) +{ + return memblock_setclr_flag(base, size, 1, MEMBLOCK_MOVABLE); +} + +/** + * memblock_clear_movable - Clear flag MEMBLOCK_MOVABLE for a specified region. + * @base: the base phys addr of the region + * @size: the size of the region + * + * Return: 0 on success, -errno on failure. + */ +int __init_memblock memblock_clear_movable(phys_addr_t base, phys_addr_t size) +{ + return memblock_setclr_flag(base, size, 0, MEMBLOCK_MOVABLE); +} + static bool should_skip_region(struct memblock_type *type, struct memblock_region *m, int nid, int flags) From patchwork Tue Sep 13 19:55:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975321 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60944C6FA8B for ; Tue, 13 Sep 2022 19:58:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EBA7C6B0089; Tue, 13 Sep 2022 15:58:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E69B86B008C; Tue, 13 Sep 2022 15:58:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CBBBA6B0093; Tue, 13 Sep 2022 15:58:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BD9D56B0089 for ; Tue, 13 Sep 2022 15:58:43 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 97EF41A086E for ; Tue, 13 Sep 2022 19:58:43 +0000 (UTC) X-FDA: 79908124926.08.486D80E Received: from mail-qk1-f174.google.com (mail-qk1-f174.google.com [209.85.222.174]) by imf25.hostedemail.com (Postfix) with ESMTP id 32618A00C5 for ; Tue, 13 Sep 2022 19:58:43 +0000 (UTC) Received: by mail-qk1-f174.google.com with SMTP id g2so6481896qkk.1 for ; Tue, 13 Sep 2022 12:58:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=WT2HgOEfMbFWghWK9yDd+nkL0DbhssI0liQaO6tZEZs=; b=NfYu0WDbln3rXPAMbeUyjmnvdIGA79s9mtSia48vz0sEDfu2+4xfWqtDXeYpTUlyRc 54Asfh9SWgfuW8gGqmBaFd7ieggdhyEGugonHr+HKUAimN2KvkcgvqS3KbvRNk7jbeiU pGj8A2IHi3i4nWgWwdQY/62/SBJ4kOo+XJKjOoujv76opKObBHDRoqHOvK9jveceTEPN xT/IY+PpVqUyvoH2+Nkw/JGST2FyicYuP52lylnWYbkXJXT6cJrrig7qu89oh2NMLK/r lFceQECtLR20FuN/NHPRY6nDYri8WtxZ1z09emAxpBR0TCKH3LCGsCDGO13oleU0JeEj NOVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=WT2HgOEfMbFWghWK9yDd+nkL0DbhssI0liQaO6tZEZs=; b=qZtkbMWtFgHnuzToocDUYjF4sYGLkhZRLr0oyaK/GfN7ycYL0yMjCioGounjsoO1/0 TqJjYqqjf5M15l5IlKACGa0Wc4W9iXl3tv0BeZxFewIhno7rdrE/QWvozLaF8mTH1G9k k6rZIxbRMRHTF0Hc4As+OZk7XdXVvSdPGQe0n5hJ35pcp8ySjIPDZHcpPDqG1KCUoJ7y ULy5/TkWDmJj0hLJ8ozVbSZduYFldH2p7Vo60fQoTGESgIy2H7T6pGEyhvXpGtFuMu9B BhITRRN82zWgZlVU/qt95IB5e2RuX62LP82YbMVVmHhPWBR7PmmN3SvutYvbjOI7vId+ mJrg== X-Gm-Message-State: ACgBeo2vOYH7ZlEIWGlmBlyRNfm9qq9+h0F9kK8dgy+GspIJiWCV2KlE cODQZNBqb32ZJYQ9obJfqJo= X-Google-Smtp-Source: AA6agR7Nra03ZBRrpuwPuFCXQWjHEJPitoW4JIxgLux1CjcxnJdmZWLlhKoopGRGctCWLJD9uLTPGQ== X-Received: by 2002:a37:aac4:0:b0:6cb:d070:7842 with SMTP id t187-20020a37aac4000000b006cbd0707842mr19201938qke.386.1663099122426; Tue, 13 Sep 2022 12:58:42 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:41 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 13/21] mm/dmb: Introduce Designated Movable Blocks Date: Tue, 13 Sep 2022 12:55:00 -0700 Message-Id: <20220913195508.3511038-14-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=NfYu0WDb; spf=pass (imf25.hostedemail.com: domain of opendmb@gmail.com designates 209.85.222.174 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099123; a=rsa-sha256; cv=none; b=7ssszpbb8//Sm49euYpxD221clCiKZ61AXPUVtVgqe5z4iuzYFZZJJfU5KUoq6RyavX2h3 AkXwUqnlh78RKUIXgeCBv+FqHkQlEdhNboW7P+vPfGUqq3Ln+zFUp0SdfRBq6flT4z9Dbk aBlj/B5CRz6fuPmHSFhB3H5OPl3YeM8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099123; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WT2HgOEfMbFWghWK9yDd+nkL0DbhssI0liQaO6tZEZs=; b=nUxoTHIIayVnW+5wQjMPm83mHf/N+Ra8UBtLxbTOCcLutQAfyspxFTKVSOvUUO3XlPfqF8 qgE1bpIYhwn2EzLy/08tThNwcCyvJPFSw5h3mMIqVCF5S9gBfc8olwhHm/apwhlhAjWbi9 MWJtOCIB8V3wrGapjWxtUpcup+P4eeo= X-Rspam-User: Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=NfYu0WDb; spf=pass (imf25.hostedemail.com: domain of opendmb@gmail.com designates 209.85.222.174 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam04 X-Stat-Signature: nx9djzif8fwzrb9mro6bo6bk3mps7fg4 X-Rspamd-Queue-Id: 32618A00C5 X-HE-Tag: 1663099123-722500 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Designated Movable Blocks are blocks of memory that are composed of one or more adjacent memblocks that have the MEMBLOCK_MOVABLE designation. These blocks must be reserved before receiving that designation and will be located in the ZONE_MOVABLE zone rather than any other zone that may span them. Signed-off-by: Doug Berger --- include/linux/dmb.h | 28 ++++++++++++++ mm/Kconfig | 12 ++++++ mm/Makefile | 1 + mm/dmb.c | 92 +++++++++++++++++++++++++++++++++++++++++++++ mm/memblock.c | 6 ++- mm/page_alloc.c | 84 ++++++++++++++++++++++++++++++++++------- 6 files changed, 209 insertions(+), 14 deletions(-) create mode 100644 include/linux/dmb.h create mode 100644 mm/dmb.c diff --git a/include/linux/dmb.h b/include/linux/dmb.h new file mode 100644 index 000000000000..eecc90e7f884 --- /dev/null +++ b/include/linux/dmb.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __DMB_H__ +#define __DMB_H__ + +#include + +/* + * the buddy -- especially pageblock merging and alloc_contig_range() + * -- can deal with only some pageblocks of a higher-order page being + * MIGRATE_MOVABLE, we can use pageblock_nr_pages. + */ +#define DMB_MIN_ALIGNMENT_PAGES pageblock_nr_pages +#define DMB_MIN_ALIGNMENT_BYTES (PAGE_SIZE * DMB_MIN_ALIGNMENT_PAGES) + +enum { + DMB_DISJOINT = 0, + DMB_INTERSECTS, + DMB_MIXED, +}; + +struct dmb; + +extern int dmb_reserve(phys_addr_t base, phys_addr_t size, + struct dmb **res_dmb); +extern int dmb_intersects(unsigned long spfn, unsigned long epfn); +extern void dmb_init_region(struct memblock_region *region); + +#endif diff --git a/mm/Kconfig b/mm/Kconfig index 0331f1461f81..7739edde5d4d 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -868,6 +868,18 @@ config CMA_AREAS If unsure, leave the default value "7" in UMA and "19" in NUMA. +config DMB_COUNT + int "Maximum count of Designated Movable Blocks" + default 19 if NUMA + default 7 + help + Designated Movable Blocks are blocks of memory that can be used + by the page allocator exclusively for movable pages. They are + managed in ZONE_MOVABLE but may overlap with other zones. This + parameter sets the maximum number of DMBs in the system. + + If unsure, leave the default value "7" in UMA and "19" in NUMA. + config MEM_SOFT_DIRTY bool "Track memory changes" depends on CHECKPOINT_RESTORE && HAVE_ARCH_SOFT_DIRTY && PROC_FS diff --git a/mm/Makefile b/mm/Makefile index 9a564f836403..d0b469a494f2 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -67,6 +67,7 @@ obj-y += page-alloc.o obj-y += init-mm.o obj-y += memblock.o obj-y += $(memory-hotplug-y) +obj-y += dmb.o ifdef CONFIG_MMU obj-$(CONFIG_ADVISE_SYSCALLS) += madvise.o diff --git a/mm/dmb.c b/mm/dmb.c new file mode 100644 index 000000000000..9d9fd31089d2 --- /dev/null +++ b/mm/dmb.c @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* + * Designated Movable Block + */ + +#define pr_fmt(fmt) "dmb: " fmt + +#include + +struct dmb { + unsigned long start_pfn; + unsigned long end_pfn; + void *owner; +}; + +static struct dmb dmb_areas[CONFIG_DMB_COUNT]; +static unsigned int dmb_area_count; + +int __init dmb_reserve(phys_addr_t base, phys_addr_t size, + struct dmb **res_dmb) +{ + struct dmb *dmb; + + /* Sanity checks */ + if (dmb_area_count == ARRAY_SIZE(dmb_areas)) { + pr_warn("Not enough slots for DMB reserved regions!\n"); + return -ENOSPC; + } + + if (!size || !memblock_is_region_reserved(base, size)) + return -EINVAL; + + /* ensure minimal alignment required by mm core */ + if (!IS_ALIGNED(base | size, DMB_MIN_ALIGNMENT_BYTES)) + return -EINVAL; + + /* + * Each reserved area must be initialised later, when more kernel + * subsystems (like slab allocator) are available. + */ + dmb = &dmb_areas[dmb_area_count++]; + + dmb->start_pfn = PFN_DOWN(base); + dmb->end_pfn = PFN_DOWN(base + size); + if (res_dmb) + *res_dmb = dmb; + + memblock_mark_movable(base, size); + return 0; +} + +int dmb_intersects(unsigned long spfn, unsigned long epfn) +{ + int i; + struct dmb *dmb; + + if (spfn >= epfn) + return DMB_DISJOINT; + + for (i = 0; i < dmb_area_count; i++) { + dmb = &dmb_areas[i]; + if (spfn >= dmb->end_pfn) + continue; + if (epfn <= dmb->start_pfn) + return DMB_DISJOINT; + if (spfn >= dmb->start_pfn && epfn <= dmb->end_pfn) + return DMB_INTERSECTS; + else + return DMB_MIXED; + } + + return DMB_DISJOINT; +} +EXPORT_SYMBOL(dmb_intersects); + +void __init dmb_init_region(struct memblock_region *region) +{ + unsigned long pfn; + int i; + + for (pfn = memblock_region_memory_base_pfn(region); + pfn < memblock_region_memory_end_pfn(region); + pfn += pageblock_nr_pages) { + struct page *page = pfn_to_page(pfn); + + for (i = 0; i < pageblock_nr_pages; i++) + set_page_zone(page + i, ZONE_MOVABLE); + + /* free reserved pageblocks to page allocator */ + init_reserved_pageblock(page); + } +} diff --git a/mm/memblock.c b/mm/memblock.c index 5d6a210d98ec..9eb91acdeb75 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include @@ -2090,13 +2091,16 @@ static void __init memmap_init_reserved_pages(void) for_each_reserved_mem_range(i, &start, &end) reserve_bootmem_region(start, end); - /* and also treat struct pages for the NOMAP regions as PageReserved */ for_each_mem_region(region) { + /* treat struct pages for the NOMAP regions as PageReserved */ if (memblock_is_nomap(region)) { start = region->base; end = start + region->size; reserve_bootmem_region(start, end); } + /* move Designated Movable Block pages to ZONE_MOVABLE */ + if (memblock_is_movable(region)) + dmb_init_region(region); } } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 1682d8815efa..e723094d1e1e 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -75,6 +75,7 @@ #include #include #include +#include #include #include #include @@ -433,6 +434,7 @@ static unsigned long required_kernelcore __initdata; static unsigned long required_kernelcore_percent __initdata; static unsigned long required_movablecore __initdata; static unsigned long required_movablecore_percent __initdata; +static unsigned long min_dmb_pfn[MAX_NUMNODES] __initdata; static unsigned long zone_movable_pfn[MAX_NUMNODES] __initdata; bool mirrored_kernelcore __initdata_memblock; @@ -2165,7 +2167,7 @@ static int __init deferred_init_memmap(void *data) } zone_empty: /* Sanity check that the next zone really is unpopulated */ - WARN_ON(++zid < MAX_NR_ZONES && populated_zone(++zone)); + WARN_ON(++zid < ZONE_MOVABLE && populated_zone(++zone)); pr_info("node %d deferred pages initialised in %ums\n", pgdat->node_id, jiffies_to_msecs(jiffies - start)); @@ -6899,6 +6901,10 @@ static void __init memmap_init_zone_range(struct zone *zone, unsigned long zone_end_pfn = zone_start_pfn + zone->spanned_pages; int nid = zone_to_nid(zone), zone_id = zone_idx(zone); + /* Skip overlap of ZONE_MOVABLE */ + if (zone_id == ZONE_MOVABLE && zone_start_pfn < *hole_pfn) + zone_start_pfn = *hole_pfn; + start_pfn = clamp(start_pfn, zone_start_pfn, zone_end_pfn); end_pfn = clamp(end_pfn, zone_start_pfn, zone_end_pfn); @@ -7348,6 +7354,9 @@ static unsigned long __init zone_spanned_pages_in_node(int nid, node_start_pfn, node_end_pfn, zone_start_pfn, zone_end_pfn); + if (zone_type == ZONE_MOVABLE && min_dmb_pfn[nid]) + *zone_start_pfn = min(*zone_start_pfn, min_dmb_pfn[nid]); + /* Check that this node has pages within the zone's required range */ if (*zone_end_pfn < node_start_pfn || *zone_start_pfn > node_end_pfn) return 0; @@ -7416,12 +7425,17 @@ static unsigned long __init zone_absent_pages_in_node(int nid, &zone_start_pfn, &zone_end_pfn); nr_absent = __absent_pages_in_range(nid, zone_start_pfn, zone_end_pfn); + if (zone_type == ZONE_MOVABLE && min_dmb_pfn[nid]) { + zone_start_pfn = min(zone_start_pfn, min_dmb_pfn[nid]); + nr_absent += zone_movable_pfn[nid] - zone_start_pfn; + } + /* * ZONE_MOVABLE handling. - * Treat pages to be ZONE_MOVABLE in ZONE_NORMAL as absent pages + * Treat pages to be ZONE_MOVABLE in other zones as absent pages * and vice versa. */ - if (mirrored_kernelcore && zone_movable_pfn[nid]) { + if (zone_movable_pfn[nid]) { unsigned long start_pfn, end_pfn; struct memblock_region *r; @@ -7431,6 +7445,21 @@ static unsigned long __init zone_absent_pages_in_node(int nid, end_pfn = clamp(memblock_region_memory_end_pfn(r), zone_start_pfn, zone_end_pfn); + if (memblock_is_movable(r)) { + if (zone_type != ZONE_MOVABLE) { + nr_absent += end_pfn - start_pfn; + continue; + } + + end_pfn = min(end_pfn, zone_movable_pfn[nid]); + if (start_pfn < zone_movable_pfn[nid]) + nr_absent -= end_pfn - start_pfn; + continue; + } + + if (!mirrored_kernelcore) + continue; + if (zone_type == ZONE_MOVABLE && memblock_is_mirror(r)) nr_absent += end_pfn - start_pfn; @@ -7450,6 +7479,15 @@ static void __init calculate_node_totalpages(struct pglist_data *pgdat, { unsigned long realtotalpages = 0; enum zone_type i; + int nid = pgdat->node_id; + + /* + * If Designated Movable Blocks are defined on this node, ensure that + * zone_movable_pfn is also defined for this node. + */ + if (min_dmb_pfn[nid] && !zone_movable_pfn[nid]) + zone_movable_pfn[nid] = min(node_end_pfn, + arch_zone_highest_possible_pfn[movable_zone]); for (i = 0; i < MAX_NR_ZONES; i++) { struct zone *zone = pgdat->node_zones + i; @@ -7457,12 +7495,12 @@ static void __init calculate_node_totalpages(struct pglist_data *pgdat, unsigned long spanned, absent; unsigned long size, real_size; - spanned = zone_spanned_pages_in_node(pgdat->node_id, i, + spanned = zone_spanned_pages_in_node(nid, i, node_start_pfn, node_end_pfn, &zone_start_pfn, &zone_end_pfn); - absent = zone_absent_pages_in_node(pgdat->node_id, i, + absent = zone_absent_pages_in_node(nid, i, node_start_pfn, node_end_pfn); @@ -7922,15 +7960,23 @@ unsigned long __init find_min_pfn_with_active_regions(void) static unsigned long __init early_calculate_totalpages(void) { unsigned long totalpages = 0; - unsigned long start_pfn, end_pfn; - int i, nid; + struct memblock_region *r; - for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, &nid) { - unsigned long pages = end_pfn - start_pfn; + for_each_mem_region(r) { + unsigned long start_pfn, end_pfn, pages; + int nid; - totalpages += pages; - if (pages) + nid = memblock_get_region_node(r); + start_pfn = memblock_region_memory_base_pfn(r); + end_pfn = memblock_region_memory_end_pfn(r); + + pages = end_pfn - start_pfn; + if (pages) { + totalpages += pages; node_set_state(nid, N_MEMORY); + if (memblock_is_movable(r) && !min_dmb_pfn[nid]) + min_dmb_pfn[nid] = start_pfn; + } } return totalpages; } @@ -7943,7 +7989,7 @@ static unsigned long __init early_calculate_totalpages(void) */ static void __init find_zone_movable_pfns_for_nodes(void) { - int i, nid; + int nid; unsigned long usable_startpfn; unsigned long kernelcore_node, kernelcore_remaining; /* save the state before borrow the nodemask */ @@ -8071,13 +8117,24 @@ static void __init find_zone_movable_pfns_for_nodes(void) kernelcore_remaining = kernelcore_node; /* Go through each range of PFNs within this node */ - for_each_mem_pfn_range(i, nid, &start_pfn, &end_pfn, NULL) { + for_each_mem_region(r) { unsigned long size_pages; + if (memblock_get_region_node(r) != nid) + continue; + + start_pfn = memblock_region_memory_base_pfn(r); + end_pfn = memblock_region_memory_end_pfn(r); start_pfn = max(start_pfn, zone_movable_pfn[nid]); if (start_pfn >= end_pfn) continue; + /* Skip over Designated Movable Blocks */ + if (memblock_is_movable(r)) { + zone_movable_pfn[nid] = end_pfn; + continue; + } + /* Account for what is only usable for kernelcore */ if (start_pfn < usable_startpfn) { unsigned long kernel_pages; @@ -8226,6 +8283,7 @@ void __init free_area_init(unsigned long *max_zone_pfn) } /* Find the PFNs that ZONE_MOVABLE begins at in each node */ + memset(min_dmb_pfn, 0, sizeof(min_dmb_pfn)); memset(zone_movable_pfn, 0, sizeof(zone_movable_pfn)); find_zone_movable_pfns_for_nodes(); From patchwork Tue Sep 13 19:55:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975322 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DBAB4C54EE9 for ; Tue, 13 Sep 2022 19:58:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 78EC36B0093; Tue, 13 Sep 2022 15:58:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 73E466B0095; Tue, 13 Sep 2022 15:58:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B8468D0001; Tue, 13 Sep 2022 15:58:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 498CA6B0093 for ; Tue, 13 Sep 2022 15:58:47 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 215A612017A for ; Tue, 13 Sep 2022 19:58:47 +0000 (UTC) X-FDA: 79908125052.28.CC663F2 Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) by imf02.hostedemail.com (Postfix) with ESMTP id D09B8800AB for ; Tue, 13 Sep 2022 19:58:46 +0000 (UTC) Received: by mail-qt1-f176.google.com with SMTP id y2so9616387qtv.5 for ; Tue, 13 Sep 2022 12:58:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=Q4ZU5cexm9Q08qLiEijmmMTjwuY3GlJ1Cn90C3bzJMU=; b=DPtm4ok3PSf82mq7JMYxEELlSbe2Rh/g3fQ51ooEmzdbhvAmX7bq1p/PDmYadgyeC1 zi9y/jcGd3D1R6ZJkZ/FQsWIwfFLD+Pb0MJITbf3fgUqnK83ONtjfJ+HMIIxaq6Be8T0 7yW02a5Gpkn6GV6C9TtbKwXcBYYQfckEhhDlRw3kQGNLccCEAsysOi1687eQCVwa1Snz UzUeXpguUoApWzimdM3XamY7zoYoXwniOBrFXnFh1C31eUhw4i9vT9n/3okZ8ZMJuUbE x2Q9CpCURykZLJagLzDO5v/o6/9ctT6KeQ6BSDQrVAvGz6WE7hB0JQGNvnw+QEyD9e4R S75w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=Q4ZU5cexm9Q08qLiEijmmMTjwuY3GlJ1Cn90C3bzJMU=; b=XYBPsAxRTfWT2k2WGcuDZXnhbifg2tmP+IHBgc9nNkG7H9tqlfJHWrMKeGkGm+x1WX XS+JYRMbEMRdTJf0lg+UiYCg6DtPI+UMZX72tawJ/gc5w72pN1pzjYrLwa0Y6RBawR+4 51HRdr2qmM5MwDz5JdvXeR9ZMHijenkBAOPDP3DxArs5erkS7FiGLmCU5ezHs76NBVE2 b+PX1/4VUlLN0Lme/deozQazC6MZ+lulPizs1jd/1eCAYN/7g80kFqZduTu+Tz77IaZC T4lEtgDdC+1T3jC08WJgIx+xU7lfzyRKBAQabU/Qd7DqdcoBQdH1PeV00q8Mgn/b21Cj VAfg== X-Gm-Message-State: ACgBeo3bjnVRQQ925bCi5P95M4Xsv1qMzDWcFTkVMs2sLMkDn+5vT6W3 sATanP6WcEzw8MdTxQDll7Q= X-Google-Smtp-Source: AA6agR79LKnSKwsuEjaHOZbqMf9sDJeff1s7ltAoeOd0yWLYcv6FI23B3yp+SehZLzBtT5Q7988TlQ== X-Received: by 2002:a05:622a:174b:b0:343:1fc:14d8 with SMTP id l11-20020a05622a174b00b0034301fc14d8mr29317695qtk.579.1663099126048; Tue, 13 Sep 2022 12:58:46 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:45 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 14/21] mm/page_alloc: make alloc_contig_pages DMB aware Date: Tue, 13 Sep 2022 12:55:01 -0700 Message-Id: <20220913195508.3511038-15-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099126; a=rsa-sha256; cv=none; b=StO7XWXScHF1l+dg15b37BY4Zl1w8nXNsl1UDPM2aKRmkf0Srja/IYS7Ut6LycasYeTmJ4 iDMhmUUBs4HcE/QqBOypWv4EUJhJKysyrEKNL2+Q3ymJWFFj2IWu/sTAYl41lav2/EG1Pf ef4DLeWvwdHiE6Jv5k3NHSa4ky/cBG4= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=DPtm4ok3; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of opendmb@gmail.com designates 209.85.160.176 as permitted sender) smtp.mailfrom=opendmb@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099126; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Q4ZU5cexm9Q08qLiEijmmMTjwuY3GlJ1Cn90C3bzJMU=; b=5O/Ju8bAS3KsFFeJLac3KqLiouI/BYI453CKw35onZlDeQdtYDVdKtVMpPIIgRPAsNzCVR hHMcrwLAEOcd4sLMR0qur+dw1W00reeGcOVl0pWmN+srMrb+lqf7TxS2HJR1dhnt82degz SFkq3IIdLyXoyKU8gQaBBTCORvOGeNY= Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=DPtm4ok3; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of opendmb@gmail.com designates 209.85.160.176 as permitted sender) smtp.mailfrom=opendmb@gmail.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: D09B8800AB X-Stat-Signature: 5sii3bkui4348zi36zqgehpyehagr5a7 X-Rspam-User: X-HE-Tag: 1663099126-925498 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Designated Movable Blocks are skipped when attempting to allocate contiguous pages. Doing per page validation across all spanned pages within a zone can be extra inefficient when Designated Movable Blocks create large overlaps between zones. Use dmb_intersects() within pfn_range_valid_contig as an early check to signal the range is not valid. The zone_movable_pfn array which represents the start of non- overlapped ZONE_MOVABLE on the node is now preserved to be used at runtime to skip over any DMB-only portion of the zone. Signed-off-by: Doug Berger --- mm/page_alloc.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e723094d1e1e..69753cc51e19 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -435,7 +435,7 @@ static unsigned long required_kernelcore_percent __initdata; static unsigned long required_movablecore __initdata; static unsigned long required_movablecore_percent __initdata; static unsigned long min_dmb_pfn[MAX_NUMNODES] __initdata; -static unsigned long zone_movable_pfn[MAX_NUMNODES] __initdata; +static unsigned long zone_movable_pfn[MAX_NUMNODES]; bool mirrored_kernelcore __initdata_memblock; /* movable_zone is the "real" zone pages in ZONE_MOVABLE are taken from */ @@ -9370,6 +9370,9 @@ static bool pfn_range_valid_contig(struct zone *z, unsigned long start_pfn, unsigned long i, end_pfn = start_pfn + nr_pages; struct page *page; + if (dmb_intersects(start_pfn, end_pfn)) + return false; + for (i = start_pfn; i < end_pfn; i++) { page = pfn_to_online_page(i); if (!page) @@ -9426,7 +9429,10 @@ struct page *alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask, gfp_zone(gfp_mask), nodemask) { spin_lock_irqsave(&zone->lock, flags); - pfn = ALIGN(zone->zone_start_pfn, nr_pages); + if (zone_idx(zone) == ZONE_MOVABLE && zone_movable_pfn[nid]) + pfn = ALIGN(zone_movable_pfn[nid], nr_pages); + else + pfn = ALIGN(zone->zone_start_pfn, nr_pages); while (zone_spans_last_pfn(zone, pfn, nr_pages)) { if (pfn_range_valid_contig(zone, pfn, nr_pages)) { /* From patchwork Tue Sep 13 19:55:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D592EECAAD8 for ; Tue, 13 Sep 2022 19:58:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7B4AB6B0074; Tue, 13 Sep 2022 15:58:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 764DD6B0096; Tue, 13 Sep 2022 15:58:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6076A8D0001; Tue, 13 Sep 2022 15:58:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 503376B0074 for ; Tue, 13 Sep 2022 15:58:54 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 315941A096F for ; Tue, 13 Sep 2022 19:58:54 +0000 (UTC) X-FDA: 79908125388.04.714C770 Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) by imf29.hostedemail.com (Postfix) with ESMTP id E5F7B1200E2 for ; Tue, 13 Sep 2022 19:58:53 +0000 (UTC) Received: by mail-qk1-f180.google.com with SMTP id c19so7402650qkm.7 for ; Tue, 13 Sep 2022 12:58:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=0ytqVsMPjuVvTMmUHQDgw++YWpDMZOL8Z97lPlE2FVY=; b=io9Nt4/RcZ2YZMphnT4cANVUYCQB6U7tpDHWWA0ymeRAMaiACQe59CDdxV3fKa+Hsn xNls20lsFyfw6rNS33zM5vtcFodTEOrCxgBbNhbWX8T1mEeMX+VwfC33CIm+xfP758LM zM7bYSmAZtJOGaBc2/pCs2H6YwEWsP5kO079m4wM+f6s/zfDbHxnmlkNGrxVR7sLq2J2 vi+KzltoxbcgQVb6NYjPdS8rs20FudQRcZ6AiEV/jd58a8RhxFomEIPJSEoUVrySG9CU fGCpRyRxdV4msiy4sXSvtH2LCG/1JOnoJrLiDkFSx8Zr9e7tZ4S5+YfhjAOUxZaXHtxy O3kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=0ytqVsMPjuVvTMmUHQDgw++YWpDMZOL8Z97lPlE2FVY=; b=WqubFxoNNTUGfXhLD67F5Ik0YgRP+6lrvXvmMO7FEyfl8JFmsTv+93gEH/JS013XpD sc07tNtusl3jmOhpUwdXpDy+bGhyN+GepSRfIWGNKDeDdcp6sx4AMh7cYHF4d6vmJf0r /7LHINZfFRjdtUhQoa6ucwHHrZpFigCG1OizI3fDO1LwEdlMv3G7Fx3ofeUqLQrrkTMG lUNF72qPIlqN64FiIAVPwCnPxVW/rVHoYHksvFpg+1WuNU3vdFf/ax3JgmhokeSSuEER 7kVtf/fV7GdoW69+FRNSS436DuiweUpSRKSd1kqNiFkFs3XWVQHPbJAAr4W8sHwTRix5 EdBg== X-Gm-Message-State: ACgBeo0dlHvp0f/AFneq58ovp527bnbKECuM2n6bwv/op0rtEem7SFiM 2SyCQZ8/L0ViywGmZxUdp0U= X-Google-Smtp-Source: AA6agR6d/FiwNOHoDQ8OKNxH916Sryp0qXJllILzqqegobMzkZRMGgPUcF2Qjw5zlXSo5I0Sdx+6eA== X-Received: by 2002:a05:620a:4407:b0:6cb:e111:32c2 with SMTP id v7-20020a05620a440700b006cbe11132c2mr18923399qkp.415.1663099133214; Tue, 13 Sep 2022 12:58:53 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:52 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 16/21] dt-bindings: reserved-memory: introduce designated-movable-block Date: Tue, 13 Sep 2022 12:55:03 -0700 Message-Id: <20220913195508.3511038-17-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099133; a=rsa-sha256; cv=none; b=nMXddA51Ql6WSU2B+q0+uq05hS9T+JIGpo2iLu7gYxsms71G/FoaUU32oDM/jDM2X5Ycum CxtwhO4c6pfkStWAyV8G2aDlt00K+mM6Rb9ZhdAJiytFDlZhMzUcz4WMzZiYVGxv2RmBjc jo51X2pMnlvk4AqU2rRt2tPGpJ3tF7E= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="io9Nt4/R"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of opendmb@gmail.com designates 209.85.222.180 as permitted sender) smtp.mailfrom=opendmb@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099133; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0ytqVsMPjuVvTMmUHQDgw++YWpDMZOL8Z97lPlE2FVY=; b=RYLqfgLrWcB9ixrsgPPkGFNzDEg5WOnM9L32zBu8iwCciMaXaORt6aTxjZU5djFefom6hU LRYHfFzZ7i1kf/sdfaCic8/w3IW53rIhOurnEWS+dcZuKzYvlym7GA2R1Oq13cQyrreAX3 u+RN+SW64cZ7uxCi/NsdWsf3sTmx7pk= Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="io9Nt4/R"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of opendmb@gmail.com designates 209.85.222.180 as permitted sender) smtp.mailfrom=opendmb@gmail.com X-Rspam-User: X-Rspamd-Server: rspam01 X-Stat-Signature: pz7zsyxe93nc7oraucaoi4i3kce1hmgr X-Rspamd-Queue-Id: E5F7B1200E2 X-HE-Tag: 1663099133-789273 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce designated-movable-block.yaml to document the devicetree binding for Designated Movable Block children of the reserved-memory node. Signed-off-by: Doug Berger --- .../designated-movable-block.yaml | 51 +++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 Documentation/devicetree/bindings/reserved-memory/designated-movable-block.yaml diff --git a/Documentation/devicetree/bindings/reserved-memory/designated-movable-block.yaml b/Documentation/devicetree/bindings/reserved-memory/designated-movable-block.yaml new file mode 100644 index 000000000000..42f846069a2e --- /dev/null +++ b/Documentation/devicetree/bindings/reserved-memory/designated-movable-block.yaml @@ -0,0 +1,51 @@ +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause) +%YAML 1.2 +--- +$id: http://devicetree.org/schemas/reserved-memory/designated-movable-block.yaml# +$schema: http://devicetree.org/meta-schemas/core.yaml# + +title: /reserved-memory Designated Movable Block node binding + +maintainers: + - devicetree-spec@vger.kernel.org + +allOf: + - $ref: "reserved-memory.yaml" + +properties: + compatible: + const: designated-movable-block + description: + This indicates a region of memory meant to be placed into + ZONE_MOVABLE. + +unevaluatedProperties: false + +required: + - compatible + - reusable + +examples: + - | + reserved-memory { + #address-cells = <0x2>; + #size-cells = <0x2>; + + DMB0@10800000 { + compatible = "designated-movable-block"; + reusable; + reg = <0x0 0x10800000 0x0 0x2d800000>; + }; + + DMB1@40000000 { + compatible = "designated-movable-block"; + reusable; + reg = <0x0 0x40000000 0x0 0x30000000>; + }; + + DMB2@80000000 { + compatible = "designated-movable-block"; + reusable; + reg = <0x0 0x80000000 0x0 0x2fc00000>; + }; + }; From patchwork Tue Sep 13 19:55:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975324 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0A01C54EE9 for ; Tue, 13 Sep 2022 19:58:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F6936B007E; Tue, 13 Sep 2022 15:58:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4A5A06B0096; Tue, 13 Sep 2022 15:58:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3201F8D0001; Tue, 13 Sep 2022 15:58:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 21D346B007E for ; Tue, 13 Sep 2022 15:58:58 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id F3D411C6645 for ; Tue, 13 Sep 2022 19:58:57 +0000 (UTC) X-FDA: 79908125514.06.A7D74E0 Received: from mail-qk1-f170.google.com (mail-qk1-f170.google.com [209.85.222.170]) by imf22.hostedemail.com (Postfix) with ESMTP id AB227C00C3 for ; Tue, 13 Sep 2022 19:58:57 +0000 (UTC) Received: by mail-qk1-f170.google.com with SMTP id q11so6264603qkc.12 for ; Tue, 13 Sep 2022 12:58:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=U4jOrSu6bqJBytWJghR6t0ljWrrZGLSB9ph/v+e9V8I=; b=D/mTJk6kZw64eyUhHPaSFfxR9Msa5UBxyOTlcsmCwhUVjiAhTk7oqnxftuemkUJs0o e/3ecILYiDZtf1ZduEhchjc87lM/UzBWANhyHjDCeSCCZZ+OsXTubHr/HBHsItOzKdH7 p/GQO/8nQMFRqeejgZsBOVzCwSXLRfYO2jhkP7CDsujDhgqd9989s5l2qAcx+CNYHM2D 3FzlZrQmr76F2aSmz5wNYnpRQxzHx962ONlutGAvPr/+dXcOvxUiHsk1k1ee7yaWbHVI 2LCJzHzqV9IGkV4gb1i+myYCzvATpnUG8sQPbUiE3I3lJUeSHhrVFH3mITfi1tyJU0b1 X7Mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=U4jOrSu6bqJBytWJghR6t0ljWrrZGLSB9ph/v+e9V8I=; b=zG2N/pU2TgrszvLmPppmQdvNmE5q178ElrONFUJja9b0CbK3dxd2B63Pw5LTVAJGrs qphwm9/+0tzVmf32xn57b+SowQAz1TcWP6HafoKxfwvj4NzSk41XIZ+TZWa3Px37JFGV EXoJDDvcTviiA3rZWtfxpoS0k1hbyCmDe5iHvKVeORGMQ2Kufu+1Euz7+BXZZJricGhM 7Ncv0PYN6i8QUgrHW6gFwf8EDW3gjC+I5wy9qhhi70bYuJ4QFc596lkrv7Q88bSwXBa9 wzY4bvlDJzDFS80DZAj0rK39rs0mddHkeQthHefVKj6fv97R8eFv5fb1NUmZzknBQDED YHfg== X-Gm-Message-State: ACgBeo1hbNAYy4N8BSl9bXUlihObEtGAbzZf6b5c/E8cwLyrINvB1j/c 7Snicpem/sWSiApRcXeEGwk= X-Google-Smtp-Source: AA6agR42tNhbqU65weBklFefGDPSfeY7AMPrK26FTuyeT6UMnlp5fPm7LJh/3FGN4dhzcFbZNJ8Esw== X-Received: by 2002:a05:620a:2452:b0:6bb:d8ba:ca65 with SMTP id h18-20020a05620a245200b006bbd8baca65mr24199159qkn.263.1663099136924; Tue, 13 Sep 2022 12:58:56 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:58:56 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 17/21] mm/dmb: introduce rmem designated-movable-block Date: Tue, 13 Sep 2022 12:55:04 -0700 Message-Id: <20220913195508.3511038-18-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099137; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=U4jOrSu6bqJBytWJghR6t0ljWrrZGLSB9ph/v+e9V8I=; b=L4tq+Piq7EXO9fpQyE8ohM/Q851Le9VmVzmIvRXDmx83pplZ8wp6O0urlzNXJR9LhrKSEK f8vd3y2UqUrc3u9NMWUlKiHoicVFTyqwyhtTR845ACYsH34kO3QLpZgoYJDwo4swnO6z0q zRBB9irEVyaHjhtV7Y5jvqh9oPVLtBo= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="D/mTJk6k"; spf=pass (imf22.hostedemail.com: domain of opendmb@gmail.com designates 209.85.222.170 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099137; a=rsa-sha256; cv=none; b=8bn6fNJtDnKuIvo5oRMFoUhQq6DW/Llxh9hvZ1hvYdozY20y6dS4YlXLk0QbIT8sAfAIyz AYuuwuP98tNQo20e7vBAGT32qQ8ZQG+D5UnFe/KECc3yKUszZYdPlR69bGjWPt+3dBH6Ww SrAIwKYoG/naDWaF+BpEsZnCfJF7OH8= X-Rspam-User: X-Stat-Signature: n15kmu6w41hy9wugdd855npu3km3u17j X-Rspamd-Queue-Id: AB227C00C3 Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="D/mTJk6k"; spf=pass (imf22.hostedemail.com: domain of opendmb@gmail.com designates 209.85.222.170 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam08 X-HE-Tag: 1663099137-76670 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This commit allows Designated Movable Blocks to be created by including reserved-memory child nodes in the device tree with the "designated-movable-block" compatible string. Signed-off-by: Doug Berger --- drivers/of/of_reserved_mem.c | 15 ++++++--- mm/dmb.c | 64 ++++++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+), 5 deletions(-) diff --git a/drivers/of/of_reserved_mem.c b/drivers/of/of_reserved_mem.c index 65f3b02a0e4e..0eb9e8898d7b 100644 --- a/drivers/of/of_reserved_mem.c +++ b/drivers/of/of_reserved_mem.c @@ -23,6 +23,7 @@ #include #include #include +#include #include "of_private.h" @@ -113,12 +114,16 @@ static int __init __reserved_mem_alloc_size(unsigned long node, nomap = of_get_flat_dt_prop(node, "no-map", NULL) != NULL; - /* Need adjust the alignment to satisfy the CMA requirement */ - if (IS_ENABLED(CONFIG_CMA) - && of_flat_dt_is_compatible(node, "shared-dma-pool") - && of_get_flat_dt_prop(node, "reusable", NULL) - && !nomap) + if (of_flat_dt_is_compatible(node, "designated-movable-block")) { + /* Need adjust the alignment to satisfy the DMB requirement */ + align = max_t(phys_addr_t, align, DMB_MIN_ALIGNMENT_BYTES); + } else if (IS_ENABLED(CONFIG_CMA) + && of_flat_dt_is_compatible(node, "shared-dma-pool") + && of_get_flat_dt_prop(node, "reusable", NULL) + && !nomap) { + /* Need adjust the alignment to satisfy the CMA requirement */ align = max_t(phys_addr_t, align, CMA_MIN_ALIGNMENT_BYTES); + } prop = of_get_flat_dt_prop(node, "alloc-ranges", &len); if (prop) { diff --git a/mm/dmb.c b/mm/dmb.c index 9d9fd31089d2..8132d18542a0 100644 --- a/mm/dmb.c +++ b/mm/dmb.c @@ -90,3 +90,67 @@ void __init dmb_init_region(struct memblock_region *region) init_reserved_pageblock(page); } } + +/* + * Support for reserved memory regions defined in device tree + */ +#ifdef CONFIG_OF_RESERVED_MEM +#include +#include +#include + +#undef pr_fmt +#define pr_fmt(fmt) fmt + +static int rmem_dmb_device_init(struct reserved_mem *rmem, struct device *dev) +{ + struct dmb *dmb; + + dmb = (struct dmb *)rmem->priv; + if (dmb->owner) + return -EBUSY; + + dmb->owner = dev; + return 0; +} + +static void rmem_dmb_device_release(struct reserved_mem *rmem, + struct device *dev) +{ + struct dmb *dmb; + + dmb = (struct dmb *)rmem->priv; + if (dmb->owner == (void *)dev) + dmb->owner = NULL; +} + +static const struct reserved_mem_ops rmem_dmb_ops = { + .device_init = rmem_dmb_device_init, + .device_release = rmem_dmb_device_release, +}; + +static int __init rmem_dmb_setup(struct reserved_mem *rmem) +{ + unsigned long node = rmem->fdt_node; + struct dmb *dmb; + int err; + + if (!of_get_flat_dt_prop(node, "reusable", NULL) || + of_get_flat_dt_prop(node, "no-map", NULL)) + return -EINVAL; + + err = dmb_reserve(rmem->base, rmem->size, &dmb); + if (err) { + pr_err("Reserved memory: unable to setup DMB region\n"); + return err; + } + + rmem->priv = dmb; + rmem->ops = &rmem_dmb_ops; + pr_info("Reserved memory: created DMB at %pa, size %ld MiB\n", + &rmem->base, (unsigned long)rmem->size / SZ_1M); + + return 0; +} +RESERVEDMEM_OF_DECLARE(dmb, "designated-movable-block", rmem_dmb_setup); +#endif From patchwork Tue Sep 13 19:55:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62187C6FA82 for ; Tue, 13 Sep 2022 19:59:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 013556B0099; Tue, 13 Sep 2022 15:59:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F2CE46B009A; Tue, 13 Sep 2022 15:59:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA79B8D0001; Tue, 13 Sep 2022 15:59:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CB7106B0099 for ; Tue, 13 Sep 2022 15:59:01 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id AE107A0BD7 for ; Tue, 13 Sep 2022 19:59:01 +0000 (UTC) X-FDA: 79908125682.11.095F15F Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) by imf22.hostedemail.com (Postfix) with ESMTP id 716C1C00C2 for ; Tue, 13 Sep 2022 19:59:01 +0000 (UTC) Received: by mail-qt1-f177.google.com with SMTP id s18so6629362qtx.6 for ; Tue, 13 Sep 2022 12:59:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=fHU5ZUx5v+GumCekHv98dt5HtboTaZeeu0Gp9AlVmvA=; b=ZkQM2Xi16fac9Kxsd6JfpJUyj9dFTpL2a9fMxPQYb5tsP1AGTq9LrS3GQ23PdeUUAq KiAJoUd0eFmdtlIe31jI3xWYF3lNu2z6iPVbkhYktSoNmKAD7mOQ11g6lhzdx8PeDKqG lPt1LWxw7/FsEvYMWeWC/86MaywQjplVNZrIXdmUyLe5Nr4lwjgNum55ZdLf4luz/yKw pKJIZBfhsG2I2a6u/v5b8+uct94R+RrovFIPHm9V4i65FXFauBq/I0QO/UcUPxNQEMMR iJaQjUMSz8SAuIngfV/EGkXgYLEYcOdGnoAVuWBboXNtLXNpt0Nqru8cUMiF0ZW7269J 6y+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=fHU5ZUx5v+GumCekHv98dt5HtboTaZeeu0Gp9AlVmvA=; b=5QQnZiVqaF84cL4Jksipdzk8BbCl7y+1r42iMM/KhEltjsgD2Ak2EjJTW8qEIImlnX 7qwgv3GpJ6BhquyDrGVhxgjxj/uNby88fxQJFIcBD/gUHJDgn3bumfrxiyEPddtS1DIc ShI1RFq2D+ISASvylgF9Cg2Pipf970sMlacyCSmTdkIQVSipPRdpaA0fQUle0SNsHMlJ HON3u+St/KSqCFlezitHUQhio6Tz6l2chfrOTyG4CauvSP8DcUABN0hJfoLpg8xIJ2+f k72IJ/Q+NvqmIK1UArEHpZBdLxZ+OhebCkxg2X6whStcfvJgCr1Z2gl6ILaK3iF4Gl+e 6RtQ== X-Gm-Message-State: ACgBeo3Rzn+5n0VO/y6d6YdsYaJ49pZxjn/2+TxN/eikSD3apWZGTbhr 9tEgamf9L4sdaaGseV/4s3w= X-Google-Smtp-Source: AA6agR7yxxGK0L7lb4UY6ElTAIAR0o3jzl9zqXW4dIwQkZBtshIlrdqdJVz3avbMktN2rc9zUTYXIg== X-Received: by 2002:ac8:570a:0:b0:343:73af:7cd8 with SMTP id 10-20020ac8570a000000b0034373af7cd8mr30203309qtw.16.1663099140678; Tue, 13 Sep 2022 12:59:00 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.58.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:59:00 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 18/21] mm/cma: support CMA in Designated Movable Blocks Date: Tue, 13 Sep 2022 12:55:05 -0700 Message-Id: <20220913195508.3511038-19-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099141; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fHU5ZUx5v+GumCekHv98dt5HtboTaZeeu0Gp9AlVmvA=; b=qz/yftUNlPoBpKWgTsGAAvlbknBIar/ih0Ihg96e//vvr0h7grv3mW1qyW1yroMu/hOTEh IgvpYt+KjZYcHXbQSDh4EnmZUWcxpr6reSCSFapY4h7vToLJ87M4syW5W6f684Yt9ZhJEM 1CE/J6jCIFQZNw3zYe2TBt268AosANc= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ZkQM2Xi1; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of opendmb@gmail.com designates 209.85.160.177 as permitted sender) smtp.mailfrom=opendmb@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099141; a=rsa-sha256; cv=none; b=iYaBytCU0l460WS2b64fg/WfknDuGKWjsDka98L8F0NrppLRAb9hC8hd77zvLTO93qxRMH W7P69UYoSttCytgY0ScCFfNm4x4569oFz4opSuvV4x3hm9vpBNevoOe3ySl+W/Fzfq/rcs TSOU/ntBSd9XcyYHnbqAwhy41GPk5jo= Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ZkQM2Xi1; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of opendmb@gmail.com designates 209.85.160.177 as permitted sender) smtp.mailfrom=opendmb@gmail.com X-Stat-Signature: r1a4xikucnzyrozmwa5i8kmnhadbc5ch X-Rspamd-Queue-Id: 716C1C00C2 X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1663099141-308227 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This commit allows for different page allocator handling for CMA areas that are within Designated Movable Blocks. Specifically, the pageblocks are allowed to remain migratetype MIGRATE_MOVABLE to allow more aggressive utilization by the page allocator. This also means that the page allocator should not consider these pages as part of the nr_free_cma metric it uses for managing MIGRATE_CMA type pageblocks. This leads to the decision to remove these areas from the CmaTotal metrics after initialization to avoid confusion. Signed-off-by: Doug Berger Reported-by: kernel test robot Reported-by: kernel test robot Reported-by: kernel test robot --- include/linux/cma.h | 13 ++++++--- mm/cma.c | 55 +++++++++++++++++++++++++----------- mm/page_alloc.c | 69 +++++++++++++++++++++++++++++---------------- 3 files changed, 92 insertions(+), 45 deletions(-) diff --git a/include/linux/cma.h b/include/linux/cma.h index 63873b93deaa..ffbb8ea2c5f8 100644 --- a/include/linux/cma.h +++ b/include/linux/cma.h @@ -31,11 +31,13 @@ extern phys_addr_t cma_get_base(const struct cma *cma); extern unsigned long cma_get_size(const struct cma *cma); extern const char *cma_get_name(const struct cma *cma); -extern int __init cma_declare_contiguous_nid(phys_addr_t base, +extern int __init __cma_declare_contiguous_nid(phys_addr_t base, phys_addr_t size, phys_addr_t limit, phys_addr_t alignment, unsigned int order_per_bit, bool fixed, const char *name, struct cma **res_cma, - int nid); + int nid, bool in_dmb); +#define cma_declare_contiguous_nid(b, s, l, a, o, f, n, r_c, nid) \ + __cma_declare_contiguous_nid(b, s, l, a, o, f, n, r_c, nid, false) static inline int __init cma_declare_contiguous(phys_addr_t base, phys_addr_t size, phys_addr_t limit, phys_addr_t alignment, unsigned int order_per_bit, @@ -44,10 +46,13 @@ static inline int __init cma_declare_contiguous(phys_addr_t base, return cma_declare_contiguous_nid(base, size, limit, alignment, order_per_bit, fixed, name, res_cma, NUMA_NO_NODE); } -extern int cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, +extern int __cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, unsigned int order_per_bit, const char *name, - struct cma **res_cma); + struct cma **res_cma, + bool in_dmb); +#define cma_init_reserved_mem(base, size, order, name, res_cma) \ + __cma_init_reserved_mem(base, size, order, name, res_cma, 0) extern struct page *cma_alloc(struct cma *cma, unsigned long count, unsigned int align, bool no_warn); extern bool cma_pages_valid(struct cma *cma, const struct page *pages, unsigned long count); diff --git a/mm/cma.c b/mm/cma.c index 6208a3e1cd9d..4f33cd54db9e 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -33,6 +33,7 @@ #include #include #include +#include #include "cma.h" @@ -98,6 +99,10 @@ static void __init cma_activate_area(struct cma *cma) { unsigned long base_pfn = cma->base_pfn, pfn; struct zone *zone; + int is_dmb = dmb_intersects(base_pfn, base_pfn + cma->count); + + if (is_dmb == DMB_MIXED) + goto out_error; cma->bitmap = bitmap_zalloc(cma_bitmap_maxno(cma), GFP_KERNEL); if (!cma->bitmap) @@ -116,13 +121,17 @@ static void __init cma_activate_area(struct cma *cma) goto not_in_zone; } - for (pfn = base_pfn; pfn < base_pfn + cma->count; - pfn += pageblock_nr_pages) { - struct page *page = pfn_to_page(pfn); + if (is_dmb == DMB_INTERSECTS) { + totalcma_pages -= cma->count; + } else { + for (pfn = base_pfn; pfn < base_pfn + cma->count; + pfn += pageblock_nr_pages) { + struct page *page = pfn_to_page(pfn); - set_pageblock_migratetype(page, MIGRATE_CMA); - init_reserved_pageblock(page); - page_zone(page)->cma_pages += pageblock_nr_pages; + set_pageblock_migratetype(page, MIGRATE_CMA); + init_reserved_pageblock(page); + page_zone(page)->cma_pages += pageblock_nr_pages; + } } spin_lock_init(&cma->lock); @@ -141,7 +150,8 @@ static void __init cma_activate_area(struct cma *cma) if (!cma->reserve_pages_on_error) { for (pfn = base_pfn; pfn < base_pfn + cma->count; pfn += pageblock_nr_pages) - init_reserved_pageblock(pfn_to_page(pfn)); + if (!dmb_intersects(pfn, pfn + pageblock_nr_pages)) + init_reserved_pageblock(pfn_to_page(pfn)); } totalcma_pages -= cma->count; cma->count = 0; @@ -166,7 +176,7 @@ void __init cma_reserve_pages_on_error(struct cma *cma) } /** - * cma_init_reserved_mem() - create custom contiguous area from reserved memory + * __cma_init_reserved_mem() - create custom contiguous area in reserved memory * @base: Base address of the reserved area * @size: Size of the reserved area (in bytes), * @order_per_bit: Order of pages represented by one bit on bitmap. @@ -174,15 +184,18 @@ void __init cma_reserve_pages_on_error(struct cma *cma) * the area will be set to "cmaN", where N is a running counter of * used areas. * @res_cma: Pointer to store the created cma region. + * @in_dmb: Designate the reserved memory as a Designated Movable Block. * * This function creates custom contiguous area from already reserved memory. */ -int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, - unsigned int order_per_bit, - const char *name, - struct cma **res_cma) +int __init __cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, + unsigned int order_per_bit, + const char *name, + struct cma **res_cma, + bool in_dmb) { struct cma *cma; + int err; /* Sanity checks */ if (cma_area_count == ARRAY_SIZE(cma_areas)) { @@ -201,6 +214,14 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, if (!IS_ALIGNED(base | size, CMA_MIN_ALIGNMENT_BYTES)) return -EINVAL; + if (in_dmb) { + err = dmb_reserve(base, size, NULL); + if (err) { + pr_err("Cannot reserve DMB for CMA!\n"); + return err; + } + } + /* * Each reserved area must be initialised later, when more kernel * subsystems (like slab allocator) are available. @@ -223,7 +244,7 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, } /** - * cma_declare_contiguous_nid() - reserve custom contiguous area + * __cma_declare_contiguous_nid() - reserve custom contiguous area * @base: Base address of the reserved area optional, use 0 for any * @size: Size of the reserved area (in bytes), * @limit: End address of the reserved memory (optional, 0 for any). @@ -233,6 +254,7 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, * @name: The name of the area. See function cma_init_reserved_mem() * @res_cma: Pointer to store the created cma region. * @nid: nid of the free area to find, %NUMA_NO_NODE for any node + * @in_dmb: Designate the reserved memory as a Designated Movable Block. * * This function reserves memory from early allocator. It should be * called by arch specific code once the early allocator (memblock or bootmem) @@ -242,11 +264,11 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, * If @fixed is true, reserve contiguous area at exactly @base. If false, * reserve in range from @base to @limit. */ -int __init cma_declare_contiguous_nid(phys_addr_t base, +int __init __cma_declare_contiguous_nid(phys_addr_t base, phys_addr_t size, phys_addr_t limit, phys_addr_t alignment, unsigned int order_per_bit, bool fixed, const char *name, struct cma **res_cma, - int nid) + int nid, bool in_dmb) { phys_addr_t memblock_end = memblock_end_of_DRAM(); phys_addr_t highmem_start; @@ -374,7 +396,8 @@ int __init cma_declare_contiguous_nid(phys_addr_t base, base = addr; } - ret = cma_init_reserved_mem(base, size, order_per_bit, name, res_cma); + ret = __cma_init_reserved_mem(base, size, order_per_bit, name, res_cma, + in_dmb); if (ret) goto free_mem; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e38dd1b32771..09d00c178bc8 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -9233,29 +9233,8 @@ int __alloc_contig_migrate_range(struct compact_control *cc, return 0; } -/** - * alloc_contig_range() -- tries to allocate given range of pages - * @start: start PFN to allocate - * @end: one-past-the-last PFN to allocate - * @migratetype: migratetype of the underlying pageblocks (either - * #MIGRATE_MOVABLE or #MIGRATE_CMA). All pageblocks - * in range must have the same migratetype and it must - * be either of the two. - * @gfp_mask: GFP mask to use during compaction - * - * The PFN range does not have to be pageblock aligned. The PFN range must - * belong to a single zone. - * - * The first thing this routine does is attempt to MIGRATE_ISOLATE all - * pageblocks in the range. Once isolated, the pageblocks should not - * be modified by others. - * - * Return: zero on success or negative error code. On success all - * pages which PFN is in [start, end) are allocated for the caller and - * need to be freed with free_contig_range(). - */ -int alloc_contig_range(unsigned long start, unsigned long end, - unsigned migratetype, gfp_t gfp_mask) +int _alloc_contig_range(unsigned long start, unsigned long end, + unsigned int migratetype, gfp_t gfp_mask) { unsigned long outer_start, outer_end; int order; @@ -9379,6 +9358,46 @@ int alloc_contig_range(unsigned long start, unsigned long end, undo_isolate_page_range(start, end, migratetype); return ret; } + +/** + * alloc_contig_range() -- tries to allocate given range of pages + * @start: start PFN to allocate + * @end: one-past-the-last PFN to allocate + * @migratetype: migratetype of the underlying pageblocks (either + * #MIGRATE_MOVABLE or #MIGRATE_CMA). All pageblocks + * in range must have the same migratetype and it must + * be either of the two. + * @gfp_mask: GFP mask to use during compaction + * + * The PFN range does not have to be pageblock aligned. The PFN range must + * belong to a single zone. + * + * The first thing this routine does is attempt to MIGRATE_ISOLATE all + * pageblocks in the range. Once isolated, the pageblocks should not + * be modified by others. + * + * Return: zero on success or negative error code. On success all + * pages which PFN is in [start, end) are allocated for the caller and + * need to be freed with free_contig_range(). + */ +int alloc_contig_range(unsigned long start, unsigned long end, + unsigned int migratetype, gfp_t gfp_mask) +{ + switch (dmb_intersects(start, end)) { + case DMB_DISJOINT: + break; + case DMB_INTERSECTS: + if (migratetype == MIGRATE_CMA) + migratetype = MIGRATE_MOVABLE; + else + return -EBUSY; + break; + default: + return -EBUSY; + } + + return _alloc_contig_range(start, end, migratetype, gfp_mask); +} EXPORT_SYMBOL(alloc_contig_range); static int __alloc_contig_pages(unsigned long start_pfn, @@ -9386,8 +9405,8 @@ static int __alloc_contig_pages(unsigned long start_pfn, { unsigned long end_pfn = start_pfn + nr_pages; - return alloc_contig_range(start_pfn, end_pfn, MIGRATE_MOVABLE, - gfp_mask); + return _alloc_contig_range(start_pfn, end_pfn, MIGRATE_MOVABLE, + gfp_mask); } static bool pfn_range_valid_contig(struct zone *z, unsigned long start_pfn, From patchwork Tue Sep 13 19:55:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975326 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41B3BECAAD8 for ; Tue, 13 Sep 2022 19:59:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B391B6B009B; Tue, 13 Sep 2022 15:59:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AE7D98D0001; Tue, 13 Sep 2022 15:59:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 960D46B009D; Tue, 13 Sep 2022 15:59:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 843226B009B for ; Tue, 13 Sep 2022 15:59:05 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 618EEA0A46 for ; Tue, 13 Sep 2022 19:59:05 +0000 (UTC) X-FDA: 79908125850.15.E9CE418 Received: from mail-qk1-f177.google.com (mail-qk1-f177.google.com [209.85.222.177]) by imf06.hostedemail.com (Postfix) with ESMTP id 0FA4518008B for ; Tue, 13 Sep 2022 19:59:04 +0000 (UTC) Received: by mail-qk1-f177.google.com with SMTP id g2so6482749qkk.1 for ; Tue, 13 Sep 2022 12:59:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=Iv6LAHuQtTXeYWyq0DEDKZbOD/1nb3ztqrePD2HtkIE=; b=YrAFBuuyJkwRgGb5yJxtoTNNKh/7y4BOHe54UWWbwGw/7zVyvD5YS3xh6wOizb8P5y WPvQ65cSNAdgAQQ5wDi83N9jAKpum/P9kqfXIHyP2JYI4r5D1linVF+pAeO7tMTGeNJd s10XvBpPDdlS4NQdOIfsc4Y86zu8zbnj/iRe9JIr+LJQn5lsr3jvpDq8YkHXv8CVoVQ9 l1Y7VxuX7aqVyF4T9IwYMUdPvuZE/+WFrgImMS1687EPItdw+VwpQNOwUoolbQ3Hf9aR Ox2qAlcyZj/+2EVV8HvGsz7p+nWwMiSQtDYItBS1H3jfaKi/gJ7kAUpfhR4jWWYZpfwT Vmow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=Iv6LAHuQtTXeYWyq0DEDKZbOD/1nb3ztqrePD2HtkIE=; b=o/HTyrsOwK3/b+x62civXHXVMylH97ZxyqJwH1HgDi/jIeAcbIyc3gVKMNa2ocPpwc Z1ezo48pdYvL8OVounkkYDeGbzOvwc5sl898eca1fw0Atld4T4SLDVEeyOgQHaAh9W0u PiiH0NRYjW5wuzSlXOCJYUUUu9lwqhg+64bmfPjSOyaohtnYB4SMhr6aYFiMHRcZewi6 wnEMAkC9QSh6HSKNF1ckqxxeuLBEqpAICTeXQuRtv4zmn5dgbTj2InEJs7f6rbsF3Bee DHHiB8628RqtktB7SS6nJGjeiWihsc/MWs016eaPxQGWJiB9vwC+vptrtcPJkuXbH0e3 GT8A== X-Gm-Message-State: ACgBeo348xzC/ybyhSXF3PIPUutBJZuaL0WmXCQryaD9z0/OYZO/yKPm Q/ithCyke0ojvisAHBpITt4= X-Google-Smtp-Source: AA6agR6fK874H1rNDTk5i/hsxGcmtY4AZlr+R557N3WVBFwFecth0fyDkTeEea5ukgS/JNJqcxXZjQ== X-Received: by 2002:a37:916:0:b0:6ce:5f1c:d5d9 with SMTP id 22-20020a370916000000b006ce5f1cd5d9mr5679706qkj.737.1663099144354; Tue, 13 Sep 2022 12:59:04 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.59.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:59:03 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 19/21] dt-bindings: reserved-memory: shared-dma-pool: support DMB Date: Tue, 13 Sep 2022 12:55:06 -0700 Message-Id: <20220913195508.3511038-20-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099145; a=rsa-sha256; cv=none; b=0bqLjK38AOVWoA2A5y6zISYRI9zRqJ270hgwdANBqww1bE1y5Q3ZhRFnyo6sjfcuMVlSea snAp37cvtQ5HwGx6dF5tacVk1SUwC9JsNFPVCr4tx+NrDv16gyZlZRnFzGcaWRT3wu9pF3 1/kx+aUKxmkxNhrl18Wm5MRGdnOklns= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=YrAFBuuy; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf06.hostedemail.com: domain of opendmb@gmail.com designates 209.85.222.177 as permitted sender) smtp.mailfrom=opendmb@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099145; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Iv6LAHuQtTXeYWyq0DEDKZbOD/1nb3ztqrePD2HtkIE=; b=v0y1Gs+CKbELEmPrXK2tyB9se+7d7cIJchYBIX4te+oWPb6rTXDeN7v5ZhDEH/bMp3EPuD lLNdWWFAn2UMyyTu/OwfYryi/zxEyM2eLiq78H/LscH+kCSAyDSCjXbYAegKPtmo/e3gIb yX7pHNF3/OrI5YKHxbvKKieQPv2A3Bw= X-Stat-Signature: ix5nc3z9pd1nesd4f8ayuyuioysajuix X-Rspamd-Queue-Id: 0FA4518008B X-Rspamd-Server: rspam03 X-Rspam-User: Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=YrAFBuuy; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf06.hostedemail.com: domain of opendmb@gmail.com designates 209.85.222.177 as permitted sender) smtp.mailfrom=opendmb@gmail.com X-HE-Tag: 1663099144-399686 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The shared-dmb-pool compatible string creates a Designated Movable Block to contain a shared pool of DMA buffers. Signed-off-by: Doug Berger --- .../bindings/reserved-memory/shared-dma-pool.yaml | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/Documentation/devicetree/bindings/reserved-memory/shared-dma-pool.yaml b/Documentation/devicetree/bindings/reserved-memory/shared-dma-pool.yaml index 618105f079be..85824fe05ac9 100644 --- a/Documentation/devicetree/bindings/reserved-memory/shared-dma-pool.yaml +++ b/Documentation/devicetree/bindings/reserved-memory/shared-dma-pool.yaml @@ -22,6 +22,14 @@ properties: operating system to instantiate the necessary pool management subsystem if necessary. + - const: shared-dmb-pool + description: > + This indicates a shared-dma-pool region that is located within + a Designated Movable Block. The operating system is free to + use unallocated memory for movable allocations in this region. + Devices need to be tolerant of allocation latency to use this + pool. + - const: restricted-dma-pool description: > This indicates a region of memory meant to be used as a pool From patchwork Tue Sep 13 19:55:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975327 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16176C6FA82 for ; Tue, 13 Sep 2022 19:59:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AC28B6B009D; Tue, 13 Sep 2022 15:59:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A717E6B009E; Tue, 13 Sep 2022 15:59:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8EC416B009F; Tue, 13 Sep 2022 15:59:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7FE5F6B009D for ; Tue, 13 Sep 2022 15:59:09 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 5D6301A04A0 for ; Tue, 13 Sep 2022 19:59:09 +0000 (UTC) X-FDA: 79908126018.04.F1D517D Received: from mail-qk1-f177.google.com (mail-qk1-f177.google.com [209.85.222.177]) by imf20.hostedemail.com (Postfix) with ESMTP id DF22D1C00C0 for ; Tue, 13 Sep 2022 19:59:08 +0000 (UTC) Received: by mail-qk1-f177.google.com with SMTP id d17so8540092qko.13 for ; Tue, 13 Sep 2022 12:59:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=lc6C5FZ3R6bpGdrTQfUMqHUthTTami+g742/pfBsTic=; b=RlsLmEiP3JVWhyXjCMSzF7A4AVjyq1jRjWCW9x5GNVTee31Y+zsIXxNeUFqHaiWHyg 8J6hP8RFMjTeETwSHV9ENC6pqQalmeYj6JweS9Tsv7OxuqvPlsMarmRSfwGBaWF+XcBE la0+kyQTMUqV4xn7LwqYCphDGQkUMF9O3ZqeG7FJEIlcBtt95RRsh5UONSCeySm6f9no qWos+YkhM9jVDPaT5oAH5RbFh07TQsfPjWCoTiWZNYZktg0UxgNqKCsIm4np7G5/9pqK Gqu0dk9U2X717zjfg5WZXwdvFki/e2iGeK+mAGJNsof/XN1i9elvjnIO7qYgQkqpgSuo A90g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=lc6C5FZ3R6bpGdrTQfUMqHUthTTami+g742/pfBsTic=; b=EdmZR4w8ZfMFC6ykFR4wfLThkFqFFlx9fQXUrQmDYrGIxeliOeWwc8oQx1VLpUMwfe OUjF8BjeNhK84qCjhOwmpZ2LmIPmEK2ATphogy2TY4SBafs+lfsGGvwRXgVwXkjAS9vE FBztZGvJ1wqtvIim2tfandYEOW+MM/zTWu6ykuog1i1wL34dbqqnQ1iCNu1M0bEfJ80P vrzLEVWPh0MDBRfQIjSDDhNgoiswuC5zxsJ5ZAscEJabtSvgcP/6RmP2T/a4Ldjxgt1R yIhGjqf9tpl0V0VSZIBWA5V0R3LHCK5ujYDlYNY5VMpf+FFctDEMY6FTVxuSj+JdHL7P VIAg== X-Gm-Message-State: ACgBeo1djVr2H0n1JqQrAIg0yRW+N2I64TnZ7BY24uQ4iC5UfTRmNg1b 9l1plN2ZESyI5/9L0S2H9cWyT2G3/VM= X-Google-Smtp-Source: AA6agR4JgnZFDc4dYiN2cHvYsrY/zIDRf84OViiKl4IHPDPBj/qqA4B2wHLv+IBu33IWNCFBvl5WRQ== X-Received: by 2002:a05:620a:22b0:b0:6cd:f06a:7978 with SMTP id p16-20020a05620a22b000b006cdf06a7978mr13621240qkh.106.1663099148142; Tue, 13 Sep 2022 12:59:08 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.59.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:59:07 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 20/21] mm/cma: introduce rmem shared-dmb-pool Date: Tue, 13 Sep 2022 12:55:07 -0700 Message-Id: <20220913195508.3511038-21-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099148; a=rsa-sha256; cv=none; b=0wvk5x3kmTkSIH9vY/lV3M8eVSxfA7rmDfkaunFL2AdBgs3gO1VY8S8gmWPPigdvUMAn0P vJOTEtoTJf4YgkiMW0qRfAOiZOEE+PqUbGCgSbCHIYOBR67x1i+fp/6kX3IK/agZx8L8ZL yBGASMzPzmETrAP37td4E4lvTZGJO7g= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=RlsLmEiP; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of opendmb@gmail.com designates 209.85.222.177 as permitted sender) smtp.mailfrom=opendmb@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099148; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lc6C5FZ3R6bpGdrTQfUMqHUthTTami+g742/pfBsTic=; b=nw/arHPZFM7UlrXoCx9bbuMbykbk3L1pVS/03bToA/Eh/zqVJV8cODonpQ0E4/z0fmBR2U 4hGI0CbqMGkeT5Mvuf0sUJQy2o2NeFJXjkxZJYn3xXsQ2Pnlfup3j+j90yeP3lecwFDBcn zJCvfeu7r0e2bTrnJ7IB1cbgOeyOw/U= Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=RlsLmEiP; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of opendmb@gmail.com designates 209.85.222.177 as permitted sender) smtp.mailfrom=opendmb@gmail.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: DF22D1C00C0 X-Stat-Signature: ecicp8jc6uysd5pb9z33r8mhtmcky6kr X-Rspam-User: X-HE-Tag: 1663099148-168704 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A 'shared-dmb-pool' reserved-memory device tree node defines a Designated Movable Block for use by an associated Contiguous Memory Allocator. Devices access the CMA region in the same manner as a 'shared- dma-pool', but the kernel page allocator is free to use the memory like any other ZONE_MOVABLE memory. Signed-off-by: Doug Berger --- drivers/of/of_reserved_mem.c | 5 +++++ kernel/dma/contiguous.c | 33 ++++++++++++++++++++++++++++----- 2 files changed, 33 insertions(+), 5 deletions(-) diff --git a/drivers/of/of_reserved_mem.c b/drivers/of/of_reserved_mem.c index 0eb9e8898d7b..700c0dc0d3b6 100644 --- a/drivers/of/of_reserved_mem.c +++ b/drivers/of/of_reserved_mem.c @@ -123,6 +123,11 @@ static int __init __reserved_mem_alloc_size(unsigned long node, && !nomap) { /* Need adjust the alignment to satisfy the CMA requirement */ align = max_t(phys_addr_t, align, CMA_MIN_ALIGNMENT_BYTES); + } else if (IS_ENABLED(CONFIG_CMA) + && of_flat_dt_is_compatible(node, "shared-dmb-pool")) { + /* Need adjust the alignment to satisfy CMA/DMB requirements */ + align = max_t(phys_addr_t, align, CMA_MIN_ALIGNMENT_BYTES); + align = max_t(phys_addr_t, align, DMB_MIN_ALIGNMENT_BYTES); } prop = of_get_flat_dt_prop(node, "alloc-ranges", &len); diff --git a/kernel/dma/contiguous.c b/kernel/dma/contiguous.c index 6ea80ae42622..65dda12752a7 100644 --- a/kernel/dma/contiguous.c +++ b/kernel/dma/contiguous.c @@ -50,6 +50,7 @@ #include #include #include +#include #ifdef CONFIG_CMA_SIZE_MBYTES #define CMA_SIZE_MBYTES CONFIG_CMA_SIZE_MBYTES @@ -397,10 +398,11 @@ static const struct reserved_mem_ops rmem_cma_ops = { .device_release = rmem_cma_device_release, }; -static int __init rmem_cma_setup(struct reserved_mem *rmem) +static int __init _rmem_cma_setup(struct reserved_mem *rmem, bool in_dmb) { unsigned long node = rmem->fdt_node; bool default_cma = of_get_flat_dt_prop(node, "linux,cma-default", NULL); + phys_addr_t align = CMA_MIN_ALIGNMENT_BYTES; struct cma *cma; int err; @@ -414,16 +416,25 @@ static int __init rmem_cma_setup(struct reserved_mem *rmem) of_get_flat_dt_prop(node, "no-map", NULL)) return -EINVAL; - if (!IS_ALIGNED(rmem->base | rmem->size, CMA_MIN_ALIGNMENT_BYTES)) { + if (in_dmb) { + if (default_cma) { + pr_err("Reserved memory: cma-default cannot be DMB\n"); + return -EINVAL; + } + align = max_t(phys_addr_t, align, DMB_MIN_ALIGNMENT_BYTES); + } + if (!IS_ALIGNED(rmem->base | rmem->size, align)) { pr_err("Reserved memory: incorrect alignment of CMA region\n"); return -EINVAL; } - err = cma_init_reserved_mem(rmem->base, rmem->size, 0, rmem->name, &cma); + err = __cma_init_reserved_mem(rmem->base, rmem->size, 0, rmem->name, + &cma, in_dmb); if (err) { pr_err("Reserved memory: unable to setup CMA region\n"); return err; } + /* Architecture specific contiguous memory fixup. */ dma_contiguous_early_fixup(rmem->base, rmem->size); @@ -433,10 +444,22 @@ static int __init rmem_cma_setup(struct reserved_mem *rmem) rmem->ops = &rmem_cma_ops; rmem->priv = cma; - pr_info("Reserved memory: created CMA memory pool at %pa, size %ld MiB\n", - &rmem->base, (unsigned long)rmem->size / SZ_1M); + pr_info("Reserved memory: created %s memory pool at %pa, size %ld MiB\n", + in_dmb ? "DMB" : "CMA", &rmem->base, + (unsigned long)rmem->size / SZ_1M); return 0; } + +static int __init rmem_cma_setup(struct reserved_mem *rmem) +{ + return _rmem_cma_setup(rmem, false); +} RESERVEDMEM_OF_DECLARE(cma, "shared-dma-pool", rmem_cma_setup); + +static int __init rmem_cma_in_dmb_setup(struct reserved_mem *rmem) +{ + return _rmem_cma_setup(rmem, true); +} +RESERVEDMEM_OF_DECLARE(cma_in_dmb, "shared-dmb-pool", rmem_cma_in_dmb_setup); #endif From patchwork Tue Sep 13 19:55:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Berger X-Patchwork-Id: 12975328 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3D6DC54EE9 for ; Tue, 13 Sep 2022 19:59:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5B9F56B009E; Tue, 13 Sep 2022 15:59:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 569126B009F; Tue, 13 Sep 2022 15:59:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E2BC6B00A0; Tue, 13 Sep 2022 15:59:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2E2F56B009E for ; Tue, 13 Sep 2022 15:59:13 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 101B8160B94 for ; Tue, 13 Sep 2022 19:59:13 +0000 (UTC) X-FDA: 79908126186.23.E224498 Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) by imf02.hostedemail.com (Postfix) with ESMTP id 7A2DB800AE for ; Tue, 13 Sep 2022 19:59:12 +0000 (UTC) Received: by mail-qv1-f46.google.com with SMTP id y9so10099710qvo.4 for ; Tue, 13 Sep 2022 12:59:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=koBwJdR4HF2Mh9OkMa3nN+ZpGb/cJotv47IHPg3aLjM=; b=H9fVlQTVY6EaU+njTimAq+ibJtvJjx2vWGkJikgtxKpK4lpRvh7Gs0dhq+SfYU7Y1l ol227J1p7+D7fHpafEcMsArwK6zpQesHNLeCyw8pBBnlfTeZXRH0gTB01wVhDLo6ZqZ0 2mt9ehwJJq5f5bEntKn/0aquyhMi35hMoNg4MgcTAFw56eJYhoioxREYotQhcXVzv4ef 26N8v6sdGt7Ksl3t6WmZRbYcEffyv4LTbLh0c7Zn4Y23wzA2w1INQgl13/vOL+Z8k7Uy rhOg4rC2RHnt2n62JjXnioNT4+kK415lG500sJWQkISGfWvSLQb972JndYtjJi8pBB0H qioA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=koBwJdR4HF2Mh9OkMa3nN+ZpGb/cJotv47IHPg3aLjM=; b=erevnZOIKAeGsZZrQHPsG2bO15Yg6vWjzQmXYRz/8bs6/Fgq2PNNc2wgXUZalCNH65 u9S11uZU32JyJZ4XYu0/FhQLS75SxkU033XWuU0p/KlXyiqAThEWF5tvOk2szimDHaoi rDtL8BNG8IuZoR4zkbeTmy/IYlLk499/8SZURzt90bHAj0RDM5QFO/MKBjeBQPx5S5O1 2Wt+xdV7/9CV5iQc9XqivndLR63WhBytxHtmuItnNNh6DBMTfwYCOmJhiFyc8f8UG4FW hutDwZLocCs1fWWGB/H3RUFM4RGQv5KxM3F/BUo2C5bXhid4X/89PWPnwLwRh74FgiDE WNow== X-Gm-Message-State: ACgBeo1I6NxIOwHu3BYQvWMYW4+vPcwrNOjWdbW7p6gQEx8meYwzpllP ZvBw02ICYlcXKYpkfCzr29s= X-Google-Smtp-Source: AA6agR4KLeMksXEqgCvXkwrSYo6YaUFxsEGC4nazCYouj2Y6nxYnhGX19vjDfEjEBR3X6wdqMlktzg== X-Received: by 2002:a05:6214:1c0a:b0:49d:87ff:e77c with SMTP id u10-20020a0562141c0a00b0049d87ffe77cmr28164232qvc.54.1663099151805; Tue, 13 Sep 2022 12:59:11 -0700 (PDT) Received: from stbirv-lnx-3.igp.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id s11-20020a05620a29cb00b006b8e049cf08sm276305qkp.2.2022.09.13.12.59.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Sep 2022 12:59:11 -0700 (PDT) From: Doug Berger To: Andrew Morton Cc: Jonathan Corbet , Rob Herring , Krzysztof Kozlowski , Frank Rowand , Mike Kravetz , Muchun Song , Mike Rapoport , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Neeraj Upadhyay , Randy Dunlap , Damien Le Moal , Doug Berger , Florian Fainelli , David Hildenbrand , Zi Yan , Oscar Salvador , Hari Bathini , Kees Cook , - , KOSAKI Motohiro , Mel Gorman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux.dev Subject: [PATCH 21/21] mm/hugetlb: introduce hugetlb_dmb Date: Tue, 13 Sep 2022 12:55:08 -0700 Message-Id: <20220913195508.3511038-22-opendmb@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220913195508.3511038-1-opendmb@gmail.com> References: <20220913195508.3511038-1-opendmb@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663099152; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=koBwJdR4HF2Mh9OkMa3nN+ZpGb/cJotv47IHPg3aLjM=; b=isP4yDFkkFJmqfhszhoa0AidkcQBbbLsQnq8NXMmjMtsjvIPsaE+Da4T9Qargi3Fept8Rd zs5Ex8QC7TxVZicM/Kkz5a+Aff88X+pdfh2PqquF4qw7t9/d4NjokI7kbwkwWY9mzzouQT KzsTqDgL8iyYNifmpJiLKaMuGRgJeBs= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=H9fVlQTV; spf=pass (imf02.hostedemail.com: domain of opendmb@gmail.com designates 209.85.219.46 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663099152; a=rsa-sha256; cv=none; b=O0l3zfrcUqA+dl/Vnw0vFzETeyOZ4pQTSLbLATCzEJU1KlYO9nMIy7NgIy70VdZkZmaW4s aWxT5J+dUzyb8CHGB9urVVBlcUCg7lrMzOZGn7NJ2pwwkaigVO9fXqol0FWzr6b3mpHVmy R/aiEL+U/zDfmfPiDKVGGaTxDi8vkAY= X-Rspam-User: X-Stat-Signature: x5jzoa5kbqoawbakr53ctizajf4fgt3u X-Rspamd-Queue-Id: 7A2DB800AE Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=H9fVlQTV; spf=pass (imf02.hostedemail.com: domain of opendmb@gmail.com designates 209.85.219.46 as permitted sender) smtp.mailfrom=opendmb@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam08 X-HE-Tag: 1663099152-517679 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If specified on the kernel command line the hugetlb_dmb parameter modifies the behavior of the hugetlb_cma parameter to use the Contiguous Memory Allocator within Designated Movable Blocks for gigantic page allocation. This allows the kernel page allocator to use the memory more agressively than traditional CMA memory pools at the cost of potentially increased allocation latency. Signed-off-by: Doug Berger --- Documentation/admin-guide/kernel-parameters.txt | 3 +++ mm/hugetlb.c | 16 +++++++++++++--- 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 8141fac7c7cb..b29d1fa253d6 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1732,6 +1732,9 @@ hugepages using the CMA allocator. If enabled, the boot-time allocation of gigantic hugepages is skipped. + hugetlb_dmb [HW,CMA] Causes hugetlb_cma to use Designated Movable + Blocks for any CMA areas it reserves. + hugetlb_free_vmemmap= [KNL] Reguires CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP enabled. diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 2f354423f50f..d3fb8b1f443f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -54,6 +54,7 @@ struct hstate hstates[HUGE_MAX_HSTATE]; #ifdef CONFIG_CMA static struct cma *hugetlb_cma[MAX_NUMNODES]; static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES] __initdata; +static bool hugetlb_dmb __initdata; static bool hugetlb_cma_page(struct page *page, unsigned int order) { return cma_pages_valid(hugetlb_cma[page_to_nid(page)], page, @@ -7321,6 +7322,14 @@ static int __init cmdline_parse_hugetlb_cma(char *p) early_param("hugetlb_cma", cmdline_parse_hugetlb_cma); +static int __init cmdline_parse_hugetlb_dmb(char *p) +{ + hugetlb_dmb = true; + return 0; +} + +early_param("hugetlb_dmb", cmdline_parse_hugetlb_dmb); + void __init hugetlb_cma_reserve(int order) { unsigned long size, reserved, per_node; @@ -7396,10 +7405,11 @@ void __init hugetlb_cma_reserve(int order) * may be returned to CMA allocator in the case of * huge page demotion. */ - res = cma_declare_contiguous_nid(0, size, 0, + res = __cma_declare_contiguous_nid(0, size, 0, PAGE_SIZE << HUGETLB_PAGE_ORDER, - 0, false, name, - &hugetlb_cma[nid], nid); + 0, false, name, + &hugetlb_cma[nid], nid, + hugetlb_dmb); if (res) { pr_warn("hugetlb_cma: reservation failed: err %d, node %d", res, nid);