From patchwork Mon Aug 10 02:27:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11706909 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7579F618 for ; Mon, 10 Aug 2020 02:30:00 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4F8122068F for ; Mon, 10 Aug 2020 02:30:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="AdgT162U"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="uzlZkjSm" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4F8122068F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=SP99ccf+W8kdpVx4TS+leG/wWREXpzM5HtyWDwZdXo4=; b=AdgT162Uj7gKakbtidzkURZCn IHTnTbavgKWdWWj3VqNGzRor4IRROEqbePKIfT44WkSF5hbei6OCesrKYk3/uQoSZ3ddf4RXcSI6W Kqq9ioFcq8L3Z05MjECIUXWMQWVtRl9Po+ShmOv58vsjStibZ5mRn9NxRrM7tK+vhWb+VOzakYXsu C9FktAUbp1ILtlc39fJusz5U7SlPq+BcAgUmWd9MdSAr3X83CCvza0a0Kglwbt2jzUsaxetvTWYZF wba4uCbbg9dtwLVgVCJ5tzjBizbKybcXo9ZM4ZsMbIKDTk0zj1WvoJxMaVTSFQyyQ+5gr+wGCMFmA LhRLftirA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k4xXY-0004yX-EY; Mon, 10 Aug 2020 02:27:56 +0000 Received: from mail-pf1-x444.google.com ([2607:f8b0:4864:20::444]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k4xXV-0004x1-2v for linux-arm-kernel@lists.infradead.org; Mon, 10 Aug 2020 02:27:55 +0000 Received: by mail-pf1-x444.google.com with SMTP id f193so4361711pfa.12 for ; Sun, 09 Aug 2020 19:27:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+wqRNPVVkxxZAVOe5cjf4OdJxPseOUgKHXYAzqLL+cs=; b=uzlZkjSm9ZGrRLlsU85yYntg5uUa6TXp4vH+ikn0xWiGnZ80ZZ37DbPaomShZJfPnE 0oiZoFL0/mDnn+SocvSfP1WMGgn8TgdaPkindydu0o+r6R7Fc+gxEdQxx7JzfznITR1q mPWsHHhc4c1dQIfvFKE2oND4B0OzSWyERbhsTrQLzubUjimvhYHoaTsE2Hcrl5Fioz3X cjcoEbIvIfuf2WYH6Ug7ZJ+aiS1YTw6ChcwZVpAwjhDuGBDKbaXSxG1MiQCJaeuwTVKI R+BAlqiU5oVlpJVYU/bdKo/Pb9s+j72tB6r8z/RoTb4CRa3k/zRgZZC76K5epCbmm40n hKrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+wqRNPVVkxxZAVOe5cjf4OdJxPseOUgKHXYAzqLL+cs=; b=PHewNCkzqVKiPbkQGWoUm52aNtXDWhIZEAp2gf55C1PbpxapirfVE87cNvXfKbRtO2 MTJp6HuSrG/oUYFV7iubxpPSaHn+lmvVkDiov1j56fl/hLPOeGL0G6ClSm+k5V4duQtZ 7PjRZur8oKkdw8hg3Y4NZUhMPKzECWPPvTRkzr2usPlk4QDURnDJOiLud/EIIuyGbMBM gfoPyAHv+IiJrkPJwZxRefkKKv6RKHH23S2l7EaIuJe5EoUqNJDSkepR7C66HdTVLsPb 5iki6oSAVmbYloQ3HCWFsBsjSJMgfVWdPdA5phP84EF9vj9Jf2vu+eCmio8GFCoH8DUS FWVw== X-Gm-Message-State: AOAM530I4wFJLt7OVSLc0VWunSXV/r8W/WSz/stjfmt526BgldjjRfTA bbWcwIKaeB2Hieba+yJElCU= X-Google-Smtp-Source: ABdhPJxJ5wOs61jPiAtdiL9OZRHbgCxw1msWorFzPuXMtNPWII0pHR8QnAVNPZXVV89m8/gFh40qtQ== X-Received: by 2002:aa7:8b18:: with SMTP id f24mr23487412pfd.301.1597026471393; Sun, 09 Aug 2020 19:27:51 -0700 (PDT) Received: from bobo.ibm.com (193-116-100-32.tpgi.com.au. [193.116.100.32]) by smtp.gmail.com with ESMTPSA id l17sm21863475pff.126.2020.08.09.19.27.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 09 Aug 2020 19:27:50 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Subject: [PATCH v3 1/8] mm/vmalloc: fix vmalloc_to_page for huge vmap mappings Date: Mon, 10 Aug 2020 12:27:25 +1000 Message-Id: <20200810022732.1150009-2-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200810022732.1150009-1-npiggin@gmail.com> References: <20200810022732.1150009-1-npiggin@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200809_222754_054718_95A192C4 X-CRM114-Status: GOOD ( 20.41 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:444 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider [npiggin[at]gmail.com] -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Zefan Li , Catalin Marinas , x86@kernel.org, linuxppc-dev@lists.ozlabs.org, Nicholas Piggin , linux-kernel@vger.kernel.org, Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , Thomas Gleixner , Will Deacon , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org vmalloc_to_page returns NULL for addresses mapped by larger pages[*]. Whether or not a vmap is huge depends on the architecture details, alignments, boot options, etc., which the caller can not be expected to know. Therefore HUGE_VMAP is a regression for vmalloc_to_page. This change teaches vmalloc_to_page about larger pages, and returns the struct page that corresponds to the offset within the large page. This makes the API agnostic to mapping implementation details. [*] As explained by commit 029c54b095995 ("mm/vmalloc.c: huge-vmap: fail gracefully on unexpected huge vmap mappings") Signed-off-by: Nicholas Piggin --- mm/vmalloc.c | 40 ++++++++++++++++++++++++++-------------- 1 file changed, 26 insertions(+), 14 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index b482d240f9a2..49f225b0f855 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -38,6 +38,7 @@ #include #include +#include #include #include @@ -343,7 +344,9 @@ int is_vmalloc_or_module_addr(const void *x) } /* - * Walk a vmap address to the struct page it maps. + * Walk a vmap address to the struct page it maps. Huge vmap mappings will + * return the tail page that corresponds to the base page address, which + * matches small vmap mappings. */ struct page *vmalloc_to_page(const void *vmalloc_addr) { @@ -363,25 +366,33 @@ struct page *vmalloc_to_page(const void *vmalloc_addr) if (pgd_none(*pgd)) return NULL; + if (WARN_ON_ONCE(pgd_leaf(*pgd))) + return NULL; /* XXX: no allowance for huge pgd */ + if (WARN_ON_ONCE(pgd_bad(*pgd))) + return NULL; + p4d = p4d_offset(pgd, addr); if (p4d_none(*p4d)) return NULL; - pud = pud_offset(p4d, addr); + if (p4d_leaf(*p4d)) + return p4d_page(*p4d) + ((addr & ~P4D_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(p4d_bad(*p4d))) + return NULL; - /* - * Don't dereference bad PUD or PMD (below) entries. This will also - * identify huge mappings, which we may encounter on architectures - * that define CONFIG_HAVE_ARCH_HUGE_VMAP=y. Such regions will be - * identified as vmalloc addresses by is_vmalloc_addr(), but are - * not [unambiguously] associated with a struct page, so there is - * no correct value to return for them. - */ - WARN_ON_ONCE(pud_bad(*pud)); - if (pud_none(*pud) || pud_bad(*pud)) + pud = pud_offset(p4d, addr); + if (pud_none(*pud)) + return NULL; + if (pud_leaf(*pud)) + return pud_page(*pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(pud_bad(*pud))) return NULL; + pmd = pmd_offset(pud, addr); - WARN_ON_ONCE(pmd_bad(*pmd)); - if (pmd_none(*pmd) || pmd_bad(*pmd)) + if (pmd_none(*pmd)) + return NULL; + if (pmd_leaf(*pmd)) + return pmd_page(*pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(pmd_bad(*pmd))) return NULL; ptep = pte_offset_map(pmd, addr); @@ -389,6 +400,7 @@ struct page *vmalloc_to_page(const void *vmalloc_addr) if (pte_present(pte)) page = pte_page(pte); pte_unmap(ptep); + return page; } EXPORT_SYMBOL(vmalloc_to_page); From patchwork Mon Aug 10 02:27:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11706907 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6A9A616B1 for ; Mon, 10 Aug 2020 02:30:00 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 40A7E20709 for ; Mon, 10 Aug 2020 02:30:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="fE7m/kl3"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WP278OOY" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 40A7E20709 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=pGmuHg23is8GOyLe7qSLAjDNFmoW569BHKHkzmrEcAc=; b=fE7m/kl3NCWMSn1j5gPah9DUD OcyJao2khIak8p6z9sOaUsniXe2UHGmFQsejVtdHNGwQyJDNQ9UkKgWIs27CtOhs1E+vYTAfDDJ+n +feZvpG49ksxTGl9Ua8LpdSG5UYK7jL7hLtPONLGGI5Fnw3wogqZ4f1GUwNuRfrRMvLu1VWh6zpUb kyNzkHgImQTzFg80keZBFN/zG2YhPV104wVjM5c9qjSPsGxM0nwjK5Goo/DaL9yTFvxv2QVVVpFl2 7vBSa2g4xzonoSYKGzQvEAUvr32kYRE8pzl4UvvtckLtCu7X9vZeCJiYpcAfIMkzyUjSDGxLaKIKx sOZxt8hKA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k4xXd-00050A-Qq; Mon, 10 Aug 2020 02:28:01 +0000 Received: from mail-pl1-x644.google.com ([2607:f8b0:4864:20::644]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k4xXb-0004yl-83 for linux-arm-kernel@lists.infradead.org; Mon, 10 Aug 2020 02:28:00 +0000 Received: by mail-pl1-x644.google.com with SMTP id t11so4040349plr.5 for ; Sun, 09 Aug 2020 19:27:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=mxm/C2mC0786WMQTE10aJFvTg/Tw0xwUgIx5I5X+hjE=; b=WP278OOYLrG+JIYsivHVGAzMTDa1eY0iFpr/M6I+aKWKr6bMnyODBgG4KETvzcgTXd ow7Jj2tsasJ8YFFWufeQv277XGisF25VPY5ZRhlBYYgpEWp1+50PYmlf923Gu/QUOITt etVT077ynKT7AWNGF/5mQcMdPJjA0gFZC+nLCGBjmku/JKs79bACOlt5k+qdWfrkM+Ui 7cIyNArYFYHVtz1M3cDImsXm9SmwTNdx2tC09A2Dter3y6Kuu2/oxvSq89h1uNaAGDuh zDBy5Pjw69YLMWfuhW7I/ekJKMRZ80wZ+GACMvHlcpOX7TEgC42KFlaiBl9YI8RZowwS DNKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mxm/C2mC0786WMQTE10aJFvTg/Tw0xwUgIx5I5X+hjE=; b=N0iuHbXyiHsKI7WafiKUKOJWlqR4K5rQHdGGzy39NuhjbizJ2CQSWEU5A5ufA6L8q0 b7fbZdwgh/yEolHLxoWS3+JIDKFWPKWqXo0LGoQsZ9QKRrmVZyWbDjQXeG+hzzk+crRH qSQmlmorf1N8qkM6EWCuuaoUJlDEgc+s11CZvpEoQerAYFkYzVX4yh4FHTgdWCSHH+t2 TOJ42EWpfQ2gC8em4Xk+Uhnkf+rcWWagJBh4BWuURDn/j8SSpjSukAgI4ANbnBxtl3Gg MOOmMIJtGF8m0GSBZm3xzkTUgwzzwBKcnfHBUVf8A41/t2cKwRJAt8M9sG/rdmEN7/gU hD6A== X-Gm-Message-State: AOAM5311nP7ptT3AWKpfg4knaInI613He4BREZ9qZ+6z6+nNtxj8hK/b V64PplJm0oeTvq4/OEWcULI= X-Google-Smtp-Source: ABdhPJzQrXlZ5GNgG6uz8L7myrJaEH5zZlbvoa/Z7fFHnyqY5iIPX6x3JpckcdyVbZxn3y4vyTxFlA== X-Received: by 2002:a17:902:7681:: with SMTP id m1mr9854542pll.62.1597026477066; Sun, 09 Aug 2020 19:27:57 -0700 (PDT) Received: from bobo.ibm.com (193-116-100-32.tpgi.com.au. [193.116.100.32]) by smtp.gmail.com with ESMTPSA id l17sm21863475pff.126.2020.08.09.19.27.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 09 Aug 2020 19:27:56 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Subject: [PATCH v3 2/8] mm: apply_to_pte_range warn and fail if a large pte is encountered Date: Mon, 10 Aug 2020 12:27:26 +1000 Message-Id: <20200810022732.1150009-3-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200810022732.1150009-1-npiggin@gmail.com> References: <20200810022732.1150009-1-npiggin@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200809_222759_307159_F251339A X-CRM114-Status: GOOD ( 14.99 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:644 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider [npiggin[at]gmail.com] -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Zefan Li , Catalin Marinas , x86@kernel.org, linuxppc-dev@lists.ozlabs.org, Nicholas Piggin , linux-kernel@vger.kernel.org, Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , Thomas Gleixner , Will Deacon , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Signed-off-by: Nicholas Piggin --- mm/memory.c | 60 +++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 44 insertions(+), 16 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index c39a13b09602..1d5f3093c249 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2260,13 +2260,20 @@ static int apply_to_pmd_range(struct mm_struct *mm, pud_t *pud, } do { next = pmd_addr_end(addr, end); - if (create || !pmd_none_or_clear_bad(pmd)) { - err = apply_to_pte_range(mm, pmd, addr, next, fn, data, - create); - if (err) - break; + if (pmd_none(*pmd) && !create) + continue; + if (WARN_ON_ONCE(pmd_leaf(*pmd))) + return -EINVAL; + if (WARN_ON_ONCE(pmd_bad(*pmd))) { + if (!create) + continue; + pmd_clear_bad(pmd); } + err = apply_to_pte_range(mm, pmd, addr, next, fn, data, create); + if (err) + break; } while (pmd++, addr = next, addr != end); + return err; } @@ -2287,13 +2294,20 @@ static int apply_to_pud_range(struct mm_struct *mm, p4d_t *p4d, } do { next = pud_addr_end(addr, end); - if (create || !pud_none_or_clear_bad(pud)) { - err = apply_to_pmd_range(mm, pud, addr, next, fn, data, - create); - if (err) - break; + if (pud_none(*pud) && !create) + continue; + if (WARN_ON_ONCE(pud_leaf(*pud))) + return -EINVAL; + if (WARN_ON_ONCE(pud_bad(*pud))) { + if (!create) + continue; + pud_clear_bad(pud); } + err = apply_to_pmd_range(mm, pud, addr, next, fn, data, create); + if (err) + break; } while (pud++, addr = next, addr != end); + return err; } @@ -2314,13 +2328,20 @@ static int apply_to_p4d_range(struct mm_struct *mm, pgd_t *pgd, } do { next = p4d_addr_end(addr, end); - if (create || !p4d_none_or_clear_bad(p4d)) { - err = apply_to_pud_range(mm, p4d, addr, next, fn, data, - create); - if (err) - break; + if (p4d_none(*p4d) && !create) + continue; + if (WARN_ON_ONCE(p4d_leaf(*p4d))) + return -EINVAL; + if (WARN_ON_ONCE(p4d_bad(*p4d))) { + if (!create) + continue; + p4d_clear_bad(p4d); } + err = apply_to_pud_range(mm, p4d, addr, next, fn, data, create); + if (err) + break; } while (p4d++, addr = next, addr != end); + return err; } @@ -2339,8 +2360,15 @@ static int __apply_to_page_range(struct mm_struct *mm, unsigned long addr, pgd = pgd_offset(mm, addr); do { next = pgd_addr_end(addr, end); - if (!create && pgd_none_or_clear_bad(pgd)) + if (pgd_none(*pgd) && !create) continue; + if (WARN_ON_ONCE(pgd_leaf(*pgd))) + return -EINVAL; + if (WARN_ON_ONCE(pgd_bad(*pgd))) { + if (!create) + continue; + pgd_clear_bad(pgd); + } err = apply_to_p4d_range(mm, pgd, addr, next, fn, data, create); if (err) break; From patchwork Mon Aug 10 02:27:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11706913 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4E1CD913 for ; Mon, 10 Aug 2020 02:30:01 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 279632068F for ; Mon, 10 Aug 2020 02:30:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="cFgvdj7c"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SMQPO073" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 279632068F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=mfV/AheEnWOxm0OR4Py+dnwmgB0q5ldjJMuL/CTQMT4=; b=cFgvdj7cEvQ5CwEAqsFLLnjWG KCqv7lpNi9mcOkvb2DsKuHw1ExK6Xpi7gZ5im+ykyIF+FVNwMBN6U9USkw+E7O2G25buOhu0Hc8Ke prCetlkmX/ooBzfD+swjmeOgYfiSYSt05qqBrFvdt1wqQDRwNk9SJeCd/R7OQlz6r5hzvJw6rpBOV rEQFscoDNxYEWeLoXKu2tmr2nfSWYCocV7J4dyWHQ04ot2JGYsTjzEVlXoQuXmq1BYt+/IXQ4taJf ZkUophngj7sHuDS8C/xQas04H1VGXb/TWAHvaBNhkB9ZQZeqidY7IR5N8vrTwsc8uyusEjIK7hexd YJ7CJSzIA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k4xXi-00051o-Cm; Mon, 10 Aug 2020 02:28:06 +0000 Received: from mail-pf1-x444.google.com ([2607:f8b0:4864:20::444]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k4xXg-00050W-G6 for linux-arm-kernel@lists.infradead.org; Mon, 10 Aug 2020 02:28:05 +0000 Received: by mail-pf1-x444.google.com with SMTP id m8so4381797pfh.3 for ; Sun, 09 Aug 2020 19:28:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xoa50mjKdO+6rEo7JFgsM1iktECeAufQruBypQNx5uU=; b=SMQPO07308HbTz5R2bXEM/3gc3ItFl6g5ePYDHuBQx4pD8Y6rKeRum7JmixNo2JTPy F8e1S2IG7aEYVgVAu8HIbxKoAlslBlF27d5piH2FeaWK1c7upqS585JDLUhCBcKzQHTn wr9L/QH+bZwdnlKEEWwiDjWifyVhnVJzNCNcb2NqGD0npXFc9c+tb2z8LGK1AJDVAsNZ XiIbv0FFtFNWpoJD4OnZR6JRnhsts/5+BHgl3O3yu2m4Rl8JzRD3sJ042T61COweBtFC EM1Y+FQW7Nrt8jT95YSaPBYerMpBnEsZSojnMJJMiWbq+fSUxZ6KLcRbExu4ZjJFbrx8 PSOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xoa50mjKdO+6rEo7JFgsM1iktECeAufQruBypQNx5uU=; b=iIaAsbfHFwxQvxBL2/o0AnplrRNSYuJ5DxIHcsCoJuRnvpzzbOfgTrGxfO9oR2KNku 3V9/muKf+LUcGpVzhSgc2n3holz3Bbikn+vkNKE0suQ1S5ECOGDINM/dGeTZwVEvPWMD XpxDfGLwsdhfXXi1DaYAdR7/GL2Amh7DRE+sV/VJnkW0AR9EDGhUKNpUfmAlhv2Hr6Fk 5UVVINumULgvWBG8J3qBR0sOV/YXUVRMDCUbupNjSLqRPXr/iDxj/jRFeNQozXPUTO9y D6AUTSzkxfjmf4XBGahlHBKfSsvEEy5V/4Hq0EYWJ0KcBM1lWcAt4NQlX8VIkJQzr15o QmEA== X-Gm-Message-State: AOAM533/w1uGQtvimMrqF8hTQVX9HBB72ohRylCJPk9/kXLS5hmfW5C7 tMihHjlNQWE+UXsFOgjD5vk= X-Google-Smtp-Source: ABdhPJyF0v331ifmF6+EZbg+clfmQWCPigoJ9Z+l4oz8cqoLCgmGsSnYR+bh1HGTwKbaJ/maQnmNDA== X-Received: by 2002:a62:6083:: with SMTP id u125mr24178457pfb.286.1597026482560; Sun, 09 Aug 2020 19:28:02 -0700 (PDT) Received: from bobo.ibm.com (193-116-100-32.tpgi.com.au. [193.116.100.32]) by smtp.gmail.com with ESMTPSA id l17sm21863475pff.126.2020.08.09.19.27.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 09 Aug 2020 19:28:02 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Subject: [PATCH v3 3/8] mm/vmalloc: rename vmap_*_range vmap_pages_*_range Date: Mon, 10 Aug 2020 12:27:27 +1000 Message-Id: <20200810022732.1150009-4-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200810022732.1150009-1-npiggin@gmail.com> References: <20200810022732.1150009-1-npiggin@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200809_222804_540227_F9A7ABD4 X-CRM114-Status: GOOD ( 14.26 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:444 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider [npiggin[at]gmail.com] -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Zefan Li , Catalin Marinas , x86@kernel.org, linuxppc-dev@lists.ozlabs.org, Nicholas Piggin , linux-kernel@vger.kernel.org, Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , Thomas Gleixner , Will Deacon , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org The vmalloc mapper operates on a struct page * array rather than a linear physical address, re-name it to make this distinction clear. Signed-off-by: Nicholas Piggin --- mm/vmalloc.c | 28 ++++++++++++---------------- 1 file changed, 12 insertions(+), 16 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 49f225b0f855..3a1e45fd1626 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -190,9 +190,8 @@ void unmap_kernel_range_noflush(unsigned long start, unsigned long size) arch_sync_kernel_mappings(start, end); } -static int vmap_pte_range(pmd_t *pmd, unsigned long addr, - unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) +static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { pte_t *pte; @@ -218,9 +217,8 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, return 0; } -static int vmap_pmd_range(pud_t *pud, unsigned long addr, - unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) +static int vmap_pages_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { pmd_t *pmd; unsigned long next; @@ -230,15 +228,14 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr, return -ENOMEM; do { next = pmd_addr_end(addr, end); - if (vmap_pte_range(pmd, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pte_range(pmd, addr, next, prot, pages, nr, mask)) return -ENOMEM; } while (pmd++, addr = next, addr != end); return 0; } -static int vmap_pud_range(p4d_t *p4d, unsigned long addr, - unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) +static int vmap_pages_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { pud_t *pud; unsigned long next; @@ -248,15 +245,14 @@ static int vmap_pud_range(p4d_t *p4d, unsigned long addr, return -ENOMEM; do { next = pud_addr_end(addr, end); - if (vmap_pmd_range(pud, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pmd_range(pud, addr, next, prot, pages, nr, mask)) return -ENOMEM; } while (pud++, addr = next, addr != end); return 0; } -static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, - unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) +static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { p4d_t *p4d; unsigned long next; @@ -266,7 +262,7 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, return -ENOMEM; do { next = p4d_addr_end(addr, end); - if (vmap_pud_range(p4d, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pud_range(p4d, addr, next, prot, pages, nr, mask)) return -ENOMEM; } while (p4d++, addr = next, addr != end); return 0; @@ -307,7 +303,7 @@ int map_kernel_range_noflush(unsigned long addr, unsigned long size, next = pgd_addr_end(addr, end); if (pgd_bad(*pgd)) mask |= PGTBL_PGD_MODIFIED; - err = vmap_p4d_range(pgd, addr, next, prot, pages, &nr, &mask); + err = vmap_pages_p4d_range(pgd, addr, next, prot, pages, &nr, &mask); if (err) return err; } while (pgd++, addr = next, addr != end); From patchwork Mon Aug 10 02:27:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11706911 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D4EEC175D for ; Mon, 10 Aug 2020 02:30:00 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AE4C62068F for ; Mon, 10 Aug 2020 02:30:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="bXXumH/+"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="tZiIwaB8" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AE4C62068F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=x40LcK1OEI6C3maRrpa5tVdXg2O92NkcSpjli/uioC0=; b=bXXumH/+IwRXCVsNcLNZxmC1U HAROk+cIVWYkFBYdSICYTul+QtlCtXmc3gLSvzJq3x3bMrHc0xT2gi3Fub66zaw5vJrEtoo+U0f1C u2zy9UMzqGqiBCnvkOyNFNmvX0I+maaTan9jmxSHeoQ955NvRPPRyuo9QRLE40Mx0hr1JkItatrdE GZ4024GxPwrgKO9BkQt5rlD9KInamViDNq3DDpRaV3z4v+nfjX2ibZiGAMtokM6hIGGYwvUaIEsTY fJm5hGmXztvkBA4+wshJDCSzzR6O6UXmMeQ2HcB1c3kfL7OVjF2pHRrJGnQh7/Wi30sb9T/HDP77x xDqB6QCWg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k4xXo-00053s-ME; Mon, 10 Aug 2020 02:28:12 +0000 Received: from mail-pj1-x1044.google.com ([2607:f8b0:4864:20::1044]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k4xXm-00052H-7G for linux-arm-kernel@lists.infradead.org; Mon, 10 Aug 2020 02:28:11 +0000 Received: by mail-pj1-x1044.google.com with SMTP id c6so4019203pje.1 for ; Sun, 09 Aug 2020 19:28:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=zgciMtrhUFyjwGUVG4/y+NlOfAX4ULiG8JBMTv5iRYk=; b=tZiIwaB8BveXvJ0qK+agLn8o7f/k9ONmRsftQ1bZ63EC63Z/XEH5pTtw77UfAZPd9n kxiiuHHhcrAVfr9Fl2k8sXr2I6N/989l3wwxLvH0bkId7wwtO4GjZcU9QK+YoEmJnu10 4ZbdSBy9Hml4gG7G+zLD4mgSI1lhMem2Nkla3FSN9oks/WzhKSP4xooacEQDGg72msep FbEdQznPWaUSu1WfjTzRbx/VXQcnK6XiCCcZxbGkSa1LcTXHYeVlsSkhs8W+5fYRiL7s MM2Bsd/g/A2yHig91GGvSj0CpLqSVy51uhfQFGwWKUEMjTxdVQHl5uHqgOyVmTTt67qN tW8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zgciMtrhUFyjwGUVG4/y+NlOfAX4ULiG8JBMTv5iRYk=; b=hHFMbL/FFNyozhfcMNQv77+4/oeynmw0IvRfHUjTu3rtVH70/KvVrmrf7/KYDIsyEE cefJhPHY5ZZlLjkyQT+k1s4PIt0S54a6kR4MZfuzcVG/NT0neHN8Le7w4LgOmOaz7qkj 4xNLVo83uzte6uPxjAB9Y3rvRf/2W0aKC4PmBgxM5e9qNCY/tKy/iLwWRZwa7LiiAAJh tcdRfkGYTL/s1ucGp53l0pylZHBGlwDdguIhYXJ/o9gBM20HJZuHbaHCa7RwXxJlIH8d A4EiyUsG5lDli8wr0AOflaNHAhSea9yCN0HE901tyQiBvW744TG9edBnucaD10O8lRNC 7eWg== X-Gm-Message-State: AOAM532bY5zc10oOVyjgKMja5BYHQf3Sj+s5DnQUz5v7cp9tuhU0IX5w ATQeOVli0U5I+Khs+EDSfq0= X-Google-Smtp-Source: ABdhPJzbhgo47H7fSvYqc7Khnd7jyS/4Sw4Oefgk6E1CVciPFOLuQaOPH7ht6fIVkgxh5qrmVHA1dA== X-Received: by 2002:a17:902:6b43:: with SMTP id g3mr20835887plt.319.1597026488157; Sun, 09 Aug 2020 19:28:08 -0700 (PDT) Received: from bobo.ibm.com (193-116-100-32.tpgi.com.au. [193.116.100.32]) by smtp.gmail.com with ESMTPSA id l17sm21863475pff.126.2020.08.09.19.28.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 09 Aug 2020 19:28:07 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Subject: [PATCH v3 4/8] lib/ioremap: rename ioremap_*_range to vmap_*_range Date: Mon, 10 Aug 2020 12:27:28 +1000 Message-Id: <20200810022732.1150009-5-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200810022732.1150009-1-npiggin@gmail.com> References: <20200810022732.1150009-1-npiggin@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200809_222810_399737_A7F5E043 X-CRM114-Status: GOOD ( 15.95 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:1044 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider [npiggin[at]gmail.com] -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Zefan Li , Catalin Marinas , x86@kernel.org, linuxppc-dev@lists.ozlabs.org, Nicholas Piggin , linux-kernel@vger.kernel.org, Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , Thomas Gleixner , Will Deacon , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org This will be moved to mm/ and used as a generic kernel virtual mapping function, so re-name it in preparation. Signed-off-by: Nicholas Piggin --- mm/ioremap.c | 55 ++++++++++++++++++++++------------------------------ 1 file changed, 23 insertions(+), 32 deletions(-) diff --git a/mm/ioremap.c b/mm/ioremap.c index 5fa1ab41d152..6016ae3227ad 100644 --- a/mm/ioremap.c +++ b/mm/ioremap.c @@ -61,9 +61,8 @@ static inline int ioremap_pud_enabled(void) { return 0; } static inline int ioremap_pmd_enabled(void) { return 0; } #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ -static int ioremap_pte_range(pmd_t *pmd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, pgtbl_mod_mask *mask) { pte_t *pte; u64 pfn; @@ -81,9 +80,8 @@ static int ioremap_pte_range(pmd_t *pmd, unsigned long addr, return 0; } -static int ioremap_try_huge_pmd(pmd_t *pmd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) +static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { if (!ioremap_pmd_enabled()) return 0; @@ -103,9 +101,8 @@ static int ioremap_try_huge_pmd(pmd_t *pmd, unsigned long addr, return pmd_set_huge(pmd, phys_addr, prot); } -static inline int ioremap_pmd_range(pud_t *pud, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, pgtbl_mod_mask *mask) { pmd_t *pmd; unsigned long next; @@ -116,20 +113,19 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long addr, do { next = pmd_addr_end(addr, end); - if (ioremap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) { *mask |= PGTBL_PMD_MODIFIED; continue; } - if (ioremap_pte_range(pmd, addr, next, phys_addr, prot, mask)) + if (vmap_pte_range(pmd, addr, next, phys_addr, prot, mask)) return -ENOMEM; } while (pmd++, phys_addr += (next - addr), addr = next, addr != end); return 0; } -static int ioremap_try_huge_pud(pud_t *pud, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) +static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { if (!ioremap_pud_enabled()) return 0; @@ -149,9 +145,8 @@ static int ioremap_try_huge_pud(pud_t *pud, unsigned long addr, return pud_set_huge(pud, phys_addr, prot); } -static inline int ioremap_pud_range(p4d_t *p4d, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, pgtbl_mod_mask *mask) { pud_t *pud; unsigned long next; @@ -162,20 +157,19 @@ static inline int ioremap_pud_range(p4d_t *p4d, unsigned long addr, do { next = pud_addr_end(addr, end); - if (ioremap_try_huge_pud(pud, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot)) { *mask |= PGTBL_PUD_MODIFIED; continue; } - if (ioremap_pmd_range(pud, addr, next, phys_addr, prot, mask)) + if (vmap_pmd_range(pud, addr, next, phys_addr, prot, mask)) return -ENOMEM; } while (pud++, phys_addr += (next - addr), addr = next, addr != end); return 0; } -static int ioremap_try_huge_p4d(p4d_t *p4d, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) +static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { if (!ioremap_p4d_enabled()) return 0; @@ -195,9 +189,8 @@ static int ioremap_try_huge_p4d(p4d_t *p4d, unsigned long addr, return p4d_set_huge(p4d, phys_addr, prot); } -static inline int ioremap_p4d_range(pgd_t *pgd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, pgtbl_mod_mask *mask) { p4d_t *p4d; unsigned long next; @@ -208,19 +201,18 @@ static inline int ioremap_p4d_range(pgd_t *pgd, unsigned long addr, do { next = p4d_addr_end(addr, end); - if (ioremap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) { + if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) { *mask |= PGTBL_P4D_MODIFIED; continue; } - if (ioremap_pud_range(p4d, addr, next, phys_addr, prot, mask)) + if (vmap_pud_range(p4d, addr, next, phys_addr, prot, mask)) return -ENOMEM; } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); return 0; } -int ioremap_page_range(unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot) +int ioremap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot) { pgd_t *pgd; unsigned long start; @@ -235,8 +227,7 @@ int ioremap_page_range(unsigned long addr, pgd = pgd_offset_k(addr); do { next = pgd_addr_end(addr, end); - err = ioremap_p4d_range(pgd, addr, next, phys_addr, prot, - &mask); + err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, &mask); if (err) break; } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); @@ -272,7 +263,7 @@ void __iomem *ioremap_prot(phys_addr_t addr, size_t size, unsigned long prot) return NULL; vaddr = (unsigned long)area->addr; - if (ioremap_page_range(vaddr, vaddr + size, addr, __pgprot(prot))) { + if (vmap_page_range(vaddr, vaddr + size, addr, __pgprot(prot))) { free_vm_area(area); return NULL; } From patchwork Mon Aug 10 02:27:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11706921 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 48BA816B1 for ; Mon, 10 Aug 2020 02:30:37 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0E9842065E for ; Mon, 10 Aug 2020 02:30:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="xHwDLcGU"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JpBNYLbx" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0E9842065E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=9F2gGxaCu7/1mIwmltY6e2RKD4mEqOCVpKDxZN368Go=; b=xHwDLcGUSc43Ule/8MWk9lzgX Q3Bgn4MLJp3hOzT1wWWiB5PensbIdS7p/Hztoot/zX0tg3mvGqxdw9DZwgZ1kA3eczkhx2XlptXDn oNORoXLtRMu68qstn6KBFLa1OgszMeSf83UDH/V5QTKcAU8pUK+P14ZQ8dJ2xAGn9hJ8W0OX8U/Jn ZH06/YbzMms0MswNfcvi9bn9lehQXQixyvBmkWubwTlLk4f3ndIZcOkACM0UpNk94NdWh/Dh1weOU gh9/YSZQ9gU76II6x7HFDiRHEgMCuDxwbsepdX1PCSruR/BOp1pb2wPDHNFaR5uhiCj5JttGGdnV1 rrBRLzBGQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k4xXv-00056I-Ns; Mon, 10 Aug 2020 02:28:19 +0000 Received: from mail-pl1-x641.google.com ([2607:f8b0:4864:20::641]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k4xXs-00054H-4O for linux-arm-kernel@lists.infradead.org; Mon, 10 Aug 2020 02:28:17 +0000 Received: by mail-pl1-x641.google.com with SMTP id t11so4040580plr.5 for ; Sun, 09 Aug 2020 19:28:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=GViDwBNAuHYIcRqNUdbY3WsI6pN6F9PmQ8RucBVMOSo=; b=JpBNYLbxxDzWTz82j5nO9BGdFD8sDiaZIrylyBR1lkYinN1Yn/bEjr2l4qdDr8+KzD pmFZle283m8IGl0byL/hLs4RhtyIFCKFWB523sCl5WzlleWKWjJxagSF5bXCuS30NpwH I+FP1aGRCV0qC2hkFduEl6nwdEWn/LI0S+FPIMiYMx2wHd/3Mo+o7pe/jqaytB+fSknY Rbu2xq4CcSto1Fu/yaLj3MLE0rzU569CR8tDw6nnODhCFu8RiqSYM5HCdlX2j8H8KV/W Jjd6s6g9piZPLW6iWEmkjC0d+ZkzdoxFbHedFvXkWKCQoVgiv1jP7ovawV0s2hiYWrFA yO9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GViDwBNAuHYIcRqNUdbY3WsI6pN6F9PmQ8RucBVMOSo=; b=dJA74JVG1ORmaFBkYQMbDYLSZmlb9hMA9/Qt2LlcDYj8TEX1ncYUs0g1jnTDIHUDeo 1+BBSG97DT2JNPSHnilJsbcUs2AjhnR8E8Jqv62VHOaFpWjSrFpAcErTBT+dKlN6XDYJ Ol5dHVwHBqtUBCZ8ijQbxeZi5Ajsry2+x5N4kK7SK0+8S9UERHmeJDkSiTkdroBHpPWo lkbwlPs3PRYZHhkaFHiiIoREbNMpfX93G7CO0+TP4LaZSW6deN76JGhk3x3Xj7c9jzDQ szLSUqvUjrnQnTvEjdh9pL2qKki2YXJq1sy7fJ6AkPkawGvIq2L4LGi8JkL4hYRg0PGS EV6Q== X-Gm-Message-State: AOAM533QElPInQ61OsClmxG/wUB1YZcuIioWT+RAXukk5Uuzkh70kF4S LbE+iig/8W6fe1OA3nv58lI= X-Google-Smtp-Source: ABdhPJyakffxZqIrUTgWsubK1L1QMR4hUZy2SqMAi4fIjOwQGJaqDFgSe6nsUArLoGBtvG2gBMs1QQ== X-Received: by 2002:a17:90a:b10e:: with SMTP id z14mr24577698pjq.8.1597026493946; Sun, 09 Aug 2020 19:28:13 -0700 (PDT) Received: from bobo.ibm.com (193-116-100-32.tpgi.com.au. [193.116.100.32]) by smtp.gmail.com with ESMTPSA id l17sm21863475pff.126.2020.08.09.19.28.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 09 Aug 2020 19:28:13 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Subject: [PATCH v3 5/8] mm: HUGE_VMAP arch support cleanup Date: Mon, 10 Aug 2020 12:27:29 +1000 Message-Id: <20200810022732.1150009-6-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200810022732.1150009-1-npiggin@gmail.com> References: <20200810022732.1150009-1-npiggin@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200809_222816_265763_8FC53CE3 X-CRM114-Status: GOOD ( 26.51 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:641 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider [npiggin[at]gmail.com] -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Zefan Li , Catalin Marinas , x86@kernel.org, linuxppc-dev@lists.ozlabs.org, Nicholas Piggin , linux-kernel@vger.kernel.org, Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , Thomas Gleixner , Will Deacon , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org This changes the awkward approach where architectures provide init functions to determine which levels they can provide large mappings for, to one where the arch is queried for each call. This removes code and indirection, and allows constant-folding of dead code for unsupported levels. This also adds a prot argument to the arch query. This is unused currently but could help with some architectures (e.g., some powerpc processors can't map uncacheable memory with large pages). Signed-off-by: Nicholas Piggin Reported-by: kernel test robot --- arch/arm64/mm/mmu.c | 10 +-- arch/powerpc/mm/book3s64/radix_pgtable.c | 8 +- arch/x86/mm/ioremap.c | 10 +-- include/linux/io.h | 9 --- include/linux/vmalloc.h | 10 +++ init/main.c | 1 - mm/ioremap.c | 96 +++++++++++------------- 7 files changed, 67 insertions(+), 77 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 75df62fea1b6..184a6e917417 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1304,12 +1304,12 @@ void *__init fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot) return dt_virt; } -int __init arch_ioremap_p4d_supported(void) +bool arch_vmap_p4d_supported(pgprot_t prot) { - return 0; + return false; } -int __init arch_ioremap_pud_supported(void) +bool arch_vmap_pud_supported(pgprot_t prot) { /* * Only 4k granule supports level 1 block mappings. @@ -1319,9 +1319,9 @@ int __init arch_ioremap_pud_supported(void) !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); } -int __init arch_ioremap_pmd_supported(void) +bool arch_vmap_pmd_supported(pgprot_t prot) { - /* See arch_ioremap_pud_supported() */ + /* See arch_vmap_pud_supported() */ return !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); } diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 28c784976bed..eca83a50bf2e 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1134,13 +1134,13 @@ void radix__ptep_modify_prot_commit(struct vm_area_struct *vma, set_pte_at(mm, addr, ptep, pte); } -int __init arch_ioremap_pud_supported(void) +bool arch_vmap_pud_supported(pgprot_t prot) { /* HPT does not cope with large pages in the vmalloc area */ return radix_enabled(); } -int __init arch_ioremap_pmd_supported(void) +bool arch_vmap_pmd_supported(pgprot_t prot) { return radix_enabled(); } @@ -1234,7 +1234,7 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) return 1; } -int __init arch_ioremap_p4d_supported(void) +bool arch_vmap_p4d_supported(pgprot_t prot) { - return 0; + return false; } diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c index 84d85dbd1dad..159bfca757b9 100644 --- a/arch/x86/mm/ioremap.c +++ b/arch/x86/mm/ioremap.c @@ -481,21 +481,21 @@ void iounmap(volatile void __iomem *addr) } EXPORT_SYMBOL(iounmap); -int __init arch_ioremap_p4d_supported(void) +bool arch_vmap_p4d_supported(pgprot_t prot) { - return 0; + return false; } -int __init arch_ioremap_pud_supported(void) +bool arch_vmap_pud_supported(pgprot_t prot) { #ifdef CONFIG_X86_64 return boot_cpu_has(X86_FEATURE_GBPAGES); #else - return 0; + return false; #endif } -int __init arch_ioremap_pmd_supported(void) +bool arch_vmap_pmd_supported(pgprot_t prot) { return boot_cpu_has(X86_FEATURE_PSE); } diff --git a/include/linux/io.h b/include/linux/io.h index 8394c56babc2..f1effd4d7a3c 100644 --- a/include/linux/io.h +++ b/include/linux/io.h @@ -31,15 +31,6 @@ static inline int ioremap_page_range(unsigned long addr, unsigned long end, } #endif -#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -void __init ioremap_huge_init(void); -int arch_ioremap_p4d_supported(void); -int arch_ioremap_pud_supported(void); -int arch_ioremap_pmd_supported(void); -#else -static inline void ioremap_huge_init(void) { } -#endif - /* * Managed iomap interface */ diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 0221f852a7e1..787d77ad7536 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -84,6 +84,16 @@ struct vmap_area { }; }; +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +bool arch_vmap_p4d_supported(pgprot_t prot); +bool arch_vmap_pud_supported(pgprot_t prot); +bool arch_vmap_pmd_supported(pgprot_t prot); +#else +static inline bool arch_vmap_p4d_supported(pgprot_t prot) { return false; } +static inline bool arch_vmap_pud_supported(pgprot_t prot) { return false; } +static inline bool arch_vmap_pmd_supported(pgprot_t prot) { return false; } +#endif + /* * Highlevel APIs for driver use */ diff --git a/init/main.c b/init/main.c index ae78fb68d231..1c89aa127b8f 100644 --- a/init/main.c +++ b/init/main.c @@ -820,7 +820,6 @@ static void __init mm_init(void) pgtable_init(); debug_objects_mem_init(); vmalloc_init(); - ioremap_huge_init(); /* Should be run before the first non-init thread is created */ init_espfix_bsp(); /* Should be run after espfix64 is set up. */ diff --git a/mm/ioremap.c b/mm/ioremap.c index 6016ae3227ad..b0032dbadaf7 100644 --- a/mm/ioremap.c +++ b/mm/ioremap.c @@ -16,49 +16,16 @@ #include "pgalloc-track.h" #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -static int __read_mostly ioremap_p4d_capable; -static int __read_mostly ioremap_pud_capable; -static int __read_mostly ioremap_pmd_capable; -static int __read_mostly ioremap_huge_disabled; +static bool __ro_after_init iomap_allow_huge = true; static int __init set_nohugeiomap(char *str) { - ioremap_huge_disabled = 1; + iomap_allow_huge = false; return 0; } early_param("nohugeiomap", set_nohugeiomap); - -void __init ioremap_huge_init(void) -{ - if (!ioremap_huge_disabled) { - if (arch_ioremap_p4d_supported()) - ioremap_p4d_capable = 1; - if (arch_ioremap_pud_supported()) - ioremap_pud_capable = 1; - if (arch_ioremap_pmd_supported()) - ioremap_pmd_capable = 1; - } -} - -static inline int ioremap_p4d_enabled(void) -{ - return ioremap_p4d_capable; -} - -static inline int ioremap_pud_enabled(void) -{ - return ioremap_pud_capable; -} - -static inline int ioremap_pmd_enabled(void) -{ - return ioremap_pmd_capable; -} - -#else /* !CONFIG_HAVE_ARCH_HUGE_VMAP */ -static inline int ioremap_p4d_enabled(void) { return 0; } -static inline int ioremap_pud_enabled(void) { return 0; } -static inline int ioremap_pmd_enabled(void) { return 0; } +#else /* CONFIG_HAVE_ARCH_HUGE_VMAP */ +static const bool iomap_allow_huge = false; #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, @@ -81,9 +48,12 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, } static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) { - if (!ioremap_pmd_enabled()) + if (max_page_shift < PMD_SHIFT) + return 0; + + if (!arch_vmap_pmd_supported(prot)) return 0; if ((end - addr) != PMD_SIZE) @@ -102,7 +72,8 @@ static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, } static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, pgtbl_mod_mask *mask) + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, + pgtbl_mod_mask *mask) { pmd_t *pmd; unsigned long next; @@ -113,7 +84,7 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, do { next = pmd_addr_end(addr, end); - if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, max_page_shift)) { *mask |= PGTBL_PMD_MODIFIED; continue; } @@ -125,9 +96,12 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, } static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) { - if (!ioremap_pud_enabled()) + if (max_page_shift < PUD_SHIFT) + return 0; + + if (!arch_vmap_pud_supported(prot)) return 0; if ((end - addr) != PUD_SIZE) @@ -146,7 +120,8 @@ static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, } static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, pgtbl_mod_mask *mask) + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, + pgtbl_mod_mask *mask) { pud_t *pud; unsigned long next; @@ -157,21 +132,24 @@ static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, do { next = pud_addr_end(addr, end); - if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot, max_page_shift)) { *mask |= PGTBL_PUD_MODIFIED; continue; } - if (vmap_pmd_range(pud, addr, next, phys_addr, prot, mask)) + if (vmap_pmd_range(pud, addr, next, phys_addr, prot, max_page_shift, mask)) return -ENOMEM; } while (pud++, phys_addr += (next - addr), addr = next, addr != end); return 0; } static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) { - if (!ioremap_p4d_enabled()) + if (max_page_shift < P4D_SHIFT) + return 0; + + if (!arch_vmap_p4d_supported(prot)) return 0; if ((end - addr) != P4D_SIZE) @@ -190,7 +168,8 @@ static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, } static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, pgtbl_mod_mask *mask) + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, + pgtbl_mod_mask *mask) { p4d_t *p4d; unsigned long next; @@ -201,18 +180,19 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, do { next = p4d_addr_end(addr, end); - if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) { + if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot, max_page_shift)) { *mask |= PGTBL_P4D_MODIFIED; continue; } - if (vmap_pud_range(p4d, addr, next, phys_addr, prot, mask)) + if (vmap_pud_range(p4d, addr, next, phys_addr, prot, max_page_shift, mask)) return -ENOMEM; } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); return 0; } -int ioremap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot) +static int vmap_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) { pgd_t *pgd; unsigned long start; @@ -227,7 +207,7 @@ int ioremap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_a pgd = pgd_offset_k(addr); do { next = pgd_addr_end(addr, end); - err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, &mask); + err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, max_page_shift, &mask); if (err) break; } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); @@ -240,6 +220,16 @@ int ioremap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_a return err; } +int ioremap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot) +{ + unsigned int max_page_shift = PAGE_SHIFT; + + if (iomap_allow_huge) + max_page_shift = P4D_SHIFT; + + return vmap_range(addr, end, phys_addr, prot, max_page_shift); +} + #ifdef CONFIG_GENERIC_IOREMAP void __iomem *ioremap_prot(phys_addr_t addr, size_t size, unsigned long prot) { From patchwork Mon Aug 10 02:27:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11706917 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BF299913 for ; Mon, 10 Aug 2020 02:30:08 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8A1632068F for ; Mon, 10 Aug 2020 02:30:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="j2Y72ITX"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nzm2mSWg" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8A1632068F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=bnGZ3iGD1n7yx6Th3SSqnNPudc0ozOH2JYRR4XFjWEk=; b=j2Y72ITXY0IgcatGnYcPP/XX4 tJ4eDpk4FnZM3AEWqjmW5gwBP8y/V2LDs0EIn8HPYZybGlIx6on9QazcnCU1jo4ynCCbRUsESNenO 83GDQAwv/vWxxNsUw/X6JmEm1vkp8RwC1u1Yc47xSnX7JQ36Y6WKVgKFmZsBz/jefAB5J0mS65Tk2 cqqWeIC3c5QNkbhLehSJnq81GNExUai/DNv7YB0Do85O/OCZ3y9RD2EkzLitdPLKitr2Ua35cH18f HtOkS3uNnLAUxOQWnM0rOqPVbrZUlAvgUcMsb7AgmNH7Ez8gcdS/5rEUmGAMJ3JBfMr84SOxZm/vk qSCYStmUw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k4xY1-00058K-PI; Mon, 10 Aug 2020 02:28:25 +0000 Received: from mail-pl1-x644.google.com ([2607:f8b0:4864:20::644]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k4xXx-00056g-ST for linux-arm-kernel@lists.infradead.org; Mon, 10 Aug 2020 02:28:23 +0000 Received: by mail-pl1-x644.google.com with SMTP id z20so4037402plo.6 for ; Sun, 09 Aug 2020 19:28:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=n6u3W3xGRvE1f9TahLPt2tr1kQk4naizqZpWLpaJwxQ=; b=nzm2mSWgcF2C9f99wzi08Ql2QMVmUWsm29f5O5eBDvlRDzhbWguvvk0MLq14Jov9pL J3gyMa5yXaJWzK0BWQO6lRlBa1bAT/MndVEYllh7nk8KvX5NPUsDu06EltSLVrFDfBXJ Et472wbuMqXYBIG9zaM6d1eD8MopnUtlFa6K1izrDGSSfVay3hgRh5mY06XRwIwemtNm Dl/LWPGUFXUFy9KiGA2KFg4ZtSs6m6WNtmEG0+CQi0ea31POZD4rulroMY1atOaH6bmn JTmyFDX0To2jEHBYI8X667VaPiCbVWrE803eS7YxuYsC4QX8PoaXRISjM3WXMiLtcIiM /K6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=n6u3W3xGRvE1f9TahLPt2tr1kQk4naizqZpWLpaJwxQ=; b=OW1b0tXJ5URlpQm5a+RZYqzDF65+oIDNk6xlr4obeRhAl0hCkRW4Qzd/oEoNcpAcS/ EGv0J/r7mdta9KQfqlPUjbrlHb42PCLfsoZn1CeS9YEEqR+M52IFcqEL6Nfz6gRWSZO2 QiYRyz3zNxwUOelmHvUOQsXHhtSsqE/zU8FzKYdhgjOmO6R3IE2KSprlcRZTzZi1Q1h5 2ScngkP8Kij+FqwJcrAy44APjUGq5HrG3I1gWkQJxTqjC7Bqa1aXsqwrr6F84BThH5GC uoYcZnE7ga0DgNXyE3Ea5x4boSBIQ3eVthb+PGyLBTJyt+Nuo8v+HaFVFwzO8TfsXsLO duKg== X-Gm-Message-State: AOAM531ToPz7bMXYJUX5oYR9/w1A1RpwTF2RBiZyPbjfDlDReqvoBCtC aPHnaUL45CQawaiH4cGmUyo= X-Google-Smtp-Source: ABdhPJysgrcVwNeIzX0wpzRpCYPA89SKM60P0U1p8K/OIM8Od7oilhb+Ai1vknhvluUYAT5wjCLY/w== X-Received: by 2002:a17:90a:ff0e:: with SMTP id ce14mr25645781pjb.128.1597026500019; Sun, 09 Aug 2020 19:28:20 -0700 (PDT) Received: from bobo.ibm.com (193-116-100-32.tpgi.com.au. [193.116.100.32]) by smtp.gmail.com with ESMTPSA id l17sm21863475pff.126.2020.08.09.19.28.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 09 Aug 2020 19:28:19 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Subject: [PATCH v3 6/8] mm: Move vmap_range from lib/ioremap.c to mm/vmalloc.c Date: Mon, 10 Aug 2020 12:27:30 +1000 Message-Id: <20200810022732.1150009-7-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200810022732.1150009-1-npiggin@gmail.com> References: <20200810022732.1150009-1-npiggin@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200809_222822_009997_51ECA2E0 X-CRM114-Status: GOOD ( 19.77 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:644 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider [npiggin[at]gmail.com] -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Zefan Li , Catalin Marinas , x86@kernel.org, linuxppc-dev@lists.ozlabs.org, Nicholas Piggin , linux-kernel@vger.kernel.org, Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , Thomas Gleixner , Will Deacon , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org This is a generic kernel virtual memory mapper, not specific to ioremap. Signed-off-by: Nicholas Piggin --- include/linux/vmalloc.h | 2 + mm/ioremap.c | 192 ---------------------------------------- mm/vmalloc.c | 191 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 193 insertions(+), 192 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 787d77ad7536..e3590e93bfff 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -181,6 +181,8 @@ extern struct vm_struct *remove_vm_area(const void *addr); extern struct vm_struct *find_vm_area(const void *addr); #ifdef CONFIG_MMU +extern int vmap_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift); extern int map_kernel_range_noflush(unsigned long start, unsigned long size, pgprot_t prot, struct page **pages); int map_kernel_range(unsigned long start, unsigned long size, pgprot_t prot, diff --git a/mm/ioremap.c b/mm/ioremap.c index b0032dbadaf7..cdda0e022740 100644 --- a/mm/ioremap.c +++ b/mm/ioremap.c @@ -28,198 +28,6 @@ early_param("nohugeiomap", set_nohugeiomap); static const bool iomap_allow_huge = false; #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ -static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, pgtbl_mod_mask *mask) -{ - pte_t *pte; - u64 pfn; - - pfn = phys_addr >> PAGE_SHIFT; - pte = pte_alloc_kernel_track(pmd, addr, mask); - if (!pte) - return -ENOMEM; - do { - BUG_ON(!pte_none(*pte)); - set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); - pfn++; - } while (pte++, addr += PAGE_SIZE, addr != end); - *mask |= PGTBL_PTE_MODIFIED; - return 0; -} - -static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) -{ - if (max_page_shift < PMD_SHIFT) - return 0; - - if (!arch_vmap_pmd_supported(prot)) - return 0; - - if ((end - addr) != PMD_SIZE) - return 0; - - if (!IS_ALIGNED(addr, PMD_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, PMD_SIZE)) - return 0; - - if (pmd_present(*pmd) && !pmd_free_pte_page(pmd, addr)) - return 0; - - return pmd_set_huge(pmd, phys_addr, prot); -} - -static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, - pgtbl_mod_mask *mask) -{ - pmd_t *pmd; - unsigned long next; - - pmd = pmd_alloc_track(&init_mm, pud, addr, mask); - if (!pmd) - return -ENOMEM; - do { - next = pmd_addr_end(addr, end); - - if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, max_page_shift)) { - *mask |= PGTBL_PMD_MODIFIED; - continue; - } - - if (vmap_pte_range(pmd, addr, next, phys_addr, prot, mask)) - return -ENOMEM; - } while (pmd++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) -{ - if (max_page_shift < PUD_SHIFT) - return 0; - - if (!arch_vmap_pud_supported(prot)) - return 0; - - if ((end - addr) != PUD_SIZE) - return 0; - - if (!IS_ALIGNED(addr, PUD_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, PUD_SIZE)) - return 0; - - if (pud_present(*pud) && !pud_free_pmd_page(pud, addr)) - return 0; - - return pud_set_huge(pud, phys_addr, prot); -} - -static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, - pgtbl_mod_mask *mask) -{ - pud_t *pud; - unsigned long next; - - pud = pud_alloc_track(&init_mm, p4d, addr, mask); - if (!pud) - return -ENOMEM; - do { - next = pud_addr_end(addr, end); - - if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot, max_page_shift)) { - *mask |= PGTBL_PUD_MODIFIED; - continue; - } - - if (vmap_pmd_range(pud, addr, next, phys_addr, prot, max_page_shift, mask)) - return -ENOMEM; - } while (pud++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) -{ - if (max_page_shift < P4D_SHIFT) - return 0; - - if (!arch_vmap_p4d_supported(prot)) - return 0; - - if ((end - addr) != P4D_SIZE) - return 0; - - if (!IS_ALIGNED(addr, P4D_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, P4D_SIZE)) - return 0; - - if (p4d_present(*p4d) && !p4d_free_pud_page(p4d, addr)) - return 0; - - return p4d_set_huge(p4d, phys_addr, prot); -} - -static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, - pgtbl_mod_mask *mask) -{ - p4d_t *p4d; - unsigned long next; - - p4d = p4d_alloc_track(&init_mm, pgd, addr, mask); - if (!p4d) - return -ENOMEM; - do { - next = p4d_addr_end(addr, end); - - if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot, max_page_shift)) { - *mask |= PGTBL_P4D_MODIFIED; - continue; - } - - if (vmap_pud_range(p4d, addr, next, phys_addr, prot, max_page_shift, mask)) - return -ENOMEM; - } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -static int vmap_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) -{ - pgd_t *pgd; - unsigned long start; - unsigned long next; - int err; - pgtbl_mod_mask mask = 0; - - might_sleep(); - BUG_ON(addr >= end); - - start = addr; - pgd = pgd_offset_k(addr); - do { - next = pgd_addr_end(addr, end); - err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, max_page_shift, &mask); - if (err) - break; - } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); - - flush_cache_vmap(start, end); - - if (mask & ARCH_PAGE_TABLE_SYNC_MASK) - arch_sync_kernel_mappings(start, end); - - return err; -} - int ioremap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot) { unsigned int max_page_shift = PAGE_SHIFT; diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 3a1e45fd1626..129f10545bb1 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -71,6 +71,197 @@ static void free_work(struct work_struct *w) } /*** Page table manipulation functions ***/ +static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, pgtbl_mod_mask *mask) +{ + pte_t *pte; + u64 pfn; + + pfn = phys_addr >> PAGE_SHIFT; + pte = pte_alloc_kernel_track(pmd, addr, mask); + if (!pte) + return -ENOMEM; + do { + BUG_ON(!pte_none(*pte)); + set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); + pfn++; + } while (pte++, addr += PAGE_SIZE, addr != end); + *mask |= PGTBL_PTE_MODIFIED; + return 0; +} + +static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) +{ + if (max_page_shift < PMD_SHIFT) + return 0; + + if (!arch_vmap_pmd_supported(prot)) + return 0; + + if ((end - addr) != PMD_SIZE) + return 0; + + if (!IS_ALIGNED(addr, PMD_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, PMD_SIZE)) + return 0; + + if (pmd_present(*pmd) && !pmd_free_pte_page(pmd, addr)) + return 0; + + return pmd_set_huge(pmd, phys_addr, prot); +} + +static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, + pgtbl_mod_mask *mask) +{ + pmd_t *pmd; + unsigned long next; + + pmd = pmd_alloc_track(&init_mm, pud, addr, mask); + if (!pmd) + return -ENOMEM; + do { + next = pmd_addr_end(addr, end); + + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, max_page_shift)) { + *mask |= PGTBL_PMD_MODIFIED; + continue; + } + + if (vmap_pte_range(pmd, addr, next, phys_addr, prot, mask)) + return -ENOMEM; + } while (pmd++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) +{ + if (max_page_shift < PUD_SHIFT) + return 0; + + if (!arch_vmap_pud_supported(prot)) + return 0; + + if ((end - addr) != PUD_SIZE) + return 0; + + if (!IS_ALIGNED(addr, PUD_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, PUD_SIZE)) + return 0; + + if (pud_present(*pud) && !pud_free_pmd_page(pud, addr)) + return 0; + + return pud_set_huge(pud, phys_addr, prot); +} + +static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, + pgtbl_mod_mask *mask) +{ + pud_t *pud; + unsigned long next; + + pud = pud_alloc_track(&init_mm, p4d, addr, mask); + if (!pud) + return -ENOMEM; + do { + next = pud_addr_end(addr, end); + + if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot, max_page_shift)) { + *mask |= PGTBL_PUD_MODIFIED; + continue; + } + + if (vmap_pmd_range(pud, addr, next, phys_addr, prot, max_page_shift, mask)) + return -ENOMEM; + } while (pud++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) +{ + if (max_page_shift < P4D_SHIFT) + return 0; + + if (!arch_vmap_p4d_supported(prot)) + return 0; + + if ((end - addr) != P4D_SIZE) + return 0; + + if (!IS_ALIGNED(addr, P4D_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, P4D_SIZE)) + return 0; + + if (p4d_present(*p4d) && !p4d_free_pud_page(p4d, addr)) + return 0; + + return p4d_set_huge(p4d, phys_addr, prot); +} + +static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, + pgtbl_mod_mask *mask) +{ + p4d_t *p4d; + unsigned long next; + + p4d = p4d_alloc_track(&init_mm, pgd, addr, mask); + if (!p4d) + return -ENOMEM; + do { + next = p4d_addr_end(addr, end); + + if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot, max_page_shift)) { + *mask |= PGTBL_P4D_MODIFIED; + continue; + } + + if (vmap_pud_range(p4d, addr, next, phys_addr, prot, max_page_shift, mask)) + return -ENOMEM; + } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +int vmap_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + pgd_t *pgd; + unsigned long start; + unsigned long next; + int err; + pgtbl_mod_mask mask = 0; + + might_sleep(); + BUG_ON(addr >= end); + + start = addr; + pgd = pgd_offset_k(addr); + do { + next = pgd_addr_end(addr, end); + err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, max_page_shift, &mask); + if (err) + break; + } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); + + flush_cache_vmap(start, end); + + if (mask & ARCH_PAGE_TABLE_SYNC_MASK) + arch_sync_kernel_mappings(start, end); + + return err; +} static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgtbl_mod_mask *mask) From patchwork Mon Aug 10 02:27:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11706915 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B2083618 for ; Mon, 10 Aug 2020 02:30:04 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8B09420709 for ; Mon, 10 Aug 2020 02:30:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="xnCw97pn"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fvgC0Iga" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8B09420709 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=pIOlb45AMBcOAifo4UQvJ3j3RQiidxVU5Y0te0suDO0=; b=xnCw97pnYXXLIX/AsXYwo3Twd CBuokwD9gvUIurbJOj/3LOF6X512BNb31O5TZFwd72mCdVlv9Jg1TggxbA1nCDmzNMRS7zVNVpB+c L4fuFOTray3Vs4l2FZN+tQKHfSZz4AWQOUYS4i5peNF2cKZ3Adk2OGK8gY3l7+lF6mx80utrG0m7d K9A2GEHfz0gwRmCznsXO7kLadQPl8RAHALoaDZVqkens4HrDRt0JunH6Li512XcV5I5Q3MJHf8+DA TrL5UsxgAaXfdHQbMVMiyqMRUknZnzbgpNxNbqQDJ646tLpJwocdYqjaSAxdCom9l6pKYGe55WD+F /YmX13amA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k4xYB-0005Bi-8a; Mon, 10 Aug 2020 02:28:35 +0000 Received: from mail-pl1-x642.google.com ([2607:f8b0:4864:20::642]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k4xY3-00058u-Nn for linux-arm-kernel@lists.infradead.org; Mon, 10 Aug 2020 02:28:28 +0000 Received: by mail-pl1-x642.google.com with SMTP id w17so4026139ply.11 for ; Sun, 09 Aug 2020 19:28:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8C8wdRPXIlS04tUD/986mX054vDuu2PdIQ/xTcAZZAA=; b=fvgC0IgaVGBBCP46Zn2yjyIZBeVrRa8NWp7JGxI+yUxhgJTd3YVhaytoaDsU/lDFxs 4iHYUOz1X6WPpfvjY4IF1lTRGWy4UjbYIjINrDDfjUjnH8Hv2o6be0jAQjOR4a4F7CLi 2RVr+OFdQqc/sDLwlZ1kM/3YAAHJN7AadRjMtJsROlN1GaMxJ8U9SxHRPbwbs9d5PTiK Aq4OFkLg9VtrOcx/FpLXPv5ctO8m9w9rrlcTOmRnfvSDCUXu9gcTF/7W4PID3Hqf3Iwu eaRu+KC82AZk5/OL9BigF/IAiD7pXYIiFsHSf6z1CuFlumZsmCM6WM4iS6Z0hakBUkZ+ //MQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8C8wdRPXIlS04tUD/986mX054vDuu2PdIQ/xTcAZZAA=; b=tzClmAW5zQ/LeBd/V/mljXCZIIlVYVVEUYZ1OwP4KWaF8fZcmXBH08OhUBPqC4OmVO bmrjPMUROMSOYbmIdSV7QZr31ET0p0K2+/srAjgWXOnwMYo0QvT1mqO1GpGDHON4KAYj qL2uiqDg+Xuy/bYOp+Zps6JeqJH4vDY+/DHZ6esy1NQ7x60ezeGAeus51rwtpTltobeo 2eiSUu74welm9FasaXX60L94mh4Z8G57nb31u0CJ+KT9vdlx3WpuCmJrMmBUxfqAoMel ptZlBeqxAmvRSw9JFM1XGPzIhk1yTmB9kk1mbg1yeHuRkdDWvZKUI9DtFRCKbcYTV5Xf iQTA== X-Gm-Message-State: AOAM533pRNeOImnPRo/l0mE2FxL2eTFopu/TBg6R6d+HBMh5Lf99r/iz EqRQCXU4P8oS1Wvoy1ji5ZA= X-Google-Smtp-Source: ABdhPJzqg/csRjQI/X8gAPnWHF45ru/8heKZxPOpRtPbcR9UPs4MpJo+AuvP7KubNHtfifF+p8MeWw== X-Received: by 2002:a17:902:8685:: with SMTP id g5mr22059716plo.201.1597026505571; Sun, 09 Aug 2020 19:28:25 -0700 (PDT) Received: from bobo.ibm.com (193-116-100-32.tpgi.com.au. [193.116.100.32]) by smtp.gmail.com with ESMTPSA id l17sm21863475pff.126.2020.08.09.19.28.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 09 Aug 2020 19:28:25 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Subject: [PATCH v3 7/8] mm/vmalloc: add vmap_range_noflush variant Date: Mon, 10 Aug 2020 12:27:31 +1000 Message-Id: <20200810022732.1150009-8-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200810022732.1150009-1-npiggin@gmail.com> References: <20200810022732.1150009-1-npiggin@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200809_222827_830317_3600B87D X-CRM114-Status: GOOD ( 14.91 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:642 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider [npiggin[at]gmail.com] -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Zefan Li , Catalin Marinas , x86@kernel.org, linuxppc-dev@lists.ozlabs.org, Nicholas Piggin , linux-kernel@vger.kernel.org, Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , Thomas Gleixner , Will Deacon , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org As a side-effect, the order of flush_cache_vmap() and arch_sync_kernel_mappings() calls are switched, but that now matches the other callers in this file. Signed-off-by: Nicholas Piggin --- mm/vmalloc.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 129f10545bb1..4e5cb7c7f780 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -234,8 +234,8 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, return 0; } -int vmap_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) +static int vmap_range_noflush(unsigned long addr, unsigned long end, phys_addr_t phys_addr, + pgprot_t prot, unsigned int max_page_shift) { pgd_t *pgd; unsigned long start; @@ -255,14 +255,23 @@ int vmap_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgp break; } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); - flush_cache_vmap(start, end); - if (mask & ARCH_PAGE_TABLE_SYNC_MASK) arch_sync_kernel_mappings(start, end); return err; } +int vmap_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + int err; + + err = vmap_range_noflush(addr, end, phys_addr, prot, max_page_shift); + flush_cache_vmap(addr, end); + + return err; +} + static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgtbl_mod_mask *mask) { From patchwork Mon Aug 10 02:27:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11706919 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 45A13913 for ; Mon, 10 Aug 2020 02:30:37 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0E9CE206C3 for ; Mon, 10 Aug 2020 02:30:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="bBW21nrL"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="tPXjNJyx" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0E9CE206C3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=XKioHpWPWbvGirVijnlHo2wRrX1L1GZDH4/5/9pyR/g=; b=bBW21nrL+sVjsiNDTifFGoFGi qetCV5YYQOu3OsKppWA2HbQLiUvErJEawDMgYsl5eYN7IA4oBV+dU9bAsDvDQU5cPwvyd+VAsfqai UvVwTphZ1udGyd/lF/Bbw3g+qClp8Jz0KFhZ4BreNOFr+eooQcGYyxXnh31LiEyDV6Kxn2kHmD3Sj BvSOgshrF5Lyy07Hkw4JZYcIc68JA49GfloSTa4Z7tWbxVBG6SjnWGUl0uykGdKCj748xnfE0N902 M5uY13pqg2vWQjAzb0VJyaLJmb7v6OYgjTJXAoUI7OVGoC3dC7lon6rfVjewC0sLmQE6r1hMbwevo vyKf08Y2A==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k4xYN-0005HR-TJ; Mon, 10 Aug 2020 02:28:47 +0000 Received: from mail-pj1-x1044.google.com ([2607:f8b0:4864:20::1044]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k4xY9-0005Bj-Js for linux-arm-kernel@lists.infradead.org; Mon, 10 Aug 2020 02:28:37 +0000 Received: by mail-pj1-x1044.google.com with SMTP id i92so8235770pje.0 for ; Sun, 09 Aug 2020 19:28:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=A0so4fQowQIABHlZeHOs4si+EurGVBhpstQxSdjCTKk=; b=tPXjNJyxl/wafOmjktA4v0Ff+6NiFDhm4uZJcRy4HkzLWih0dPUV2CRwh3DGpMwIHF aItzNDc9IbTLdzEljsN6mYtxCizmXQgXcjBf74sg5ANtkyq97tfSnq/Mp3zNE3px8GCM 3OMirQTYfqlY0iL/dLFiUOZank9oTo3Cg5tkddIODGDb6Nnjw3juULR5rfKwd+vAv/ep G+69QNC/hMz2nPxMhGOqwj+FydQc9PrVOGvCVAxahSdjvzzoma50VHV2BuXkN1rJqV4v SALbUpMz6Dfgs7owjWTFbxm6f8wGV6BKVrpRMvnH3Oq4Ay84cmpHJr45w7ow2VTEsST6 Ds4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=A0so4fQowQIABHlZeHOs4si+EurGVBhpstQxSdjCTKk=; b=cziOPKRc51lQnBLctp9iqZ/Er1Y/pke8sSqDZzVGQCDUC6aiEWOFrsaFt5H6IUrvVI 4TSL2TXwz7AiYkqyeli7j6yNHXyAWW3WmegeO1WAtB/13KYY7Ln+mhYl851UvpNOPIvI xwp5SuJhFEyqlGFy9Kp5KSz2aT5cSvct/geB4MRWEtRy2zi7tam5DGxYOUn4FtFUx+Ij xwo2EyuwJTWpgaHLhEBvSYofGQe2+xzDi3pkbbNhW3o/hi2ALW6oe6Zk7synS5jkMbsZ mOPVdrfizbFFLWDHbeDvZUnBmsDF0/kMCAQSvg4w1Y17AMziv3a1JQSq6BXBVpr/1krm u/KQ== X-Gm-Message-State: AOAM530voRMUOaW1K3x+OVdqbHxK/uu/NpfIzUzd51xltCfmqoNkN8dz ALqGH6rLlD1kW4h08/wn7io= X-Google-Smtp-Source: ABdhPJy2cbTkCgVj+7r3lbrOmme2kgKLyxtJ3KbxYuQ7Nh6zJNk1/S9jERNxlg1hhtGQ8/P+Younzg== X-Received: by 2002:a17:90a:884:: with SMTP id v4mr22909813pjc.27.1597026511406; Sun, 09 Aug 2020 19:28:31 -0700 (PDT) Received: from bobo.ibm.com (193-116-100-32.tpgi.com.au. [193.116.100.32]) by smtp.gmail.com with ESMTPSA id l17sm21863475pff.126.2020.08.09.19.28.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 09 Aug 2020 19:28:31 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Subject: [PATCH v3 8/8] mm/vmalloc: Hugepage vmalloc mappings Date: Mon, 10 Aug 2020 12:27:32 +1000 Message-Id: <20200810022732.1150009-9-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200810022732.1150009-1-npiggin@gmail.com> References: <20200810022732.1150009-1-npiggin@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200809_222834_238915_222AE273 X-CRM114-Status: GOOD ( 32.16 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:1044 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider [npiggin[at]gmail.com] -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Zefan Li , Catalin Marinas , x86@kernel.org, linuxppc-dev@lists.ozlabs.org, Nicholas Piggin , linux-kernel@vger.kernel.org, Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , Thomas Gleixner , Will Deacon , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org On platforms that define HAVE_ARCH_HUGE_VMAP and support PMD vmaps, vmalloc will attempt to allocate PMD-sized pages first, before falling back to small pages. Allocations which use something other than PAGE_KERNEL protections are not permitted to use huge pages yet, not all callers expect this (e.g., module allocations vs strict module rwx). This reduces TLB misses by nearly 30x on a `git diff` workload on a 2-node POWER9 (59,800 -> 2,100) and reduces CPU cycles by 0.54%. This can result in more internal fragmentation and memory overhead for a given allocation, an option nohugevmap is added to disable at boot. Signed-off-by: Nicholas Piggin --- .../admin-guide/kernel-parameters.txt | 2 + include/linux/vmalloc.h | 1 + mm/vmalloc.c | 174 +++++++++++++----- 3 files changed, 135 insertions(+), 42 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 98ea67f27809..eaef176c597f 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3190,6 +3190,8 @@ nohugeiomap [KNL,x86,PPC] Disable kernel huge I/O mappings. + nohugevmap [KNL,x86,PPC] Disable kernel huge vmalloc mappings. + nosmt [KNL,S390] Disable symmetric multithreading (SMT). Equivalent to smt=1. diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index e3590e93bfff..8f25dbaca0a1 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -58,6 +58,7 @@ struct vm_struct { unsigned long size; unsigned long flags; struct page **pages; + unsigned int page_order; unsigned int nr_pages; phys_addr_t phys_addr; const void *caller; diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 4e5cb7c7f780..a7728c7086bc 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -45,6 +45,19 @@ #include "internal.h" #include "pgalloc-track.h" +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +static bool __ro_after_init vmap_allow_huge = true; + +static int __init set_nohugevmap(char *str) +{ + vmap_allow_huge = false; + return 0; +} +early_param("nohugevmap", set_nohugevmap); +#else /* CONFIG_HAVE_ARCH_HUGE_VMAP */ +static const bool vmap_allow_huge = false; +#endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ + bool is_vmalloc_addr(const void *x) { unsigned long addr = (unsigned long)x; @@ -468,31 +481,12 @@ static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long en return 0; } -/** - * map_kernel_range_noflush - map kernel VM area with the specified pages - * @addr: start of the VM area to map - * @size: size of the VM area to map - * @prot: page protection flags to use - * @pages: pages to map - * - * Map PFN_UP(@size) pages at @addr. The VM area @addr and @size specify should - * have been allocated using get_vm_area() and its friends. - * - * NOTE: - * This function does NOT do any cache flushing. The caller is responsible for - * calling flush_cache_vmap() on to-be-mapped areas before calling this - * function. - * - * RETURNS: - * 0 on success, -errno on failure. - */ -int map_kernel_range_noflush(unsigned long addr, unsigned long size, - pgprot_t prot, struct page **pages) +static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages) { unsigned long start = addr; - unsigned long end = addr + size; - unsigned long next; pgd_t *pgd; + unsigned long next; int err = 0; int nr = 0; pgtbl_mod_mask mask = 0; @@ -514,6 +508,65 @@ int map_kernel_range_noflush(unsigned long addr, unsigned long size, return 0; } +static int vmap_pages_range_noflush(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift) +{ + WARN_ON(page_shift < PAGE_SHIFT); + + if (page_shift == PAGE_SHIFT) { + return vmap_small_pages_range_noflush(addr, end, prot, pages); + } else { + unsigned int i, nr = (end - addr) >> page_shift; + + for (i = 0; i < nr; i++) { + int err; + + err = vmap_range_noflush(addr, addr + (1UL << page_shift), + __pa(page_address(pages[i])), prot, page_shift); + if (err) + return err; + + addr += 1UL << page_shift; + } + + return 0; + } +} + +static int vmap_pages_range(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift) +{ + int err; + + err = vmap_pages_range_noflush(addr, end, prot, pages, page_shift); + flush_cache_vmap(addr, end); + return err; +} + +/** + * map_kernel_range_noflush - map kernel VM area with the specified pages + * @addr: start of the VM area to map + * @size: size of the VM area to map + * @prot: page protection flags to use + * @pages: pages to map + * + * Map PFN_UP(@size) pages at @addr. The VM area @addr and @size specify should + * have been allocated using get_vm_area() and its friends. + * + * NOTE: + * This function does NOT do any cache flushing. The caller is responsible for + * calling flush_cache_vmap() on to-be-mapped areas before calling this + * function. + * + * RETURNS: + * 0 on success, -errno on failure. + */ +int map_kernel_range_noflush(unsigned long addr, unsigned long size, + pgprot_t prot, struct page **pages) +{ + return vmap_pages_range_noflush(addr, addr + size, prot, pages, PAGE_SHIFT); +} + int map_kernel_range(unsigned long start, unsigned long size, pgprot_t prot, struct page **pages) { @@ -2274,9 +2327,11 @@ static struct vm_struct *__get_vm_area_node(unsigned long size, if (unlikely(!size)) return NULL; - if (flags & VM_IOREMAP) - align = 1ul << clamp_t(int, get_count_order_long(size), - PAGE_SHIFT, IOREMAP_MAX_ORDER); + if (flags & VM_IOREMAP) { + align = max(align, + 1ul << clamp_t(int, get_count_order_long(size), + PAGE_SHIFT, IOREMAP_MAX_ORDER)); + } area = kzalloc_node(sizeof(*area), gfp_mask & GFP_RECLAIM_MASK, node); if (unlikely(!area)) @@ -2475,7 +2530,8 @@ static void __vunmap(const void *addr, int deallocate_pages) struct page *page = area->pages[i]; BUG_ON(!page); - __free_pages(page, 0); + __free_pages(page, area->page_order); + i += 1 << area->page_order; } atomic_long_sub(area->nr_pages, &nr_vmalloc_pages); @@ -2614,9 +2670,12 @@ void *vmap(struct page **pages, unsigned int count, EXPORT_SYMBOL(vmap); static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, - pgprot_t prot, int node) + pgprot_t prot, unsigned int page_shift, int node) { struct page **pages; + unsigned long addr = (unsigned long)area->addr; + unsigned long size = get_vm_area_size(area); + unsigned int page_order = page_shift - PAGE_SHIFT; unsigned int nr_pages, array_size, i; const gfp_t nested_gfp = (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO; const gfp_t alloc_mask = gfp_mask | __GFP_NOWARN; @@ -2624,7 +2683,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, 0 : __GFP_HIGHMEM; - nr_pages = get_vm_area_size(area) >> PAGE_SHIFT; + nr_pages = size >> PAGE_SHIFT; array_size = (nr_pages * sizeof(struct page *)); /* Please note that the recursion is strictly bounded. */ @@ -2643,29 +2702,29 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, area->pages = pages; area->nr_pages = nr_pages; + area->page_order = page_order; - for (i = 0; i < area->nr_pages; i++) { + for (i = 0; i < area->nr_pages; i += 1 << page_order) { struct page *page; + int p; - if (node == NUMA_NO_NODE) - page = alloc_page(alloc_mask|highmem_mask); - else - page = alloc_pages_node(node, alloc_mask|highmem_mask, 0); - + page = alloc_pages_node(node, alloc_mask|highmem_mask, page_order); if (unlikely(!page)) { /* Successfully allocated i pages, free them in __vunmap() */ area->nr_pages = i; atomic_long_add(area->nr_pages, &nr_vmalloc_pages); goto fail; } - area->pages[i] = page; + + for (p = 0; p < (1 << page_order); p++) + area->pages[i + p] = page + p; + if (gfpflags_allow_blocking(gfp_mask)) cond_resched(); } atomic_long_add(area->nr_pages, &nr_vmalloc_pages); - if (map_kernel_range((unsigned long)area->addr, get_vm_area_size(area), - prot, pages) < 0) + if (vmap_pages_range(addr, addr + size, prot, pages, page_shift) < 0) goto fail; return area->addr; @@ -2701,22 +2760,45 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, pgprot_t prot, unsigned long vm_flags, int node, const void *caller) { - struct vm_struct *area; + struct vm_struct *area = NULL; void *addr; unsigned long real_size = size; + unsigned long real_align = align; + unsigned int shift = PAGE_SHIFT; size = PAGE_ALIGN(size); if (!size || (size >> PAGE_SHIFT) > totalram_pages()) goto fail; - area = __get_vm_area_node(real_size, align, VM_ALLOC | VM_UNINITIALIZED | + if (vmap_allow_huge && (pgprot_val(prot) == pgprot_val(PAGE_KERNEL))) { + unsigned long size_per_node; + + /* + * Try huge pages. Only try for PAGE_KERNEL allocations, + * others like modules don't yet expect huge pages in + * their allocations due to apply_to_page_range not + * supporting them. + */ + + size_per_node = size; + if (node == NUMA_NO_NODE) + size_per_node /= num_online_nodes(); + if (size_per_node >= PMD_SIZE) + shift = PMD_SHIFT; + } + +again: + align = max(real_align, 1UL << shift); + size = ALIGN(real_size, align); + + area = __get_vm_area_node(size, align, VM_ALLOC | VM_UNINITIALIZED | vm_flags, start, end, node, gfp_mask, caller); if (!area) goto fail; - addr = __vmalloc_area_node(area, gfp_mask, prot, node); + addr = __vmalloc_area_node(area, gfp_mask, prot, shift, node); if (!addr) - return NULL; + goto fail; /* * In this function, newly allocated vm_struct has VM_UNINITIALIZED @@ -2730,8 +2812,16 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, return addr; fail: - warn_alloc(gfp_mask, NULL, + if (shift > PAGE_SHIFT) { + shift = PAGE_SHIFT; + goto again; + } + + if (!area) { + /* Warn for area allocation, page allocations already warn */ + warn_alloc(gfp_mask, NULL, "vmalloc: allocation failure: %lu bytes", real_size); + } return NULL; }