From patchwork Fri Aug 21 15:12:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11729723 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4DBC0109B for ; Fri, 21 Aug 2020 15:12:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1AF22207DF for ; Fri, 21 Aug 2020 15:12:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="qvO/x3Lt" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1AF22207DF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3D8A08D0027; Fri, 21 Aug 2020 11:12:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 389698D0022; Fri, 21 Aug 2020 11:12:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 251748D0027; Fri, 21 Aug 2020 11:12:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0227.hostedemail.com [216.40.44.227]) by kanga.kvack.org (Postfix) with ESMTP id 0CDE98D0022 for ; Fri, 21 Aug 2020 11:12:36 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id BF0A0181AEF1A for ; Fri, 21 Aug 2020 15:12:35 +0000 (UTC) X-FDA: 77174917470.25.books77_0a011ef2703a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id 854D41804E3A1 for ; Fri, 21 Aug 2020 15:12:35 +0000 (UTC) X-Spam-Summary: 1,0,0,1a10914ff3f5df75,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1535:1543:1711:1730:1747:1777:1792:2393:2553:2559:2562:2693:3138:3139:3140:3141:3142:3308:3355:3865:3867:3868:3871:3872:3874:4117:4250:5007:6119:6261:6653:7514:7903:9036:9413:9592:10004:11026:11232:11473:11658:11914:12043:12219:12291:12295:12296:12297:12438:12517:12519:12555:12683:12895:13161:13229:13894:14181:14394:14687:14721:21080:21433:21444:21451:21627:21666:21990:30003:30054:30062:30070:30074:30090,0,RBL:209.85.215.194:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100;04y8tofyk67or4wxp9ha3zs8it8yuopwu1uepyi5f98ekzhbpbjw988g7yuwf7q.16pf537emigd1cnfmenaarp9iquwipafthxmuuio8amwyantu1zntonahicpsyh.1-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: books77_0a011ef2703a X-Filterd-Recvd-Size: 6461 Received: from mail-pg1-f194.google.com (mail-pg1-f194.google.com [209.85.215.194]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Fri, 21 Aug 2020 15:12:35 +0000 (UTC) Received: by mail-pg1-f194.google.com with SMTP id d19so1124344pgl.10 for ; Fri, 21 Aug 2020 08:12:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6FG53WANq96sGtoLSKrRzKN8EzzN/tCT8YeKUFryJxY=; b=qvO/x3LtuaWlzcRhfPRLNEAe+GmybLzPFd7TEDFfOiyJmw+WP/6BCuv7hyJuIxepte O8KDlYVX3ZQPdFw9zdlj57D8yw9Z6HeZXTn+Jm4YJA/kS7GU+MqF9x79PhCSvZcsEMVb XgfFKtwdZXMQ8ZoHAgH1Cl71OsLEN8hbyJrGrMDMb395P2issbN1XnfwEJo2JTlI/SXx DNXc1HnnxQz1u0FAKVB/p99YAbpu3gxrWalr7KOCWmuhD4u+uU4rpsLt+9ZiDXMoumHs L0Gm3rnnXdrnqMcXUEpiEzcVuknJM0Wov1F9YY7hzA01QFOErb5x3ckuVNqV9DoQ/+wn iqhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6FG53WANq96sGtoLSKrRzKN8EzzN/tCT8YeKUFryJxY=; b=R+Und1hndgAeSTYcWrHX6agBLrRx2MA83aQj2e94SD3R0Om43xPG0VjnLy0Si2eapT +K1jLMebqno/yMHbQ3YN36h4rDLwCSwaQodFpBKnYDRnaMfjahdpHqRs1BvwGHSGzKNO MSNHBl71tQffPHS6SpfsF7LZlYUHDjrooJ6DsSGDwI2E3TZnREJUhqi3KdI3jnsRAwUK yXgioizg1vNvOpuH9m2e1A4RIxfhEsacmB/Eod25T4XwyhWp01oghoWD7UX1r4jUJ0UE sXhbw7NfgmCktjHNDU2r/r9DNRFLHHHod9C4NYtBiGhcWxHCdqNrlup3K3mxmdy9Mt/X 2Fjw== X-Gm-Message-State: AOAM533RE0EaBBe7P/TBydIxf2BUQcu6dGjMz89g/fgcqYUwLpMMuzF6 ey2REHk+hPBVQhoX+2LI07SwhOH7vSI= X-Google-Smtp-Source: ABdhPJy2Q9rer+Gnc26wp0mgrvwzTqrrWkAV8Nt9IOzjPIJnUZWKJPJYiQTyxAK2BfQo5m1wdj6VAg== X-Received: by 2002:a63:2324:: with SMTP id j36mr2705340pgj.221.1598022753623; Fri, 21 Aug 2020 08:12:33 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id s8sm3126985pfc.122.2020.08.21.08.12.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 08:12:32 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy Subject: [PATCH v6 01/12] mm/vmalloc: fix vmalloc_to_page for huge vmap mappings Date: Sat, 22 Aug 2020 01:12:05 +1000 Message-Id: <20200821151216.1005117-2-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200821151216.1005117-1-npiggin@gmail.com> References: <20200821151216.1005117-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 854D41804E3A1 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vmalloc_to_page returns NULL for addresses mapped by larger pages[*]. Whether or not a vmap is huge depends on the architecture details, alignments, boot options, etc., which the caller can not be expected to know. Therefore HUGE_VMAP is a regression for vmalloc_to_page. This change teaches vmalloc_to_page about larger pages, and returns the struct page that corresponds to the offset within the large page. This makes the API agnostic to mapping implementation details. [*] As explained by commit 029c54b095995 ("mm/vmalloc.c: huge-vmap: fail gracefully on unexpected huge vmap mappings") Signed-off-by: Nicholas Piggin --- mm/vmalloc.c | 41 ++++++++++++++++++++++++++--------------- 1 file changed, 26 insertions(+), 15 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index b482d240f9a2..4e9b21adc73d 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -36,7 +36,7 @@ #include #include #include - +#include #include #include #include @@ -343,7 +343,9 @@ int is_vmalloc_or_module_addr(const void *x) } /* - * Walk a vmap address to the struct page it maps. + * Walk a vmap address to the struct page it maps. Huge vmap mappings will + * return the tail page that corresponds to the base page address, which + * matches small vmap mappings. */ struct page *vmalloc_to_page(const void *vmalloc_addr) { @@ -363,25 +365,33 @@ struct page *vmalloc_to_page(const void *vmalloc_addr) if (pgd_none(*pgd)) return NULL; + if (WARN_ON_ONCE(pgd_leaf(*pgd))) + return NULL; /* XXX: no allowance for huge pgd */ + if (WARN_ON_ONCE(pgd_bad(*pgd))) + return NULL; + p4d = p4d_offset(pgd, addr); if (p4d_none(*p4d)) return NULL; - pud = pud_offset(p4d, addr); + if (p4d_leaf(*p4d)) + return p4d_page(*p4d) + ((addr & ~P4D_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(p4d_bad(*p4d))) + return NULL; - /* - * Don't dereference bad PUD or PMD (below) entries. This will also - * identify huge mappings, which we may encounter on architectures - * that define CONFIG_HAVE_ARCH_HUGE_VMAP=y. Such regions will be - * identified as vmalloc addresses by is_vmalloc_addr(), but are - * not [unambiguously] associated with a struct page, so there is - * no correct value to return for them. - */ - WARN_ON_ONCE(pud_bad(*pud)); - if (pud_none(*pud) || pud_bad(*pud)) + pud = pud_offset(p4d, addr); + if (pud_none(*pud)) + return NULL; + if (pud_leaf(*pud)) + return pud_page(*pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(pud_bad(*pud))) return NULL; + pmd = pmd_offset(pud, addr); - WARN_ON_ONCE(pmd_bad(*pmd)); - if (pmd_none(*pmd) || pmd_bad(*pmd)) + if (pmd_none(*pmd)) + return NULL; + if (pmd_leaf(*pmd)) + return pmd_page(*pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(pmd_bad(*pmd))) return NULL; ptep = pte_offset_map(pmd, addr); @@ -389,6 +399,7 @@ struct page *vmalloc_to_page(const void *vmalloc_addr) if (pte_present(pte)) page = pte_page(pte); pte_unmap(ptep); + return page; } EXPORT_SYMBOL(vmalloc_to_page); From patchwork Fri Aug 21 15:12:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11729727 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 39603138C for ; Fri, 21 Aug 2020 15:12:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 06387207DE for ; Fri, 21 Aug 2020 15:12:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VBgyPhP1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 06387207DE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 315F28D0028; Fri, 21 Aug 2020 11:12:41 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 29EEE8D0022; Fri, 21 Aug 2020 11:12:41 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1671B8D0028; Fri, 21 Aug 2020 11:12:41 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0165.hostedemail.com [216.40.44.165]) by kanga.kvack.org (Postfix) with ESMTP id F09028D0022 for ; Fri, 21 Aug 2020 11:12:40 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id B5E501DFC for ; Fri, 21 Aug 2020 15:12:40 +0000 (UTC) X-FDA: 77174917680.25.lunch08_5c1433b2703a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id 6BF551804E3A8 for ; Fri, 21 Aug 2020 15:12:40 +0000 (UTC) X-Spam-Summary: 1,0,0,7e9a77b26b880339,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1542:1711:1730:1747:1777:1792:2393:2559:2562:2898:3138:3139:3140:3141:3142:3354:3870:3872:3876:4117:4321:5007:6119:6261:6653:7514:9413:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13255:13894:14096:14110:14181:14394:14687:14721:21080:21444:21451:21627:21666:21990:30054:30070,0,RBL:209.85.210.193:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100;04yfa4bdnc7r7u44iu5k1w1eiazriyc8ha73weaf89ri6wxbpwnor6cnz96zhuc.k5zb8tn96h53dqjjz8hkp78kt4b6bke8qzh9q9bohhy3mrobuwdfjzmmw19tdmr.o-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: lunch08_5c1433b2703a X-Filterd-Recvd-Size: 6180 Received: from mail-pf1-f193.google.com (mail-pf1-f193.google.com [209.85.210.193]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Fri, 21 Aug 2020 15:12:39 +0000 (UTC) Received: by mail-pf1-f193.google.com with SMTP id d188so1213990pfd.2 for ; Fri, 21 Aug 2020 08:12:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UY3nHiN8pgkQXggDgWgnBUQkjwXub5T9CsJzFX8rSpk=; b=VBgyPhP1+Sl/ttwJwVQxq7xmnvzliKnqHCmayEmnyxdcPFsr1octp/TFFnWKIEgvRt 294S6aSmYr6+teDwU/0oFVzm4wO8+Ym1+jXMpbnFTPBYO+2xzGlknVi7FAcl88wBr/Kn jSZkAndm5y1hxR9jtTL3QmJ/7YVnKPSdfMQyRZU97E1JDVGr3HV4RCBEpAOa/BCvA9ew PzNoQdId4cgj4lGg/uVYeDkwqO1XRV789Gm96zPIqWeFF2ms2ljOEpvTkeA0JKPRXdMQ yE1d22vcwtb7CcTa4Q/ntF/Qt3GIPcHPi4IxgVwof5EEpCPKIlnt9saulwv9E1MnoSxk SlGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UY3nHiN8pgkQXggDgWgnBUQkjwXub5T9CsJzFX8rSpk=; b=CAwoqGdxdxQsJvcpqGMuXcR1rRzv6R6l9CFfR0NMBTOGrrS5VHhZpmJ99jt2bd6tRw /SQWhKWw51ndNFn2hTL374d2tRkv0f5IhAa1Vm2C4CGB3a6rKejvN4u55LYDQUV1E50s FP4K7uW3/IsMRqTInS311+5HapgnZmItWikBe5lLQkzEZ45VbFSXiS4JwQV7RkyO0mn2 ggPOzr8vphhXX8/pasklj1xafXCGhcz+fypiWRLdGQScQUiTyIpVPAG1SxDUxwRag+XD DIMzdwGyYxqTAUIr1ERc3yJXV67hx8j8VzQLFplQYUS1J2xtTw2e5DajY4P1k0Z9lBgg qMbg== X-Gm-Message-State: AOAM532iILjEOXEOvcMyqGJh6O7SXRpdSH1DZkCu77dxM7EkbWWSqMMO 1Xawa56G1KER89kluZx1tqZi+wpJvO8= X-Google-Smtp-Source: ABdhPJyCXdSqBEAC0kFcsNvD5EqD53KjatKAQFKOQvMLJcOoi8k3fLMlmfj8qhPEMS+C8bwc9rHuyw== X-Received: by 2002:aa7:9556:: with SMTP id w22mr2775906pfq.245.1598022758914; Fri, 21 Aug 2020 08:12:38 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id s8sm3126985pfc.122.2020.08.21.08.12.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 08:12:38 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy Subject: [PATCH v6 02/12] mm: apply_to_pte_range warn and fail if a large pte is encountered Date: Sat, 22 Aug 2020 01:12:06 +1000 Message-Id: <20200821151216.1005117-3-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200821151216.1005117-1-npiggin@gmail.com> References: <20200821151216.1005117-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 6BF551804E3A8 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently this might mistake a large pte for bad, or treat it as a page table, resulting in a crash or corruption. Signed-off-by: Nicholas Piggin --- mm/memory.c | 60 +++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 44 insertions(+), 16 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 602f4283122f..3a39a47920e2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2262,13 +2262,20 @@ static int apply_to_pmd_range(struct mm_struct *mm, pud_t *pud, } do { next = pmd_addr_end(addr, end); - if (create || !pmd_none_or_clear_bad(pmd)) { - err = apply_to_pte_range(mm, pmd, addr, next, fn, data, - create); - if (err) - break; + if (pmd_none(*pmd) && !create) + continue; + if (WARN_ON_ONCE(pmd_leaf(*pmd))) + return -EINVAL; + if (WARN_ON_ONCE(pmd_bad(*pmd))) { + if (!create) + continue; + pmd_clear_bad(pmd); } + err = apply_to_pte_range(mm, pmd, addr, next, fn, data, create); + if (err) + break; } while (pmd++, addr = next, addr != end); + return err; } @@ -2289,13 +2296,20 @@ static int apply_to_pud_range(struct mm_struct *mm, p4d_t *p4d, } do { next = pud_addr_end(addr, end); - if (create || !pud_none_or_clear_bad(pud)) { - err = apply_to_pmd_range(mm, pud, addr, next, fn, data, - create); - if (err) - break; + if (pud_none(*pud) && !create) + continue; + if (WARN_ON_ONCE(pud_leaf(*pud))) + return -EINVAL; + if (WARN_ON_ONCE(pud_bad(*pud))) { + if (!create) + continue; + pud_clear_bad(pud); } + err = apply_to_pmd_range(mm, pud, addr, next, fn, data, create); + if (err) + break; } while (pud++, addr = next, addr != end); + return err; } @@ -2316,13 +2330,20 @@ static int apply_to_p4d_range(struct mm_struct *mm, pgd_t *pgd, } do { next = p4d_addr_end(addr, end); - if (create || !p4d_none_or_clear_bad(p4d)) { - err = apply_to_pud_range(mm, p4d, addr, next, fn, data, - create); - if (err) - break; + if (p4d_none(*p4d) && !create) + continue; + if (WARN_ON_ONCE(p4d_leaf(*p4d))) + return -EINVAL; + if (WARN_ON_ONCE(p4d_bad(*p4d))) { + if (!create) + continue; + p4d_clear_bad(p4d); } + err = apply_to_pud_range(mm, p4d, addr, next, fn, data, create); + if (err) + break; } while (p4d++, addr = next, addr != end); + return err; } @@ -2341,8 +2362,15 @@ static int __apply_to_page_range(struct mm_struct *mm, unsigned long addr, pgd = pgd_offset(mm, addr); do { next = pgd_addr_end(addr, end); - if (!create && pgd_none_or_clear_bad(pgd)) + if (pgd_none(*pgd) && !create) continue; + if (WARN_ON_ONCE(pgd_leaf(*pgd))) + return -EINVAL; + if (WARN_ON_ONCE(pgd_bad(*pgd))) { + if (!create) + continue; + pgd_clear_bad(pgd); + } err = apply_to_p4d_range(mm, pgd, addr, next, fn, data, create); if (err) break; From patchwork Fri Aug 21 15:12:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11729729 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CAA9B109B for ; Fri, 21 Aug 2020 15:12:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 94469207DF for ; Fri, 21 Aug 2020 15:12:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="GSNKWe4s" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 94469207DF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B2C438D002A; Fri, 21 Aug 2020 11:12:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AB5AC8D0022; Fri, 21 Aug 2020 11:12:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 97CAA8D002A; Fri, 21 Aug 2020 11:12:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0054.hostedemail.com [216.40.44.54]) by kanga.kvack.org (Postfix) with ESMTP id 7B9918D0022 for ; Fri, 21 Aug 2020 11:12:46 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 4CACA8245571 for ; Fri, 21 Aug 2020 15:12:46 +0000 (UTC) X-FDA: 77174917932.14.aunt43_0a151662703a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id 0E24418229837 for ; Fri, 21 Aug 2020 15:12:46 +0000 (UTC) X-Spam-Summary: 1,0,0,15e39caae1f1c729,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3354:3865:3867:3872:4117:5007:6261:6653:7514:8603:9413:10004:11026:11473:11658:11914:12043:12297:12438:12517:12519:12555:12895:13161:13229:13894:14096:14181:14394:14687:14721:21080:21444:21451:21627:21666:21990:30003:30054,0,RBL:209.85.210.196:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100;04y8sgwpxdq1ytc4ntyphsmg11hheychuuix8ukqnpyogge9qkahmqmyfz3kn5o.ay1wftg5tsr63ccox4ah67f3znc6zmf4q3cn7bkbgzn15pm77mmcrdt17h8813u.c-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:27,LUA_SUMMARY:none X-HE-Tag: aunt43_0a151662703a X-Filterd-Recvd-Size: 6237 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) by imf20.hostedemail.com (Postfix) with ESMTP for ; Fri, 21 Aug 2020 15:12:45 +0000 (UTC) Received: by mail-pf1-f196.google.com with SMTP id u128so1205969pfb.6 for ; Fri, 21 Aug 2020 08:12:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sv/9eriXs4YMtuUqEg5bncZ0rlcJBEbL+hLm5+QBb40=; b=GSNKWe4sKvin/No7SBzCc6TLryWUHXwgfTbEx2FE6v6Qty5pCQrFFPx5gnqy/TMrZ0 FhhPfnO1bjKpB4mpbA2Zbxb9SnDJDwQaEqzIh/veS4HnSfjR3rbv9N9XL2owb+is8FBp U8tWqJVuLXxEfds8/KLwA4IOscmZGRMKdB06OQO7XdHheYKQrpMNaMFkXftauA9SdKrT lW1CdeEv5KN8HhDLkiTqMOuJxGW/FRMC5XelBXGxC4PtVgAegGzDna0egFgASOCs/cMD 9ZO6lXh6LIZdusjsisbkJtFSMIfznqn5HCpt0SnfiGnceZS3t5zZrhGdIB1XZ3XVbxe3 86KQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sv/9eriXs4YMtuUqEg5bncZ0rlcJBEbL+hLm5+QBb40=; b=jM0SYdLq68epmeDl3aE1O82ElJVfTDTnWtzD3oCVT1kEzuSrBwlht0zWNcKeyGU0EG NYxjWLrhLSQWxAU8kzXjioVEW3sDqgePXL+n+1W1Udq9g8ExqLsVQoKQ60YLQew5Rfwe ff0f3db+vqlYSBUqpahAg7YCFLeRDDn0u0aZWnjVnzqeKtsRv7HgPlBTXVSF0dfWlni7 GA8ha0wC3bjckTlYFotDIdsB82/yqR+XGvCjBbsbqR1wmstRGOeDBkGjKY4s7E6JkB3d UshlQ7WhGeX+DY7ttEkJjXj9BOXQHD7jmeDB3I+bEDBg4gtyu3l7eHryTRpdflcf18BZ pYMA== X-Gm-Message-State: AOAM533Xf2fXtMOafjzrVfuQ9bhgbaLJAPkhpZUlboiB7pLrYrjaniaf kyA2fL1Ju47XNjh9RFhVEnKvpA0ZwAQ= X-Google-Smtp-Source: ABdhPJyVcrLzuP+hh//+EDBR3EBQ4EbV3sHIVDVex2GP8JpaDxXcsB6PkbGOFarFI53/CgzL2OUHFA== X-Received: by 2002:a63:af51:: with SMTP id s17mr2699253pgo.286.1598022764313; Fri, 21 Aug 2020 08:12:44 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id s8sm3126985pfc.122.2020.08.21.08.12.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 08:12:43 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy Subject: [PATCH v6 03/12] mm/vmalloc: rename vmap_*_range vmap_pages_*_range Date: Sat, 22 Aug 2020 01:12:07 +1000 Message-Id: <20200821151216.1005117-4-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200821151216.1005117-1-npiggin@gmail.com> References: <20200821151216.1005117-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0E24418229837 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The vmalloc mapper operates on a struct page * array rather than a linear physical address, re-name it to make this distinction clear. Signed-off-by: Nicholas Piggin --- mm/vmalloc.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 4e9b21adc73d..45cd80ec7eeb 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -189,7 +189,7 @@ void unmap_kernel_range_noflush(unsigned long start, unsigned long size) arch_sync_kernel_mappings(start, end); } -static int vmap_pte_range(pmd_t *pmd, unsigned long addr, +static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { @@ -217,7 +217,7 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, return 0; } -static int vmap_pmd_range(pud_t *pud, unsigned long addr, +static int vmap_pages_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { @@ -229,13 +229,13 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr, return -ENOMEM; do { next = pmd_addr_end(addr, end); - if (vmap_pte_range(pmd, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pte_range(pmd, addr, next, prot, pages, nr, mask)) return -ENOMEM; } while (pmd++, addr = next, addr != end); return 0; } -static int vmap_pud_range(p4d_t *p4d, unsigned long addr, +static int vmap_pages_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { @@ -247,13 +247,13 @@ static int vmap_pud_range(p4d_t *p4d, unsigned long addr, return -ENOMEM; do { next = pud_addr_end(addr, end); - if (vmap_pmd_range(pud, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pmd_range(pud, addr, next, prot, pages, nr, mask)) return -ENOMEM; } while (pud++, addr = next, addr != end); return 0; } -static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, +static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { @@ -265,7 +265,7 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, return -ENOMEM; do { next = p4d_addr_end(addr, end); - if (vmap_pud_range(p4d, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pud_range(p4d, addr, next, prot, pages, nr, mask)) return -ENOMEM; } while (p4d++, addr = next, addr != end); return 0; @@ -306,7 +306,7 @@ int map_kernel_range_noflush(unsigned long addr, unsigned long size, next = pgd_addr_end(addr, end); if (pgd_bad(*pgd)) mask |= PGTBL_PGD_MODIFIED; - err = vmap_p4d_range(pgd, addr, next, prot, pages, &nr, &mask); + err = vmap_pages_p4d_range(pgd, addr, next, prot, pages, &nr, &mask); if (err) return err; } while (pgd++, addr = next, addr != end); From patchwork Fri Aug 21 15:12:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11729731 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7A9A4109B for ; Fri, 21 Aug 2020 15:12:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3A239207DE for ; Fri, 21 Aug 2020 15:12:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bIpiMwmt" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3A239207DE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 49C0B8D002B; Fri, 21 Aug 2020 11:12:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 425178D0022; Fri, 21 Aug 2020 11:12:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 314088D002B; Fri, 21 Aug 2020 11:12:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0242.hostedemail.com [216.40.44.242]) by kanga.kvack.org (Postfix) with ESMTP id 14BF78D0022 for ; Fri, 21 Aug 2020 11:12:52 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id A5358180AD80F for ; Fri, 21 Aug 2020 15:12:51 +0000 (UTC) X-FDA: 77174918142.16.slave15_180e66a2703a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin16.hostedemail.com (Postfix) with ESMTP id 7357A100E690C for ; Fri, 21 Aug 2020 15:12:51 +0000 (UTC) X-Spam-Summary: 1,0,0,9b9b87375212bcb2,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:2:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1730:1747:1777:1792:1981:2194:2199:2393:2559:2562:2693:2898:3138:3139:3140:3141:3142:3870:3872:3876:4049:4120:4605:5007:6119:6261:6653:7514:7875:8603:9413:10004:11026:11473:11658:11914:12043:12291:12297:12438:12517:12519:12555:12683:12895:13894:14096:14110:14394:14687:21080:21444:21451:21611:21627:21666:21990:30054:30070,0,RBL:209.85.216.67:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100;04yr1tmpfq8t4rm9x6th8hmhahyn1opbn15mdcfei4rxy1xzjbaqk81xn86w7mc.bdhg4d7ts474bbqebnahn3yio4op7jorn1yebef5b3cepqwaadmz3cmmcmhgr9g.6-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: slave15_180e66a2703a X-Filterd-Recvd-Size: 9332 Received: from mail-pj1-f67.google.com (mail-pj1-f67.google.com [209.85.216.67]) by imf06.hostedemail.com (Postfix) with ESMTP for ; Fri, 21 Aug 2020 15:12:50 +0000 (UTC) Received: by mail-pj1-f67.google.com with SMTP id ep8so945026pjb.3 for ; Fri, 21 Aug 2020 08:12:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0Y/hT12bvV3dyctfA7PHWDL72BJBoDG5OIVFWS0o/KA=; b=bIpiMwmt/XIM38Vr6L1TlzZOPMgWYxGOwBAbhA3oYMBCFWPqTUMnGA6po81aipEVWy LCzFlAAclsAz7y8oeClqiLMycBmdgJrtZNCffPgv0t18TWGbtbWL4O/w8TBHO5Uic51y CcVw4H7gUfF2TgiwlgtpnG5NeVps3SCC6n7JJGmLyAxqQ74b8CpYowythc5hnQtKhhGh d65SUZmCu5xQaap6bORtBpaRA3BvXze/2Kxi4MaMPEEhua3o3AgpmGu+2wgTLHvTr6jv yofXOJT1FjbXSBYVmn+aGcHvCNOByG4AaDH9kcJ4M7oOQdrR5dK+pBxT22iS6bFYv6SY poCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0Y/hT12bvV3dyctfA7PHWDL72BJBoDG5OIVFWS0o/KA=; b=sxzxeE59TTLAuRoI47+YatY751SP2nDPfnkXu9kHXAe5k8TZoeMDc8GQv96gPSYVpa HqhXaDees0efiuvVfDBuwkBJdQhvGOcD8kFsQB0C/OZhQeMncKGtI5YvDHNmATamE3NU cmvkeoRg8lEKL0yKpQT2PFpyqamDtLvlhvsquLx3eLL86mEC+5Zo8wCbEf63pTLmasxG n5Ou9ITnwaqjvvfvEPf7Q947Iqj3Xdx4Xnz2nm0aVcrjf2+J43xxhCs+KZKR3+/HpoxW UtXv/22jQZv5n1n6EStIpPTYAkSSR3RAAhPdjlmT61kDaj655/4ByTkkE65NuKK2fDow BEAA== X-Gm-Message-State: AOAM531P59AFCMsFpfUefQAkyxqFIqIEp2NwltX3ARy8LKb8mblU/O2g sm3uASxdPSTEGs8Bx3+c6l4BcYDPMBk= X-Google-Smtp-Source: ABdhPJwM5fcpqmto8P3F5fSFr0rvGDBKsbzEu5uwFG787PaO4Qmy+4/1b58cB+92h3UZOd8VH4aIog== X-Received: by 2002:a17:902:8609:: with SMTP id f9mr2765135plo.324.1598022769517; Fri, 21 Aug 2020 08:12:49 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id s8sm3126985pfc.122.2020.08.21.08.12.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 08:12:48 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy Subject: [PATCH v6 04/12] mm/ioremap: rename ioremap_*_range to vmap_*_range Date: Sat, 22 Aug 2020 01:12:08 +1000 Message-Id: <20200821151216.1005117-5-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200821151216.1005117-1-npiggin@gmail.com> References: <20200821151216.1005117-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 7357A100E690C X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This will be used as a generic kernel virtual mapping function, so re-name it in preparation. Signed-off-by: Nicholas Piggin --- mm/ioremap.c | 64 +++++++++++++++++++++++++++------------------------- 1 file changed, 33 insertions(+), 31 deletions(-) diff --git a/mm/ioremap.c b/mm/ioremap.c index 5fa1ab41d152..3f4d36f9745a 100644 --- a/mm/ioremap.c +++ b/mm/ioremap.c @@ -61,9 +61,9 @@ static inline int ioremap_pud_enabled(void) { return 0; } static inline int ioremap_pmd_enabled(void) { return 0; } #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ -static int ioremap_pte_range(pmd_t *pmd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) { pte_t *pte; u64 pfn; @@ -81,9 +81,8 @@ static int ioremap_pte_range(pmd_t *pmd, unsigned long addr, return 0; } -static int ioremap_try_huge_pmd(pmd_t *pmd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) +static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { if (!ioremap_pmd_enabled()) return 0; @@ -103,9 +102,9 @@ static int ioremap_try_huge_pmd(pmd_t *pmd, unsigned long addr, return pmd_set_huge(pmd, phys_addr, prot); } -static inline int ioremap_pmd_range(pud_t *pud, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) { pmd_t *pmd; unsigned long next; @@ -116,20 +115,19 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long addr, do { next = pmd_addr_end(addr, end); - if (ioremap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) { *mask |= PGTBL_PMD_MODIFIED; continue; } - if (ioremap_pte_range(pmd, addr, next, phys_addr, prot, mask)) + if (vmap_pte_range(pmd, addr, next, phys_addr, prot, mask)) return -ENOMEM; } while (pmd++, phys_addr += (next - addr), addr = next, addr != end); return 0; } -static int ioremap_try_huge_pud(pud_t *pud, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) +static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { if (!ioremap_pud_enabled()) return 0; @@ -149,9 +147,9 @@ static int ioremap_try_huge_pud(pud_t *pud, unsigned long addr, return pud_set_huge(pud, phys_addr, prot); } -static inline int ioremap_pud_range(p4d_t *p4d, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) { pud_t *pud; unsigned long next; @@ -162,20 +160,19 @@ static inline int ioremap_pud_range(p4d_t *p4d, unsigned long addr, do { next = pud_addr_end(addr, end); - if (ioremap_try_huge_pud(pud, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot)) { *mask |= PGTBL_PUD_MODIFIED; continue; } - if (ioremap_pmd_range(pud, addr, next, phys_addr, prot, mask)) + if (vmap_pmd_range(pud, addr, next, phys_addr, prot, mask)) return -ENOMEM; } while (pud++, phys_addr += (next - addr), addr = next, addr != end); return 0; } -static int ioremap_try_huge_p4d(p4d_t *p4d, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) +static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { if (!ioremap_p4d_enabled()) return 0; @@ -195,9 +192,9 @@ static int ioremap_try_huge_p4d(p4d_t *p4d, unsigned long addr, return p4d_set_huge(p4d, phys_addr, prot); } -static inline int ioremap_p4d_range(pgd_t *pgd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) { p4d_t *p4d; unsigned long next; @@ -208,19 +205,19 @@ static inline int ioremap_p4d_range(pgd_t *pgd, unsigned long addr, do { next = p4d_addr_end(addr, end); - if (ioremap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) { + if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) { *mask |= PGTBL_P4D_MODIFIED; continue; } - if (ioremap_pud_range(p4d, addr, next, phys_addr, prot, mask)) + if (vmap_pud_range(p4d, addr, next, phys_addr, prot, mask)) return -ENOMEM; } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); return 0; } -int ioremap_page_range(unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot) +static int vmap_range(unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { pgd_t *pgd; unsigned long start; @@ -235,8 +232,7 @@ int ioremap_page_range(unsigned long addr, pgd = pgd_offset_k(addr); do { next = pgd_addr_end(addr, end); - err = ioremap_p4d_range(pgd, addr, next, phys_addr, prot, - &mask); + err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, &mask); if (err) break; } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); @@ -249,6 +245,12 @@ int ioremap_page_range(unsigned long addr, return err; } +int ioremap_page_range(unsigned long addr, + unsigned long end, phys_addr_t phys_addr, pgprot_t prot) +{ + return vmap_range(addr, end, phys_addr, prot); +} + #ifdef CONFIG_GENERIC_IOREMAP void __iomem *ioremap_prot(phys_addr_t addr, size_t size, unsigned long prot) { From patchwork Fri Aug 21 15:12:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11729735 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AAE16109B for ; Fri, 21 Aug 2020 15:13:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5E323207DE for ; Fri, 21 Aug 2020 15:13:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mGkVJOBB" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5E323207DE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 314878D002D; Fri, 21 Aug 2020 11:13:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2A49D8D0022; Fri, 21 Aug 2020 11:13:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F20E8D002D; Fri, 21 Aug 2020 11:13:00 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0114.hostedemail.com [216.40.44.114]) by kanga.kvack.org (Postfix) with ESMTP id DBF868D0022 for ; Fri, 21 Aug 2020 11:13:00 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 6772A362A for ; Fri, 21 Aug 2020 15:13:00 +0000 (UTC) X-FDA: 77174918520.06.rock49_05006ba2703a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id 2A8B81003A86F for ; Fri, 21 Aug 2020 15:13:00 +0000 (UTC) X-Spam-Summary: 1,0,0,9b234b4ca0636507,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:4:41:69:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1605:1730:1747:1777:1792:1801:2194:2196:2199:2200:2393:2559:2562:2639:2693:2898:3138:3139:3140:3141:3142:3167:3865:3866:3867:3868:3870:3871:4321:4385:4605:5007:6119:6261:6653:6742:7514:7875:8603:8660:9413:9592:10004:11026:11473:11657:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:12986:13148:13230:13846:13894:14096:14394:14687:21080:21444:21451:21627:21666:21939:21966:21990:30012:30054:30070,0,RBL:209.85.210.193:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100;04yfme5j66aaw3i3q7ccxx1r7fuwdoc6187wn9pemjmja8a7q8onr9rmna7eqm6.fuwez7iqj1opey5oc9bds8rtndbdbpbbuij84u4fptzga3yz9yeujagqd91etsf.w-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: rock49_05006ba2703a X-Filterd-Recvd-Size: 17279 Received: from mail-pf1-f193.google.com (mail-pf1-f193.google.com [209.85.210.193]) by imf25.hostedemail.com (Postfix) with ESMTP for ; Fri, 21 Aug 2020 15:12:59 +0000 (UTC) Received: by mail-pf1-f193.google.com with SMTP id f193so1193211pfa.12 for ; Fri, 21 Aug 2020 08:12:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FfMdahxzyz8aJR5iMN6j/1TYRMKCpmX1Y7qmlhtnFyU=; b=mGkVJOBBkKsBWYb2KJ9FGxeojGVGiTEn2ivs8qZsClKR7hWVZKAKr9qRixwdUQb3jM 7e6PIpLSRq4KzITiMeTTiXVgK+UfrXJh+kK31sbwfxBfvM8uCjgLugUVnh/XmGdwo+YX qziKZcn/3Lasfc6MDixbl2YGWREZGLw4UM5BtiiSZf5pJgXk5MrFoc+EZHLJS5Z54iDS og4ggV7nKklYwro0z9bOfF/Y174p3ecZqpCK9gSEWklK9NVRrP/8Fv4MxTPUwMpzoRAJ rjh40WYtByEModlfUgRJMxafMxG5CEaupHiwblA3M3WZOSp5uvP0xogE/RhyKjG6DtNr 8Rgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FfMdahxzyz8aJR5iMN6j/1TYRMKCpmX1Y7qmlhtnFyU=; b=T2iuw8lMYY9RtZKkyYdfKSEHYeSkXEoSsQRBvIjvEMGSW7YGjfCcy2sefApzqE/5QU 9xZ9WFonyjkT3AA71hI8dwjzCdK6ga2ewXO8jVXS7l4VLceotpxtDN8x4STdADjI96E/ wPzrawc5Mg/t0YGMeS1pWgi077cIUSdxTjDY9nk7UJD+GG5YUj29NX9inTb/IxXD8xgd exzd9xoEbbEZAP/MsIsBM7/mal3kwmL6R4TEeknXNlVzktMPXPleufLu5MHpseK0rlk5 X3g8VU+2E/oszQhKO57dvgwiVJFx4tYQ5iM6OtHIfVlDHk12PXs6gsjKK/u100zQ4TEz n+Xg== X-Gm-Message-State: AOAM531+1iNUCuqOFbnojd7iFBLb2A6KBmWLPG6PuCIshy/Ed/M7xDGX Vd4WwrcXDdxnce4TmBx/3H9jH057uWY= X-Google-Smtp-Source: ABdhPJz+aTWKALvPTvcEoD1iuNfeTRAHqWXld/X0KGJyfEeOr5wexMmXLqxD/dzmWGQXtqSXvAW4Ag== X-Received: by 2002:a63:354:: with SMTP id 81mr2406362pgd.300.1598022777711; Fri, 21 Aug 2020 08:12:57 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id s8sm3126985pfc.122.2020.08.21.08.12.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 08:12:56 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" Subject: [PATCH v6 05/12] mm: HUGE_VMAP arch support cleanup Date: Sat, 22 Aug 2020 01:12:09 +1000 Message-Id: <20200821151216.1005117-6-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200821151216.1005117-1-npiggin@gmail.com> References: <20200821151216.1005117-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2A8B81003A86F X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This changes the awkward approach where architectures provide init functions to determine which levels they can provide large mappings for, to one where the arch is queried for each call. This removes code and indirection, and allows constant-folding of dead code for unsupported levels. This also adds a prot argument to the arch query. This is unused currently but could help with some architectures (e.g., some powerpc processors can't map uncacheable memory with large pages). Cc: linuxppc-dev@lists.ozlabs.org Cc: Catalin Marinas Cc: Will Deacon Cc: linux-arm-kernel@lists.infradead.org Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: x86@kernel.org Cc: "H. Peter Anvin" Signed-off-by: Nicholas Piggin --- arch/arm64/include/asm/vmalloc.h | 8 +++ arch/arm64/mm/mmu.c | 10 +-- arch/powerpc/include/asm/vmalloc.h | 8 +++ arch/powerpc/mm/book3s64/radix_pgtable.c | 8 +-- arch/x86/include/asm/vmalloc.h | 7 ++ arch/x86/mm/ioremap.c | 10 +-- include/linux/io.h | 9 --- include/linux/vmalloc.h | 6 ++ init/main.c | 1 - mm/ioremap.c | 88 +++++++++--------------- 10 files changed, 77 insertions(+), 78 deletions(-) diff --git a/arch/arm64/include/asm/vmalloc.h b/arch/arm64/include/asm/vmalloc.h index 2ca708ab9b20..597b40405319 100644 --- a/arch/arm64/include/asm/vmalloc.h +++ b/arch/arm64/include/asm/vmalloc.h @@ -1,4 +1,12 @@ #ifndef _ASM_ARM64_VMALLOC_H #define _ASM_ARM64_VMALLOC_H +#include + +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +bool arch_vmap_p4d_supported(pgprot_t prot); +bool arch_vmap_pud_supported(pgprot_t prot); +bool arch_vmap_pmd_supported(pgprot_t prot); +#endif + #endif /* _ASM_ARM64_VMALLOC_H */ diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 75df62fea1b6..9df7e0058c78 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1304,12 +1304,12 @@ void *__init fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot) return dt_virt; } -int __init arch_ioremap_p4d_supported(void) +bool arch_vmap_p4d_supported(pgprot_t prot) { - return 0; + return false; } -int __init arch_ioremap_pud_supported(void) +bool arch_vmap_pud_supported(pgprot_t prot); { /* * Only 4k granule supports level 1 block mappings. @@ -1319,9 +1319,9 @@ int __init arch_ioremap_pud_supported(void) !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); } -int __init arch_ioremap_pmd_supported(void) +bool arch_vmap_pmd_supported(pgprot_t prot) { - /* See arch_ioremap_pud_supported() */ + /* See arch_vmap_pud_supported() */ return !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); } diff --git a/arch/powerpc/include/asm/vmalloc.h b/arch/powerpc/include/asm/vmalloc.h index b992dfaaa161..105abb73f075 100644 --- a/arch/powerpc/include/asm/vmalloc.h +++ b/arch/powerpc/include/asm/vmalloc.h @@ -1,4 +1,12 @@ #ifndef _ASM_POWERPC_VMALLOC_H #define _ASM_POWERPC_VMALLOC_H +#include + +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +bool arch_vmap_p4d_supported(pgprot_t prot); +bool arch_vmap_pud_supported(pgprot_t prot); +bool arch_vmap_pmd_supported(pgprot_t prot); +#endif + #endif /* _ASM_POWERPC_VMALLOC_H */ diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 28c784976bed..eca83a50bf2e 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1134,13 +1134,13 @@ void radix__ptep_modify_prot_commit(struct vm_area_struct *vma, set_pte_at(mm, addr, ptep, pte); } -int __init arch_ioremap_pud_supported(void) +bool arch_vmap_pud_supported(pgprot_t prot) { /* HPT does not cope with large pages in the vmalloc area */ return radix_enabled(); } -int __init arch_ioremap_pmd_supported(void) +bool arch_vmap_pmd_supported(pgprot_t prot) { return radix_enabled(); } @@ -1234,7 +1234,7 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) return 1; } -int __init arch_ioremap_p4d_supported(void) +bool arch_vmap_p4d_supported(pgprot_t prot) { - return 0; + return false; } diff --git a/arch/x86/include/asm/vmalloc.h b/arch/x86/include/asm/vmalloc.h index 29837740b520..094ea2b565f3 100644 --- a/arch/x86/include/asm/vmalloc.h +++ b/arch/x86/include/asm/vmalloc.h @@ -1,6 +1,13 @@ #ifndef _ASM_X86_VMALLOC_H #define _ASM_X86_VMALLOC_H +#include #include +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +bool arch_vmap_p4d_supported(pgprot_t prot); +bool arch_vmap_pud_supported(pgprot_t prot); +bool arch_vmap_pmd_supported(pgprot_t prot); +#endif + #endif /* _ASM_X86_VMALLOC_H */ diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c index 84d85dbd1dad..159bfca757b9 100644 --- a/arch/x86/mm/ioremap.c +++ b/arch/x86/mm/ioremap.c @@ -481,21 +481,21 @@ void iounmap(volatile void __iomem *addr) } EXPORT_SYMBOL(iounmap); -int __init arch_ioremap_p4d_supported(void) +bool arch_vmap_p4d_supported(pgprot_t prot) { - return 0; + return false; } -int __init arch_ioremap_pud_supported(void) +bool arch_vmap_pud_supported(pgprot_t prot) { #ifdef CONFIG_X86_64 return boot_cpu_has(X86_FEATURE_GBPAGES); #else - return 0; + return false; #endif } -int __init arch_ioremap_pmd_supported(void) +bool arch_vmap_pmd_supported(pgprot_t prot) { return boot_cpu_has(X86_FEATURE_PSE); } diff --git a/include/linux/io.h b/include/linux/io.h index 8394c56babc2..f1effd4d7a3c 100644 --- a/include/linux/io.h +++ b/include/linux/io.h @@ -31,15 +31,6 @@ static inline int ioremap_page_range(unsigned long addr, unsigned long end, } #endif -#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -void __init ioremap_huge_init(void); -int arch_ioremap_p4d_supported(void); -int arch_ioremap_pud_supported(void); -int arch_ioremap_pmd_supported(void); -#else -static inline void ioremap_huge_init(void) { } -#endif - /* * Managed iomap interface */ diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 0221f852a7e1..3f6bba4cc9bc 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -84,6 +84,12 @@ struct vmap_area { }; }; +#ifndef CONFIG_HAVE_ARCH_HUGE_VMAP +static inline bool arch_vmap_p4d_supported(pgprot_t prot) { return false; } +static inline bool arch_vmap_pud_supported(pgprot_t prot) { return false; } +static inline bool arch_vmap_pmd_supported(pgprot_t prot) { return false; } +#endif + /* * Highlevel APIs for driver use */ diff --git a/init/main.c b/init/main.c index ae78fb68d231..1c89aa127b8f 100644 --- a/init/main.c +++ b/init/main.c @@ -820,7 +820,6 @@ static void __init mm_init(void) pgtable_init(); debug_objects_mem_init(); vmalloc_init(); - ioremap_huge_init(); /* Should be run before the first non-init thread is created */ init_espfix_bsp(); /* Should be run after espfix64 is set up. */ diff --git a/mm/ioremap.c b/mm/ioremap.c index 3f4d36f9745a..c67f91164401 100644 --- a/mm/ioremap.c +++ b/mm/ioremap.c @@ -16,49 +16,16 @@ #include "pgalloc-track.h" #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -static int __read_mostly ioremap_p4d_capable; -static int __read_mostly ioremap_pud_capable; -static int __read_mostly ioremap_pmd_capable; -static int __read_mostly ioremap_huge_disabled; +static bool __ro_after_init iomap_max_page_shift = PAGE_SHIFT; static int __init set_nohugeiomap(char *str) { - ioremap_huge_disabled = 1; + iomap_max_page_shift = P4D_SHIFT; return 0; } early_param("nohugeiomap", set_nohugeiomap); - -void __init ioremap_huge_init(void) -{ - if (!ioremap_huge_disabled) { - if (arch_ioremap_p4d_supported()) - ioremap_p4d_capable = 1; - if (arch_ioremap_pud_supported()) - ioremap_pud_capable = 1; - if (arch_ioremap_pmd_supported()) - ioremap_pmd_capable = 1; - } -} - -static inline int ioremap_p4d_enabled(void) -{ - return ioremap_p4d_capable; -} - -static inline int ioremap_pud_enabled(void) -{ - return ioremap_pud_capable; -} - -static inline int ioremap_pmd_enabled(void) -{ - return ioremap_pmd_capable; -} - -#else /* !CONFIG_HAVE_ARCH_HUGE_VMAP */ -static inline int ioremap_p4d_enabled(void) { return 0; } -static inline int ioremap_pud_enabled(void) { return 0; } -static inline int ioremap_pmd_enabled(void) { return 0; } +#else /* CONFIG_HAVE_ARCH_HUGE_VMAP */ +static const bool iomap_max_page_shift = PAGE_SHIFT; #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, @@ -82,9 +49,13 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, } static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) { - if (!ioremap_pmd_enabled()) + if (max_page_shift < PMD_SHIFT) + return 0; + + if (!arch_vmap_pmd_supported(prot)) return 0; if ((end - addr) != PMD_SIZE) @@ -104,7 +75,7 @@ static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) + unsigned int max_page_shift, pgtbl_mod_mask *mask) { pmd_t *pmd; unsigned long next; @@ -115,7 +86,7 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, do { next = pmd_addr_end(addr, end); - if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, max_page_shift)) { *mask |= PGTBL_PMD_MODIFIED; continue; } @@ -127,9 +98,13 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, } static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) { - if (!ioremap_pud_enabled()) + if (max_page_shift < PUD_SHIFT) + return 0; + + if (!arch_vmap_pud_supported(prot)) return 0; if ((end - addr) != PUD_SIZE) @@ -149,7 +124,7 @@ static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) + unsigned int max_page_shift, pgtbl_mod_mask *mask) { pud_t *pud; unsigned long next; @@ -160,21 +135,25 @@ static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, do { next = pud_addr_end(addr, end); - if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot, max_page_shift)) { *mask |= PGTBL_PUD_MODIFIED; continue; } - if (vmap_pmd_range(pud, addr, next, phys_addr, prot, mask)) + if (vmap_pmd_range(pud, addr, next, phys_addr, prot, max_page_shift, mask)) return -ENOMEM; } while (pud++, phys_addr += (next - addr), addr = next, addr != end); return 0; } static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) { - if (!ioremap_p4d_enabled()) + if (max_page_shift < P4D_SHIFT) + return 0; + + if (!arch_vmap_p4d_supported(prot)) return 0; if ((end - addr) != P4D_SIZE) @@ -194,7 +173,7 @@ static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) + unsigned int max_page_shift, pgtbl_mod_mask *mask) { p4d_t *p4d; unsigned long next; @@ -205,19 +184,20 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, do { next = p4d_addr_end(addr, end); - if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) { + if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot, max_page_shift)) { *mask |= PGTBL_P4D_MODIFIED; continue; } - if (vmap_pud_range(p4d, addr, next, phys_addr, prot, mask)) + if (vmap_pud_range(p4d, addr, next, phys_addr, prot, max_page_shift, mask)) return -ENOMEM; } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); return 0; } static int vmap_range(unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) { pgd_t *pgd; unsigned long start; @@ -232,7 +212,7 @@ static int vmap_range(unsigned long addr, unsigned long end, pgd = pgd_offset_k(addr); do { next = pgd_addr_end(addr, end); - err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, &mask); + err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, max_page_shift, &mask); if (err) break; } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); @@ -248,7 +228,7 @@ static int vmap_range(unsigned long addr, unsigned long end, int ioremap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot) { - return vmap_range(addr, end, phys_addr, prot); + return vmap_range(addr, end, phys_addr, prot, iomap_max_page_shift); } #ifdef CONFIG_GENERIC_IOREMAP From patchwork Fri Aug 21 15:12:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11729749 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E7E29138C for ; Fri, 21 Aug 2020 15:14:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AAF9A2076E for ; Fri, 21 Aug 2020 15:14:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Dxuzt8Gq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AAF9A2076E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E11EB8D0048; Fri, 21 Aug 2020 11:14:05 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DE8F88D0022; Fri, 21 Aug 2020 11:14:05 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CB1BF8D0048; Fri, 21 Aug 2020 11:14:05 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0243.hostedemail.com [216.40.44.243]) by kanga.kvack.org (Postfix) with ESMTP id B10C88D0022 for ; Fri, 21 Aug 2020 11:14:05 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 6C126362A for ; Fri, 21 Aug 2020 15:14:05 +0000 (UTC) X-FDA: 77174921250.17.gate05_3b112f02703a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id 63C7218022CE3 for ; Fri, 21 Aug 2020 15:13:05 +0000 (UTC) X-Spam-Summary: 1,0,0,b0f5839786e28925,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1535:1542:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:3138:3139:3140:3141:3142:3353:3865:3867:3868:3870:3872:4321:4385:4605:5007:6119:6261:6653:7514:9413:9592:10004:11026:11473:11657:11658:11914:12043:12297:12438:12517:12519:12555:12895:12986:13894:14096:14181:14394:14687:14721:21080:21444:21451:21627:21666:21990:30054,0,RBL:209.85.214.196:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100;04yrsawyhw9hy9ffihc1m3etetf3yopy57epfzhd5zrawg36q1u1gyyp7b4iatg.gu4xc9pjbpgu9mw9rpsajt5xhh7k65xytk44nno5jxy3esmxrdmeon54oecgztg.o-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: gate05_3b112f02703a X-Filterd-Recvd-Size: 5488 Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Fri, 21 Aug 2020 15:13:04 +0000 (UTC) Received: by mail-pl1-f196.google.com with SMTP id y6so996368plk.10 for ; Fri, 21 Aug 2020 08:13:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=mshGO+fS8Fvu8aEJg35ArnuOW0kj8WJXDS3j67fcQcE=; b=Dxuzt8Gq7VFN+A3EE1DxMPxeR9USeuD5Ig74qLhRJfNfR1CmVPxrcgV8gPhNHj2A8P quPQ5BBpTYZyqfVksB15bO0M5UDQGUg2/RLxDTyK+Jo40A3mOU5yASraaNxMTRw9S0OZ JN/wNNkHe44ywg52mENRYGVDjuigU4bPuvrNsGcn/ef5dP1I5n/TKYfxM9lVuHicwz6s Zw/5/ic/iqoZQf//idHiVEb0OpRo1LT4zDCNXeQ+gcZ8ESpEwAwKL9N6H+yqcJSx8YM0 URv+AuGrWbPWMx5M1EKzJeHJmDN2ARnD48B0zdNvmgnYzHz2Jn8n6nsRPZQUsqV4Q4nK axrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mshGO+fS8Fvu8aEJg35ArnuOW0kj8WJXDS3j67fcQcE=; b=B799DFxf/AM3Wp2SgHejm2cKsmvxpWd37TbjjwGHPtx8FiGBA/dM1VspKEc3xoCqnj fvRaX1YqQ5y7uGisUOIUNv5fPbG15sWFrMcOuOmsQShkbgTR7kX8rOb3YBdrRW/a0O3O gs3PLTjIa9FumYvYh8x7GFMAQOAQiYrFMM5nIaOvScpROUh75us3WHCxLepD9UxBSLBa HJDUiK+eQvOcsqShwf2oU/g2jyHhad0PM1SZHXOcfhHkRO0D4CBeGoejH1wBsVvlbhcd 2lZmS0Vx2PE4yBEcs0EtqclrXcBVWqC+w9xcLU9Ly7Ztr5xdkKxCq2ZSVqTUUlz2LBGZ o0IQ== X-Gm-Message-State: AOAM532LUcxQnHwIJtmwlI4vu9jOaiSH7hdATioqtrejTQDPEcGvx3Ea g/aM6yJhKAj3pVBDQvjeVMo6UlrMUuA= X-Google-Smtp-Source: ABdhPJyI4dN8FzkV0FOxrBa1THMuB+8nPxJGWft+AXElncM0Z7aiq7OcjvvT8WAe06i2OCzHEasr8Q== X-Received: by 2002:a17:90a:cb0b:: with SMTP id z11mr2882310pjt.77.1598022783284; Fri, 21 Aug 2020 08:13:03 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id s8sm3126985pfc.122.2020.08.21.08.12.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 08:13:02 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy Subject: [PATCH v6 06/12] powerpc: inline huge vmap supported functions Date: Sat, 22 Aug 2020 01:12:10 +1000 Message-Id: <20200821151216.1005117-7-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200821151216.1005117-1-npiggin@gmail.com> References: <20200821151216.1005117-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 63C7218022CE3 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This allows unsupported levels to be constant folded away, and so p4d_free_pud_page can be removed because it's no longer linked to. Cc: linuxppc-dev@lists.ozlabs.org Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/vmalloc.h | 19 ++++++++++++++++--- arch/powerpc/mm/book3s64/radix_pgtable.c | 21 --------------------- 2 files changed, 16 insertions(+), 24 deletions(-) diff --git a/arch/powerpc/include/asm/vmalloc.h b/arch/powerpc/include/asm/vmalloc.h index 105abb73f075..3f0c153befb0 100644 --- a/arch/powerpc/include/asm/vmalloc.h +++ b/arch/powerpc/include/asm/vmalloc.h @@ -1,12 +1,25 @@ #ifndef _ASM_POWERPC_VMALLOC_H #define _ASM_POWERPC_VMALLOC_H +#include #include #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -bool arch_vmap_p4d_supported(pgprot_t prot); -bool arch_vmap_pud_supported(pgprot_t prot); -bool arch_vmap_pmd_supported(pgprot_t prot); +static inline bool arch_vmap_p4d_supported(pgprot_t prot) +{ + return false; +} + +static inline bool arch_vmap_pud_supported(pgprot_t prot) +{ + /* HPT does not cope with large pages in the vmalloc area */ + return radix_enabled(); +} + +static inline bool arch_vmap_pmd_supported(pgprot_t prot) +{ + return radix_enabled(); +} #endif #endif /* _ASM_POWERPC_VMALLOC_H */ diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index eca83a50bf2e..27f5837cf145 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1134,22 +1134,6 @@ void radix__ptep_modify_prot_commit(struct vm_area_struct *vma, set_pte_at(mm, addr, ptep, pte); } -bool arch_vmap_pud_supported(pgprot_t prot) -{ - /* HPT does not cope with large pages in the vmalloc area */ - return radix_enabled(); -} - -bool arch_vmap_pmd_supported(pgprot_t prot) -{ - return radix_enabled(); -} - -int p4d_free_pud_page(p4d_t *p4d, unsigned long addr) -{ - return 0; -} - int pud_set_huge(pud_t *pud, phys_addr_t addr, pgprot_t prot) { pte_t *ptep = (pte_t *)pud; @@ -1233,8 +1217,3 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) return 1; } - -bool arch_vmap_p4d_supported(pgprot_t prot) -{ - return false; -} From patchwork Fri Aug 21 15:12:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11729737 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4D504109B for ; Fri, 21 Aug 2020 15:13:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1A3D620855 for ; Fri, 21 Aug 2020 15:13:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="IqkomYLy" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1A3D620855 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 204578D002F; Fri, 21 Aug 2020 11:13:11 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1B3D08D0022; Fri, 21 Aug 2020 11:13:11 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 07C938D002F; Fri, 21 Aug 2020 11:13:11 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0225.hostedemail.com [216.40.44.225]) by kanga.kvack.org (Postfix) with ESMTP id E2ABB8D0022 for ; Fri, 21 Aug 2020 11:13:10 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id AF8E31E19 for ; Fri, 21 Aug 2020 15:13:10 +0000 (UTC) X-FDA: 77174918940.07.cause34_241739a2703a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id 7DA111803FD61 for ; Fri, 21 Aug 2020 15:13:10 +0000 (UTC) X-Spam-Summary: 1,0,0,9b74fabeac023eaf,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1535:1542:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:2693:3138:3139:3140:3141:3142:3353:3866:3867:3868:3872:4117:4321:4385:4605:5007:6261:6653:6742:7514:8603:9389:9413:9592:10004:11026:11473:11657:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12895:12986:13894:14096:14181:14394:14687:14721:21080:21444:21451:21627:21666:21990:30054,0,RBL:209.85.215.195:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100;04yrgr9wdfwat31cjd96381gw96bsypwducz1yinat5si4b97rah8q1f7hrophw.nx8nj4ma3jezddmtk9kc6j9n1z146cwjg311fxfiy76zag6ksms9knka3wffocj.g-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: cause34_241739a2703a X-Filterd-Recvd-Size: 6080 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Fri, 21 Aug 2020 15:13:09 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id 189so1120429pgg.13 for ; Fri, 21 Aug 2020 08:13:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5XA+NVPhwbbllPG3xUWDGiDWcwCyUaHTGopG9dY3tOQ=; b=IqkomYLyWOWMOqPurIrzg99tZuO52VxFInUZTbikhVRxM4PXfxkwGHmlLO+XS0zZSk Pr7Jd2rs7vQ185N1QdUaYkoZ4efSqpJzxD94EJrzKx8kIqjycx8mvql3EtLZ04XD3PAv +mN7jYy17uEBwVMXBv+3QBaSIDwxz6zgeYjlPRuBkvdkuCiXh+vxSMcls6+ydgc6gluz WkRQ9CbCHTwvp0L5Wq+43+aFGNYPR0WtTbRIskVC8QnIMlAQeaDA6mbIgiGQ3bghONDn zxSAZVTZqpMLebzKsAXqaVzCJEbINtXsTKreC3UxJIdgVmZfEImtFPVyNKgk8QoG/nxi Cnvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5XA+NVPhwbbllPG3xUWDGiDWcwCyUaHTGopG9dY3tOQ=; b=NXFRxWr2QUFGtb2kE8LX76loyWYWPrE2f7pT4nodvIoVeKDCozol/WFW+/Y69hurad ZzCwH7AEsffWgWMRdWrDkvjg3KSg0eE82bjEYb9TTDgsEyft5MVm2W1j38c5CFcuHHSV QtHf54XH5+n+of5ZoeA18NjWzIUCIUxJhhi/7GVjM40eV1pTX/6fCExTYjoQuBmU1t8C hdgV+TKDpR/wMLoH5UwsVN3KW+0v0m6XedWPU5cR++/MikjC3pWYJw/uPpcuVwV5LPel 47iRRx30BfG5UA2+PHWIOZbCVYfBotp7nIVmbI7Dq7SNZACl8Qcz2X1FPVziD4BbXQwW Kiow== X-Gm-Message-State: AOAM530eEm5GKdhzFCpsSM43g+YudC1XOCFPZZu1DgaCpHno7rkG1AaD hW129cnzxmUGB8Zqv6C9eNXYceIfpCI= X-Google-Smtp-Source: ABdhPJwMltqvuSSQ4qdpsDYl7lU9jXM0/s7ZBmlyh9r6bV9pBXP6Cdk0H0vc9dugPEDfhsI32WoXkg== X-Received: by 2002:a63:7505:: with SMTP id q5mr2561816pgc.312.1598022788965; Fri, 21 Aug 2020 08:13:08 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id s8sm3126985pfc.122.2020.08.21.08.13.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 08:13:08 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org Subject: [PATCH v6 07/12] arm64: inline huge vmap supported functions Date: Sat, 22 Aug 2020 01:12:11 +1000 Message-Id: <20200821151216.1005117-8-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200821151216.1005117-1-npiggin@gmail.com> References: <20200821151216.1005117-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 7DA111803FD61 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This allows unsupported levels to be constant folded away, and so p4d_free_pud_page can be removed because it's no longer linked to. Cc: Catalin Marinas Cc: Will Deacon Cc: linux-arm-kernel@lists.infradead.org Signed-off-by: Nicholas Piggin --- arch/arm64/include/asm/vmalloc.h | 23 ++++++++++++++++++++--- arch/arm64/mm/mmu.c | 26 -------------------------- 2 files changed, 20 insertions(+), 29 deletions(-) diff --git a/arch/arm64/include/asm/vmalloc.h b/arch/arm64/include/asm/vmalloc.h index 597b40405319..fc9a12d6cc1a 100644 --- a/arch/arm64/include/asm/vmalloc.h +++ b/arch/arm64/include/asm/vmalloc.h @@ -4,9 +4,26 @@ #include #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -bool arch_vmap_p4d_supported(pgprot_t prot); -bool arch_vmap_pud_supported(pgprot_t prot); -bool arch_vmap_pmd_supported(pgprot_t prot); +static inline bool arch_vmap_p4d_supported(pgprot_t prot) +{ + return false; +} + +static inline bool arch_vmap_pud_supported(pgprot_t prot) +{ + /* + * Only 4k granule supports level 1 block mappings. + * SW table walks can't handle removal of intermediate entries. + */ + return IS_ENABLED(CONFIG_ARM64_4K_PAGES) && + !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); +} + +static inline bool arch_vmap_pmd_supported(pgprot_t prot) +{ + /* See arch_vmap_pud_supported() */ + return !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); +} #endif #endif /* _ASM_ARM64_VMALLOC_H */ diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 9df7e0058c78..07093e148957 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1304,27 +1304,6 @@ void *__init fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot) return dt_virt; } -bool arch_vmap_p4d_supported(pgprot_t prot) -{ - return false; -} - -bool arch_vmap_pud_supported(pgprot_t prot); -{ - /* - * Only 4k granule supports level 1 block mappings. - * SW table walks can't handle removal of intermediate entries. - */ - return IS_ENABLED(CONFIG_ARM64_4K_PAGES) && - !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); -} - -bool arch_vmap_pmd_supported(pgprot_t prot) -{ - /* See arch_vmap_pud_supported() */ - return !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); -} - int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t prot) { pud_t new_pud = pfn_pud(__phys_to_pfn(phys), mk_pud_sect_prot(prot)); @@ -1416,11 +1395,6 @@ int pud_free_pmd_page(pud_t *pudp, unsigned long addr) return 1; } -int p4d_free_pud_page(p4d_t *p4d, unsigned long addr) -{ - return 0; /* Don't attempt a block mapping */ -} - #ifdef CONFIG_MEMORY_HOTPLUG static void __remove_pgd_mapping(pgd_t *pgdir, unsigned long start, u64 size) { From patchwork Fri Aug 21 15:12:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11729739 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F2EFB109B for ; Fri, 21 Aug 2020 15:13:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B66DC2086A for ; Fri, 21 Aug 2020 15:13:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="C61EU6HX" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B66DC2086A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C8B8F8D0031; Fri, 21 Aug 2020 11:13:17 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C160E8D0022; Fri, 21 Aug 2020 11:13:17 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ADC7F8D0031; Fri, 21 Aug 2020 11:13:17 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0081.hostedemail.com [216.40.44.81]) by kanga.kvack.org (Postfix) with ESMTP id 9309D8D0022 for ; Fri, 21 Aug 2020 11:13:17 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 53365824556B for ; Fri, 21 Aug 2020 15:13:17 +0000 (UTC) X-FDA: 77174919234.29.shade79_5c154292703a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id E949E18086CC7 for ; Fri, 21 Aug 2020 15:13:16 +0000 (UTC) X-Spam-Summary: 1,0,0,d061bbd16d8f1a64,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1535:1543:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:2895:2901:3138:3139:3140:3141:3142:3167:3354:3865:3867:3868:3870:3872:3874:4117:4385:4605:5007:6119:6261:6630:6653:6742:7514:7875:7903:9413:9592:10004:11026:11473:11657:11658:11914:12043:12297:12438:12517:12519:12555:12895:13846:13894:14096:14181:14394:14687:14721:21080:21433:21444:21451:21627:21666:21990:30054:30070,0,RBL:209.85.214.195:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100;04ygsgpkxgzmu8x9nmbp64i8ufm8aycynskrrbopbhr8r58uwaa9amkdwhqfeqd.rads6qqbbsakmwbwcjrq1jxpac5hfykfc7ibs7parsd1kjgs87pwko5deqgdtyh.y-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: shade79_5c154292703a X-Filterd-Recvd-Size: 6404 Received: from mail-pl1-f195.google.com (mail-pl1-f195.google.com [209.85.214.195]) by imf29.hostedemail.com (Postfix) with ESMTP for ; Fri, 21 Aug 2020 15:13:16 +0000 (UTC) Received: by mail-pl1-f195.google.com with SMTP id h2so1013909plr.0 for ; Fri, 21 Aug 2020 08:13:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/TNFv6HMLiWXamsCOEc67C9ud98FI7LShW37Zy1NZ3Q=; b=C61EU6HXNNiIE03XtFMUUqH0afKGOIVNbSDjGdx057g8dEiIh5RJ37CBcBRlcqMGp2 AqxrungEYkWu+oIKtkmSUOzy/+Au53BUprQpyGz/LZJqP4A/KwsYAATREHIm/WefPvqd pu2o5GkG+bynDn1hswfdToNcyF2ifZfTP7XHQXdD/TNqeDCMlpxPcvj2sJ0YFaLLZGCq XN6VgMZIrg74vq/qsWlwaKAK0JFLaf9nckuEf2X2UbXEDapC1I136yAjpO7dmUGU4/6C PJYr7n4Cgio2Zpo2a7rN8G3HXn2ofcfKbFBH6cjmSGoNCRkS6dI7nw9yqEVwS/tRwOr0 hRjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/TNFv6HMLiWXamsCOEc67C9ud98FI7LShW37Zy1NZ3Q=; b=LAdl69s9+3qbfoG+6DAejWjfw3O6i1gApZyQk42IRNoZZsFHtLIC3BZFm7ctORGbQ0 CXpxzfmsDKOpCRApjVujK0R2XcMjOaluMw8eYzsFdG4O9sfYAeSw2Di1p3q3yNPh9ZSi 2lGiVxz4aXBt0f/SSicOYZT6Q/qhun0PsxInlHJ+UhtQ6xxUW/ud1q54sjWzKSztewEv NvLjL+IRZtolIpRe1l+F3kx2ozM8O5MabdqZggrW0JFxFGJtW9yodrMoCT8Hzj26r2AK l1ADUHWShbntIo32mw+0gxfI6YQ+ntoo/xpimS4i1Ud8fdRRCqkefDkPNopUnChWcGZV 8cNQ== X-Gm-Message-State: AOAM532+lXGy/xpolH9Wz4PIq3YE2CvabkuAESnAoJhVZPyvrgUvkJnp 0D0qv3DALK8a1t+HpCAzIGCcchqzLWo= X-Google-Smtp-Source: ABdhPJxJ42RePcnQK55+NHJSCK6N+XE6cpbW5VmwJeO028VLoSugON1xyR8CJ90w/V7P+0Hx+E2elg== X-Received: by 2002:a17:902:8e85:: with SMTP id bg5mr2686443plb.306.1598022795348; Fri, 21 Aug 2020 08:13:15 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id s8sm3126985pfc.122.2020.08.21.08.13.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 08:13:14 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" Subject: [PATCH v6 08/12] x86: inline huge vmap supported functions Date: Sat, 22 Aug 2020 01:12:12 +1000 Message-Id: <20200821151216.1005117-9-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200821151216.1005117-1-npiggin@gmail.com> References: <20200821151216.1005117-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: E949E18086CC7 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This allows unsupported levels to be constant folded away, and so p4d_free_pud_page can be removed because it's no longer linked to. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: x86@kernel.org Cc: "H. Peter Anvin" Signed-off-by: Nicholas Piggin --- arch/x86/include/asm/vmalloc.h | 22 +++++++++++++++++++--- arch/x86/mm/ioremap.c | 19 ------------------- arch/x86/mm/pgtable.c | 13 ------------- 3 files changed, 19 insertions(+), 35 deletions(-) diff --git a/arch/x86/include/asm/vmalloc.h b/arch/x86/include/asm/vmalloc.h index 094ea2b565f3..e714b00fc0ca 100644 --- a/arch/x86/include/asm/vmalloc.h +++ b/arch/x86/include/asm/vmalloc.h @@ -1,13 +1,29 @@ #ifndef _ASM_X86_VMALLOC_H #define _ASM_X86_VMALLOC_H +#include #include #include #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -bool arch_vmap_p4d_supported(pgprot_t prot); -bool arch_vmap_pud_supported(pgprot_t prot); -bool arch_vmap_pmd_supported(pgprot_t prot); +static inline bool arch_vmap_p4d_supported(pgprot_t prot) +{ + return false; +} + +static inline bool arch_vmap_pud_supported(pgprot_t prot) +{ +#ifdef CONFIG_X86_64 + return boot_cpu_has(X86_FEATURE_GBPAGES); +#else + return false; +#endif +} + +static inline bool arch_vmap_pmd_supported(pgprot_t prot) +{ + return boot_cpu_has(X86_FEATURE_PSE); +} #endif #endif /* _ASM_X86_VMALLOC_H */ diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c index 159bfca757b9..1465a22a9bfb 100644 --- a/arch/x86/mm/ioremap.c +++ b/arch/x86/mm/ioremap.c @@ -481,25 +481,6 @@ void iounmap(volatile void __iomem *addr) } EXPORT_SYMBOL(iounmap); -bool arch_vmap_p4d_supported(pgprot_t prot) -{ - return false; -} - -bool arch_vmap_pud_supported(pgprot_t prot) -{ -#ifdef CONFIG_X86_64 - return boot_cpu_has(X86_FEATURE_GBPAGES); -#else - return false; -#endif -} - -bool arch_vmap_pmd_supported(pgprot_t prot) -{ - return boot_cpu_has(X86_FEATURE_PSE); -} - /* * Convert a physical pointer to a virtual kernel pointer for /dev/mem * access diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index dfd82f51ba66..801c418ee97d 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -780,14 +780,6 @@ int pmd_clear_huge(pmd_t *pmd) return 0; } -/* - * Until we support 512GB pages, skip them in the vmap area. - */ -int p4d_free_pud_page(p4d_t *p4d, unsigned long addr) -{ - return 0; -} - #ifdef CONFIG_X86_64 /** * pud_free_pmd_page - Clear pud entry and free pmd page. @@ -859,11 +851,6 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) #else /* !CONFIG_X86_64 */ -int pud_free_pmd_page(pud_t *pud, unsigned long addr) -{ - return pud_none(*pud); -} - /* * Disable free page handling on x86-PAE. This assures that ioremap() * does not update sync'd pmd entries. See vmalloc_sync_one(). From patchwork Fri Aug 21 15:12:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11729741 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CEE85138C for ; Fri, 21 Aug 2020 15:13:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 81DA9207DF for ; Fri, 21 Aug 2020 15:13:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mObTXcNO" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 81DA9207DF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5AEE98D0032; Fri, 21 Aug 2020 11:13:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 55DD88D0022; Fri, 21 Aug 2020 11:13:23 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 426DD8D0032; Fri, 21 Aug 2020 11:13:23 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0095.hostedemail.com [216.40.44.95]) by kanga.kvack.org (Postfix) with ESMTP id 282428D0022 for ; Fri, 21 Aug 2020 11:13:23 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id D7DF4824556B for ; Fri, 21 Aug 2020 15:13:22 +0000 (UTC) X-FDA: 77174919444.07.spade26_15017fa2703a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id 76E4D1803F9AD for ; Fri, 21 Aug 2020 15:13:22 +0000 (UTC) X-Spam-Summary: 1,0,0,cdbb8e052c6d1c71,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:1:41:69:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1605:1730:1747:1777:1792:1981:2194:2196:2199:2200:2393:2559:2562:2637:2898:3138:3139:3140:3141:3142:3867:3871:4321:4385:4605:5007:6119:6261:6653:7514:7875:8603:9036:9413:9592:10004:11026:11232:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:12986:13894:14096:14110:14394:14687:21080:21324:21444:21451:21611:21627:21666:21740:21990:30054:30070,0,RBL:209.85.215.194:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100;04ygu4696axih1eieyhw6ktbsonjyopg37etga1q7iqtiwh4kj9ya3a3sn4gjz9.3fa5bjc6urpxgua57byd4psmn9sic57ew38fahi3yadbhj7x35y7111osympjtc.1-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: spade26_15017fa2703a X-Filterd-Recvd-Size: 14903 Received: from mail-pg1-f194.google.com (mail-pg1-f194.google.com [209.85.215.194]) by imf10.hostedemail.com (Postfix) with ESMTP for ; Fri, 21 Aug 2020 15:13:22 +0000 (UTC) Received: by mail-pg1-f194.google.com with SMTP id p37so1140032pgl.3 for ; Fri, 21 Aug 2020 08:13:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FErZs3hZSGjvqwNcDn/CTTFNbpmYi2cg1JR35qAXZTs=; b=mObTXcNOhhA8U1rtS9mrKXAffB+UTmkGengrtRhUbVapk7J2QjjqdgEEYlZ2UOhQn6 58Gp/+PUjNNZIxcRMWvcNlWTJbl5fYMMfrT6N4oeTgNfrboS6AaFq9QmalPxR46ZfUCn bW/nx9aBS1cWbDttrIfwej5X2RV7sNcp6c+KDA73dK/T5wgU/Fd2a128NR5WSFwuH+YQ JlK5ftPY6ovhVjMYDpbRxPmbM2WwZvtAb0zNBVtvF7otP+YW/oTvAf5MZ6R9s+rSaYG2 xmaCcPVtv7B+h1tVALqbHJgBZNAzk1ojycA+Q+ERxCFfKGxaoHjijkx3ek1FsjsWlBuu FS0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FErZs3hZSGjvqwNcDn/CTTFNbpmYi2cg1JR35qAXZTs=; b=RG995DBvt5kht/lSTH3a2gesFEwKtTPBUaqSK3489j5YiIVUNLbW7IYjxhllhmc6Tb aleFTDrf0XolJ16pSH1I7zM8jRAF9X+5g9cNIuKl7zJGIYPfG85UQ0LQz4y/Eyne5F7S g4CcqxFVFtwgtFJgxZZ4HY73skbOqVVLE/3pMOPneCu67Tvu5fQHy6JvESTn42dDV7VJ 1y35eJSZzFGUAi3EZMsdzst8ksI2ZBvDHKPo7+v81LhVG4WY+fledBm0FjeX4ttu9Ou5 dfJva2on4TS7AlITfyHLvdIbbqbwLhV6VeUAYau/VdGXN6uUr0FhwpA9ViW8bAe0tcIE 1GfQ== X-Gm-Message-State: AOAM5314GM+UdjmCqvkBKpf8hHxR0IF+zXpyNk/S308sCkgCUSLZJr4R PzXfhfjYO5r/lHg0Upcifri7eIbtzmA= X-Google-Smtp-Source: ABdhPJy7aXmzfWVcfn2hCNJVY3PmZt1ompR3e5JLEO+7k4LSFTuTrRiXWMf/vVTVKPU/IDT4yiQlvQ== X-Received: by 2002:a63:7707:: with SMTP id s7mr2597560pgc.407.1598022800172; Fri, 21 Aug 2020 08:13:20 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id s8sm3126985pfc.122.2020.08.21.08.13.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 08:13:19 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy Subject: [PATCH v6 09/12] mm: Move vmap_range from mm/ioremap.c to mm/vmalloc.c Date: Sat, 22 Aug 2020 01:12:13 +1000 Message-Id: <20200821151216.1005117-10-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200821151216.1005117-1-npiggin@gmail.com> References: <20200821151216.1005117-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 76E4D1803F9AD X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a generic kernel virtual memory mapper, not specific to ioremap. Signed-off-by: Nicholas Piggin --- include/linux/vmalloc.h | 3 + mm/ioremap.c | 197 ---------------------------------------- mm/vmalloc.c | 196 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 199 insertions(+), 197 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 3f6bba4cc9bc..15adb9a14fb6 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -177,6 +177,9 @@ extern struct vm_struct *remove_vm_area(const void *addr); extern struct vm_struct *find_vm_area(const void *addr); #ifdef CONFIG_MMU +int vmap_range(unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift); extern int map_kernel_range_noflush(unsigned long start, unsigned long size, pgprot_t prot, struct page **pages); int map_kernel_range(unsigned long start, unsigned long size, pgprot_t prot, diff --git a/mm/ioremap.c b/mm/ioremap.c index c67f91164401..d1dcc7e744ac 100644 --- a/mm/ioremap.c +++ b/mm/ioremap.c @@ -28,203 +28,6 @@ early_param("nohugeiomap", set_nohugeiomap); static const bool iomap_max_page_shift = PAGE_SHIFT; #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ -static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) -{ - pte_t *pte; - u64 pfn; - - pfn = phys_addr >> PAGE_SHIFT; - pte = pte_alloc_kernel_track(pmd, addr, mask); - if (!pte) - return -ENOMEM; - do { - BUG_ON(!pte_none(*pte)); - set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); - pfn++; - } while (pte++, addr += PAGE_SIZE, addr != end); - *mask |= PGTBL_PTE_MODIFIED; - return 0; -} - -static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) -{ - if (max_page_shift < PMD_SHIFT) - return 0; - - if (!arch_vmap_pmd_supported(prot)) - return 0; - - if ((end - addr) != PMD_SIZE) - return 0; - - if (!IS_ALIGNED(addr, PMD_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, PMD_SIZE)) - return 0; - - if (pmd_present(*pmd) && !pmd_free_pte_page(pmd, addr)) - return 0; - - return pmd_set_huge(pmd, phys_addr, prot); -} - -static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift, pgtbl_mod_mask *mask) -{ - pmd_t *pmd; - unsigned long next; - - pmd = pmd_alloc_track(&init_mm, pud, addr, mask); - if (!pmd) - return -ENOMEM; - do { - next = pmd_addr_end(addr, end); - - if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, max_page_shift)) { - *mask |= PGTBL_PMD_MODIFIED; - continue; - } - - if (vmap_pte_range(pmd, addr, next, phys_addr, prot, mask)) - return -ENOMEM; - } while (pmd++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) -{ - if (max_page_shift < PUD_SHIFT) - return 0; - - if (!arch_vmap_pud_supported(prot)) - return 0; - - if ((end - addr) != PUD_SIZE) - return 0; - - if (!IS_ALIGNED(addr, PUD_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, PUD_SIZE)) - return 0; - - if (pud_present(*pud) && !pud_free_pmd_page(pud, addr)) - return 0; - - return pud_set_huge(pud, phys_addr, prot); -} - -static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift, pgtbl_mod_mask *mask) -{ - pud_t *pud; - unsigned long next; - - pud = pud_alloc_track(&init_mm, p4d, addr, mask); - if (!pud) - return -ENOMEM; - do { - next = pud_addr_end(addr, end); - - if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot, max_page_shift)) { - *mask |= PGTBL_PUD_MODIFIED; - continue; - } - - if (vmap_pmd_range(pud, addr, next, phys_addr, prot, max_page_shift, mask)) - return -ENOMEM; - } while (pud++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) -{ - if (max_page_shift < P4D_SHIFT) - return 0; - - if (!arch_vmap_p4d_supported(prot)) - return 0; - - if ((end - addr) != P4D_SIZE) - return 0; - - if (!IS_ALIGNED(addr, P4D_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, P4D_SIZE)) - return 0; - - if (p4d_present(*p4d) && !p4d_free_pud_page(p4d, addr)) - return 0; - - return p4d_set_huge(p4d, phys_addr, prot); -} - -static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift, pgtbl_mod_mask *mask) -{ - p4d_t *p4d; - unsigned long next; - - p4d = p4d_alloc_track(&init_mm, pgd, addr, mask); - if (!p4d) - return -ENOMEM; - do { - next = p4d_addr_end(addr, end); - - if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot, max_page_shift)) { - *mask |= PGTBL_P4D_MODIFIED; - continue; - } - - if (vmap_pud_range(p4d, addr, next, phys_addr, prot, max_page_shift, mask)) - return -ENOMEM; - } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -static int vmap_range(unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) -{ - pgd_t *pgd; - unsigned long start; - unsigned long next; - int err; - pgtbl_mod_mask mask = 0; - - might_sleep(); - BUG_ON(addr >= end); - - start = addr; - pgd = pgd_offset_k(addr); - do { - next = pgd_addr_end(addr, end); - err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, max_page_shift, &mask); - if (err) - break; - } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); - - flush_cache_vmap(start, end); - - if (mask & ARCH_PAGE_TABLE_SYNC_MASK) - arch_sync_kernel_mappings(start, end); - - return err; -} - int ioremap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot) { diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 45cd80ec7eeb..256554d598e6 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -70,6 +70,202 @@ static void free_work(struct work_struct *w) } /*** Page table manipulation functions ***/ +static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) +{ + pte_t *pte; + u64 pfn; + + pfn = phys_addr >> PAGE_SHIFT; + pte = pte_alloc_kernel_track(pmd, addr, mask); + if (!pte) + return -ENOMEM; + do { + BUG_ON(!pte_none(*pte)); + set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); + pfn++; + } while (pte++, addr += PAGE_SIZE, addr != end); + *mask |= PGTBL_PTE_MODIFIED; + return 0; +} + +static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + if (max_page_shift < PMD_SHIFT) + return 0; + + if (!arch_vmap_pmd_supported(prot)) + return 0; + + if ((end - addr) != PMD_SIZE) + return 0; + + if (!IS_ALIGNED(addr, PMD_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, PMD_SIZE)) + return 0; + + if (pmd_present(*pmd) && !pmd_free_pte_page(pmd, addr)) + return 0; + + return pmd_set_huge(pmd, phys_addr, prot); +} + +static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift, pgtbl_mod_mask *mask) +{ + pmd_t *pmd; + unsigned long next; + + pmd = pmd_alloc_track(&init_mm, pud, addr, mask); + if (!pmd) + return -ENOMEM; + do { + next = pmd_addr_end(addr, end); + + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, max_page_shift)) { + *mask |= PGTBL_PMD_MODIFIED; + continue; + } + + if (vmap_pte_range(pmd, addr, next, phys_addr, prot, mask)) + return -ENOMEM; + } while (pmd++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + if (max_page_shift < PUD_SHIFT) + return 0; + + if (!arch_vmap_pud_supported(prot)) + return 0; + + if ((end - addr) != PUD_SIZE) + return 0; + + if (!IS_ALIGNED(addr, PUD_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, PUD_SIZE)) + return 0; + + if (pud_present(*pud) && !pud_free_pmd_page(pud, addr)) + return 0; + + return pud_set_huge(pud, phys_addr, prot); +} + +static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift, pgtbl_mod_mask *mask) +{ + pud_t *pud; + unsigned long next; + + pud = pud_alloc_track(&init_mm, p4d, addr, mask); + if (!pud) + return -ENOMEM; + do { + next = pud_addr_end(addr, end); + + if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot, max_page_shift)) { + *mask |= PGTBL_PUD_MODIFIED; + continue; + } + + if (vmap_pmd_range(pud, addr, next, phys_addr, prot, max_page_shift, mask)) + return -ENOMEM; + } while (pud++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + if (max_page_shift < P4D_SHIFT) + return 0; + + if (!arch_vmap_p4d_supported(prot)) + return 0; + + if ((end - addr) != P4D_SIZE) + return 0; + + if (!IS_ALIGNED(addr, P4D_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, P4D_SIZE)) + return 0; + + if (p4d_present(*p4d) && !p4d_free_pud_page(p4d, addr)) + return 0; + + return p4d_set_huge(p4d, phys_addr, prot); +} + +static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift, pgtbl_mod_mask *mask) +{ + p4d_t *p4d; + unsigned long next; + + p4d = p4d_alloc_track(&init_mm, pgd, addr, mask); + if (!p4d) + return -ENOMEM; + do { + next = p4d_addr_end(addr, end); + + if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot, max_page_shift)) { + *mask |= PGTBL_P4D_MODIFIED; + continue; + } + + if (vmap_pud_range(p4d, addr, next, phys_addr, prot, max_page_shift, mask)) + return -ENOMEM; + } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +int vmap_range(unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + pgd_t *pgd; + unsigned long start; + unsigned long next; + int err; + pgtbl_mod_mask mask = 0; + + might_sleep(); + BUG_ON(addr >= end); + + start = addr; + pgd = pgd_offset_k(addr); + do { + next = pgd_addr_end(addr, end); + err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, max_page_shift, &mask); + if (err) + break; + } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); + + flush_cache_vmap(start, end); + + if (mask & ARCH_PAGE_TABLE_SYNC_MASK) + arch_sync_kernel_mappings(start, end); + + return err; +} static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgtbl_mod_mask *mask) From patchwork Fri Aug 21 15:12:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11729743 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 06DDD109B for ; Fri, 21 Aug 2020 15:13:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C72C5207DF for ; Fri, 21 Aug 2020 15:13:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="u6yhubFc" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C72C5207DF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BA0658D0035; Fri, 21 Aug 2020 11:13:27 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B2AC48D0022; Fri, 21 Aug 2020 11:13:27 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9F18B8D0035; Fri, 21 Aug 2020 11:13:27 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0072.hostedemail.com [216.40.44.72]) by kanga.kvack.org (Postfix) with ESMTP id 8661C8D0022 for ; Fri, 21 Aug 2020 11:13:27 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 430DF824556B for ; Fri, 21 Aug 2020 15:13:27 +0000 (UTC) X-FDA: 77174919654.16.slip27_5515b0a2703a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin16.hostedemail.com (Postfix) with ESMTP id 0C855100E6912 for ; Fri, 21 Aug 2020 15:13:27 +0000 (UTC) X-Spam-Summary: 1,0,0,b3e82260f0628319,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2731:3138:3139:3140:3141:3142:3352:3865:3866:3868:3870:3874:4321:5007:6261:6653:7514:7875:9413:9592:10004:11026:11473:11658:11914:12043:12297:12438:12517:12519:12555:12895:13069:13161:13229:13311:13357:13894:14096:14181:14384:14394:14687:14721:21080:21444:21451:21627:21666:21990:30054,0,RBL:209.85.210.195:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04yfnyrwe4khdwytcdxgomq3agoinyc71khjzgrret6jjxbm968kb1o7tqbr6oh.wayfyuxek4dsc75hxrzzi1fs4ai6dohy185ep76t6xabuhrhf38t46hw8jmmjho.6-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: slip27_5515b0a2703a X-Filterd-Recvd-Size: 4661 Received: from mail-pf1-f195.google.com (mail-pf1-f195.google.com [209.85.210.195]) by imf43.hostedemail.com (Postfix) with ESMTP for ; Fri, 21 Aug 2020 15:13:26 +0000 (UTC) Received: by mail-pf1-f195.google.com with SMTP id r11so1193146pfl.11 for ; Fri, 21 Aug 2020 08:13:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Ik3l57t0FyOClvLjEe+sb/BhE/y0hhY0gnEqux751Uc=; b=u6yhubFcXihID2fWq+QiE0CcG9dvjSoF59E015bcoBinPTWoTc6VB4OCRmb/y5KRsT qJkbjXob1gFUX91mcQBSG3U26lx8vo9W5U8g0dpaAxpbW8ExWeJzgZEfBwuwalftHFZn 0wke4MbGjU2ojD8aQbwZOkVbPO2SYtKoHSz12zM/hQNNjeYeWZF7jE5g2AHPO0aB+DYw 8C6ylDWaxR7KccTt7GD+eLNOYbuhrkEsjTe/Yv3CvgZ8RSQ10W7DxFdo98xlAAYx2wTT XzgU57hcg/H50XvMaweP1v5brBYXdmpavwyQpD5YyHUNi2KfejW5mx/+gd2ao61mGiJ9 ZR+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ik3l57t0FyOClvLjEe+sb/BhE/y0hhY0gnEqux751Uc=; b=QgFOcYJN3LgDMOl9ZmOD1DS4x/yown5g31YR02dy4CkhlI2FDgL8JR0iCfxX2Togd2 QlsnFIVc0eYxv8/ueADXU5LtNlHpG/jk14g7NiLkCWLbjtI9oJQIsDQd/WYXRojEt4j5 n/ZOJfYALWdcupsdUGyBGqEXo/tTPxF9D6miT/jSerXbz5Yusxm4pLfp3Vj2yXN74Fc/ 1koTBBjBLFSV8gbIlQHnXlyakLUcjkOksVJpSstsatSyYx3Pv9oA9qbSEge5ou8JxE1E MXnm8vozP9tQ2Xvejlqu41tjRBgRckcoauuwt/8QL+9T94NFiPtji8tf7DitjokqQBD2 DolQ== X-Gm-Message-State: AOAM533rtHTCK13Ufs9ui+ENPcqPG881/R61+33UoyKqu5WWtQgSBJxR 2bowtoIJPlyHZBV6JThVOhR0w2dkEnM= X-Google-Smtp-Source: ABdhPJzOk8LezUqei41Rt5u2ralsl0Vc1m91oWRQQQjcIB5tNWY2WeZkdOIgpy1cX1Su0SK2t1NhyA== X-Received: by 2002:a63:5256:: with SMTP id s22mr2774026pgl.325.1598022805468; Fri, 21 Aug 2020 08:13:25 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id s8sm3126985pfc.122.2020.08.21.08.13.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 08:13:24 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy Subject: [PATCH v6 10/12] mm/vmalloc: add vmap_range_noflush variant Date: Sat, 22 Aug 2020 01:12:14 +1000 Message-Id: <20200821151216.1005117-11-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200821151216.1005117-1-npiggin@gmail.com> References: <20200821151216.1005117-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0C855100E6912 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As a side-effect, the order of flush_cache_vmap() and arch_sync_kernel_mappings() calls are switched, but that now matches the other callers in this file. Signed-off-by: Nicholas Piggin --- mm/vmalloc.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 256554d598e6..1d6cad16bda3 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -237,7 +237,7 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, return 0; } -int vmap_range(unsigned long addr, unsigned long end, +static int vmap_range_noflush(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) { @@ -259,14 +259,24 @@ int vmap_range(unsigned long addr, unsigned long end, break; } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); - flush_cache_vmap(start, end); - if (mask & ARCH_PAGE_TABLE_SYNC_MASK) arch_sync_kernel_mappings(start, end); return err; } +int vmap_range(unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + int err; + + err = vmap_range_noflush(addr, end, phys_addr, prot, max_page_shift); + flush_cache_vmap(addr, end); + + return err; +} + static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgtbl_mod_mask *mask) { From patchwork Fri Aug 21 15:12:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11729745 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F3019138C for ; Fri, 21 Aug 2020 15:13:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A5301207C3 for ; Fri, 21 Aug 2020 15:13:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Z8zEtgRK" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A5301207C3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id ADED68D0036; Fri, 21 Aug 2020 11:13:33 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A8E048D0022; Fri, 21 Aug 2020 11:13:33 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9560A8D0036; Fri, 21 Aug 2020 11:13:33 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0043.hostedemail.com [216.40.44.43]) by kanga.kvack.org (Postfix) with ESMTP id 79EEF8D0022 for ; Fri, 21 Aug 2020 11:13:33 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 15182180AD820 for ; Fri, 21 Aug 2020 15:13:33 +0000 (UTC) X-FDA: 77174919906.09.dolls43_32030872703a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin09.hostedemail.com (Postfix) with ESMTP id E1AAD180AD802 for ; Fri, 21 Aug 2020 15:13:32 +0000 (UTC) X-Spam-Summary: 1,0,0,5ca799c4446aca61,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:4:41:69:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1605:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2553:2559:2562:2638:2689:2693:2731:2897:2899:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3874:4250:4321:4385:4605:4641:5007:6119:6261:6653:7514:7875:7903:7904:8603:9036:9413:9592:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12679:12683:12895:12986:13894:14096:14394:14687:21080:21324:21444:21451:21611:21627:21666:21740:21990:30054:30056:30090,0,RBL:209.85.210.196:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100;04yfqxkunh4rs7rj4uzab5e44kicpycfg6btwtzq9hep1kcqo6waghiexx1ogpm.97ariu64b4bgc8ie99d8acg3iw6qhyp9ej1hgd3y6xwz4juixxjjbmgp87akoid.c-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFt ime:23,L X-HE-Tag: dolls43_32030872703a X-Filterd-Recvd-Size: 16170 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) by imf44.hostedemail.com (Postfix) with ESMTP for ; Fri, 21 Aug 2020 15:13:32 +0000 (UTC) Received: by mail-pf1-f196.google.com with SMTP id x25so1209410pff.4 for ; Fri, 21 Aug 2020 08:13:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6KJeQP8NI0Gmq28+Z9mGp8SWZDiLwVnET1WJkFSU9fg=; b=Z8zEtgRKVXUKEafq+BytMBdAjJWq2DwMBV20kRlLX5910YO0R7cKvUsIcL8v1pRYKU f3Pl6F79/hwufCJXg45xBzbla+S5DozM+HIRjqgZ4PHTEj7MzbyVUPLu7UpMi/zmdIU/ VNrkrLEiCD8Cw2llSTxosbpmwffD2TnRjeWhPiUJQZHXJyb6AQx2vVa0VOVwQEwJ2Zk7 xMP8bH6RNgY8ShKkJthiCFS9QRx2G/C2fdXbkRyh0cbiN+2fC0QC0jwWueRJUCxk3hrK xTIfLi1r4nATNuQZjM5FRvJJFcDQ2rI387nTdTYz8HW8k/+bPgZVVcyg5q2bXzNmE7eX nyyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6KJeQP8NI0Gmq28+Z9mGp8SWZDiLwVnET1WJkFSU9fg=; b=tlR/uffy2FfljqNQkjJg1CEXDA0MUOJ11SyQM+V+PQzqf5MzPYuzh75wWra6lEeqw6 fCYW0izoFUKskmui/ftkMBAjE7mOV1eXs3KUEINd4Rte67snRJm/6STJJp72di1LWJ0J mv599Tc66u0OeuAeP9vy1nm6toV4Mp7WVFHCvsMEsyFShfwzRMFd8+wtHHVcRodzvR9s XqD0Ly/aji6SF76pqQYF4oxblQzxq+D1c5O99wLVkiYG7YniToIpXAGvCf3Eg+t/grjL clelheU1N1ixXS/dL5UMGS+O1LpsGCDFKW6X5n75ekLrdNz6UZ9tDgA+0kueJbafuyZ4 7NJg== X-Gm-Message-State: AOAM533MEj0PeSv8ROM416KN1wZ6T0M4iMJ40uzIzxaF1fpekMLplq5G AfV8n6S5oRp/SSqure+ErvaPoan/zrE= X-Google-Smtp-Source: ABdhPJyRBkCseiTqr7oIuBuYcSqCIamQ8jFoTXrJZMGETK3wzfRu2eOl4lTHIp9RIC12bd+rxOYlIQ== X-Received: by 2002:a05:6a00:7c8:: with SMTP id n8mr2869467pfu.123.1598022810717; Fri, 21 Aug 2020 08:13:30 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id s8sm3126985pfc.122.2020.08.21.08.13.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 08:13:30 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy Subject: [PATCH v6 11/12] mm/vmalloc: Hugepage vmalloc mappings Date: Sat, 22 Aug 2020 01:12:15 +1000 Message-Id: <20200821151216.1005117-12-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200821151216.1005117-1-npiggin@gmail.com> References: <20200821151216.1005117-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: E1AAD180AD802 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Support huge page vmalloc mappings. Config option HAVE_ARCH_HUGE_VMALLOC enables support on architectures that define HAVE_ARCH_HUGE_VMAP and supports PMD sized vmap mappings. vmalloc will attempt to allocate PMD-sized pages if allocating PMD size or larger, and fall back to small pages if that was unsuccessful. Allocations that do not use PAGE_KERNEL prot are not permitted to use huge pages, because not all callers expect this (e.g., module allocations vs strict module rwx). This reduces TLB misses by nearly 30x on a `git diff` workload on a 2-node POWER9 (59,800 -> 2,100) and reduces CPU cycles by 0.54%. This can result in more internal fragmentation and memory overhead for a given allocation, an option nohugevmalloc is added to disable at boot. Signed-off-by: Nicholas Piggin --- arch/Kconfig | 4 + include/linux/vmalloc.h | 1 + mm/page_alloc.c | 5 +- mm/vmalloc.c | 180 ++++++++++++++++++++++++++++++---------- 4 files changed, 145 insertions(+), 45 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index af14a567b493..b2b89d629317 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -616,6 +616,10 @@ config HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD config HAVE_ARCH_HUGE_VMAP bool +config HAVE_ARCH_HUGE_VMALLOC + depends on HAVE_ARCH_HUGE_VMAP + bool + config ARCH_WANT_HUGE_PMD_SHARE bool diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 15adb9a14fb6..a7449064fe35 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -58,6 +58,7 @@ struct vm_struct { unsigned long size; unsigned long flags; struct page **pages; + unsigned int page_order; unsigned int nr_pages; phys_addr_t phys_addr; const void *caller; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0e2bab486fea..b6427cc7b838 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -69,6 +69,7 @@ #include #include #include +#include #include #include @@ -8102,6 +8103,7 @@ void *__init alloc_large_system_hash(const char *tablename, void *table = NULL; gfp_t gfp_flags; bool virt; + bool huge; /* allow the kernel cmdline to have a say */ if (!numentries) { @@ -8169,6 +8171,7 @@ void *__init alloc_large_system_hash(const char *tablename, } else if (get_order(size) >= MAX_ORDER || hashdist) { table = __vmalloc(size, gfp_flags); virt = true; + huge = (find_vm_area(table)->page_order > 0); } else { /* * If bucketsize is not a power-of-two, we may free @@ -8185,7 +8188,7 @@ void *__init alloc_large_system_hash(const char *tablename, pr_info("%s hash table entries: %ld (order: %d, %lu bytes, %s)\n", tablename, 1UL << log2qty, ilog2(size) - PAGE_SHIFT, size, - virt ? "vmalloc" : "linear"); + virt ? (huge ? "vmalloc hugepage" : "vmalloc") : "linear"); if (_hash_shift) *_hash_shift = log2qty; diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 1d6cad16bda3..8db53c2d7f72 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -44,6 +44,19 @@ #include "internal.h" #include "pgalloc-track.h" +#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC +static bool __ro_after_init vmap_allow_huge = true; + +static int __init set_nohugevmalloc(char *str) +{ + vmap_allow_huge = false; + return 0; +} +early_param("nohugevmalloc", set_nohugevmalloc); +#else /* CONFIG_HAVE_ARCH_HUGE_VMALLOC */ +static const bool vmap_allow_huge = false; +#endif /* CONFIG_HAVE_ARCH_HUGE_VMALLOC */ + bool is_vmalloc_addr(const void *x) { unsigned long addr = (unsigned long)x; @@ -477,31 +490,12 @@ static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr, return 0; } -/** - * map_kernel_range_noflush - map kernel VM area with the specified pages - * @addr: start of the VM area to map - * @size: size of the VM area to map - * @prot: page protection flags to use - * @pages: pages to map - * - * Map PFN_UP(@size) pages at @addr. The VM area @addr and @size specify should - * have been allocated using get_vm_area() and its friends. - * - * NOTE: - * This function does NOT do any cache flushing. The caller is responsible for - * calling flush_cache_vmap() on to-be-mapped areas before calling this - * function. - * - * RETURNS: - * 0 on success, -errno on failure. - */ -int map_kernel_range_noflush(unsigned long addr, unsigned long size, - pgprot_t prot, struct page **pages) +static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages) { unsigned long start = addr; - unsigned long end = addr + size; - unsigned long next; pgd_t *pgd; + unsigned long next; int err = 0; int nr = 0; pgtbl_mod_mask mask = 0; @@ -523,6 +517,65 @@ int map_kernel_range_noflush(unsigned long addr, unsigned long size, return 0; } +static int vmap_pages_range_noflush(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift) +{ + unsigned int i, nr = (end - addr) >> PAGE_SHIFT; + + WARN_ON(page_shift < PAGE_SHIFT); + + if (page_shift == PAGE_SHIFT) + return vmap_small_pages_range_noflush(addr, end, prot, pages); + + for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) { + int err; + + err = vmap_range_noflush(addr, addr + (1UL << page_shift), + __pa(page_address(pages[i])), prot, + page_shift); + if (err) + return err; + + addr += 1UL << page_shift; + } + + return 0; +} + +static int vmap_pages_range(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift) +{ + int err; + + err = vmap_pages_range_noflush(addr, end, prot, pages, page_shift); + flush_cache_vmap(addr, end); + return err; +} + +/** + * map_kernel_range_noflush - map kernel VM area with the specified pages + * @addr: start of the VM area to map + * @size: size of the VM area to map + * @prot: page protection flags to use + * @pages: pages to map + * + * Map PFN_UP(@size) pages at @addr. The VM area @addr and @size specify should + * have been allocated using get_vm_area() and its friends. + * + * NOTE: + * This function does NOT do any cache flushing. The caller is responsible for + * calling flush_cache_vmap() on to-be-mapped areas before calling this + * function. + * + * RETURNS: + * 0 on success, -errno on failure. + */ +int map_kernel_range_noflush(unsigned long addr, unsigned long size, + pgprot_t prot, struct page **pages) +{ + return vmap_pages_range_noflush(addr, addr + size, prot, pages, PAGE_SHIFT); +} + int map_kernel_range(unsigned long start, unsigned long size, pgprot_t prot, struct page **pages) { @@ -2400,6 +2453,7 @@ static inline void set_area_direct_map(const struct vm_struct *area, { int i; + /* HUGE_VMALLOC passes small pages to set_direct_map */ for (i = 0; i < area->nr_pages; i++) if (page_address(area->pages[i])) set_direct_map(area->pages[i]); @@ -2433,11 +2487,12 @@ static void vm_remove_mappings(struct vm_struct *area, int deallocate_pages) * map. Find the start and end range of the direct mappings to make sure * the vm_unmap_aliases() flush includes the direct map. */ - for (i = 0; i < area->nr_pages; i++) { + for (i = 0; i < area->nr_pages; i += 1U << area->page_order) { unsigned long addr = (unsigned long)page_address(area->pages[i]); if (addr) { + unsigned long page_size = PAGE_SIZE << area->page_order; start = min(addr, start); - end = max(addr + PAGE_SIZE, end); + end = max(addr + page_size, end); flush_dmap = 1; } } @@ -2480,11 +2535,11 @@ static void __vunmap(const void *addr, int deallocate_pages) if (deallocate_pages) { int i; - for (i = 0; i < area->nr_pages; i++) { + for (i = 0; i < area->nr_pages; i += 1U << area->page_order) { struct page *page = area->pages[i]; BUG_ON(!page); - __free_pages(page, 0); + __free_pages(page, area->page_order); } atomic_long_sub(area->nr_pages, &nr_vmalloc_pages); @@ -2623,9 +2678,12 @@ void *vmap(struct page **pages, unsigned int count, EXPORT_SYMBOL(vmap); static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, - pgprot_t prot, int node) + pgprot_t prot, unsigned int page_shift, int node) { struct page **pages; + unsigned long addr = (unsigned long)area->addr; + unsigned long size = get_vm_area_size(area); + unsigned int page_order = page_shift - PAGE_SHIFT; unsigned int nr_pages, array_size, i; const gfp_t nested_gfp = (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO; const gfp_t alloc_mask = gfp_mask | __GFP_NOWARN; @@ -2633,7 +2691,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, 0 : __GFP_HIGHMEM; - nr_pages = get_vm_area_size(area) >> PAGE_SHIFT; + nr_pages = size >> PAGE_SHIFT; array_size = (nr_pages * sizeof(struct page *)); /* Please note that the recursion is strictly bounded. */ @@ -2652,29 +2710,29 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, area->pages = pages; area->nr_pages = nr_pages; + area->page_order = page_order; - for (i = 0; i < area->nr_pages; i++) { + for (i = 0; i < area->nr_pages; i += 1U << page_order) { struct page *page; + int p; - if (node == NUMA_NO_NODE) - page = alloc_page(alloc_mask|highmem_mask); - else - page = alloc_pages_node(node, alloc_mask|highmem_mask, 0); - + page = alloc_pages_node(node, alloc_mask|highmem_mask, page_order); if (unlikely(!page)) { /* Successfully allocated i pages, free them in __vunmap() */ area->nr_pages = i; atomic_long_add(area->nr_pages, &nr_vmalloc_pages); goto fail; } - area->pages[i] = page; + + for (p = 0; p < (1U << page_order); p++) + area->pages[i + p] = page + p; + if (gfpflags_allow_blocking(gfp_mask)) cond_resched(); } atomic_long_add(area->nr_pages, &nr_vmalloc_pages); - if (map_kernel_range((unsigned long)area->addr, get_vm_area_size(area), - prot, pages) < 0) + if (vmap_pages_range(addr, addr + size, prot, pages, page_shift) < 0) goto fail; return area->addr; @@ -2682,7 +2740,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, fail: warn_alloc(gfp_mask, NULL, "vmalloc: allocation failure, allocated %ld of %ld bytes", - (area->nr_pages*PAGE_SIZE), area->size); + (area->nr_pages*PAGE_SIZE), size); __vfree(area->addr); return NULL; } @@ -2713,19 +2771,42 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, struct vm_struct *area; void *addr; unsigned long real_size = size; + unsigned long real_align = align; + unsigned int shift = PAGE_SHIFT; - size = PAGE_ALIGN(size); if (!size || (size >> PAGE_SHIFT) > totalram_pages()) goto fail; - area = __get_vm_area_node(real_size, align, VM_ALLOC | VM_UNINITIALIZED | + if (vmap_allow_huge && (pgprot_val(prot) == pgprot_val(PAGE_KERNEL))) { + unsigned long size_per_node; + + /* + * Try huge pages. Only try for PAGE_KERNEL allocations, + * others like modules don't yet expect huge pages in + * their allocations due to apply_to_page_range not + * supporting them. + */ + + size_per_node = size; + if (node == NUMA_NO_NODE) + size_per_node /= num_online_nodes(); + if (size_per_node >= PMD_SIZE) { + shift = PMD_SHIFT; + align = max(real_align, 1UL << shift); + size = ALIGN(real_size, 1UL << shift); + } + } + +again: + size = PAGE_ALIGN(size); + area = __get_vm_area_node(size, align, VM_ALLOC | VM_UNINITIALIZED | vm_flags, start, end, node, gfp_mask, caller); if (!area) goto fail; - addr = __vmalloc_area_node(area, gfp_mask, prot, node); + addr = __vmalloc_area_node(area, gfp_mask, prot, shift, node); if (!addr) - return NULL; + goto fail; /* * In this function, newly allocated vm_struct has VM_UNINITIALIZED @@ -2739,8 +2820,19 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, return addr; fail: - warn_alloc(gfp_mask, NULL, + if (shift > PAGE_SHIFT) { + free_vm_area(area); + shift = PAGE_SHIFT; + align = real_align; + size = real_size; + goto again; + } + + if (!area) { + /* Warn for area allocation, page allocations already warn */ + warn_alloc(gfp_mask, NULL, "vmalloc: allocation failure: %lu bytes", real_size); + } return NULL; } @@ -3739,7 +3831,7 @@ static int s_show(struct seq_file *m, void *p) seq_printf(m, " %pS", v->caller); if (v->nr_pages) - seq_printf(m, " pages=%d", v->nr_pages); + seq_printf(m, " pages=%d order=%d", v->nr_pages, v->page_order); if (v->phys_addr) seq_printf(m, " phys=%pa", &v->phys_addr); From patchwork Fri Aug 21 15:12:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11729747 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6476D109B for ; Fri, 21 Aug 2020 15:13:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3186C207C3 for ; Fri, 21 Aug 2020 15:13:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ABKo10dj" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3186C207C3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5FB288D003B; Fri, 21 Aug 2020 11:13:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 584D18D0022; Fri, 21 Aug 2020 11:13:42 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 55DD18D003B; Fri, 21 Aug 2020 11:13:41 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0093.hostedemail.com [216.40.44.93]) by kanga.kvack.org (Postfix) with ESMTP id 37F028D0022 for ; Fri, 21 Aug 2020 11:13:41 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 7EEDC180AD83B for ; Fri, 21 Aug 2020 15:13:40 +0000 (UTC) X-FDA: 77174920200.27.tiger54_161529a2703a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin27.hostedemail.com (Postfix) with ESMTP id EB8A83D679 for ; Fri, 21 Aug 2020 15:13:37 +0000 (UTC) X-Spam-Summary: 1,0,0,fc608a1f7ee2e5c7,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:1801:2194:2199:2393:2559:2562:3138:3139:3140:3141:3142:3352:3867:4250:4321:4605:5007:6119:6261:6642:6653:7514:9413:10004:11026:11232:11473:11658:11914:12043:12297:12438:12517:12519:12555:12895:13069:13311:13357:13894:14181:14384:14394:14581:14687:14721:21080:21444:21451:21627:21666:30054:30089,0,RBL:209.85.215.193:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04y8rhj1q4yf3ppqx55k68w8gm9qrycddu3htbxez758sx193u6u1y5gmd487si.g7zq36yz9assntd9zsmgnr44kf8w9u1bgjdsqnd1m5774imzmteebkyo1dbns3g.h-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: tiger54_161529a2703a X-Filterd-Recvd-Size: 4484 Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com [209.85.215.193]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Fri, 21 Aug 2020 15:13:36 +0000 (UTC) Received: by mail-pg1-f193.google.com with SMTP id 189so1120989pgg.13 for ; Fri, 21 Aug 2020 08:13:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Gv+PP/wjBFW/QutbN7MDl66u1IlF9/srIH5l1WMUz8Y=; b=ABKo10djrnU4pGClhFKavAXdh54V4mnbGrFg88b4z5GAyvOzXK9Eatymo9uLkZeCkl 2KeYeAal4TC74hqFqtqWVYr+S9XoSTUei+yPXR1nwbKG2HskYDeK468qveJL66dPV04W CRRc30oZdDSysRKYpri0vtoFTrTe8+adB1CPOGf48fZZSRnz4o0sJ5mf6x5a6cA0ofL4 19YSrxZrL+fkPQUkLYIOkcXIc3/ENsTQpySXISI/iwWDCu4uxqzoye8zVjzI82fpwiuK 4hZfxA7JJWC/dklgIjV7wahwIdIgahUUgl+QKLx6L3g/KymHmFwij0sKPdvlpJphHxxv 4L4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Gv+PP/wjBFW/QutbN7MDl66u1IlF9/srIH5l1WMUz8Y=; b=uAeksGoHi6X6HJ6Ir907EuR+VSCvTyrrp0so984iGFOcBp2PHCYAJJlLNCHO+3CEJU VU3fJeubShd3gwbYCziofj8tb8icRPuQUbwh60TFk48jchGNK1a6IgmxXYLgFE983X7A nLgdY5JoNPgWYdIOTrnZbIIAaWQQO/c1qfezH7Ji3dGFGU1O38FC8G8mwcp2hMMdO9kq W26FvFDmmEfLCDWRR+vi85zQrA4PqtdwJSm8CJQaj6LQrZk/1uMsCUYjkM4k7yyeOW8E 73l7DeVVu531Z6h0nAyIqwCcXNjxbz+cTmSaLEmAsScMuR35Fl1v83meRvt3UXzv2dV+ DKLw== X-Gm-Message-State: AOAM53371kbHXNmoCTds4Vfs6tJt56txij1TPO7xfy6GXQvoSeEnG2kZ a8l1DfqmkAVgh1D+kh9a29HRTdnbgDU= X-Google-Smtp-Source: ABdhPJwItWdQvR8DxTON4DygzvNWnRtG1VJd3vYg6XhNeG9Dmt4XVNKDtCB8WfPdSWvlVqPVfP71sA== X-Received: by 2002:a63:584e:: with SMTP id i14mr2637896pgm.433.1598022815728; Fri, 21 Aug 2020 08:13:35 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id s8sm3126985pfc.122.2020.08.21.08.13.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Aug 2020 08:13:35 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy Subject: [PATCH v6 12/12] powerpc/64s/radix: Enable huge vmalloc mappings Date: Sat, 22 Aug 2020 01:12:16 +1000 Message-Id: <20200821151216.1005117-13-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200821151216.1005117-1-npiggin@gmail.com> References: <20200821151216.1005117-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: EB8A83D679 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Cc: linuxppc-dev@lists.ozlabs.org Signed-off-by: Nicholas Piggin --- Documentation/admin-guide/kernel-parameters.txt | 2 ++ arch/powerpc/Kconfig | 1 + 2 files changed, 3 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index bdc1f33fd3d1..6f0b41289a90 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3190,6 +3190,8 @@ nohugeiomap [KNL,X86,PPC] Disable kernel huge I/O mappings. + nohugevmalloc [PPC] Disable kernel huge vmalloc mappings. + nosmt [KNL,S390] Disable symmetric multithreading (SMT). Equivalent to smt=1. diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 1f48bbfb3ce9..9171d25ad7dc 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -175,6 +175,7 @@ config PPC select GENERIC_TIME_VSYSCALL select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMAP if PPC_BOOK3S_64 && PPC_RADIX_MMU + select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_KASAN if PPC32 && PPC_PAGE_SHIFT <= 14 select HAVE_ARCH_KASAN_VMALLOC if PPC32 && PPC_PAGE_SHIFT <= 14