From patchwork Sun Aug 16 09:08:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11716025 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 85A9F138C for ; Sun, 16 Aug 2020 09:09:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5277D20674 for ; Sun, 16 Aug 2020 09:09:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="EA7yKwrJ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5277D20674 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 814F56B0003; Sun, 16 Aug 2020 05:09:24 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 79B726B0005; Sun, 16 Aug 2020 05:09:24 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 663E36B0007; Sun, 16 Aug 2020 05:09:24 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0129.hostedemail.com [216.40.44.129]) by kanga.kvack.org (Postfix) with ESMTP id 4CFB36B0003 for ; Sun, 16 Aug 2020 05:09:24 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 10340364B for ; Sun, 16 Aug 2020 09:09:24 +0000 (UTC) X-FDA: 77155858248.06.trade35_5602af22700c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id DA68C10043AB5 for ; Sun, 16 Aug 2020 09:09:23 +0000 (UTC) X-Spam-Summary: 1,0,0,7ebeba319f860fa7,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1535:1543:1711:1730:1747:1777:1792:2393:2553:2559:2562:2693:3138:3139:3140:3141:3142:3308:3355:3865:3867:3868:3871:3872:3874:4117:4250:4321:5007:6119:6261:6653:6742:7514:7903:9036:9413:10004:11026:11232:11473:11658:11914:12043:12219:12291:12295:12296:12297:12438:12517:12519:12555:12683:12895:12986:13161:13229:13894:14181:14394:14687:14721:21080:21433:21444:21451:21627:21666:21990:30003:30054:30062:30070:30074:30090,0,RBL:209.85.210.194:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100;04yr4eq31r6z8o9c6mtyzr59qbzteypwu1uepyi5f98ekzhbpbjw988g7yuwf7q.16pf537emigd1cnrgfxw4da7nsuwipafthxmuuio8amwygrfniwwyrnahicpsyh.1-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: trade35_5602af22700c X-Filterd-Recvd-Size: 6533 Received: from mail-pf1-f194.google.com (mail-pf1-f194.google.com [209.85.210.194]) by imf43.hostedemail.com (Postfix) with ESMTP for ; Sun, 16 Aug 2020 09:09:23 +0000 (UTC) Received: by mail-pf1-f194.google.com with SMTP id u128so6673319pfb.6 for ; Sun, 16 Aug 2020 02:09:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+wqRNPVVkxxZAVOe5cjf4OdJxPseOUgKHXYAzqLL+cs=; b=EA7yKwrJ6XMClXzMrMaldEGD6T0ONt/b7yucodcpi2bwP1vk+eiWi4GJ0TmCJsOLNB p7Wlwa96VnXwdnOrBaddEuriy3Eb0sUQMuHy7YjQbUmgTiqkk5LCJYeZIY0v0vOYGTY1 3cPDR77QB9eua4GSDy4zUL730tY9ZDbMg+SXtYUKtQUZ/+DeffhpRACd2dta2Z9+KrZI k/ygx8c5qU1iWmEliqszYl8TQ4EmYUlW7RGM+AXjaj6FHIliHnQSkqB7tFI1LLnZFbAr znQlNJBbdlX0bJq6EixSSIs1uju+X6XZDrd3oZDKOLOo7FkuglAmK3MDwSptKgUZ5ceI Qieg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+wqRNPVVkxxZAVOe5cjf4OdJxPseOUgKHXYAzqLL+cs=; b=naQVD9VN2i6Q0ztN48hy4P0qHzfo6tJnX+Bi8zUZCUVB+/oa+MugD7aLyCHVIf3Yy8 5aTrpV4D9w85Q2oLJ0UJGlWx51aqVHm8OMntjDWsgydXctlldxMUMK4VU8r7RuWgDeo/ 5rayB8XZRmQb3kkD4S9yTaV3NHHWo6rLFxfA8a/7V7AxAUahGwc/Cz5+sj/B1Mg93WLd 75Ha+FuR2CQ6uDMphzMrBgg8lhsjrwk75K2HNe+HsxJUqHvYlf6GoeMdXU2VmDBHepJ1 9G4sTpJC0uF0VdO39i4GCDTbY6nV/GIMx5CAE+roGl6hGWoy+sHp8moM2nt2Z+nYwEGB +6uQ== X-Gm-Message-State: AOAM533129mBBC2HIaavhf47rYvd6SMbnYlHR1/LWUI1cDbRBDxbqhsD lovIeQDykPThtp1ePCFEKmmrXhny2Bs= X-Google-Smtp-Source: ABdhPJxXLdCsefTkkLAS/8ct2D5KIK9jTaH4NgtRhJ5fzMvhgKO+ZM9cbHGzYfxmkSGIyTTIUuOt/A== X-Received: by 2002:a63:e245:: with SMTP id y5mr2171726pgj.51.1597568962365; Sun, 16 Aug 2020 02:09:22 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (193-116-193-175.tpgi.com.au. [193.116.193.175]) by smtp.gmail.com with ESMTPSA id o19sm12768369pjs.8.2020.08.16.02.09.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Aug 2020 02:09:21 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" , Zefan Li , Jonathan Cameron Subject: [PATCH v4 1/8] mm/vmalloc: fix vmalloc_to_page for huge vmap mappings Date: Sun, 16 Aug 2020 19:08:57 +1000 Message-Id: <20200816090904.83947-2-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200816090904.83947-1-npiggin@gmail.com> References: <20200816090904.83947-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: DA68C10043AB5 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vmalloc_to_page returns NULL for addresses mapped by larger pages[*]. Whether or not a vmap is huge depends on the architecture details, alignments, boot options, etc., which the caller can not be expected to know. Therefore HUGE_VMAP is a regression for vmalloc_to_page. This change teaches vmalloc_to_page about larger pages, and returns the struct page that corresponds to the offset within the large page. This makes the API agnostic to mapping implementation details. [*] As explained by commit 029c54b095995 ("mm/vmalloc.c: huge-vmap: fail gracefully on unexpected huge vmap mappings") Signed-off-by: Nicholas Piggin --- mm/vmalloc.c | 40 ++++++++++++++++++++++++++-------------- 1 file changed, 26 insertions(+), 14 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index b482d240f9a2..49f225b0f855 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -38,6 +38,7 @@ #include #include +#include #include #include @@ -343,7 +344,9 @@ int is_vmalloc_or_module_addr(const void *x) } /* - * Walk a vmap address to the struct page it maps. + * Walk a vmap address to the struct page it maps. Huge vmap mappings will + * return the tail page that corresponds to the base page address, which + * matches small vmap mappings. */ struct page *vmalloc_to_page(const void *vmalloc_addr) { @@ -363,25 +366,33 @@ struct page *vmalloc_to_page(const void *vmalloc_addr) if (pgd_none(*pgd)) return NULL; + if (WARN_ON_ONCE(pgd_leaf(*pgd))) + return NULL; /* XXX: no allowance for huge pgd */ + if (WARN_ON_ONCE(pgd_bad(*pgd))) + return NULL; + p4d = p4d_offset(pgd, addr); if (p4d_none(*p4d)) return NULL; - pud = pud_offset(p4d, addr); + if (p4d_leaf(*p4d)) + return p4d_page(*p4d) + ((addr & ~P4D_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(p4d_bad(*p4d))) + return NULL; - /* - * Don't dereference bad PUD or PMD (below) entries. This will also - * identify huge mappings, which we may encounter on architectures - * that define CONFIG_HAVE_ARCH_HUGE_VMAP=y. Such regions will be - * identified as vmalloc addresses by is_vmalloc_addr(), but are - * not [unambiguously] associated with a struct page, so there is - * no correct value to return for them. - */ - WARN_ON_ONCE(pud_bad(*pud)); - if (pud_none(*pud) || pud_bad(*pud)) + pud = pud_offset(p4d, addr); + if (pud_none(*pud)) + return NULL; + if (pud_leaf(*pud)) + return pud_page(*pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(pud_bad(*pud))) return NULL; + pmd = pmd_offset(pud, addr); - WARN_ON_ONCE(pmd_bad(*pmd)); - if (pmd_none(*pmd) || pmd_bad(*pmd)) + if (pmd_none(*pmd)) + return NULL; + if (pmd_leaf(*pmd)) + return pmd_page(*pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(pmd_bad(*pmd))) return NULL; ptep = pte_offset_map(pmd, addr); @@ -389,6 +400,7 @@ struct page *vmalloc_to_page(const void *vmalloc_addr) if (pte_present(pte)) page = pte_page(pte); pte_unmap(ptep); + return page; } EXPORT_SYMBOL(vmalloc_to_page); From patchwork Sun Aug 16 09:08:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11716027 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 79F3414E3 for ; Sun, 16 Aug 2020 09:09:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 46E05206DA for ; Sun, 16 Aug 2020 09:09:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="b9r2FrUb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 46E05206DA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 88BDF6B0005; Sun, 16 Aug 2020 05:09:30 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 811546B0007; Sun, 16 Aug 2020 05:09:30 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 664116B0008; Sun, 16 Aug 2020 05:09:30 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0177.hostedemail.com [216.40.44.177]) by kanga.kvack.org (Postfix) with ESMTP id 4A9EC6B0005 for ; Sun, 16 Aug 2020 05:09:30 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 0A9A5180AD820 for ; Sun, 16 Aug 2020 09:09:30 +0000 (UTC) X-FDA: 77155858500.28.blade28_0f130522700c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id D214A6C36 for ; Sun, 16 Aug 2020 09:09:29 +0000 (UTC) X-Spam-Summary: 1,0,0,a9a3bbeff89958b8,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1515:1535:1542:1711:1730:1747:1777:1792:2393:2559:2562:2898:3138:3139:3140:3141:3142:3354:3876:3877:4117:4321:5007:6114:6119:6261:6642:6653:6742:7514:9413:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:12986:13255:13894:14096:14110:14181:14394:14687:14721:21080:21444:21451:21611:21627:21666:21990:30054:30070,0,RBL:209.85.215.196:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04y88o3o3q41f688hih77dp4gn94eoc8ha73weafrcdj8f1an8pmb7rddtuhwaf.mtrcedn96h53dqjjmd51asrhr4b6bke8qzhcsgttjw738robuwdfjzmmw19tdmr.o-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: blade28_0f130522700c X-Filterd-Recvd-Size: 6200 Received: from mail-pg1-f196.google.com (mail-pg1-f196.google.com [209.85.215.196]) by imf23.hostedemail.com (Postfix) with ESMTP for ; Sun, 16 Aug 2020 09:09:29 +0000 (UTC) Received: by mail-pg1-f196.google.com with SMTP id s15so6599707pgc.8 for ; Sun, 16 Aug 2020 02:09:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=mxm/C2mC0786WMQTE10aJFvTg/Tw0xwUgIx5I5X+hjE=; b=b9r2FrUb9Vyhol+l3W9WoRcRqwuEhJjICuC8voZ71tdHvf9AxDjiBeHUPoJzDXz918 GfC0Lqqv1aLzAMXtaXVGWxJ6//psWOIy2I4sji++lJ4BdiP0PWs10DaVhUxrs/zwIvhB kwIE3+NTA1aXJvKR4AsAcVvlRBup95ZJs0o8J9ggH+EDr5FqvguEbk6SCdoc6r18RYLb qMc3ECitpN/FuWjc9PL9bKG4zy+91aFxbDPlqpmB6w4xTa9WdDDN2iuZL1gIDQqtr6Ak fX7yiUhYUaJ841lhLFcKD1xkRiYDVw4mI1xoWm//U3az5Xq+hLgWbkAGx8JSu9AnOfQ7 JqvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mxm/C2mC0786WMQTE10aJFvTg/Tw0xwUgIx5I5X+hjE=; b=W4MDJtRJolVyfkkjVc5NjwgmFj4Jwx3Z6cZMCnDsEDPw4TJCgnDIpFwCFlhMUYv8rH o4h/eI44sOE8+C/60hjn3hejFV1BKhug0xs991P1+Ou1UKfx83ILEAMbKDk9u7+1ymf5 l3rVCfujd5JiQunGASlTmoAsyaFHNq7R5Hgl+wY80jP/KkR83wRZ7nWx4totGUo79d6v 3+SIzfxpir1Ymm2WwwJZ1/dTVc9Q8YuAaw3TRGUwztux5fRClc6Ed+Hj1V9cARc3tZ07 7CMy64ulpHLHckg3COJWTw3mDZh0XiHnFc3Nc0yTjBYEcQD7QDApS05EC9kphGfBmaFx nYRQ== X-Gm-Message-State: AOAM532stqa501BVYeGSm4sBE//GwCeEwW/AUnHYFcfA1qHQ/4ZkOjWr gl+zLt3sMU5mwFoTTaLTWvgYwo05wxA= X-Google-Smtp-Source: ABdhPJwxOvFbG/ZjwUp2j/jXOU/b9ImNMZIMN5ReacxCD9gMRBxoKCQ/eh7ex/628GY5c5MYbRsgNg== X-Received: by 2002:aa7:9a1c:: with SMTP id w28mr4426784pfj.116.1597568968280; Sun, 16 Aug 2020 02:09:28 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (193-116-193-175.tpgi.com.au. [193.116.193.175]) by smtp.gmail.com with ESMTPSA id o19sm12768369pjs.8.2020.08.16.02.09.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Aug 2020 02:09:27 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" , Zefan Li , Jonathan Cameron Subject: [PATCH v4 2/8] mm: apply_to_pte_range warn and fail if a large pte is encountered Date: Sun, 16 Aug 2020 19:08:58 +1000 Message-Id: <20200816090904.83947-3-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200816090904.83947-1-npiggin@gmail.com> References: <20200816090904.83947-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: D214A6C36 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Signed-off-by: Nicholas Piggin --- mm/memory.c | 60 +++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 44 insertions(+), 16 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index c39a13b09602..1d5f3093c249 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2260,13 +2260,20 @@ static int apply_to_pmd_range(struct mm_struct *mm, pud_t *pud, } do { next = pmd_addr_end(addr, end); - if (create || !pmd_none_or_clear_bad(pmd)) { - err = apply_to_pte_range(mm, pmd, addr, next, fn, data, - create); - if (err) - break; + if (pmd_none(*pmd) && !create) + continue; + if (WARN_ON_ONCE(pmd_leaf(*pmd))) + return -EINVAL; + if (WARN_ON_ONCE(pmd_bad(*pmd))) { + if (!create) + continue; + pmd_clear_bad(pmd); } + err = apply_to_pte_range(mm, pmd, addr, next, fn, data, create); + if (err) + break; } while (pmd++, addr = next, addr != end); + return err; } @@ -2287,13 +2294,20 @@ static int apply_to_pud_range(struct mm_struct *mm, p4d_t *p4d, } do { next = pud_addr_end(addr, end); - if (create || !pud_none_or_clear_bad(pud)) { - err = apply_to_pmd_range(mm, pud, addr, next, fn, data, - create); - if (err) - break; + if (pud_none(*pud) && !create) + continue; + if (WARN_ON_ONCE(pud_leaf(*pud))) + return -EINVAL; + if (WARN_ON_ONCE(pud_bad(*pud))) { + if (!create) + continue; + pud_clear_bad(pud); } + err = apply_to_pmd_range(mm, pud, addr, next, fn, data, create); + if (err) + break; } while (pud++, addr = next, addr != end); + return err; } @@ -2314,13 +2328,20 @@ static int apply_to_p4d_range(struct mm_struct *mm, pgd_t *pgd, } do { next = p4d_addr_end(addr, end); - if (create || !p4d_none_or_clear_bad(p4d)) { - err = apply_to_pud_range(mm, p4d, addr, next, fn, data, - create); - if (err) - break; + if (p4d_none(*p4d) && !create) + continue; + if (WARN_ON_ONCE(p4d_leaf(*p4d))) + return -EINVAL; + if (WARN_ON_ONCE(p4d_bad(*p4d))) { + if (!create) + continue; + p4d_clear_bad(p4d); } + err = apply_to_pud_range(mm, p4d, addr, next, fn, data, create); + if (err) + break; } while (p4d++, addr = next, addr != end); + return err; } @@ -2339,8 +2360,15 @@ static int __apply_to_page_range(struct mm_struct *mm, unsigned long addr, pgd = pgd_offset(mm, addr); do { next = pgd_addr_end(addr, end); - if (!create && pgd_none_or_clear_bad(pgd)) + if (pgd_none(*pgd) && !create) continue; + if (WARN_ON_ONCE(pgd_leaf(*pgd))) + return -EINVAL; + if (WARN_ON_ONCE(pgd_bad(*pgd))) { + if (!create) + continue; + pgd_clear_bad(pgd); + } err = apply_to_p4d_range(mm, pgd, addr, next, fn, data, create); if (err) break; From patchwork Sun Aug 16 09:08:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11716029 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6FFEF138C for ; Sun, 16 Aug 2020 09:09:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 32CCB206DA for ; Sun, 16 Aug 2020 09:09:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="a9eH2Qud" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 32CCB206DA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6DA4D6B0007; Sun, 16 Aug 2020 05:09:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 663A56B0008; Sun, 16 Aug 2020 05:09:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 535B56B000A; Sun, 16 Aug 2020 05:09:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0240.hostedemail.com [216.40.44.240]) by kanga.kvack.org (Postfix) with ESMTP id 3B69F6B0007 for ; Sun, 16 Aug 2020 05:09:36 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id F37E0181AEF30 for ; Sun, 16 Aug 2020 09:09:35 +0000 (UTC) X-FDA: 77155858752.22.men58_55022902700c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin22.hostedemail.com (Postfix) with ESMTP id C728C18038C3C for ; Sun, 16 Aug 2020 09:09:35 +0000 (UTC) X-Spam-Summary: 1,0,0,a38618af589dd5d8,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3355:3865:3867:3872:4117:4321:5007:6261:6653:6742:7514:8603:9413:10004:11026:11473:11658:11914:12043:12297:12438:12517:12519:12555:12895:12986:13161:13229:13894:14096:14181:14394:14687:14721:21080:21444:21451:21627:21666:21990:30003:30054,0,RBL:209.85.216.68:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100;04ygd8icnzrzehoc7yu6ddc34z31gypm7h3jcjuakakfct3okjhqtqdb3qdxepp.cdos6t96pq8egchuruwbnqq8qf15tnqupi97ffyegf4mpp4rhoeu9bc6e3cxk35.e-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: men58_55022902700c X-Filterd-Recvd-Size: 6880 Received: from mail-pj1-f68.google.com (mail-pj1-f68.google.com [209.85.216.68]) by imf48.hostedemail.com (Postfix) with ESMTP for ; Sun, 16 Aug 2020 09:09:35 +0000 (UTC) Received: by mail-pj1-f68.google.com with SMTP id e4so6320939pjd.0 for ; Sun, 16 Aug 2020 02:09:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xoa50mjKdO+6rEo7JFgsM1iktECeAufQruBypQNx5uU=; b=a9eH2Qud8eaxaDEyJ6Ox3nV2dEQGPtlOBvhHiUtqVX8FJWPdpCeQ/7Z+MH3ABFUlzY hGmX62DEx+PK4neClmvN49t6HjShSviRro/UAGDtJPSR6V3MbsPNXrDz8S4WZTYDrlfr bCc4QjNnpMrwwl0bn98+LNDPdIcDqScjmU09UuXnxnCUFPo1/sZPQWzrH75XUEL2HCmT 7zVlApSM3TvG4rV6nJQRTMHPF7VgTZL/WOdO4pwkBBtsA41oUEymfx9+I+gV7FC0GiBD FRtkqpWRR3h3zLZqRnoL0QTqf6iqBTeliFHIGWSiG4+f+zGXSw2HK5l/O3HF53m4tE+M HQaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xoa50mjKdO+6rEo7JFgsM1iktECeAufQruBypQNx5uU=; b=WyFkpTRXK7Gl522pPYUkyUrrHb/MY2jKKePV/zYY+hx+RdB65cKsqYM5sDD9II9/3V 49jwDQFjzaChaovt5mKZKpHltMacgl5ZemPsfqQT4tflzDYbG8u2q2ARbtPeuMSt9VRw oriGQI43jL0pS3b6ois+njOge0FyDitXENJs9nDu8DvWCH6s9CuBGgCyTzVxKFsKxL5f 8oiMWhpLww1F9ZuGE7nFvRfPhDCpBsI6mIoqYB85uv3OhypJHMHpJIzFZcaN/yeYGlga j9lZS2Mq6yK5zrP0cLdvM6/CeR2RxUm2D5pSWO6wnkO5ROLQIS7bvjr07bKv99bDQSDf ++RA== X-Gm-Message-State: AOAM533R8CGnJMWhVRNsJgFg8ecKy4i57Locz6yG2IuAKAzSl+aAegit ELx8hn/W1jg0dM8s2iF9urrbzMg+ZnE= X-Google-Smtp-Source: ABdhPJxjxbOaxbaEbI6j9pxumC5wRGiGlNLcDB22u2QtqVpRvdt2rZbK1BklnxLXaykWubkHnlaE0Q== X-Received: by 2002:a17:902:cb0f:: with SMTP id c15mr7427207ply.85.1597568974143; Sun, 16 Aug 2020 02:09:34 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (193-116-193-175.tpgi.com.au. [193.116.193.175]) by smtp.gmail.com with ESMTPSA id o19sm12768369pjs.8.2020.08.16.02.09.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Aug 2020 02:09:33 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" , Zefan Li , Jonathan Cameron Subject: [PATCH v4 3/8] mm/vmalloc: rename vmap_*_range vmap_pages_*_range Date: Sun, 16 Aug 2020 19:08:59 +1000 Message-Id: <20200816090904.83947-4-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200816090904.83947-1-npiggin@gmail.com> References: <20200816090904.83947-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: C728C18038C3C X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The vmalloc mapper operates on a struct page * array rather than a linear physical address, re-name it to make this distinction clear. Signed-off-by: Nicholas Piggin --- mm/vmalloc.c | 28 ++++++++++++---------------- 1 file changed, 12 insertions(+), 16 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 49f225b0f855..3a1e45fd1626 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -190,9 +190,8 @@ void unmap_kernel_range_noflush(unsigned long start, unsigned long size) arch_sync_kernel_mappings(start, end); } -static int vmap_pte_range(pmd_t *pmd, unsigned long addr, - unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) +static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { pte_t *pte; @@ -218,9 +217,8 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, return 0; } -static int vmap_pmd_range(pud_t *pud, unsigned long addr, - unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) +static int vmap_pages_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { pmd_t *pmd; unsigned long next; @@ -230,15 +228,14 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr, return -ENOMEM; do { next = pmd_addr_end(addr, end); - if (vmap_pte_range(pmd, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pte_range(pmd, addr, next, prot, pages, nr, mask)) return -ENOMEM; } while (pmd++, addr = next, addr != end); return 0; } -static int vmap_pud_range(p4d_t *p4d, unsigned long addr, - unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) +static int vmap_pages_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { pud_t *pud; unsigned long next; @@ -248,15 +245,14 @@ static int vmap_pud_range(p4d_t *p4d, unsigned long addr, return -ENOMEM; do { next = pud_addr_end(addr, end); - if (vmap_pmd_range(pud, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pmd_range(pud, addr, next, prot, pages, nr, mask)) return -ENOMEM; } while (pud++, addr = next, addr != end); return 0; } -static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, - unsigned long end, pgprot_t prot, struct page **pages, int *nr, - pgtbl_mod_mask *mask) +static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { p4d_t *p4d; unsigned long next; @@ -266,7 +262,7 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, return -ENOMEM; do { next = p4d_addr_end(addr, end); - if (vmap_pud_range(p4d, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pud_range(p4d, addr, next, prot, pages, nr, mask)) return -ENOMEM; } while (p4d++, addr = next, addr != end); return 0; @@ -307,7 +303,7 @@ int map_kernel_range_noflush(unsigned long addr, unsigned long size, next = pgd_addr_end(addr, end); if (pgd_bad(*pgd)) mask |= PGTBL_PGD_MODIFIED; - err = vmap_p4d_range(pgd, addr, next, prot, pages, &nr, &mask); + err = vmap_pages_p4d_range(pgd, addr, next, prot, pages, &nr, &mask); if (err) return err; } while (pgd++, addr = next, addr != end); From patchwork Sun Aug 16 09:09:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11716033 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 60C0214E3 for ; Sun, 16 Aug 2020 09:09:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1F78E20738 for ; Sun, 16 Aug 2020 09:09:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Sx4pg6cb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1F78E20738 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4B7806B0008; Sun, 16 Aug 2020 05:09:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 441006B000A; Sun, 16 Aug 2020 05:09:42 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3089A6B000C; Sun, 16 Aug 2020 05:09:42 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0211.hostedemail.com [216.40.44.211]) by kanga.kvack.org (Postfix) with ESMTP id 1468E6B0008 for ; Sun, 16 Aug 2020 05:09:42 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id D58A93635 for ; Sun, 16 Aug 2020 09:09:41 +0000 (UTC) X-FDA: 77155858962.06.pig26_3717b572700c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id A951410043AB5 for ; Sun, 16 Aug 2020 09:09:41 +0000 (UTC) X-Spam-Summary: 1,0,0,7a5cc57bd9380f7e,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:2:41:69:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1730:1747:1777:1792:1981:2194:2196:2199:2200:2393:2559:2562:2693:2898:3138:3139:3140:3141:3142:3867:3868:3870:3872:4049:4120:4385:4605:5007:6119:6261:6653:6742:7514:7875:8603:9413:10004:11026:11473:11658:11914:12043:12291:12297:12438:12517:12519:12555:12683:12895:13894:14096:14110:14394:14687:21080:21444:21451:21627:21666:21990:30054:30070,0,RBL:209.85.210.196:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100;04ygiy1pzcew1ispw1jhujx5btdraopkk8j1fcj1p4rxy1xzjbaqk81xn86zx35.4qoe7fpts474bbqebnahn3yioauzz4fg1omebef5b3cepqwaadmz3cmmcmhgr9g.6-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: pig26_3717b572700c X-Filterd-Recvd-Size: 9435 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) by imf19.hostedemail.com (Postfix) with ESMTP for ; Sun, 16 Aug 2020 09:09:41 +0000 (UTC) Received: by mail-pf1-f196.google.com with SMTP id x25so6680980pff.4 for ; Sun, 16 Aug 2020 02:09:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=zgciMtrhUFyjwGUVG4/y+NlOfAX4ULiG8JBMTv5iRYk=; b=Sx4pg6cbpRc7TLtLsYf6QS9/V3eSzJw243m7oVuutLFfRoykNyppQoVF+KP8ckVRZI wfQE8vKHCtcxI6nu1EvkQH9aabReq7AOvteeKC4uYRn0ImOcz1tXyHIQGdJqzz3fqqFd 8WS2qRtCkI35FPj0Q8b6xqwc+dEQGCLoAb1zJGeAQS/YvJ8twi2P2VGEUDW2FAxa+TFV fUCtPV+VGosZf9ILnCMdzVlKauYa05gMYcc/5RJ62/Bd9xnriIdTcFEanDwSZKAj7//V C2G4WPqjN+6gu3/NQvy0GD7A0xIQhAPEEF43yQ7Xw9+LwYYDETkpaG84bxIMCb0WJbUE LBnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zgciMtrhUFyjwGUVG4/y+NlOfAX4ULiG8JBMTv5iRYk=; b=gX8hAOYkstxTh34xGcuIyAa6JvK35ElGw/nIkbCsbKhmpViqMNrDxwia3cx5YC939R 0b42ExD+/LZp7vqZq7gYCwmD4oFEr0gwK6eFntMQyruOJ5JJmHn8ZURyJOuZtdSr2T55 ANQZ2sbPbCvu98A18Y1iC2rnpmvZrj5Pj7HvmCWq5CZADGAbwaZWKRozoiqx+IWJInyA m213i2FU6YtNeR0egfk5Y4TRdE/WMqNOynoX4xPmhSQ5nM3x1RRJw8961LEzw6HcWuZM CsrQCBdvmrTDuMw0MZQbTvJLJ8BnjdPRalYMvwl0Hh1YPkqofARLG5uO3AnkW700SqcT 6lzw== X-Gm-Message-State: AOAM5308n4tlAbr5QLeI3w7G5OQaGjNIst3bxzEomI4yMufMscXeyS9M cr2z7GXu7ZNqfn5MvcE9Uv3W3jF7JuQ= X-Google-Smtp-Source: ABdhPJzKawRgG36izl7nSxDOZkM0Of2POzBfKZla6VMvJ5K5Q5uX/ljQ/tfcoEX+aTZsibK3iGuCtQ== X-Received: by 2002:a63:8f1c:: with SMTP id n28mr3822208pgd.330.1597568980154; Sun, 16 Aug 2020 02:09:40 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (193-116-193-175.tpgi.com.au. [193.116.193.175]) by smtp.gmail.com with ESMTPSA id o19sm12768369pjs.8.2020.08.16.02.09.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Aug 2020 02:09:39 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" , Zefan Li , Jonathan Cameron Subject: [PATCH v4 4/8] lib/ioremap: rename ioremap_*_range to vmap_*_range Date: Sun, 16 Aug 2020 19:09:00 +1000 Message-Id: <20200816090904.83947-5-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200816090904.83947-1-npiggin@gmail.com> References: <20200816090904.83947-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: A951410043AB5 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This will be moved to mm/ and used as a generic kernel virtual mapping function, so re-name it in preparation. Signed-off-by: Nicholas Piggin --- mm/ioremap.c | 55 ++++++++++++++++++++++------------------------------ 1 file changed, 23 insertions(+), 32 deletions(-) diff --git a/mm/ioremap.c b/mm/ioremap.c index 5fa1ab41d152..6016ae3227ad 100644 --- a/mm/ioremap.c +++ b/mm/ioremap.c @@ -61,9 +61,8 @@ static inline int ioremap_pud_enabled(void) { return 0; } static inline int ioremap_pmd_enabled(void) { return 0; } #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ -static int ioremap_pte_range(pmd_t *pmd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, pgtbl_mod_mask *mask) { pte_t *pte; u64 pfn; @@ -81,9 +80,8 @@ static int ioremap_pte_range(pmd_t *pmd, unsigned long addr, return 0; } -static int ioremap_try_huge_pmd(pmd_t *pmd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) +static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { if (!ioremap_pmd_enabled()) return 0; @@ -103,9 +101,8 @@ static int ioremap_try_huge_pmd(pmd_t *pmd, unsigned long addr, return pmd_set_huge(pmd, phys_addr, prot); } -static inline int ioremap_pmd_range(pud_t *pud, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, pgtbl_mod_mask *mask) { pmd_t *pmd; unsigned long next; @@ -116,20 +113,19 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long addr, do { next = pmd_addr_end(addr, end); - if (ioremap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) { *mask |= PGTBL_PMD_MODIFIED; continue; } - if (ioremap_pte_range(pmd, addr, next, phys_addr, prot, mask)) + if (vmap_pte_range(pmd, addr, next, phys_addr, prot, mask)) return -ENOMEM; } while (pmd++, phys_addr += (next - addr), addr = next, addr != end); return 0; } -static int ioremap_try_huge_pud(pud_t *pud, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) +static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { if (!ioremap_pud_enabled()) return 0; @@ -149,9 +145,8 @@ static int ioremap_try_huge_pud(pud_t *pud, unsigned long addr, return pud_set_huge(pud, phys_addr, prot); } -static inline int ioremap_pud_range(p4d_t *p4d, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, pgtbl_mod_mask *mask) { pud_t *pud; unsigned long next; @@ -162,20 +157,19 @@ static inline int ioremap_pud_range(p4d_t *p4d, unsigned long addr, do { next = pud_addr_end(addr, end); - if (ioremap_try_huge_pud(pud, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot)) { *mask |= PGTBL_PUD_MODIFIED; continue; } - if (ioremap_pmd_range(pud, addr, next, phys_addr, prot, mask)) + if (vmap_pmd_range(pud, addr, next, phys_addr, prot, mask)) return -ENOMEM; } while (pud++, phys_addr += (next - addr), addr = next, addr != end); return 0; } -static int ioremap_try_huge_p4d(p4d_t *p4d, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) +static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { if (!ioremap_p4d_enabled()) return 0; @@ -195,9 +189,8 @@ static int ioremap_try_huge_p4d(p4d_t *p4d, unsigned long addr, return p4d_set_huge(p4d, phys_addr, prot); } -static inline int ioremap_p4d_range(pgd_t *pgd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, pgtbl_mod_mask *mask) { p4d_t *p4d; unsigned long next; @@ -208,19 +201,18 @@ static inline int ioremap_p4d_range(pgd_t *pgd, unsigned long addr, do { next = p4d_addr_end(addr, end); - if (ioremap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) { + if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) { *mask |= PGTBL_P4D_MODIFIED; continue; } - if (ioremap_pud_range(p4d, addr, next, phys_addr, prot, mask)) + if (vmap_pud_range(p4d, addr, next, phys_addr, prot, mask)) return -ENOMEM; } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); return 0; } -int ioremap_page_range(unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot) +int ioremap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot) { pgd_t *pgd; unsigned long start; @@ -235,8 +227,7 @@ int ioremap_page_range(unsigned long addr, pgd = pgd_offset_k(addr); do { next = pgd_addr_end(addr, end); - err = ioremap_p4d_range(pgd, addr, next, phys_addr, prot, - &mask); + err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, &mask); if (err) break; } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); @@ -272,7 +263,7 @@ void __iomem *ioremap_prot(phys_addr_t addr, size_t size, unsigned long prot) return NULL; vaddr = (unsigned long)area->addr; - if (ioremap_page_range(vaddr, vaddr + size, addr, __pgprot(prot))) { + if (vmap_page_range(vaddr, vaddr + size, addr, __pgprot(prot))) { free_vm_area(area); return NULL; } From patchwork Sun Aug 16 09:09:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11716039 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0019514E3 for ; Sun, 16 Aug 2020 09:09:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 992FB206F4 for ; Sun, 16 Aug 2020 09:09:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="TJhZgnnv" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 992FB206F4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B956E6B000A; Sun, 16 Aug 2020 05:09:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B1CB66B000C; Sun, 16 Aug 2020 05:09:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9BEDC8D0001; Sun, 16 Aug 2020 05:09:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0070.hostedemail.com [216.40.44.70]) by kanga.kvack.org (Postfix) with ESMTP id 83DF46B000A for ; Sun, 16 Aug 2020 05:09:48 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 456142479 for ; Sun, 16 Aug 2020 09:09:48 +0000 (UTC) X-FDA: 77155859256.20.joke49_5f0be182700c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id 1E3ED180C07A3 for ; Sun, 16 Aug 2020 09:09:48 +0000 (UTC) X-Spam-Summary: 1,0,0,d307380db0300e44,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:4:41:69:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1605:1730:1747:1777:1792:1801:2194:2196:2199:2200:2393:2559:2562:2638:2898:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:4321:4385:4605:5007:6119:6261:6653:6742:7514:7875:8603:8660:9413:9592:10004:11026:11473:11657:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:12986:13148:13230:13894:14096:14394:14687:21080:21433:21444:21451:21611:21627:21666:21939:21966:21990:30012:30054:30070,0,RBL:209.85.210.196:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04yrsyfej3i9wyadfno76tcbjbdfyycfe4ng4ycrxyxjuqpm69h45hk3qkyoqm6.fuwez7ihcrqa8w4w44pqqexc3wccbpf36rdfwk789oxgoj4zg5o1rimpdzzfpa4.g-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: joke49_5f0be182700c X-Filterd-Recvd-Size: 15935 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Sun, 16 Aug 2020 09:09:47 +0000 (UTC) Received: by mail-pf1-f196.google.com with SMTP id x25so6681037pff.4 for ; Sun, 16 Aug 2020 02:09:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=aEToTD5VYBwgrCywfBxzxRiL8d+HP5Dpc3Q6Q4EbEw4=; b=TJhZgnnvjrutV8Q4xybwxo9polcXq7vG2zqYA63JlfiYAX+gDqO0LUlW0fpELlwZXG O2OpD8iJdualirqQgWC8IdE+pIEp53utOBUez8MNCaVhA3Hh+xSliUlLl9kyNehelVf7 w53lt358xpduKazKqu/4fz4hvvKulYq057DN++kTwHIPH5vur5OKneJmkpF3rH7Li79n 16PtUCua8V+lbv/ZbULT0VPz5/2Yf1HVIvcG3eP2n3leSkZ6uAkQlVJfe8PIRp4o0IA6 9/z90n9smDcJDzArV8lK0YV6nkW7u9GJwXzziIWsg2jRzORXFaW9goTEE3J4un4EMjGI d1VA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aEToTD5VYBwgrCywfBxzxRiL8d+HP5Dpc3Q6Q4EbEw4=; b=GY1Wyb6yHYnxxfRomOFMVI2w8HVj71Gswsc0axPmk4/zhmL8Mmc2TxGuE81l9oQ81A h3/0TI/lzCPiUiJWYCuEZGknWfU+VJBa1WHZGs7W8UfWOB8LS1mD7L8mWY23IPk1VtrR 8Bb8jhEuIbxGRGLoiJrzBiZo3Vst2niosKBlP7ZHEI4uRrZ6sa+mK4laCyK6rdtcMnWp t8WTg9RTbBeknLgXlibOAk+qWV5SrUf7AsY+LOwEZtRmSGCvac/z17qiPJls8PCWjfja M5vi+PpXt3vgRMvoxptsh3LZnVHrpbp3ucv9pz5y+gCyRd3QjAXoHuther+x53mnuxge x4/g== X-Gm-Message-State: AOAM532yv5jq4eoU+WRces4aAVhQWfvHHXioVfg0uSoquSCJAdcoDYgt hR7ds9xEMRQJ97oIJ4EEqtNjcK4Qs7I= X-Google-Smtp-Source: ABdhPJys+zcULBlpjhklWA6npK9XqDndU0z5pKbAEsggx/Fi4vDh0NedcOYe8BO+rY8RrQYbSdV9Hw== X-Received: by 2002:a63:7f50:: with SMTP id p16mr6190271pgn.451.1597568986212; Sun, 16 Aug 2020 02:09:46 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (193-116-193-175.tpgi.com.au. [193.116.193.175]) by smtp.gmail.com with ESMTPSA id o19sm12768369pjs.8.2020.08.16.02.09.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Aug 2020 02:09:45 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" , Zefan Li , Jonathan Cameron Subject: [PATCH v4 5/8] mm: HUGE_VMAP arch support cleanup Date: Sun, 16 Aug 2020 19:09:01 +1000 Message-Id: <20200816090904.83947-6-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200816090904.83947-1-npiggin@gmail.com> References: <20200816090904.83947-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1E3ED180C07A3 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This changes the awkward approach where architectures provide init functions to determine which levels they can provide large mappings for, to one where the arch is queried for each call. This removes code and indirection, and allows constant-folding of dead code for unsupported levels. This also adds a prot argument to the arch query. This is unused currently but could help with some architectures (e.g., some powerpc processors can't map uncacheable memory with large pages). Signed-off-by: Nicholas Piggin --- arch/arm64/mm/mmu.c | 12 +-- arch/powerpc/mm/book3s64/radix_pgtable.c | 10 ++- arch/x86/mm/ioremap.c | 12 +-- include/linux/io.h | 9 --- include/linux/vmalloc.h | 10 +++ init/main.c | 1 - mm/ioremap.c | 96 +++++++++++------------- 7 files changed, 73 insertions(+), 77 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 75df62fea1b6..bbb3ccf6a7ce 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1304,12 +1304,13 @@ void *__init fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot) return dt_virt; } -int __init arch_ioremap_p4d_supported(void) +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +bool arch_vmap_p4d_supported(pgprot_t prot) { - return 0; + return false; } -int __init arch_ioremap_pud_supported(void) +bool arch_vmap_pud_supported(pgprot_t prot) { /* * Only 4k granule supports level 1 block mappings. @@ -1319,11 +1320,12 @@ int __init arch_ioremap_pud_supported(void) !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); } -int __init arch_ioremap_pmd_supported(void) +bool arch_vmap_pmd_supported(pgprot_t prot) { - /* See arch_ioremap_pud_supported() */ + /* See arch_vmap_pud_supported() */ return !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); } +#endif int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t prot) { diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 28c784976bed..eeb0e8451176 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1134,13 +1134,14 @@ void radix__ptep_modify_prot_commit(struct vm_area_struct *vma, set_pte_at(mm, addr, ptep, pte); } -int __init arch_ioremap_pud_supported(void) +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +bool arch_vmap_pud_supported(pgprot_t prot) { /* HPT does not cope with large pages in the vmalloc area */ return radix_enabled(); } -int __init arch_ioremap_pmd_supported(void) +bool arch_vmap_pmd_supported(pgprot_t prot) { return radix_enabled(); } @@ -1149,6 +1150,7 @@ int p4d_free_pud_page(p4d_t *p4d, unsigned long addr) { return 0; } +#endif int pud_set_huge(pud_t *pud, phys_addr_t addr, pgprot_t prot) { @@ -1234,7 +1236,7 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) return 1; } -int __init arch_ioremap_p4d_supported(void) +bool arch_vmap_p4d_supported(pgprot_t prot) { - return 0; + return false; } diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c index 84d85dbd1dad..5b8b495ab4ed 100644 --- a/arch/x86/mm/ioremap.c +++ b/arch/x86/mm/ioremap.c @@ -481,24 +481,26 @@ void iounmap(volatile void __iomem *addr) } EXPORT_SYMBOL(iounmap); -int __init arch_ioremap_p4d_supported(void) +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +bool arch_vmap_p4d_supported(pgprot_t prot) { - return 0; + return false; } -int __init arch_ioremap_pud_supported(void) +bool arch_vmap_pud_supported(pgprot_t prot) { #ifdef CONFIG_X86_64 return boot_cpu_has(X86_FEATURE_GBPAGES); #else - return 0; + return false; #endif } -int __init arch_ioremap_pmd_supported(void) +bool arch_vmap_pmd_supported(pgprot_t prot) { return boot_cpu_has(X86_FEATURE_PSE); } +#endif /* * Convert a physical pointer to a virtual kernel pointer for /dev/mem diff --git a/include/linux/io.h b/include/linux/io.h index 8394c56babc2..f1effd4d7a3c 100644 --- a/include/linux/io.h +++ b/include/linux/io.h @@ -31,15 +31,6 @@ static inline int ioremap_page_range(unsigned long addr, unsigned long end, } #endif -#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -void __init ioremap_huge_init(void); -int arch_ioremap_p4d_supported(void); -int arch_ioremap_pud_supported(void); -int arch_ioremap_pmd_supported(void); -#else -static inline void ioremap_huge_init(void) { } -#endif - /* * Managed iomap interface */ diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 0221f852a7e1..787d77ad7536 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -84,6 +84,16 @@ struct vmap_area { }; }; +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +bool arch_vmap_p4d_supported(pgprot_t prot); +bool arch_vmap_pud_supported(pgprot_t prot); +bool arch_vmap_pmd_supported(pgprot_t prot); +#else +static inline bool arch_vmap_p4d_supported(pgprot_t prot) { return false; } +static inline bool arch_vmap_pud_supported(pgprot_t prot) { return false; } +static inline bool arch_vmap_pmd_supported(pgprot_t prot) { return false; } +#endif + /* * Highlevel APIs for driver use */ diff --git a/init/main.c b/init/main.c index ae78fb68d231..1c89aa127b8f 100644 --- a/init/main.c +++ b/init/main.c @@ -820,7 +820,6 @@ static void __init mm_init(void) pgtable_init(); debug_objects_mem_init(); vmalloc_init(); - ioremap_huge_init(); /* Should be run before the first non-init thread is created */ init_espfix_bsp(); /* Should be run after espfix64 is set up. */ diff --git a/mm/ioremap.c b/mm/ioremap.c index 6016ae3227ad..b0032dbadaf7 100644 --- a/mm/ioremap.c +++ b/mm/ioremap.c @@ -16,49 +16,16 @@ #include "pgalloc-track.h" #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -static int __read_mostly ioremap_p4d_capable; -static int __read_mostly ioremap_pud_capable; -static int __read_mostly ioremap_pmd_capable; -static int __read_mostly ioremap_huge_disabled; +static bool __ro_after_init iomap_allow_huge = true; static int __init set_nohugeiomap(char *str) { - ioremap_huge_disabled = 1; + iomap_allow_huge = false; return 0; } early_param("nohugeiomap", set_nohugeiomap); - -void __init ioremap_huge_init(void) -{ - if (!ioremap_huge_disabled) { - if (arch_ioremap_p4d_supported()) - ioremap_p4d_capable = 1; - if (arch_ioremap_pud_supported()) - ioremap_pud_capable = 1; - if (arch_ioremap_pmd_supported()) - ioremap_pmd_capable = 1; - } -} - -static inline int ioremap_p4d_enabled(void) -{ - return ioremap_p4d_capable; -} - -static inline int ioremap_pud_enabled(void) -{ - return ioremap_pud_capable; -} - -static inline int ioremap_pmd_enabled(void) -{ - return ioremap_pmd_capable; -} - -#else /* !CONFIG_HAVE_ARCH_HUGE_VMAP */ -static inline int ioremap_p4d_enabled(void) { return 0; } -static inline int ioremap_pud_enabled(void) { return 0; } -static inline int ioremap_pmd_enabled(void) { return 0; } +#else /* CONFIG_HAVE_ARCH_HUGE_VMAP */ +static const bool iomap_allow_huge = false; #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, @@ -81,9 +48,12 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, } static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) { - if (!ioremap_pmd_enabled()) + if (max_page_shift < PMD_SHIFT) + return 0; + + if (!arch_vmap_pmd_supported(prot)) return 0; if ((end - addr) != PMD_SIZE) @@ -102,7 +72,8 @@ static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, } static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, pgtbl_mod_mask *mask) + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, + pgtbl_mod_mask *mask) { pmd_t *pmd; unsigned long next; @@ -113,7 +84,7 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, do { next = pmd_addr_end(addr, end); - if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, max_page_shift)) { *mask |= PGTBL_PMD_MODIFIED; continue; } @@ -125,9 +96,12 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, } static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) { - if (!ioremap_pud_enabled()) + if (max_page_shift < PUD_SHIFT) + return 0; + + if (!arch_vmap_pud_supported(prot)) return 0; if ((end - addr) != PUD_SIZE) @@ -146,7 +120,8 @@ static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, } static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, pgtbl_mod_mask *mask) + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, + pgtbl_mod_mask *mask) { pud_t *pud; unsigned long next; @@ -157,21 +132,24 @@ static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, do { next = pud_addr_end(addr, end); - if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot, max_page_shift)) { *mask |= PGTBL_PUD_MODIFIED; continue; } - if (vmap_pmd_range(pud, addr, next, phys_addr, prot, mask)) + if (vmap_pmd_range(pud, addr, next, phys_addr, prot, max_page_shift, mask)) return -ENOMEM; } while (pud++, phys_addr += (next - addr), addr = next, addr != end); return 0; } static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) { - if (!ioremap_p4d_enabled()) + if (max_page_shift < P4D_SHIFT) + return 0; + + if (!arch_vmap_p4d_supported(prot)) return 0; if ((end - addr) != P4D_SIZE) @@ -190,7 +168,8 @@ static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, } static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, pgtbl_mod_mask *mask) + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, + pgtbl_mod_mask *mask) { p4d_t *p4d; unsigned long next; @@ -201,18 +180,19 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, do { next = p4d_addr_end(addr, end); - if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) { + if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot, max_page_shift)) { *mask |= PGTBL_P4D_MODIFIED; continue; } - if (vmap_pud_range(p4d, addr, next, phys_addr, prot, mask)) + if (vmap_pud_range(p4d, addr, next, phys_addr, prot, max_page_shift, mask)) return -ENOMEM; } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); return 0; } -int ioremap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot) +static int vmap_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) { pgd_t *pgd; unsigned long start; @@ -227,7 +207,7 @@ int ioremap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_a pgd = pgd_offset_k(addr); do { next = pgd_addr_end(addr, end); - err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, &mask); + err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, max_page_shift, &mask); if (err) break; } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); @@ -240,6 +220,16 @@ int ioremap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_a return err; } +int ioremap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot) +{ + unsigned int max_page_shift = PAGE_SHIFT; + + if (iomap_allow_huge) + max_page_shift = P4D_SHIFT; + + return vmap_range(addr, end, phys_addr, prot, max_page_shift); +} + #ifdef CONFIG_GENERIC_IOREMAP void __iomem *ioremap_prot(phys_addr_t addr, size_t size, unsigned long prot) { From patchwork Sun Aug 16 09:09:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11716043 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C795A138C for ; Sun, 16 Aug 2020 09:09:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 79886206F4 for ; Sun, 16 Aug 2020 09:09:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="NVGD++6T" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 79886206F4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 860846B000C; Sun, 16 Aug 2020 05:09:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7E8E38D0001; Sun, 16 Aug 2020 05:09:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6B1A46B000E; Sun, 16 Aug 2020 05:09:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0134.hostedemail.com [216.40.44.134]) by kanga.kvack.org (Postfix) with ESMTP id 53E536B000C for ; Sun, 16 Aug 2020 05:09:54 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 196DD181AEF30 for ; Sun, 16 Aug 2020 09:09:54 +0000 (UTC) X-FDA: 77155859508.22.nail36_0108f042700c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin22.hostedemail.com (Postfix) with ESMTP id E423318038E97 for ; Sun, 16 Aug 2020 09:09:53 +0000 (UTC) X-Spam-Summary: 1,0,0,20d77850a547901b,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:4:41:69:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1605:1730:1747:1777:1792:1981:2194:2196:2199:2200:2393:2559:2562:2637:2898:3138:3139:3140:3141:3142:3867:3871:4321:4385:4605:5007:6119:6261:6653:6742:7514:7875:8603:9036:9413:9592:10004:11026:11232:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:12986:13894:14096:14110:14394:14687:21080:21324:21444:21451:21627:21666:21740:21990:30054:30070,0,RBL:209.85.216.68:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100;04yrrhrafzjtbo9exhy69k4shh6o8opg37etga1q7iqtiwh4kj9ya3a3sn4gjz9.3fa5bjc6urpxgua57byd4psmn9sic57ew38fahi3yadbhj7x35y7111osympjtc.1-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: nail36_0108f042700c X-Filterd-Recvd-Size: 15018 Received: from mail-pj1-f68.google.com (mail-pj1-f68.google.com [209.85.216.68]) by imf37.hostedemail.com (Postfix) with ESMTP for ; Sun, 16 Aug 2020 09:09:53 +0000 (UTC) Received: by mail-pj1-f68.google.com with SMTP id t6so6246381pjr.0 for ; Sun, 16 Aug 2020 02:09:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=n6u3W3xGRvE1f9TahLPt2tr1kQk4naizqZpWLpaJwxQ=; b=NVGD++6Tj0lMk2z9wNq4nJ10f0IMpkeXEtio2bhJMTlw9Dmcjnf/o63R7KSh03veQc NToPC6Hmrex+3m/B92Iyi25twwC4savpdhDKxi//9FiGCauuQ2YGjD0K//71+2xOSzKa HFC7IEclgGS2A9gPZo8rTHUj2W/MOZTaobaq3jLvdjyjMqE0rd4Vyn6QOa0vs2Yxxzt7 g6ifkka8zaR3NSTaJng5wVIxFbGB+ZXbBjSvJJs3SgC0oVAh2QUEO9Xh5B3+IgkxM5Rx i9sVElWkpbJIYuJ5OEmqOZggSNLPow/F6Vms9tIO8gj7ZUhTXlE2HMSvrJmOEusq1afq 39xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=n6u3W3xGRvE1f9TahLPt2tr1kQk4naizqZpWLpaJwxQ=; b=lgSuMfm5BRHUi2/Pk36dyq0ZAZOOoDQxEm2ThlfuQZJnhz8rpWSv/gtO7g0WKCQFQl eCAdNlxNt2eUYBj3ruvQt5yM2kw0wU5S8/7dydu4XHq391TtGuxmJ2mRRgPgJdtIbjuB 1m9XLQceDJ1/6sdh/bWw4o4NiX5eRUUku40lkoaciVmQLj88m1Q+6KODjtAtwjMrxL+8 LeUQc0uWoLjATnDnl4S9b/mV7Enhq14F/yA8Yn7Xu7mIHdhMmvIfX32zLUL0GB1iAUKe /cDH1jBJ1SVNvMpVdkwVAa5hEjba2jd1lpn45ojE4kYmHTJfUJ698eHj5fT2nrzVLoea CMdg== X-Gm-Message-State: AOAM530/2rsU4ufrZQsLxHXUWLCVNKTVM8gH98oOC+QFu5jFrmBKBJrs GenqHJU8rL0k2PmiJpT/DQwLiE02p04= X-Google-Smtp-Source: ABdhPJwEOv2tQfJABdqUlNbhrJ4kA0cCE72Kz9fQNeNQlnkwEIlF+5yHcfYk6AgPwIP09/kTPpG05g== X-Received: by 2002:a17:90a:c208:: with SMTP id e8mr8235623pjt.73.1597568992318; Sun, 16 Aug 2020 02:09:52 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (193-116-193-175.tpgi.com.au. [193.116.193.175]) by smtp.gmail.com with ESMTPSA id o19sm12768369pjs.8.2020.08.16.02.09.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Aug 2020 02:09:51 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" , Zefan Li , Jonathan Cameron Subject: [PATCH v4 6/8] mm: Move vmap_range from lib/ioremap.c to mm/vmalloc.c Date: Sun, 16 Aug 2020 19:09:02 +1000 Message-Id: <20200816090904.83947-7-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200816090904.83947-1-npiggin@gmail.com> References: <20200816090904.83947-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: E423318038E97 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a generic kernel virtual memory mapper, not specific to ioremap. Signed-off-by: Nicholas Piggin --- include/linux/vmalloc.h | 2 + mm/ioremap.c | 192 ---------------------------------------- mm/vmalloc.c | 191 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 193 insertions(+), 192 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 787d77ad7536..e3590e93bfff 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -181,6 +181,8 @@ extern struct vm_struct *remove_vm_area(const void *addr); extern struct vm_struct *find_vm_area(const void *addr); #ifdef CONFIG_MMU +extern int vmap_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift); extern int map_kernel_range_noflush(unsigned long start, unsigned long size, pgprot_t prot, struct page **pages); int map_kernel_range(unsigned long start, unsigned long size, pgprot_t prot, diff --git a/mm/ioremap.c b/mm/ioremap.c index b0032dbadaf7..cdda0e022740 100644 --- a/mm/ioremap.c +++ b/mm/ioremap.c @@ -28,198 +28,6 @@ early_param("nohugeiomap", set_nohugeiomap); static const bool iomap_allow_huge = false; #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ -static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, pgtbl_mod_mask *mask) -{ - pte_t *pte; - u64 pfn; - - pfn = phys_addr >> PAGE_SHIFT; - pte = pte_alloc_kernel_track(pmd, addr, mask); - if (!pte) - return -ENOMEM; - do { - BUG_ON(!pte_none(*pte)); - set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); - pfn++; - } while (pte++, addr += PAGE_SIZE, addr != end); - *mask |= PGTBL_PTE_MODIFIED; - return 0; -} - -static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) -{ - if (max_page_shift < PMD_SHIFT) - return 0; - - if (!arch_vmap_pmd_supported(prot)) - return 0; - - if ((end - addr) != PMD_SIZE) - return 0; - - if (!IS_ALIGNED(addr, PMD_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, PMD_SIZE)) - return 0; - - if (pmd_present(*pmd) && !pmd_free_pte_page(pmd, addr)) - return 0; - - return pmd_set_huge(pmd, phys_addr, prot); -} - -static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, - pgtbl_mod_mask *mask) -{ - pmd_t *pmd; - unsigned long next; - - pmd = pmd_alloc_track(&init_mm, pud, addr, mask); - if (!pmd) - return -ENOMEM; - do { - next = pmd_addr_end(addr, end); - - if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, max_page_shift)) { - *mask |= PGTBL_PMD_MODIFIED; - continue; - } - - if (vmap_pte_range(pmd, addr, next, phys_addr, prot, mask)) - return -ENOMEM; - } while (pmd++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) -{ - if (max_page_shift < PUD_SHIFT) - return 0; - - if (!arch_vmap_pud_supported(prot)) - return 0; - - if ((end - addr) != PUD_SIZE) - return 0; - - if (!IS_ALIGNED(addr, PUD_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, PUD_SIZE)) - return 0; - - if (pud_present(*pud) && !pud_free_pmd_page(pud, addr)) - return 0; - - return pud_set_huge(pud, phys_addr, prot); -} - -static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, - pgtbl_mod_mask *mask) -{ - pud_t *pud; - unsigned long next; - - pud = pud_alloc_track(&init_mm, p4d, addr, mask); - if (!pud) - return -ENOMEM; - do { - next = pud_addr_end(addr, end); - - if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot, max_page_shift)) { - *mask |= PGTBL_PUD_MODIFIED; - continue; - } - - if (vmap_pmd_range(pud, addr, next, phys_addr, prot, max_page_shift, mask)) - return -ENOMEM; - } while (pud++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) -{ - if (max_page_shift < P4D_SHIFT) - return 0; - - if (!arch_vmap_p4d_supported(prot)) - return 0; - - if ((end - addr) != P4D_SIZE) - return 0; - - if (!IS_ALIGNED(addr, P4D_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, P4D_SIZE)) - return 0; - - if (p4d_present(*p4d) && !p4d_free_pud_page(p4d, addr)) - return 0; - - return p4d_set_huge(p4d, phys_addr, prot); -} - -static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, - pgtbl_mod_mask *mask) -{ - p4d_t *p4d; - unsigned long next; - - p4d = p4d_alloc_track(&init_mm, pgd, addr, mask); - if (!p4d) - return -ENOMEM; - do { - next = p4d_addr_end(addr, end); - - if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot, max_page_shift)) { - *mask |= PGTBL_P4D_MODIFIED; - continue; - } - - if (vmap_pud_range(p4d, addr, next, phys_addr, prot, max_page_shift, mask)) - return -ENOMEM; - } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -static int vmap_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) -{ - pgd_t *pgd; - unsigned long start; - unsigned long next; - int err; - pgtbl_mod_mask mask = 0; - - might_sleep(); - BUG_ON(addr >= end); - - start = addr; - pgd = pgd_offset_k(addr); - do { - next = pgd_addr_end(addr, end); - err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, max_page_shift, &mask); - if (err) - break; - } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); - - flush_cache_vmap(start, end); - - if (mask & ARCH_PAGE_TABLE_SYNC_MASK) - arch_sync_kernel_mappings(start, end); - - return err; -} - int ioremap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot) { unsigned int max_page_shift = PAGE_SHIFT; diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 3a1e45fd1626..129f10545bb1 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -71,6 +71,197 @@ static void free_work(struct work_struct *w) } /*** Page table manipulation functions ***/ +static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, pgtbl_mod_mask *mask) +{ + pte_t *pte; + u64 pfn; + + pfn = phys_addr >> PAGE_SHIFT; + pte = pte_alloc_kernel_track(pmd, addr, mask); + if (!pte) + return -ENOMEM; + do { + BUG_ON(!pte_none(*pte)); + set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); + pfn++; + } while (pte++, addr += PAGE_SIZE, addr != end); + *mask |= PGTBL_PTE_MODIFIED; + return 0; +} + +static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) +{ + if (max_page_shift < PMD_SHIFT) + return 0; + + if (!arch_vmap_pmd_supported(prot)) + return 0; + + if ((end - addr) != PMD_SIZE) + return 0; + + if (!IS_ALIGNED(addr, PMD_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, PMD_SIZE)) + return 0; + + if (pmd_present(*pmd) && !pmd_free_pte_page(pmd, addr)) + return 0; + + return pmd_set_huge(pmd, phys_addr, prot); +} + +static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, + pgtbl_mod_mask *mask) +{ + pmd_t *pmd; + unsigned long next; + + pmd = pmd_alloc_track(&init_mm, pud, addr, mask); + if (!pmd) + return -ENOMEM; + do { + next = pmd_addr_end(addr, end); + + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, max_page_shift)) { + *mask |= PGTBL_PMD_MODIFIED; + continue; + } + + if (vmap_pte_range(pmd, addr, next, phys_addr, prot, mask)) + return -ENOMEM; + } while (pmd++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) +{ + if (max_page_shift < PUD_SHIFT) + return 0; + + if (!arch_vmap_pud_supported(prot)) + return 0; + + if ((end - addr) != PUD_SIZE) + return 0; + + if (!IS_ALIGNED(addr, PUD_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, PUD_SIZE)) + return 0; + + if (pud_present(*pud) && !pud_free_pmd_page(pud, addr)) + return 0; + + return pud_set_huge(pud, phys_addr, prot); +} + +static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, + pgtbl_mod_mask *mask) +{ + pud_t *pud; + unsigned long next; + + pud = pud_alloc_track(&init_mm, p4d, addr, mask); + if (!pud) + return -ENOMEM; + do { + next = pud_addr_end(addr, end); + + if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot, max_page_shift)) { + *mask |= PGTBL_PUD_MODIFIED; + continue; + } + + if (vmap_pmd_range(pud, addr, next, phys_addr, prot, max_page_shift, mask)) + return -ENOMEM; + } while (pud++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) +{ + if (max_page_shift < P4D_SHIFT) + return 0; + + if (!arch_vmap_p4d_supported(prot)) + return 0; + + if ((end - addr) != P4D_SIZE) + return 0; + + if (!IS_ALIGNED(addr, P4D_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, P4D_SIZE)) + return 0; + + if (p4d_present(*p4d) && !p4d_free_pud_page(p4d, addr)) + return 0; + + return p4d_set_huge(p4d, phys_addr, prot); +} + +static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, + pgtbl_mod_mask *mask) +{ + p4d_t *p4d; + unsigned long next; + + p4d = p4d_alloc_track(&init_mm, pgd, addr, mask); + if (!p4d) + return -ENOMEM; + do { + next = p4d_addr_end(addr, end); + + if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot, max_page_shift)) { + *mask |= PGTBL_P4D_MODIFIED; + continue; + } + + if (vmap_pud_range(p4d, addr, next, phys_addr, prot, max_page_shift, mask)) + return -ENOMEM; + } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +int vmap_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + pgd_t *pgd; + unsigned long start; + unsigned long next; + int err; + pgtbl_mod_mask mask = 0; + + might_sleep(); + BUG_ON(addr >= end); + + start = addr; + pgd = pgd_offset_k(addr); + do { + next = pgd_addr_end(addr, end); + err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, max_page_shift, &mask); + if (err) + break; + } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); + + flush_cache_vmap(start, end); + + if (mask & ARCH_PAGE_TABLE_SYNC_MASK) + arch_sync_kernel_mappings(start, end); + + return err; +} static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgtbl_mod_mask *mask) From patchwork Sun Aug 16 09:09:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11716047 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 83A28138C for ; Sun, 16 Aug 2020 09:10:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4E8642078D for ; Sun, 16 Aug 2020 09:10:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="oQlIsW4j" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4E8642078D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 79DBA6B000D; Sun, 16 Aug 2020 05:10:00 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 727216B000E; Sun, 16 Aug 2020 05:10:00 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5CC2B6B0010; Sun, 16 Aug 2020 05:10:00 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0234.hostedemail.com [216.40.44.234]) by kanga.kvack.org (Postfix) with ESMTP id 438176B000D for ; Sun, 16 Aug 2020 05:10:00 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 03584180AD80F for ; Sun, 16 Aug 2020 09:10:00 +0000 (UTC) X-FDA: 77155859760.11.snail65_3500a562700c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id C9197180F8B80 for ; Sun, 16 Aug 2020 09:09:59 +0000 (UTC) X-Spam-Summary: 1,0,0,7d21970e6d30d201,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2731:3138:3139:3140:3141:3142:3352:3865:3866:3868:3870:3874:4321:5007:6119:6261:6653:6742:7514:7875:9413:9592:10004:11026:11473:11658:11914:12043:12297:12438:12517:12519:12555:12895:12986:13069:13161:13229:13311:13357:13894:14096:14181:14384:14394:14687:14721:21080:21444:21451:21627:21666:21990:30054,0,RBL:209.85.216.66:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04ygardnhnroo8edkpp51zfuhtc56ocr9fix7m89shg5c1o6xwjojwo7djb75td.ir6cy6dga9bykxkjxoj31dn3sei6dohy1857cqsagdd9qpxmu8g4mmzzhzimrqp.e-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: snail65_3500a562700c X-Filterd-Recvd-Size: 4927 Received: from mail-pj1-f66.google.com (mail-pj1-f66.google.com [209.85.216.66]) by imf15.hostedemail.com (Postfix) with ESMTP for ; Sun, 16 Aug 2020 09:09:59 +0000 (UTC) Received: by mail-pj1-f66.google.com with SMTP id e4so6321140pjd.0 for ; Sun, 16 Aug 2020 02:09:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8C8wdRPXIlS04tUD/986mX054vDuu2PdIQ/xTcAZZAA=; b=oQlIsW4jf6QdaM41v1DAzwkm/rUCXyiKKAHM7uRLQyLEF7lxXjR7GmEvQPShvtKNQf dETlAFV1UwJxM+uJ3hw4p+i6HjdZ1KaUBRDFnyZ1LZfv7LViQGsume3ZFDlou062UCRW JERsTRYkRNxB8GTDfdMkpO6z8LDG5x9/cYyiDL0NPA8A8r/rv4NwzlGCwJiFTznJYcLi a9+GxQoBiIh/oEFboZbnFS+PIOXYCRnGyGpdqYXeMHtOssz4U0nXw/L2tDv8F5qCy41R eQO0EaYIzsJnXoToRJzFrrJ1hrJkMkk9481LETndvWuLQcDVqKpHKM0dWPQNXgse3SQy uL6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8C8wdRPXIlS04tUD/986mX054vDuu2PdIQ/xTcAZZAA=; b=k4WyHsz6cdRO/GqwXEOeGq0gVHQQfZaEJhvOHXCEqO4Q3Wek9xVPi2Y4/8wvbHVlkz Y8TufdAP9Gr5uekMfT5wcojqQ0Rj9fUQjCEecx92TCZh7U0tvFpxlR0lOuRVZJ8XirGo HAmBWNDWXlxfOQfZuRmt2ZIIqZCU6rOP+qiboTshF73pfyIQDs7vPIEaGleZtFSifM25 m5HN3iA7SQVvrIibEr+fBnL1/m/7ofYFL3C5Sis5Gx/7CwYgUHDcXdlv14TjgseFRWfu CJUqPGgu8/677O2OmEbKSjbzwMpLQtd0vNkPRq1a1qTioecIQqYC0FgZGw7UkqVieKxO J1Fw== X-Gm-Message-State: AOAM530G0sbKKRMYAWGcjnbvdIImGYHkymI7sDnavsIWx3bTWQuqRh4p ye0vauD35RKXCRSTUb6KAKjCMn3s2w0= X-Google-Smtp-Source: ABdhPJzRG648jeLeZZ2Z4oujD6qX3vm8KDLEE34yBusy/W9VPlVLdok2Quap0oZrYbchmPfC1Ozb0A== X-Received: by 2002:a17:90a:f68b:: with SMTP id cl11mr8361244pjb.68.1597568998489; Sun, 16 Aug 2020 02:09:58 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (193-116-193-175.tpgi.com.au. [193.116.193.175]) by smtp.gmail.com with ESMTPSA id o19sm12768369pjs.8.2020.08.16.02.09.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Aug 2020 02:09:58 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" , Zefan Li , Jonathan Cameron Subject: [PATCH v4 7/8] mm/vmalloc: add vmap_range_noflush variant Date: Sun, 16 Aug 2020 19:09:03 +1000 Message-Id: <20200816090904.83947-8-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200816090904.83947-1-npiggin@gmail.com> References: <20200816090904.83947-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: C9197180F8B80 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As a side-effect, the order of flush_cache_vmap() and arch_sync_kernel_mappings() calls are switched, but that now matches the other callers in this file. Signed-off-by: Nicholas Piggin --- mm/vmalloc.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 129f10545bb1..4e5cb7c7f780 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -234,8 +234,8 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, return 0; } -int vmap_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) +static int vmap_range_noflush(unsigned long addr, unsigned long end, phys_addr_t phys_addr, + pgprot_t prot, unsigned int max_page_shift) { pgd_t *pgd; unsigned long start; @@ -255,14 +255,23 @@ int vmap_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgp break; } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); - flush_cache_vmap(start, end); - if (mask & ARCH_PAGE_TABLE_SYNC_MASK) arch_sync_kernel_mappings(start, end); return err; } +int vmap_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + int err; + + err = vmap_range_noflush(addr, end, phys_addr, prot, max_page_shift); + flush_cache_vmap(addr, end); + + return err; +} + static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgtbl_mod_mask *mask) { From patchwork Sun Aug 16 09:09:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11716049 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2859914E3 for ; Sun, 16 Aug 2020 09:10:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CC79220674 for ; Sun, 16 Aug 2020 09:10:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="R5qOCbE0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CC79220674 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EA3716B000E; Sun, 16 Aug 2020 05:10:06 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E530C6B0010; Sun, 16 Aug 2020 05:10:06 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA97B6B0022; Sun, 16 Aug 2020 05:10:06 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0127.hostedemail.com [216.40.44.127]) by kanga.kvack.org (Postfix) with ESMTP id A463B6B000E for ; Sun, 16 Aug 2020 05:10:06 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 63E9F3635 for ; Sun, 16 Aug 2020 09:10:06 +0000 (UTC) X-FDA: 77155860012.16.quill02_260a0a32700c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin16.hostedemail.com (Postfix) with ESMTP id 3B30E100E690C for ; Sun, 16 Aug 2020 09:10:06 +0000 (UTC) X-Spam-Summary: 1,0,0,504b6f9a2f4b1972,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:1:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1605:1730:1747:1777:1792:1801:2194:2196:2198:2199:2200:2201:2393:2559:2562:2636:2689:2693:2897:2899:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3874:4250:4321:4385:4605:4641:5007:6119:6261:6653:6742:7514:7875:7903:7904:8603:9036:9413:9592:10004:11026:11233:11473:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:13894:14096:14394:14687:21080:21444:21451:21627:21666:21740:21990:30054:30056:30089,0,RBL:209.85.214.196:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100;04yrfaswqof5q5g9z4f3uocdb5kf6oca15kgy7j8ajrna4wc85uapknjzig8ynu.whdz437w4cyygjesroxk5e4ohg5cxkkjfa939pjqg6qtwtik63hxyrpdotpykfc.n-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: quill02_260a0a32700c X-Filterd-Recvd-Size: 14089 Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Sun, 16 Aug 2020 09:10:05 +0000 (UTC) Received: by mail-pl1-f196.google.com with SMTP id t11so6039653plr.5 for ; Sun, 16 Aug 2020 02:10:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9NFhkMBJW82v5R2qq0DXUlpjk/Z0vbs/aoPPNmBeGGI=; b=R5qOCbE0HWQPPquvSxVOEDrtcnseyPIwuRJP6nFU9h/k9Vq94VIBHVamKYNEQcx84D /4ZM2lVZZ0NRbTaCQ13VFr0BUY1/pGFoP0twisLTrZ2Nar2VbfnZYZ+qKY/9IqMPB0R+ 7ilWcEX8R1uLjEs8tEEaFPJhkSbA+PA1yQ4X/ySQGgTDXfIIBx53ZuFXWEReFPcUHyZL 7Oec3jZe/0YDgjgQTE6WXGrqbhcjrGQg5DD9QiG2dcwyFGvtJhS/PXY6Dts6+Ft6/DRA uOwFfYBo8T2LmcYi0HvSS0TvAd31qwPJm9lHasKK+V38tFTSS6ofy34MbUB6h/JRSbVo D4Bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9NFhkMBJW82v5R2qq0DXUlpjk/Z0vbs/aoPPNmBeGGI=; b=N6x15zZC7+H7P4dpjmnaUZ9qEvM6XGOpVXtAyF9FvChsCROobo3XQdXNXUTepSwaiY UjwgC+fF5J9NA38r6fSVOIQaey4FTNmL+wAqhvZ/udggHDqHpCTEi12fSJvb5+KtOPmZ GgnazkgFu3emA2KljC+LgvJlrHDyQxanm3QgBhr8gzpKtOSkYNsmJne086DhaL9RT74m WSIs/4FeaotlmgOVQwSGjH5BYhor0uARf6venutX3mP7aMxTRdmkl+/OfXnIFfs/PNEG mvDInpaSHggHlkHUc6DSB5UAtvd030Ba08IhQ5NP4j2V3ljUlIUTKbsZ4oTT66ovKMbK fo1g== X-Gm-Message-State: AOAM530Qmrpef3O/U5hF2apn7QTiNLkr0FsMjBbmdKSOwQuE2NFe4p/q 8LMje5+U//zge7QkivhV3y2ESPYYHzI= X-Google-Smtp-Source: ABdhPJzYPmi5So9wweRK6VLaklObcWMLYwgxfwLsGfXoZAFZVfS3BesQaYpbzaDloikWs4g5q0sNhQ== X-Received: by 2002:a17:90a:e381:: with SMTP id b1mr8260896pjz.218.1597569004509; Sun, 16 Aug 2020 02:10:04 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (193-116-193-175.tpgi.com.au. [193.116.193.175]) by smtp.gmail.com with ESMTPSA id o19sm12768369pjs.8.2020.08.16.02.09.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Aug 2020 02:10:04 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" , Zefan Li , Jonathan Cameron Subject: [PATCH v4 8/8] mm/vmalloc: Hugepage vmalloc mappings Date: Sun, 16 Aug 2020 19:09:04 +1000 Message-Id: <20200816090904.83947-9-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200816090904.83947-1-npiggin@gmail.com> References: <20200816090904.83947-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 3B30E100E690C X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On platforms that define HAVE_ARCH_HUGE_VMAP and support PMD vmaps, vmalloc will attempt to allocate PMD-sized pages first, before falling back to small pages. Allocations which use something other than PAGE_KERNEL protections are not permitted to use huge pages yet, not all callers expect this (e.g., module allocations vs strict module rwx). This reduces TLB misses by nearly 30x on a `git diff` workload on a 2-node POWER9 (59,800 -> 2,100) and reduces CPU cycles by 0.54%. This can result in more internal fragmentation and memory overhead for a given allocation, an option nohugevmap is added to disable at boot. Signed-off-by: Nicholas Piggin Reported-by: kernel test robot --- .../admin-guide/kernel-parameters.txt | 2 + include/linux/vmalloc.h | 1 + mm/vmalloc.c | 177 +++++++++++++----- 3 files changed, 137 insertions(+), 43 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 98ea67f27809..eaef176c597f 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3190,6 +3190,8 @@ nohugeiomap [KNL,x86,PPC] Disable kernel huge I/O mappings. + nohugevmap [KNL,x86,PPC] Disable kernel huge vmalloc mappings. + nosmt [KNL,S390] Disable symmetric multithreading (SMT). Equivalent to smt=1. diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index e3590e93bfff..8f25dbaca0a1 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -58,6 +58,7 @@ struct vm_struct { unsigned long size; unsigned long flags; struct page **pages; + unsigned int page_order; unsigned int nr_pages; phys_addr_t phys_addr; const void *caller; diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 4e5cb7c7f780..c3595d87261c 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -45,6 +45,19 @@ #include "internal.h" #include "pgalloc-track.h" +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +static bool __ro_after_init vmap_allow_huge = true; + +static int __init set_nohugevmap(char *str) +{ + vmap_allow_huge = false; + return 0; +} +early_param("nohugevmap", set_nohugevmap); +#else /* CONFIG_HAVE_ARCH_HUGE_VMAP */ +static const bool vmap_allow_huge = false; +#endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ + bool is_vmalloc_addr(const void *x) { unsigned long addr = (unsigned long)x; @@ -468,31 +481,12 @@ static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long en return 0; } -/** - * map_kernel_range_noflush - map kernel VM area with the specified pages - * @addr: start of the VM area to map - * @size: size of the VM area to map - * @prot: page protection flags to use - * @pages: pages to map - * - * Map PFN_UP(@size) pages at @addr. The VM area @addr and @size specify should - * have been allocated using get_vm_area() and its friends. - * - * NOTE: - * This function does NOT do any cache flushing. The caller is responsible for - * calling flush_cache_vmap() on to-be-mapped areas before calling this - * function. - * - * RETURNS: - * 0 on success, -errno on failure. - */ -int map_kernel_range_noflush(unsigned long addr, unsigned long size, - pgprot_t prot, struct page **pages) +static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages) { unsigned long start = addr; - unsigned long end = addr + size; - unsigned long next; pgd_t *pgd; + unsigned long next; int err = 0; int nr = 0; pgtbl_mod_mask mask = 0; @@ -514,6 +508,65 @@ int map_kernel_range_noflush(unsigned long addr, unsigned long size, return 0; } +static int vmap_pages_range_noflush(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift) +{ + WARN_ON(page_shift < PAGE_SHIFT); + + if (page_shift == PAGE_SHIFT) { + return vmap_small_pages_range_noflush(addr, end, prot, pages); + } else { + unsigned int i, nr = (end - addr) >> page_shift; + + for (i = 0; i < nr; i++) { + int err; + + err = vmap_range_noflush(addr, addr + (1UL << page_shift), + __pa(page_address(pages[i])), prot, page_shift); + if (err) + return err; + + addr += 1UL << page_shift; + } + + return 0; + } +} + +static int vmap_pages_range(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift) +{ + int err; + + err = vmap_pages_range_noflush(addr, end, prot, pages, page_shift); + flush_cache_vmap(addr, end); + return err; +} + +/** + * map_kernel_range_noflush - map kernel VM area with the specified pages + * @addr: start of the VM area to map + * @size: size of the VM area to map + * @prot: page protection flags to use + * @pages: pages to map + * + * Map PFN_UP(@size) pages at @addr. The VM area @addr and @size specify should + * have been allocated using get_vm_area() and its friends. + * + * NOTE: + * This function does NOT do any cache flushing. The caller is responsible for + * calling flush_cache_vmap() on to-be-mapped areas before calling this + * function. + * + * RETURNS: + * 0 on success, -errno on failure. + */ +int map_kernel_range_noflush(unsigned long addr, unsigned long size, + pgprot_t prot, struct page **pages) +{ + return vmap_pages_range_noflush(addr, addr + size, prot, pages, PAGE_SHIFT); +} + int map_kernel_range(unsigned long start, unsigned long size, pgprot_t prot, struct page **pages) { @@ -2274,9 +2327,11 @@ static struct vm_struct *__get_vm_area_node(unsigned long size, if (unlikely(!size)) return NULL; - if (flags & VM_IOREMAP) - align = 1ul << clamp_t(int, get_count_order_long(size), - PAGE_SHIFT, IOREMAP_MAX_ORDER); + if (flags & VM_IOREMAP) { + align = max(align, + 1ul << clamp_t(int, get_count_order_long(size), + PAGE_SHIFT, IOREMAP_MAX_ORDER)); + } area = kzalloc_node(sizeof(*area), gfp_mask & GFP_RECLAIM_MASK, node); if (unlikely(!area)) @@ -2471,11 +2526,11 @@ static void __vunmap(const void *addr, int deallocate_pages) if (deallocate_pages) { int i; - for (i = 0; i < area->nr_pages; i++) { + for (i = 0; i < area->nr_pages; i += 1 << area->page_order) { struct page *page = area->pages[i]; BUG_ON(!page); - __free_pages(page, 0); + __free_pages(page, area->page_order); } atomic_long_sub(area->nr_pages, &nr_vmalloc_pages); @@ -2614,9 +2669,12 @@ void *vmap(struct page **pages, unsigned int count, EXPORT_SYMBOL(vmap); static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, - pgprot_t prot, int node) + pgprot_t prot, unsigned int page_shift, int node) { struct page **pages; + unsigned long addr = (unsigned long)area->addr; + unsigned long size = get_vm_area_size(area); + unsigned int page_order = page_shift - PAGE_SHIFT; unsigned int nr_pages, array_size, i; const gfp_t nested_gfp = (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO; const gfp_t alloc_mask = gfp_mask | __GFP_NOWARN; @@ -2624,7 +2682,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, 0 : __GFP_HIGHMEM; - nr_pages = get_vm_area_size(area) >> PAGE_SHIFT; + nr_pages = size >> PAGE_SHIFT; array_size = (nr_pages * sizeof(struct page *)); /* Please note that the recursion is strictly bounded. */ @@ -2643,29 +2701,29 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, area->pages = pages; area->nr_pages = nr_pages; + area->page_order = page_order; - for (i = 0; i < area->nr_pages; i++) { + for (i = 0; i < area->nr_pages; i += 1 << page_order) { struct page *page; + int p; - if (node == NUMA_NO_NODE) - page = alloc_page(alloc_mask|highmem_mask); - else - page = alloc_pages_node(node, alloc_mask|highmem_mask, 0); - + page = alloc_pages_node(node, alloc_mask|highmem_mask, page_order); if (unlikely(!page)) { /* Successfully allocated i pages, free them in __vunmap() */ area->nr_pages = i; atomic_long_add(area->nr_pages, &nr_vmalloc_pages); goto fail; } - area->pages[i] = page; + + for (p = 0; p < (1 << page_order); p++) + area->pages[i + p] = page + p; + if (gfpflags_allow_blocking(gfp_mask)) cond_resched(); } atomic_long_add(area->nr_pages, &nr_vmalloc_pages); - if (map_kernel_range((unsigned long)area->addr, get_vm_area_size(area), - prot, pages) < 0) + if (vmap_pages_range(addr, addr + size, prot, pages, page_shift) < 0) goto fail; return area->addr; @@ -2701,22 +2759,45 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, pgprot_t prot, unsigned long vm_flags, int node, const void *caller) { - struct vm_struct *area; + struct vm_struct *area = NULL; void *addr; unsigned long real_size = size; + unsigned long real_align = align; + unsigned int shift = PAGE_SHIFT; size = PAGE_ALIGN(size); if (!size || (size >> PAGE_SHIFT) > totalram_pages()) goto fail; - area = __get_vm_area_node(real_size, align, VM_ALLOC | VM_UNINITIALIZED | + if (vmap_allow_huge && (pgprot_val(prot) == pgprot_val(PAGE_KERNEL))) { + unsigned long size_per_node; + + /* + * Try huge pages. Only try for PAGE_KERNEL allocations, + * others like modules don't yet expect huge pages in + * their allocations due to apply_to_page_range not + * supporting them. + */ + + size_per_node = size; + if (node == NUMA_NO_NODE) + size_per_node /= num_online_nodes(); + if (size_per_node >= PMD_SIZE) { + shift = PMD_SHIFT; + align = max(real_align, 1UL << shift); + size = ALIGN(real_size, 1UL << shift); + } + } + +again: + area = __get_vm_area_node(size, align, VM_ALLOC | VM_UNINITIALIZED | vm_flags, start, end, node, gfp_mask, caller); if (!area) goto fail; - addr = __vmalloc_area_node(area, gfp_mask, prot, node); + addr = __vmalloc_area_node(area, gfp_mask, prot, shift, node); if (!addr) - return NULL; + goto fail; /* * In this function, newly allocated vm_struct has VM_UNINITIALIZED @@ -2730,8 +2811,18 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, return addr; fail: - warn_alloc(gfp_mask, NULL, + if (shift > PAGE_SHIFT) { + shift = PAGE_SHIFT; + align = real_align; + size = real_size; + goto again; + } + + if (!area) { + /* Warn for area allocation, page allocations already warn */ + warn_alloc(gfp_mask, NULL, "vmalloc: allocation failure: %lu bytes", real_size); + } return NULL; }