From patchwork Wed May 27 15:19:03 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Toshi Kani X-Patchwork-Id: 6490621 Return-Path: X-Original-To: patchwork-linux-nvdimm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 2D06A9F1CC for ; Wed, 27 May 2015 15:39:26 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 04EC12068F for ; Wed, 27 May 2015 15:39:25 +0000 (UTC) Received: from ml01.01.org (ml01.01.org [198.145.21.10]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C4259206B1 for ; Wed, 27 May 2015 15:39:23 +0000 (UTC) Received: from ml01.vlan14.01.org (localhost [IPv6:::1]) by ml01.01.org (Postfix) with ESMTP id B73381827E6; Wed, 27 May 2015 08:39:23 -0700 (PDT) X-Original-To: linux-nvdimm@lists.01.org Delivered-To: linux-nvdimm@lists.01.org Received: from g2t2354.austin.hp.com (g2t2354.austin.hp.com [15.217.128.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 1EE031827CC for ; Wed, 27 May 2015 08:39:22 -0700 (PDT) Received: from g5t1633.atlanta.hp.com (g5t1633.atlanta.hp.com [16.201.144.132]) by g2t2354.austin.hp.com (Postfix) with ESMTP id 32598C3; Wed, 27 May 2015 15:39:21 +0000 (UTC) Received: from misato.fc.hp.com (misato.fc.hp.com [16.78.168.61]) by g5t1633.atlanta.hp.com (Postfix) with ESMTP id 83B9A97; Wed, 27 May 2015 15:39:09 +0000 (UTC) From: Toshi Kani To: hpa@zytor.com, tglx@linutronix.de, mingo@redhat.com, akpm@linux-foundation.org, arnd@arndb.de Subject: [PATCH v10 11/12] x86, mm, pat: Refactor !pat_enabled handling Date: Wed, 27 May 2015 09:19:03 -0600 Message-Id: <1432739944-22633-12-git-send-email-toshi.kani@hp.com> X-Mailer: git-send-email 1.9.3 In-Reply-To: <1432739944-22633-1-git-send-email-toshi.kani@hp.com> References: <1432739944-22633-1-git-send-email-toshi.kani@hp.com> Cc: jgross@suse.com, konrad.wilk@oracle.com, linux-nvdimm@lists.01.org, mcgrof@suse.com, x86@kernel.org, linux-kernel@vger.kernel.org, stefan.bader@canonical.com, luto@amacapital.net, linux-mm@kvack.org, hmh@hmh.eng.br, hch@lst.de X-BeenThere: linux-nvdimm@lists.01.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "Linux-nvdimm developer list." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch refactors the !pat_enabled code paths and integrates them into the PAT abstraction code. The PAT table is emulated by corresponding to the two cache attribute bits, PWT (Write Through) and PCD (Cache Disable). The emulated PAT table is the same as the BIOS default setup when the system has PAT but the "nopat" boot option is specified. The emulated PAT table is also used when MSR_IA32_CR_PAT returns 0 (9d34cfdf4). Signed-off-by: Toshi Kani Reviewed-by: Juergen Gross --- arch/x86/mm/init.c | 6 ++-- arch/x86/mm/iomap_32.c | 12 ++++--- arch/x86/mm/ioremap.c | 10 +----- arch/x86/mm/pageattr.c | 6 ---- arch/x86/mm/pat.c | 77 +++++++++++++++++++++++++++++------------------- 5 files changed, 57 insertions(+), 54 deletions(-) diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index 1d55318..8533b46 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -40,7 +40,7 @@ */ uint16_t __cachemode2pte_tbl[_PAGE_CACHE_MODE_NUM] = { [_PAGE_CACHE_MODE_WB ] = 0 | 0 , - [_PAGE_CACHE_MODE_WC ] = _PAGE_PWT | 0 , + [_PAGE_CACHE_MODE_WC ] = 0 | _PAGE_PCD, [_PAGE_CACHE_MODE_UC_MINUS] = 0 | _PAGE_PCD, [_PAGE_CACHE_MODE_UC ] = _PAGE_PWT | _PAGE_PCD, [_PAGE_CACHE_MODE_WT ] = 0 | _PAGE_PCD, @@ -50,11 +50,11 @@ EXPORT_SYMBOL(__cachemode2pte_tbl); uint8_t __pte2cachemode_tbl[8] = { [__pte2cm_idx( 0 | 0 | 0 )] = _PAGE_CACHE_MODE_WB, - [__pte2cm_idx(_PAGE_PWT | 0 | 0 )] = _PAGE_CACHE_MODE_WC, + [__pte2cm_idx(_PAGE_PWT | 0 | 0 )] = _PAGE_CACHE_MODE_UC_MINUS, [__pte2cm_idx( 0 | _PAGE_PCD | 0 )] = _PAGE_CACHE_MODE_UC_MINUS, [__pte2cm_idx(_PAGE_PWT | _PAGE_PCD | 0 )] = _PAGE_CACHE_MODE_UC, [__pte2cm_idx( 0 | 0 | _PAGE_PAT)] = _PAGE_CACHE_MODE_WB, - [__pte2cm_idx(_PAGE_PWT | 0 | _PAGE_PAT)] = _PAGE_CACHE_MODE_WC, + [__pte2cm_idx(_PAGE_PWT | 0 | _PAGE_PAT)] = _PAGE_CACHE_MODE_UC_MINUS, [__pte2cm_idx(0 | _PAGE_PCD | _PAGE_PAT)] = _PAGE_CACHE_MODE_UC_MINUS, [__pte2cm_idx(_PAGE_PWT | _PAGE_PCD | _PAGE_PAT)] = _PAGE_CACHE_MODE_UC, }; diff --git a/arch/x86/mm/iomap_32.c b/arch/x86/mm/iomap_32.c index 9ca35fc..2c51a2b 100644 --- a/arch/x86/mm/iomap_32.c +++ b/arch/x86/mm/iomap_32.c @@ -77,13 +77,13 @@ void __iomem * iomap_atomic_prot_pfn(unsigned long pfn, pgprot_t prot) { /* - * For non-PAT systems, promote PAGE_KERNEL_WC to PAGE_KERNEL_UC_MINUS. - * PAGE_KERNEL_WC maps to PWT, which translates to uncached if the - * MTRR is UC or WC. UC_MINUS gets the real intention, of the - * user, which is "WC if the MTRR is WC, UC if you can't do that." + * For non-PAT systems, translate non-WB request to UC- just in + * case the caller set the PWT bit to prot directly without using + * pgprot_writecombine(). UC- translates to uncached if the MTRR + * is UC or WC. UC- gets the real intention, of the user, which is + * "WC if the MTRR is WC, UC if you can't do that." */ - if (!pat_enabled && pgprot_val(prot) == - (__PAGE_KERNEL | cachemode2protval(_PAGE_CACHE_MODE_WC))) + if (!pat_enabled && pgprot2cachemode(prot) != _PAGE_CACHE_MODE_WB) prot = __pgprot(__PAGE_KERNEL | cachemode2protval(_PAGE_CACHE_MODE_UC_MINUS)); diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c index 7e702dc..f966129 100644 --- a/arch/x86/mm/ioremap.c +++ b/arch/x86/mm/ioremap.c @@ -265,11 +265,8 @@ EXPORT_SYMBOL(ioremap_nocache); */ void __iomem *ioremap_wc(resource_size_t phys_addr, unsigned long size) { - if (pat_enabled) - return __ioremap_caller(phys_addr, size, _PAGE_CACHE_MODE_WC, + return __ioremap_caller(phys_addr, size, _PAGE_CACHE_MODE_WC, __builtin_return_address(0)); - else - return ioremap_nocache(phys_addr, size); } EXPORT_SYMBOL(ioremap_wc); @@ -285,11 +282,8 @@ EXPORT_SYMBOL(ioremap_wc); */ void __iomem *ioremap_wt(resource_size_t phys_addr, unsigned long size) { - if (pat_enabled) - return __ioremap_caller(phys_addr, size, _PAGE_CACHE_MODE_WT, + return __ioremap_caller(phys_addr, size, _PAGE_CACHE_MODE_WT, __builtin_return_address(0)); - else - return ioremap_nocache(phys_addr, size); } EXPORT_SYMBOL(ioremap_wt); diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c index 6427273..5a25e95 100644 --- a/arch/x86/mm/pageattr.c +++ b/arch/x86/mm/pageattr.c @@ -1578,9 +1578,6 @@ int set_memory_wc(unsigned long addr, int numpages) { int ret; - if (!pat_enabled) - return set_memory_uc(addr, numpages); - ret = reserve_memtype(__pa(addr), __pa(addr) + numpages * PAGE_SIZE, _PAGE_CACHE_MODE_WC, NULL); if (ret) @@ -1604,9 +1601,6 @@ int set_memory_wt(unsigned long addr, int numpages) { int ret; - if (!pat_enabled) - return set_memory_uc(addr, numpages); - ret = reserve_memtype(__pa(addr), __pa(addr) + numpages * PAGE_SIZE, _PAGE_CACHE_MODE_WT, NULL); if (ret) diff --git a/arch/x86/mm/pat.c b/arch/x86/mm/pat.c index 7cfd995..0533867 100644 --- a/arch/x86/mm/pat.c +++ b/arch/x86/mm/pat.c @@ -186,7 +186,11 @@ void pat_init_cache_modes(void) char pat_msg[33]; u64 pat; - rdmsrl(MSR_IA32_CR_PAT, pat); + if (pat_enabled) + rdmsrl(MSR_IA32_CR_PAT, pat); + else + pat = boot_pat_state; + pat_msg[32] = 0; for (i = 7; i >= 0; i--) { cache = pat_get_cache_mode((pat >> (i * 8)) & 7, @@ -204,21 +208,16 @@ void pat_init(void) struct cpuinfo_x86 *c = &boot_cpu_data; static bool boot_cpu_done; - if (!pat_enabled) - return; - if (!boot_cpu_done) { - if (!cpu_has_pat) { + if (!cpu_has_pat) pat_disable("PAT not supported by CPU."); - return; - } - rdmsrl(MSR_IA32_CR_PAT, boot_pat_state); - if (!boot_pat_state) { - pat_disable("PAT read returns always zero, disabled."); - return; + if (pat_enabled) { + rdmsrl(MSR_IA32_CR_PAT, boot_pat_state); + if (!boot_pat_state) + pat_disable("PAT read returns always zero, disabled."); } - } else if (!cpu_has_pat) { + } else if (!cpu_has_pat && pat_enabled) { /* * If this happens we are on a secondary CPU, but * switched to PAT on the boot CPU. We have no way to @@ -228,9 +227,35 @@ void pat_init(void) BUG(); } - if ((c->x86_vendor == X86_VENDOR_INTEL) && - (((c->x86 == 0x6) && (c->x86_model <= 0xd)) || - ((c->x86 == 0xf) && (c->x86_model <= 0x6)))) { + if (!pat_enabled) { + /* + * No PAT. Emulate the PAT table that corresponds to the two + * cache bits, PWT (Write Through) and PCD (Cache Disable). + * This setup is the same as the BIOS default setup when the + * system has PAT but the "nopat" boot option is specified. + * This emulated PAT table is also used when MSR_IA32_CR_PAT + * returns 0. + * + * PTE encoding used in Linux: + * PCD + * |PWT PAT + * || slot + * 00 0 WB : _PAGE_CACHE_MODE_WB + * 01 1 WT : _PAGE_CACHE_MODE_WT + * 10 2 UC-: _PAGE_CACHE_MODE_UC_MINUS + * 11 3 UC : _PAGE_CACHE_MODE_UC + * + * NOTE: When WC or WP is used, it is redirected to UC- per + * the default setup in __cachemode2pte_tbl[]. + */ + pat = PAT(0, WB) | PAT(1, WT) | PAT(2, UC_MINUS) | PAT(3, UC) | + PAT(4, WB) | PAT(5, WT) | PAT(6, UC_MINUS) | PAT(7, UC); + if (!boot_pat_state) + boot_pat_state = pat; + + } else if ((c->x86_vendor == X86_VENDOR_INTEL) && + (((c->x86 == 0x6) && (c->x86_model <= 0xd)) || + ((c->x86 == 0xf) && (c->x86_model <= 0x6)))) { /* * PAT support with the lower four entries. Intel Pentium 2, * 3, M, and 4 are affected by PAT errata, which makes the @@ -284,7 +309,8 @@ void pat_init(void) PAT(4, WB) | PAT(5, WC) | PAT(6, UC_MINUS) | PAT(7, WT); } - wrmsrl(MSR_IA32_CR_PAT, pat); + if (pat_enabled) + wrmsrl(MSR_IA32_CR_PAT, pat); if (!boot_cpu_done) { pat_init_cache_modes(); @@ -457,13 +483,8 @@ int reserve_memtype(u64 start, u64 end, enum page_cache_mode req_type, BUG_ON(start >= end); /* end is exclusive */ if (!pat_enabled) { - /* WB and UC- are the only types supported without PAT */ - if (new_type) { - if (req_type == _PAGE_CACHE_MODE_WB) - *new_type = _PAGE_CACHE_MODE_WB; - else - *new_type = _PAGE_CACHE_MODE_UC_MINUS; - } + if (new_type) + *new_type = req_type; return 0; } @@ -962,21 +983,15 @@ void untrack_pfn(struct vm_area_struct *vma, unsigned long pfn, pgprot_t pgprot_writecombine(pgprot_t prot) { - if (pat_enabled) - return __pgprot(pgprot_val(prot) | + return __pgprot(pgprot_val(prot) | cachemode2protval(_PAGE_CACHE_MODE_WC)); - else - return pgprot_noncached(prot); } EXPORT_SYMBOL_GPL(pgprot_writecombine); pgprot_t pgprot_writethrough(pgprot_t prot) { - if (pat_enabled) - return __pgprot(pgprot_val(prot) | + return __pgprot(pgprot_val(prot) | cachemode2protval(_PAGE_CACHE_MODE_WT)); - else - return pgprot_noncached(prot); } EXPORT_SYMBOL_GPL(pgprot_writethrough);