From patchwork Mon Apr 17 19:11:21 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 9684289 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id F3CB5600F6 for ; Mon, 17 Apr 2017 19:17:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E3FCE223A6 for ; Mon, 17 Apr 2017 19:17:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D8BFF27D4D; Mon, 17 Apr 2017 19:17:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 0BE39223A6 for ; Mon, 17 Apr 2017 19:17:56 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 13F5EC052505; Mon, 17 Apr 2017 19:17:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 13F5EC052505 Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=intel.com Authentication-Results: ext-mx07.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=dm-devel-bounces@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 13F5EC052505 Received: from colo-mx.corp.redhat.com (unknown [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E48A560461; Mon, 17 Apr 2017 19:17:54 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id AB1AA5ED65; Mon, 17 Apr 2017 19:17:54 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id v3HJHqOX026727 for ; Mon, 17 Apr 2017 15:17:52 -0400 Received: by smtp.corp.redhat.com (Postfix) id 7C08C60461; Mon, 17 Apr 2017 19:17:52 +0000 (UTC) Delivered-To: dm-devel@redhat.com Received: from mx1.redhat.com (ext-mx05.extmail.prod.ext.phx2.redhat.com [10.5.110.29]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 721691F8; Mon, 17 Apr 2017 19:17:50 +0000 (UTC) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7DC4A2EF165; Mon, 17 Apr 2017 19:17:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 7DC4A2EF165 Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=intel.com Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=dan.j.williams@intel.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 7DC4A2EF165 Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Apr 2017 12:17:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos; i="5.37,215,1488873600"; d="scan'208"; a="1136849211" Received: from dwillia2-desk3.jf.intel.com (HELO dwillia2-desk3.amr.corp.intel.com) ([10.54.39.125]) by fmsmga001.fm.intel.com with ESMTP; 17 Apr 2017 12:17:04 -0700 From: Dan Williams To: linux-nvdimm@ml01.01.org Date: Mon, 17 Apr 2017 12:11:21 -0700 Message-ID: <149245628098.10206.16202709432908862265.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <149245612770.10206.15496018295337908594.stgit@dwillia2-desk3.amr.corp.intel.com> References: <149245612770.10206.15496018295337908594.stgit@dwillia2-desk3.amr.corp.intel.com> User-Agent: StGit/0.17.1-9-g687f MIME-Version: 1.0 X-Greylist: Sender passed SPF test, Sender IP whitelisted by DNSRBL, ACL 203 matched, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Mon, 17 Apr 2017 19:17:37 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Mon, 17 Apr 2017 19:17:37 +0000 (UTC) for IP:'134.134.136.65' DOMAIN:'mga03.intel.com' HELO:'mga03.intel.com' FROM:'dan.j.williams@intel.com' RCPT:'' X-RedHat-Spam-Score: 0.698 (BAYES_50, DCC_REPUT_13_19, RP_MATCHES_RCVD, SPF_PASS) 134.134.136.65 mga03.intel.com 134.134.136.65 mga03.intel.com X-Scanned-By: MIMEDefang 2.78 on 10.5.110.29 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-loop: dm-devel@redhat.com Cc: Tony Luck , Jan Kara , Ross Zwisler , Toshi Kani , Matthew Wilcox , x86@kernel.org, linux-kernel@vger.kernel.org, Jeff Moyer , linux-block@vger.kernel.org, dm-devel@redhat.com, Ingo Molnar , Oliver O'Halloran , Al Viro , "H. Peter Anvin" , linux-fsdevel@vger.kernel.org, Thomas Gleixner , Linus Torvalds , hch@lst.de Subject: [dm-devel] [resend PATCH v2 28/33] x86, libnvdimm, dax: stop abusing __copy_user_nocache X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Mon, 17 Apr 2017 19:17:55 +0000 (UTC) X-Virus-Scanned: ClamAV using ClamSMTP The pmem and nd_blk drivers both have need to copy data through the cpu cache to persistent memory. To date they have been abusing __copy_user_nocache through the memcpy_to_pmem abstraction, but this has several problems: * __copy_user_nocache does not guarantee that it will always avoid the cache. While we have fixed the cases where the pmem usage might trigger that behavior it's a fragile assumption and burdens the uaccess.h implementation with worrying about the distinction between 'nocache' and the stricter write-through semantic needed by pmem. Quoting Linus: "Quite frankly, the whole "memcpy_nocache()" idea or (ab-)using copy_user_nocache() just needs to die. ... If some driver ends up using "movnt" by hand, that is up to that *driver*." * It implements SMAP (supervisor mode access protection) which is only meant for user copies. * It expects faults. For in-kernel copies, faults are fatal and we should not be coding for exception handling in that case. __arch_memcpy_to_pmem() is effectively a copy of __copy_user_nocache() minus SMAP, unaligned support, and exception handling. The configuration symbol ARCH_HAS_PMEM_API is also moved local to libnvdimm to be next to the implementation. Cc: Cc: Jan Kara Cc: Jeff Moyer Cc: Ingo Molnar Cc: Christoph Hellwig Cc: Toshi Kani Cc: Tony Luck Cc: "H. Peter Anvin" Cc: Al Viro Cc: Thomas Gleixner Cc: Oliver O'Halloran Cc: Matthew Wilcox Cc: Ross Zwisler Cc: Linus Torvalds Signed-off-by: Dan Williams --- MAINTAINERS | 2 - arch/x86/Kconfig | 1 - arch/x86/include/asm/pmem.h | 48 ----------------------------- drivers/acpi/nfit/core.c | 3 +- drivers/nvdimm/Kconfig | 4 ++ drivers/nvdimm/claim.c | 4 +- drivers/nvdimm/namespace_devs.c | 1 - drivers/nvdimm/pmem.c | 4 +- drivers/nvdimm/region_devs.c | 1 - drivers/nvdimm/x86.c | 65 +++++++++++++++++++++++++++++++++++++++ fs/dax.c | 1 - include/linux/libnvdimm.h | 9 +++++ include/linux/pmem.h | 59 ----------------------------------- lib/Kconfig | 3 -- 14 files changed, 83 insertions(+), 122 deletions(-) delete mode 100644 arch/x86/include/asm/pmem.h delete mode 100644 include/linux/pmem.h -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel diff --git a/MAINTAINERS b/MAINTAINERS index 819d5e8b668a..1c4da1bebd7c 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -7458,8 +7458,6 @@ L: linux-nvdimm@lists.01.org Q: https://patchwork.kernel.org/project/linux-nvdimm/list/ S: Supported F: drivers/nvdimm/pmem.c -F: include/linux/pmem.h -F: arch/*/include/asm/pmem.h LIGHTNVM PLATFORM SUPPORT M: Matias Bjorling diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index cc98d5a294ee..d377da696903 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -53,7 +53,6 @@ config X86 select ARCH_HAS_GCOV_PROFILE_ALL select ARCH_HAS_KCOV if X86_64 select ARCH_HAS_MMIO_FLUSH - select ARCH_HAS_PMEM_API if X86_64 select ARCH_HAS_SET_MEMORY select ARCH_HAS_SG_CHAIN select ARCH_HAS_STRICT_KERNEL_RWX diff --git a/arch/x86/include/asm/pmem.h b/arch/x86/include/asm/pmem.h deleted file mode 100644 index ded2541a7ba9..000000000000 --- a/arch/x86/include/asm/pmem.h +++ /dev/null @@ -1,48 +0,0 @@ -/* - * Copyright(c) 2015 Intel Corporation. All rights reserved. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of version 2 of the GNU General Public License as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ -#ifndef __ASM_X86_PMEM_H__ -#define __ASM_X86_PMEM_H__ - -#include -#include -#include -#include - -#ifdef CONFIG_ARCH_HAS_PMEM_API -/** - * arch_memcpy_to_pmem - copy data to persistent memory - * @dst: destination buffer for the copy - * @src: source buffer for the copy - * @n: length of the copy in bytes - * - * Copy data to persistent memory media via non-temporal stores so that - * a subsequent pmem driver flush operation will drain posted write queues. - */ -static inline void arch_memcpy_to_pmem(void *dst, const void *src, size_t n) -{ - int rem; - - /* - * We are copying between two kernel buffers, if - * __copy_from_user_inatomic_nocache() returns an error (page - * fault) we would have already reported a general protection fault - * before the WARN+BUG. - */ - rem = __copy_from_user_inatomic_nocache(dst, (void __user *) src, n); - if (WARN(rem, "%s: fault copying %p <- %p unwritten: %d\n", - __func__, dst, src, rem)) - BUG(); -} - -#endif /* CONFIG_ARCH_HAS_PMEM_API */ -#endif /* __ASM_X86_PMEM_H__ */ diff --git a/drivers/acpi/nfit/core.c b/drivers/acpi/nfit/core.c index d0c07b2344e4..8b4c6212737c 100644 --- a/drivers/acpi/nfit/core.c +++ b/drivers/acpi/nfit/core.c @@ -20,7 +20,6 @@ #include #include #include -#include #include #include #include @@ -1776,7 +1775,7 @@ static int acpi_nfit_blk_single_io(struct nfit_blk *nfit_blk, } if (rw) - memcpy_to_pmem(mmio->addr.aperture + offset, + arch_memcpy_to_pmem(mmio->addr.aperture + offset, iobuf + copied, c); else { if (nfit_blk->dimm_flags & NFIT_BLK_READ_FLUSH) diff --git a/drivers/nvdimm/Kconfig b/drivers/nvdimm/Kconfig index 5bdd499b5f4f..4d45196d6f94 100644 --- a/drivers/nvdimm/Kconfig +++ b/drivers/nvdimm/Kconfig @@ -36,6 +36,10 @@ config BLK_DEV_PMEM Say Y if you want to use an NVDIMM +config ARCH_HAS_PMEM_API + depends on X86_64 + def_bool y + config ND_BLK tristate "BLK: Block data window (aperture) device support" default LIBNVDIMM diff --git a/drivers/nvdimm/claim.c b/drivers/nvdimm/claim.c index 1e13a196ce4b..0b222c8ce1d8 100644 --- a/drivers/nvdimm/claim.c +++ b/drivers/nvdimm/claim.c @@ -10,9 +10,9 @@ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * General Public License for more details. */ +#include #include #include -#include #include "nd-core.h" #include "pmem.h" #include "pfn.h" @@ -267,7 +267,7 @@ static int nsio_rw_bytes(struct nd_namespace_common *ndns, rc = -EIO; } - memcpy_to_pmem(nsio->addr + offset, buf, size); + arch_memcpy_to_pmem(nsio->addr + offset, buf, size); nvdimm_flush(to_nd_region(ndns->dev.parent)); return rc; diff --git a/drivers/nvdimm/namespace_devs.c b/drivers/nvdimm/namespace_devs.c index 1b481a5fb966..2580f6655bea 100644 --- a/drivers/nvdimm/namespace_devs.c +++ b/drivers/nvdimm/namespace_devs.c @@ -14,7 +14,6 @@ #include #include #include -#include #include #include #include "nd-core.h" diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c index 769a510c20e8..329895ca88e1 100644 --- a/drivers/nvdimm/pmem.c +++ b/drivers/nvdimm/pmem.c @@ -27,8 +27,8 @@ #include #include #include -#include #include +#include #include #include "pmem.h" #include "pfn.h" @@ -79,7 +79,7 @@ static void write_pmem(void *pmem_addr, struct page *page, { void *mem = kmap_atomic(page); - memcpy_to_pmem(pmem_addr, mem + off, len); + arch_memcpy_to_pmem(pmem_addr, mem + off, len); kunmap_atomic(mem); } diff --git a/drivers/nvdimm/region_devs.c b/drivers/nvdimm/region_devs.c index b7cb5066d961..307a48060aa3 100644 --- a/drivers/nvdimm/region_devs.c +++ b/drivers/nvdimm/region_devs.c @@ -15,7 +15,6 @@ #include #include #include -#include #include #include #include diff --git a/drivers/nvdimm/x86.c b/drivers/nvdimm/x86.c index 07478ed7ce97..d99b452332a9 100644 --- a/drivers/nvdimm/x86.c +++ b/drivers/nvdimm/x86.c @@ -40,3 +40,68 @@ void arch_invalidate_pmem(void *addr, size_t size) clflush_cache_range(addr, size); } EXPORT_SYMBOL_GPL(arch_invalidate_pmem); + +void arch_memcpy_to_pmem(void *_dst, void *_src, unsigned size) +{ + unsigned long dest = (unsigned long) _dst; + unsigned long source = (unsigned long) _src; + + /* cache copy and flush to align dest */ + if (!IS_ALIGNED(dest, 8)) { + unsigned len = min_t(unsigned, size, ALIGN(dest, 8) - dest); + + memcpy((void *) dest, (void *) source, len); + arch_wb_cache_pmem((void *) dest, len); + dest += len; + source += len; + size -= len; + if (!size) + return; + } + + /* 4x8 movnti loop */ + while (size >= 32) { + asm("movq (%0), %%r8\n" + "movq 8(%0), %%r9\n" + "movq 16(%0), %%r10\n" + "movq 24(%0), %%r11\n" + "movnti %%r8, (%1)\n" + "movnti %%r9, 8(%1)\n" + "movnti %%r10, 16(%1)\n" + "movnti %%r11, 24(%1)\n" + :: "r" (source), "r" (dest) + : "memory", "r8", "r9", "r10", "r11"); + dest += 32; + source += 32; + size -= 32; + } + + /* 1x8 movnti loop */ + while (size >= 8) { + asm("movq (%0), %%r8\n" + "movnti %%r8, (%1)\n" + :: "r" (source), "r" (dest) + : "memory", "r8"); + dest += 8; + source += 8; + size -= 8; + } + + /* 1x4 movnti loop */ + while (size >= 4) { + asm("movl (%0), %%r8d\n" + "movnti %%r8d, (%1)\n" + :: "r" (source), "r" (dest) + : "memory", "r8"); + dest += 4; + source += 4; + size -= 4; + } + + /* cache copy for remaining bytes */ + if (size) { + memcpy((void *) dest, (void *) source, size); + arch_wb_cache_pmem((void *) dest, size); + } +} +EXPORT_SYMBOL_GPL(arch_memcpy_to_pmem); diff --git a/fs/dax.c b/fs/dax.c index edee7e8298bc..f37ed21e4093 100644 --- a/fs/dax.c +++ b/fs/dax.c @@ -25,7 +25,6 @@ #include #include #include -#include #include #include #include diff --git a/include/linux/libnvdimm.h b/include/linux/libnvdimm.h index 77e7af32543f..a98004745768 100644 --- a/include/linux/libnvdimm.h +++ b/include/linux/libnvdimm.h @@ -162,4 +162,13 @@ void nd_region_release_lane(struct nd_region *nd_region, unsigned int lane); u64 nd_fletcher64(void *addr, size_t len, bool le); void nvdimm_flush(struct nd_region *nd_region); int nvdimm_has_flush(struct nd_region *nd_region); +#ifdef CONFIG_ARCH_HAS_PMEM_API +void arch_memcpy_to_pmem(void *dst, void *src, unsigned size); +#define ARCH_MEMREMAP_PMEM MEMREMAP_WB +#else +static inline void arch_memcpy_to_pmem(void *dst, void *src, unsigned size) +{ +} +#define ARCH_MEMREMAP_PMEM MEMREMAP_WT +#endif /* CONFIG_ARCH_HAS_PMEM_API */ #endif /* __LIBNVDIMM_H__ */ diff --git a/include/linux/pmem.h b/include/linux/pmem.h deleted file mode 100644 index 559c00848583..000000000000 --- a/include/linux/pmem.h +++ /dev/null @@ -1,59 +0,0 @@ -/* - * Copyright(c) 2015 Intel Corporation. All rights reserved. - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of version 2 of the GNU General Public License as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License for more details. - */ -#ifndef __PMEM_H__ -#define __PMEM_H__ - -#include -#include - -#ifdef CONFIG_ARCH_HAS_PMEM_API -#define ARCH_MEMREMAP_PMEM MEMREMAP_WB -#include -#else -#define ARCH_MEMREMAP_PMEM MEMREMAP_WT -/* - * These are simply here to enable compilation, all call sites gate - * calling these symbols with arch_has_pmem_api() and redirect to the - * implementation in asm/pmem.h. - */ -static inline void arch_memcpy_to_pmem(void *dst, const void *src, size_t n) -{ - BUG(); -} -#endif - -static inline bool arch_has_pmem_api(void) -{ - return IS_ENABLED(CONFIG_ARCH_HAS_PMEM_API); -} - -/** - * memcpy_to_pmem - copy data to persistent memory - * @dst: destination buffer for the copy - * @src: source buffer for the copy - * @n: length of the copy in bytes - * - * Perform a memory copy that results in the destination of the copy - * being effectively evicted from, or never written to, the processor - * cache hierarchy after the copy completes. After memcpy_to_pmem() - * data may still reside in cpu or platform buffers, so this operation - * must be followed by a blkdev_issue_flush() on the pmem block device. - */ -static inline void memcpy_to_pmem(void *dst, const void *src, size_t n) -{ - if (arch_has_pmem_api()) - arch_memcpy_to_pmem(dst, src, n); - else - memcpy(dst, src, n); -} -#endif /* __PMEM_H__ */ diff --git a/lib/Kconfig b/lib/Kconfig index 0c8b78a9ae2e..0c4aac6ef394 100644 --- a/lib/Kconfig +++ b/lib/Kconfig @@ -545,9 +545,6 @@ config SG_POOL config ARCH_HAS_SG_CHAIN def_bool n -config ARCH_HAS_PMEM_API - bool - config ARCH_HAS_MMIO_FLUSH bool