From patchwork Sun Jun 6 09:04:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 12301945 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7541C47096 for ; Sun, 6 Jun 2021 09:05:47 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9DE436142E for ; Sun, 6 Jun 2021 09:05:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9DE436142E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=vSr/jYZCiFB1WZ8f19JKuLJ3SGvHXGrMGgjddt1bcK4=; b=GcHxPPL5en3CHC /LeHKDPqN5FJmqwGMpAcJ8yx5JmuSbwTkqYBMXUc3WVAr/gdUy2lIaXb0DyGX0byx/v8lnl5Le3vG K2eNJD+I4lmfPGsYRz9jm48MlbHmfPGaPmW4nMjVWWDdDk1t4OaI2vEes/NW5LQGzs1ia3qOtuE9z QxqgOJoXyhrhXqAjhnW3O0JrZszpXoSsPUD579ImLGE7ueZFDobmJLY8zUYJLN6P4KbBLxI++Eop+ T815ijnMUaPnx0ZxShm4+VNKatCbJsUiQIybNopwOmn2dXsqLxuLSY6DLuGRGnrwmTu2DjwuyjBc/ 4X4HTvEIUS9dwemFtLEg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lpoiw-0002oT-AD; Sun, 06 Jun 2021 09:05:38 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lpoit-0002nB-G5 for linux-riscv@lists.infradead.org; Sun, 06 Jun 2021 09:05:37 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 459E561420; Sun, 6 Jun 2021 09:05:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1622970335; bh=HSjK8sb9rNp8fRAH1dUnUoJsw31olmcJ8Vy6Ym+Boxs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ToFie3H4Ftu8Li8fo8KSiPRR2dPZnYOYsbB3/3iMWbEhu4jX2HNne30ldox9bzZgQ phI7JSwoclyv+6K2NVR6/U9VOQuGNVgkF3+jPBjSqv7aXm+EeuYOtP2JgutKKUV1No 58cEYOzFiRE59XdeqHhCV8ivTlDY1wEnLGeZx9iDz6WF0++8afEUxzCOLZPyHExBKA gmbGqvWBLvdlQjPxQ5axvV6wnScCTrJMKqgTC3o7XHnBHmI3HH0COPzCi1fceCtGSl LCqxdc2nTX4ewqFS4BPX3MBjDURNBL7dm3AN94b/4vuSZfqkRvbhkmzsVWic0/LqeQ AN2oYO9JmiRhA== From: guoren@kernel.org To: guoren@kernel.org, anup.patel@wdc.com, palmerdabbelt@google.com, arnd@arndb.de, wens@csie.org, maxime@cerno.tech, drew@beagleboard.org, liush@allwinnertech.com, lazyparser@gmail.com, wefu@redhat.com Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-sunxi@lists.linux.dev, Guo Ren , Christoph Hellwig Subject: [RFC PATCH v2 07/11] riscv: cmo: Add dma-noncoherency support Date: Sun, 6 Jun 2021 09:04:05 +0000 Message-Id: <1622970249-50770-11-git-send-email-guoren@kernel.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1622970249-50770-1-git-send-email-guoren@kernel.org> References: <1622970249-50770-1-git-send-email-guoren@kernel.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210606_020535_598165_24EDF2D7 X-CRM114-Status: GOOD ( 20.95 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Guo Ren To support DMA device in a non-coherent interconnect SOC system, we need the below facilities: - Changing a virtual memory mapping region attributes from cacheable to noncache + strong order which used in DMA descriptors. - Add noncache + weakorder virtual memory attributes for dma mapping. - Syncing the cache with memory before DMA start and after DMA end with vendor custom CMO instructions. This patch enables linux kernel generic dma-noncoherency infrastructure and introduces new sbi_ecall API for dma_sync. @@ -27,6 +27,7 @@ enum sbi_ext_id { + SBI_EXT_DMA = 0xAB150401, Signed-off-by: Guo Ren Signed-off-by: Liu Shaohua Cc: Palmer Dabbelt Cc: Christoph Hellwig Cc: Anup Patel Cc: Arnd Bergmann Cc: Drew Fustini Cc: Wei Fu Cc: Wei Wu Cc: Chen-Yu Tsai Cc: Maxime Ripard --- arch/riscv/Kconfig | 5 ++++ arch/riscv/include/asm/pgtable.h | 26 ++++++++++++++++++++ arch/riscv/include/asm/sbi.h | 15 ++++++++++++ arch/riscv/kernel/sbi.c | 19 ++++++++++++++ arch/riscv/mm/Makefile | 1 + arch/riscv/mm/dma-mapping.c | 53 ++++++++++++++++++++++++++++++++++++++++ 6 files changed, 119 insertions(+) create mode 100644 arch/riscv/mm/dma-mapping.c diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 05c4976..817a9bb 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -20,6 +20,10 @@ config RISCV select ARCH_HAS_DEBUG_VM_PGTABLE select ARCH_HAS_DEBUG_VIRTUAL if MMU select ARCH_HAS_DEBUG_WX + select ARCH_HAS_DMA_PREP_COHERENT + select ARCH_HAS_SYNC_DMA_FOR_CPU + select ARCH_HAS_SYNC_DMA_FOR_DEVICE + select ARCH_HAS_DMA_WRITE_COMBINE select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_GCOV_PROFILE_ALL select ARCH_HAS_GIGANTIC_PAGE @@ -43,6 +47,7 @@ config RISCV select CLONE_BACKWARDS select CLINT_TIMER if !MMU select COMMON_CLK + select DMA_DIRECT_REMAP select EDAC_SUPPORT select GENERIC_ARCH_TOPOLOGY if SMP select GENERIC_ATOMIC64 if !64BIT diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 6ddeb49..e1a82b6 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -462,6 +462,32 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, return ptep_test_and_clear_young(vma, address, ptep); } +#define pgprot_noncached pgprot_noncached +static inline pgprot_t pgprot_noncached(pgprot_t _prot) +{ + unsigned long prot = pgprot_val(_prot); + + prot &= ~_PAGE_DMA_MASK; + prot |= _PAGE_DMA_IO; + + return __pgprot(prot); +} + +#define pgprot_writecombine pgprot_writecombine +static inline pgprot_t pgprot_writecombine(pgprot_t _prot) +{ + unsigned long prot = pgprot_val(_prot); + + prot &= ~_PAGE_DMA_MASK; + prot |= _PAGE_DMA_WC; + + return __pgprot(prot); +} + +#define __HAVE_PHYS_MEM_ACCESS_PROT +extern pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, + unsigned long size, pgprot_t vma_prot); + /* * Encode and decode a swap entry * diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index 0d42693..133e88a 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -27,6 +27,7 @@ enum sbi_ext_id { SBI_EXT_IPI = 0x735049, SBI_EXT_RFENCE = 0x52464E43, SBI_EXT_HSM = 0x48534D, + SBI_EXT_DMA = 0xAB150401, }; enum sbi_ext_base_fid { @@ -63,6 +64,17 @@ enum sbi_ext_hsm_fid { SBI_EXT_HSM_HART_STATUS, }; +enum sbi_ext_dma_fid { + SBI_DMA_SYNC = 0, +}; + +enum sbi_dma_sync_data_direction { + SBI_DMA_BIDIRECTIONAL = 0, + SBI_DMA_TO_DEVICE = 1, + SBI_DMA_FROM_DEVICE = 2, + SBI_DMA_NONE = 3, +}; + enum sbi_hsm_hart_status { SBI_HSM_HART_STATUS_STARTED = 0, SBI_HSM_HART_STATUS_STOPPED, @@ -128,6 +140,9 @@ int sbi_remote_hfence_vvma_asid(const unsigned long *hart_mask, unsigned long size, unsigned long asid); int sbi_probe_extension(int ext); +void sbi_dma_sync(unsigned long start, + unsigned long size, + enum sbi_dma_sync_data_direction dir); /* Check if current SBI specification version is 0.1 or not */ static inline int sbi_spec_is_0_1(void) diff --git a/arch/riscv/kernel/sbi.c b/arch/riscv/kernel/sbi.c index 7402a41..c936019 100644 --- a/arch/riscv/kernel/sbi.c +++ b/arch/riscv/kernel/sbi.c @@ -521,6 +521,25 @@ int sbi_probe_extension(int extid) } EXPORT_SYMBOL(sbi_probe_extension); +void sbi_dma_sync(unsigned long start, + unsigned long size, + enum sbi_dma_sync_data_direction dir) +{ +#if 0 + sbi_ecall(SBI_EXT_DMA, SBI_DMA_SYNC, start, size, dir, + 0, 0, 0); +#else + /* Just for try, it should be in sbi ecall and will be removed before merged */ + register unsigned long i asm("a0") = start & ~(L1_CACHE_BYTES - 1); + + for (; i < ALIGN(start + size, L1_CACHE_BYTES); i += L1_CACHE_BYTES) + __asm__ __volatile__(".long 0x02b5000b"); + + __asm__ __volatile__(".long 0x01b0000b"); +#endif +} +EXPORT_SYMBOL(sbi_dma_sync); + static long __sbi_base_ecall(int fid) { struct sbiret ret; diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile index 7ebaef1..ca0ff90 100644 --- a/arch/riscv/mm/Makefile +++ b/arch/riscv/mm/Makefile @@ -13,6 +13,7 @@ obj-y += extable.o obj-$(CONFIG_MMU) += fault.o pageattr.o obj-y += cacheflush.o obj-y += context.o +obj-y += dma-mapping.o ifeq ($(CONFIG_MMU),y) obj-$(CONFIG_SMP) += tlbflush.o diff --git a/arch/riscv/mm/dma-mapping.c b/arch/riscv/mm/dma-mapping.c new file mode 100644 index 00000000..4afd9dc --- /dev/null +++ b/arch/riscv/mm/dma-mapping.c @@ -0,0 +1,53 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include + +void arch_dma_prep_coherent(struct page *page, size_t size) +{ + void *ptr = page_address(page); + + memset(ptr, 0, size); + sbi_dma_sync(page_to_phys(page), size, SBI_DMA_BIDIRECTIONAL); +} + +void arch_sync_dma_for_device(phys_addr_t paddr, size_t size, + enum dma_data_direction dir) +{ + switch (dir) { + case DMA_TO_DEVICE: + case DMA_FROM_DEVICE: + case DMA_BIDIRECTIONAL: + sbi_dma_sync(paddr, size, dir); + break; + default: + BUG(); + } +} + +void arch_sync_dma_for_cpu(phys_addr_t paddr, size_t size, + enum dma_data_direction dir) +{ + switch (dir) { + case DMA_TO_DEVICE: + return; + case DMA_FROM_DEVICE: + case DMA_BIDIRECTIONAL: + sbi_dma_sync(paddr, size, dir); + break; + default: + BUG(); + } +} + +pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, + unsigned long size, pgprot_t vma_prot) +{ + if (!pfn_valid(pfn)) + return pgprot_noncached(vma_prot); + else if (file->f_flags & O_SYNC) + return pgprot_writecombine(vma_prot); + + return vma_prot; +} +EXPORT_SYMBOL(phys_mem_access_prot);