From patchwork Tue Oct 29 06:40:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhangfei Gao X-Patchwork-Id: 11217081 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0DFCC1747 for ; Tue, 29 Oct 2019 06:41:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DF2B4217F9 for ; Tue, 29 Oct 2019 06:41:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="bIjXxvDe" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732589AbfJ2GlJ (ORCPT ); Tue, 29 Oct 2019 02:41:09 -0400 Received: from mail-pl1-f193.google.com ([209.85.214.193]:36563 "EHLO mail-pl1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732576AbfJ2GlJ (ORCPT ); Tue, 29 Oct 2019 02:41:09 -0400 Received: by mail-pl1-f193.google.com with SMTP id g9so6471448plp.3 for ; Mon, 28 Oct 2019 23:41:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=KIkOGoEfD3sC7Lm1oZeLGLIevyyonnKw4CrTMTarj0A=; b=bIjXxvDeD2rWDSR6i985axak0TB8AlPFUcMAra8oKx8XhpAMLYgs03sHaaSUpqLxjJ GanF9IeQ9rcB7++fygVqlfUnt37XvO3LrlfTsw+0IgOkYY0Pa2fjK90vCRoJOVn1w+vK mQtOxJWYZctxQDXBOkOfslpDMwzIwsaydaFyUlhIZFLgSRqMbojBJ8PylPshovYPpXXk E7S4r9vuutLLOnm3PZ6H1kJiYGP+LrLZytPYO3UI17Itj1KFpChgA2MDBGHEZ6iYxRm7 RGSeHUEWyoNCcuJXshHSghG8Ge1Uw8MAi2hWEwZZFwTdA1b+UebAeRjOt4/luSzAKrnn GssA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=KIkOGoEfD3sC7Lm1oZeLGLIevyyonnKw4CrTMTarj0A=; b=N7VovGdL57GmlxqDZE0UVYbmu0pnG6aHSuZ7ZOwLeuutg6CeOggiQjNSYrvcThRY47 jVKLP79Smm5uefyUFfzwL3TEmnqDp44JDM7mdVnimO5/B3FD9LkfPiqwtN+/mJYva4hc L+3zmw38xSwwnLQ5sc+YZrQdPFmb/mAVdlJ8tmHplwwY9vtyKgfiTVacVSwOg7IIiYHS DiDx9pXel2vK0YmY5K5Irsl//yd497Twxi+nhYWpNSdo4QzAPDba5fB9C2YFGymozIgJ 3LWT2usVCcVdPo+8YGm0nOjgBRQK9UG4+Ht6YSYszTvMi3obUptrROsHOasiXg5puYDf XM0Q== X-Gm-Message-State: APjAAAU1TGOs4h13jS5I2Asmipbj72DRR54QHPanx77BZ6DCaO3Xa/sH nFthtCHmdVa4YNt8xt1M7ertFA== X-Google-Smtp-Source: APXvYqxNJ6Iu2A2JI6uBY0yAy/aq0E7dwPsiKMRw1Hi9k/q/qJ7hc/cmkRsr6ucYNgP6Ims5/HCwyw== X-Received: by 2002:a17:902:8647:: with SMTP id y7mr2069000plt.75.1572331266831; Mon, 28 Oct 2019 23:41:06 -0700 (PDT) Received: from localhost.localdomain ([240e:362:4dc:3a00:892e:70f7:f486:8f02]) by smtp.gmail.com with ESMTPSA id e23sm13421834pgh.84.2019.10.28.23.40.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 28 Oct 2019 23:41:06 -0700 (PDT) From: Zhangfei Gao To: Greg Kroah-Hartman , Arnd Bergmann , Herbert Xu , jonathan.cameron@huawei.com, grant.likely@arm.com, jean-philippe , Jerome Glisse , ilias.apalodimas@linaro.org, francois.ozog@linaro.org, kenneth-lee-2012@foxmail.com, Wangzhou , "haojian . zhuang" , guodong.xu@linaro.org Cc: linux-accelerators@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, iommu@lists.linux-foundation.org, Kenneth Lee , Zaibo Xu , Zhangfei Gao Subject: [PATCH v7 1/3] uacce: Add documents for uacce Date: Tue, 29 Oct 2019 14:40:14 +0800 Message-Id: <1572331216-9503-2-git-send-email-zhangfei.gao@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1572331216-9503-1-git-send-email-zhangfei.gao@linaro.org> References: <1572331216-9503-1-git-send-email-zhangfei.gao@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: Kenneth Lee Uacce (Unified/User-space-access-intended Accelerator Framework) is a kernel module targets to provide Shared Virtual Addressing (SVA) between the accelerator and process. This patch add document to explain how it works. Signed-off-by: Kenneth Lee Signed-off-by: Zaibo Xu Signed-off-by: Zhou Wang Signed-off-by: Zhangfei Gao --- Documentation/misc-devices/uacce.rst | 160 +++++++++++++++++++++++++++++++++++ 1 file changed, 160 insertions(+) create mode 100644 Documentation/misc-devices/uacce.rst diff --git a/Documentation/misc-devices/uacce.rst b/Documentation/misc-devices/uacce.rst new file mode 100644 index 0000000..ecd5d8b --- /dev/null +++ b/Documentation/misc-devices/uacce.rst @@ -0,0 +1,160 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Introduction of Uacce +========================= + +Uacce (Unified/User-space-access-intended Accelerator Framework) targets to +provide Shared Virtual Addressing (SVA) between accelerators and processes. +So accelerator can access any data structure of the main cpu. +This differs from the data sharing between cpu and io device, which share +data content rather than address. +Because of the unified address, hardware and user space of process can +share the same virtual address in the communication. +Uacce takes the hardware accelerator as a heterogeneous processor, while +IOMMU share the same CPU page tables and as a result the same translation +from va to pa. + + __________________________ __________________________ + | | | | + | User application (CPU) | | Hardware Accelerator | + |__________________________| |__________________________| + + | | + | va | va + V V + __________ __________ + | | | | + | MMU | | IOMMU | + |__________| |__________| + | | + | | + V pa V pa + _______________________________________ + | | + | Memory | + |_______________________________________| + + + +Architecture +------------ + +Uacce is the kernel module, taking charge of iommu and address sharing. +The user drivers and libraries are called WarpDrive. + +The uacce device, built around the IOMMU SVA API, can access multiple +address spaces, including the one without PASID. + +A virtual concept, queue, is used for the communication. It provides a +FIFO-like interface. And it maintains a unified address space between the +application and all involved hardware. + + ___________________ ________________ + | | user API | | + | WarpDrive library | ------------> | user driver | + |___________________| |________________| + | | + | | + | queue fd | + | | + | | + v | + ___________________ _________ | + | | | | | mmap memory + | Other framework | | uacce | | r/w interface + | crypto/nic/others | |_________| | + |___________________| | + | | | + | register | register | + | | | + | | | + | _________________ __________ | + | | | | | | + ------------- | Device Driver | | IOMMU | | + |_________________| |__________| | + | | + | V + | ___________________ + | | | + -------------------------- | Device(Hardware) | + |___________________| + + +How does it work +================ + +Uacce uses mmap and IOMMU to play the trick. + +Uacce create a chrdev for every device registered to it. New queue is +created when user application open the chrdev. The file descriptor is used +as the user handle of the queue. +The accelerator device present itself as an Uacce object, which exports as +chrdev to the user space. The user application communicates with the +hardware by ioctl (as control path) or share memory (as data path). + +The control path to the hardware is via file operation, while data path is +via mmap space of the queue fd. + +The queue file address space: +/** + * enum uacce_qfrt: qfrt type + * @UACCE_QFRT_MMIO: device mmio region + * @UACCE_QFRT_DUS: device user share region + */ +enum uacce_qfrt { + UACCE_QFRT_MMIO = 0, + UACCE_QFRT_DUS = 1, +}; + +All regions are optional and differ from device type to type. The +communication protocol is wrapped by the user driver. + +The device mmio region is mapped to the hardware mmio space. It is generally +used for doorbell or other notification to the hardware. It is not fast enough +as data channel. + +The device user share region is used for share data buffer between user process +and device. + + +The Uacce register API +----------------------- +The register API is defined in uacce.h. + +struct uacce_interface { + char name[UACCE_MAX_NAME_SIZE]; + enum uacce_dev_flag flags; + struct uacce_ops *ops; +}; + +According to the IOMMU capability, uacce_interface flags can be: + +/** + * enum uacce_dev_flag: Device flags: + * @UACCE_DEV_SVA: Shared Virtual Addresses + * Support PASID + * Support device page faults (PCI PRI or SMMU Stall) + */ +enum uacce_dev_flag { + UACCE_DEV_SVA = BIT(0), +}; + +struct uacce_device *uacce_register(struct device *parent, + struct uacce_interface *interface); +void uacce_unregister(struct uacce_device *uacce); + +uacce_register results can be: +a. If uacce module is not compiled, ERR_PTR(-ENODEV) +b. Succeed with the desired flags +c. Succeed with the negotiated flags, for example + uacce_interface.flags = UACCE_DEV_SVA but uacce->flags = ~UACCE_DEV_SVA +So user driver need check return value as well as the negotiated uacce->flags. + + +The user driver +--------------- + +The queue file mmap space will need a user driver to wrap the communication +protocol. Uacce provides some attributes in sysfs for the user driver to +match the right accelerator accordingly. +More details in Documentation/ABI/testing/sysfs-driver-uacce. From patchwork Tue Oct 29 06:40:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhangfei Gao X-Patchwork-Id: 11217083 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 659E015AB for ; Tue, 29 Oct 2019 06:41:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 24205217F9 for ; Tue, 29 Oct 2019 06:41:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="I+KCGxg3" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732612AbfJ2Gl2 (ORCPT ); Tue, 29 Oct 2019 02:41:28 -0400 Received: from mail-pf1-f180.google.com ([209.85.210.180]:42440 "EHLO mail-pf1-f180.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732630AbfJ2Gl1 (ORCPT ); Tue, 29 Oct 2019 02:41:27 -0400 Received: by mail-pf1-f180.google.com with SMTP id 21so8811851pfj.9 for ; Mon, 28 Oct 2019 23:41:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=XrWvBQ9sNdycj3useBLDFrDqzpLizWXRsjet9itNF9I=; b=I+KCGxg3tbTzqMYyF/aMg0g7jq113szfhYDcZEA6V6bFleup26IIV8JysSkOt6ermp SlD16gE5nWbxTiy8/9HNL2YhE+X3QEymHrPm7U33gJC4s5nSeKxtstuKmK6eE5OkbUrh paY4u4PphOdbmxCEOqXg0P51YRacGoTzX0/3KXmfWbYsCtRaNq4jnXo/CsAezr9niWBO iUjxSz5cZ5E8YHIKoA6l4nuKyRZCyWeaj8w8p13RNwbrdFK/S6rCH2yQ+DmXI77JcVPi xyeCzg8y+bm7+BeOaKXx2Fx6MMz84Tj4es9x3xJFrUgDZfi7z/fnXBhqm8yV4ff0BQuM BiUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=XrWvBQ9sNdycj3useBLDFrDqzpLizWXRsjet9itNF9I=; b=BLQUr4WVnEy4q0p5dU4b975VuAC9AeP2QYp9Gz/LWbXGhDa1/d7kv8CnUmTdOikMPv jxYgP5Zq9DYIzwhkqKyEbBVQwcRPX6RcdJ5Mtm3hHFBy075+5aiZMe/SVJdvjUvku8l3 kNg4ZByEydoWsV4Y/v16gpBgO967S1NGBfpg87V9b1gjtjS7N+J1cQkMJR13oye9MNir oa9PNQAFaTdqo+qm+pDCz4gsWOJanCOpnlQW3v/OubQMy1fTtdan9jbYzJ+X+upg3EYO ryP+FQbiwaJcqwqi0UFClVluU2OmTV1BDARpCjfANi6Up8LMy/oo45/w4TacN8O5LmVt Torg== X-Gm-Message-State: APjAAAVkQ1ratw2F424tQ5Auuf6LpqzMGWTiFgBgM43yC0Z0L2Z6uKoB iXNyVli9AOfOWwwzQ7XPLkiEgA== X-Google-Smtp-Source: APXvYqyyfeXdDNGUFrc/X4bocd//NLUY/RtNqCOT5svJtU5LeWoEIZKCCgxwzsmblhr4N4Vg4q2PuA== X-Received: by 2002:aa7:8421:: with SMTP id q1mr25551836pfn.174.1572331284959; Mon, 28 Oct 2019 23:41:24 -0700 (PDT) Received: from localhost.localdomain ([240e:362:4dc:3a00:892e:70f7:f486:8f02]) by smtp.gmail.com with ESMTPSA id e23sm13421834pgh.84.2019.10.28.23.41.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 28 Oct 2019 23:41:24 -0700 (PDT) From: Zhangfei Gao To: Greg Kroah-Hartman , Arnd Bergmann , Herbert Xu , jonathan.cameron@huawei.com, grant.likely@arm.com, jean-philippe , Jerome Glisse , ilias.apalodimas@linaro.org, francois.ozog@linaro.org, kenneth-lee-2012@foxmail.com, Wangzhou , "haojian . zhuang" , guodong.xu@linaro.org Cc: linux-accelerators@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, iommu@lists.linux-foundation.org, Kenneth Lee , Zaibo Xu , Zhangfei Gao Subject: [PATCH v7 2/3] uacce: add uacce driver Date: Tue, 29 Oct 2019 14:40:15 +0800 Message-Id: <1572331216-9503-3-git-send-email-zhangfei.gao@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1572331216-9503-1-git-send-email-zhangfei.gao@linaro.org> References: <1572331216-9503-1-git-send-email-zhangfei.gao@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: Kenneth Lee Uacce (Unified/User-space-access-intended Accelerator Framework) targets to provide Shared Virtual Addressing (SVA) between accelerators and processes. So accelerator can access any data structure of the main cpu. This differs from the data sharing between cpu and io device, which share data content rather than address. Since unified address, hardware and user space of process can share the same virtual address in the communication. Uacce create a chrdev for every registration, the queue is allocated to the process when the chrdev is opened. Then the process can access the hardware resource by interact with the queue file. By mmap the queue file space to user space, the process can directly put requests to the hardware without syscall to the kernel space. Signed-off-by: Kenneth Lee Signed-off-by: Zaibo Xu Signed-off-by: Zhou Wang Signed-off-by: Zhangfei Gao Signed-off-by: Jean-Philippe Brucker --- Documentation/ABI/testing/sysfs-driver-uacce | 53 +++ drivers/misc/Kconfig | 1 + drivers/misc/Makefile | 1 + drivers/misc/uacce/Kconfig | 13 + drivers/misc/uacce/Makefile | 2 + drivers/misc/uacce/uacce.c | 574 +++++++++++++++++++++++++++ include/linux/uacce.h | 163 ++++++++ include/uapi/misc/uacce/uacce.h | 38 ++ 8 files changed, 845 insertions(+) create mode 100644 Documentation/ABI/testing/sysfs-driver-uacce create mode 100644 drivers/misc/uacce/Kconfig create mode 100644 drivers/misc/uacce/Makefile create mode 100644 drivers/misc/uacce/uacce.c create mode 100644 include/linux/uacce.h create mode 100644 include/uapi/misc/uacce/uacce.h diff --git a/Documentation/ABI/testing/sysfs-driver-uacce b/Documentation/ABI/testing/sysfs-driver-uacce new file mode 100644 index 0000000..35699dc --- /dev/null +++ b/Documentation/ABI/testing/sysfs-driver-uacce @@ -0,0 +1,53 @@ +What: /sys/class/uacce//id +Date: Oct 2019 +KernelVersion: 5.5 +Contact: linux-accelerators@lists.ozlabs.org +Description: Id of the device. + +What: /sys/class/uacce//api +Date: Oct 2019 +KernelVersion: 5.5 +Contact: linux-accelerators@lists.ozlabs.org +Description: Api of the device, used by application to match the correct driver + +What: /sys/class/uacce//flags +Date: Oct 2019 +KernelVersion: 5.5 +Contact: linux-accelerators@lists.ozlabs.org +Description: Attributes of the device, see UACCE_DEV_xxx flag defined in uacce.h + +What: /sys/class/uacce//available_instances +Date: Oct 2019 +KernelVersion: 5.5 +Contact: linux-accelerators@lists.ozlabs.org +Description: Available instances left of the device + +What: /sys/class/uacce//algorithms +Date: Oct 2019 +KernelVersion: 5.5 +Contact: linux-accelerators@lists.ozlabs.org +Description: Algorithms supported by this accelerator + +What: /sys/class/uacce//qfrt_mmio_size +Date: Oct 2019 +KernelVersion: 5.5 +Contact: linux-accelerators@lists.ozlabs.org +Description: Page size of mmio region queue file + +What: /sys/class/uacce//qfrt_dus_size +Date: Oct 2019 +KernelVersion: 5.5 +Contact: linux-accelerators@lists.ozlabs.org +Description: Page size of dus region queue file + +What: /sys/class/uacce//numa_distance +Date: Oct 2019 +KernelVersion: 5.5 +Contact: linux-accelerators@lists.ozlabs.org +Description: Distance of device node to cpu node + +What: /sys/class/uacce//node_id +Date: Oct 2019 +KernelVersion: 5.5 +Contact: linux-accelerators@lists.ozlabs.org +Description: Id of the numa node diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig index c55b637..929feb0 100644 --- a/drivers/misc/Kconfig +++ b/drivers/misc/Kconfig @@ -481,4 +481,5 @@ source "drivers/misc/cxl/Kconfig" source "drivers/misc/ocxl/Kconfig" source "drivers/misc/cardreader/Kconfig" source "drivers/misc/habanalabs/Kconfig" +source "drivers/misc/uacce/Kconfig" endmenu diff --git a/drivers/misc/Makefile b/drivers/misc/Makefile index c1860d3..9abf292 100644 --- a/drivers/misc/Makefile +++ b/drivers/misc/Makefile @@ -56,4 +56,5 @@ obj-$(CONFIG_OCXL) += ocxl/ obj-y += cardreader/ obj-$(CONFIG_PVPANIC) += pvpanic.o obj-$(CONFIG_HABANA_AI) += habanalabs/ +obj-$(CONFIG_UACCE) += uacce/ obj-$(CONFIG_XILINX_SDFEC) += xilinx_sdfec.o diff --git a/drivers/misc/uacce/Kconfig b/drivers/misc/uacce/Kconfig new file mode 100644 index 0000000..5e39b60 --- /dev/null +++ b/drivers/misc/uacce/Kconfig @@ -0,0 +1,13 @@ +config UACCE + tristate "Accelerator Framework for User Land" + depends on IOMMU_API + help + UACCE provides interface for the user process to access the hardware + without interaction with the kernel space in data path. + + The user-space interface is described in + include/uapi/misc/uacce/uacce.h + + See Documentation/misc-devices/uacce.rst for more details. + + If you don't know what to do here, say N. diff --git a/drivers/misc/uacce/Makefile b/drivers/misc/uacce/Makefile new file mode 100644 index 0000000..5b4374e --- /dev/null +++ b/drivers/misc/uacce/Makefile @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +obj-$(CONFIG_UACCE) += uacce.o diff --git a/drivers/misc/uacce/uacce.c b/drivers/misc/uacce/uacce.c new file mode 100644 index 0000000..2b6b038 --- /dev/null +++ b/drivers/misc/uacce/uacce.c @@ -0,0 +1,574 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +#include +#include +#include +#include +#include + +static struct class *uacce_class; +static dev_t uacce_devt; +static DEFINE_MUTEX(uacce_mutex); +static DEFINE_XARRAY_ALLOC(uacce_xa); + +static int uacce_start_queue(struct uacce_queue *q) +{ + int ret = -EINVAL; + + mutex_lock(&uacce_mutex); + + if (q->state != UACCE_Q_INIT) + goto out_with_lock; + + if (q->uacce->ops->start_queue) { + ret = q->uacce->ops->start_queue(q); + if (ret < 0) + goto out_with_lock; + } + + q->state = UACCE_Q_STARTED; + mutex_unlock(&uacce_mutex); + + return 0; + +out_with_lock: + mutex_unlock(&uacce_mutex); + return ret; +} + +static int uacce_put_queue(struct uacce_queue *q) +{ + struct uacce_device *uacce = q->uacce; + + mutex_lock(&uacce_mutex); + + if (q->state == UACCE_Q_ZOMBIE) + goto out; + + if ((q->state == UACCE_Q_STARTED) && uacce->ops->stop_queue) + uacce->ops->stop_queue(q); + + if ((q->state == UACCE_Q_INIT || q->state == UACCE_Q_STARTED) && + uacce->ops->put_queue) + uacce->ops->put_queue(q); + + q->state = UACCE_Q_ZOMBIE; +out: + mutex_unlock(&uacce_mutex); + + return 0; +} + +static long uacce_fops_unl_ioctl(struct file *filep, + unsigned int cmd, unsigned long arg) +{ + struct uacce_queue *q = filep->private_data; + struct uacce_device *uacce = q->uacce; + + switch (cmd) { + case UACCE_CMD_START_Q: + return uacce_start_queue(q); + + case UACCE_CMD_PUT_Q: + return uacce_put_queue(q); + + default: + if (!uacce->ops->ioctl) + return -EINVAL; + + return uacce->ops->ioctl(q, cmd, arg); + } +} + +#ifdef CONFIG_COMPAT +static long uacce_fops_compat_ioctl(struct file *filep, + unsigned int cmd, unsigned long arg) +{ + arg = (unsigned long)compat_ptr(arg); + + return uacce_fops_unl_ioctl(filep, cmd, arg); +} +#endif + +static int uacce_sva_exit(struct device *dev, struct iommu_sva *handle, + void *data) +{ + struct uacce_device *uacce = data; + struct uacce_queue *q; + + mutex_lock(&uacce->q_lock); + list_for_each_entry(q, &uacce->qs, list) { + if (q->pid == task_pid_nr(current)) + uacce_put_queue(q); + } + mutex_unlock(&uacce->q_lock); + + return 0; +} + +static struct iommu_sva_ops uacce_sva_ops = { + .mm_exit = uacce_sva_exit, +}; + +static int uacce_fops_open(struct inode *inode, struct file *filep) +{ + struct iommu_sva *handle = NULL; + struct uacce_device *uacce; + struct uacce_queue *q; + int ret = 0; + int pasid = 0; + + uacce = xa_load(&uacce_xa, iminor(inode)); + if (!uacce) + return -ENODEV; + + if (!try_module_get(uacce->pdev->driver->owner)) + return -ENODEV; + + q = kzalloc(sizeof(struct uacce_queue), GFP_KERNEL); + if (!q) { + ret = -ENOMEM; + goto out_with_module; + } + + if (uacce->flags & UACCE_DEV_SVA) { + handle = iommu_sva_bind_device(uacce->pdev, current->mm, uacce); + if (IS_ERR(handle)) + goto out_with_mem; + + ret = iommu_sva_set_ops(handle, &uacce_sva_ops); + if (ret) + goto out_unbind; + + pasid = iommu_sva_get_pasid(handle); + if (pasid == IOMMU_PASID_INVALID) + goto out_unbind; + } + + if (uacce->ops->get_queue) { + ret = uacce->ops->get_queue(uacce, pasid, q); + if (ret < 0) + goto out_unbind; + } + + q->pid = task_pid_nr(current); + q->pasid = pasid; + q->handle = handle; + q->uacce = uacce; + memset(q->qfrs, 0, sizeof(q->qfrs)); + init_waitqueue_head(&q->wait); + filep->private_data = q; + q->state = UACCE_Q_INIT; + + mutex_lock(&uacce->q_lock); + list_add(&q->list, &uacce->qs); + mutex_unlock(&uacce->q_lock); + + return 0; + +out_unbind: + if (uacce->flags & UACCE_DEV_SVA) + iommu_sva_unbind_device(handle); +out_with_mem: + kfree(q); +out_with_module: + module_put(uacce->pdev->driver->owner); + return ret; +} + +static int uacce_fops_release(struct inode *inode, struct file *filep) +{ + struct uacce_queue *q = filep->private_data; + struct uacce_device *uacce = q->uacce; + + uacce_put_queue(q); + + if (uacce->flags & UACCE_DEV_SVA) + iommu_sva_unbind_device(q->handle); + + mutex_lock(&uacce->q_lock); + list_del(&q->list); + mutex_unlock(&uacce->q_lock); + kfree(q); + module_put(uacce->pdev->driver->owner); + + return 0; +} + +static void uacce_vma_close(struct vm_area_struct *vma) +{ + struct uacce_queue *q = vma->vm_private_data; + enum uacce_qfrt type = 0; + + if (vma->vm_pgoff < UACCE_QFRT_MAX) + type = vma->vm_pgoff; + + kfree(q->qfrs[type]); +} + +static const struct vm_operations_struct uacce_vm_ops = { + .close = uacce_vma_close, +}; + +static struct uacce_qfile_region * +uacce_create_region(struct uacce_queue *q, struct vm_area_struct *vma, + enum uacce_qfrt type, unsigned int flags) +{ + struct uacce_device *uacce = q->uacce; + struct uacce_qfile_region *qfr; + int ret = -ENOMEM; + + qfr = kzalloc(sizeof(*qfr), GFP_KERNEL); + if (!qfr) + return ERR_PTR(-ENOMEM); + + qfr->type = type; + qfr->flags = flags; + + if (vma->vm_flags & VM_READ) + qfr->prot |= IOMMU_READ; + + if (vma->vm_flags & VM_WRITE) + qfr->prot |= IOMMU_WRITE; + + if (flags & UACCE_QFRF_SELFMT) { + if (!uacce->ops->mmap) { + ret = -EINVAL; + goto err_with_qfr; + } + + ret = uacce->ops->mmap(q, vma, qfr); + if (ret) + goto err_with_qfr; + return qfr; + } + + return qfr; + +err_with_qfr: + kfree(qfr); + return ERR_PTR(ret); +} + +static int uacce_fops_mmap(struct file *filep, struct vm_area_struct *vma) +{ + struct uacce_queue *q = filep->private_data; + struct uacce_device *uacce = q->uacce; + struct uacce_qfile_region *qfr; + enum uacce_qfrt type = 0; + unsigned int flags = 0; + int ret; + + if (vma->vm_pgoff < UACCE_QFRT_MAX) + type = vma->vm_pgoff; + + vma->vm_flags |= VM_DONTCOPY | VM_DONTEXPAND | VM_WIPEONFORK; + vma->vm_ops = &uacce_vm_ops; + vma->vm_private_data = q; + + mutex_lock(&uacce_mutex); + + if (q->qfrs[type]) { + ret = -EEXIST; + goto out_with_lock; + } + + switch (type) { + case UACCE_QFRT_MMIO: + flags = UACCE_QFRF_SELFMT; + break; + + case UACCE_QFRT_DUS: + if (uacce->flags & UACCE_DEV_SVA) { + flags = UACCE_QFRF_SELFMT; + break; + } + break; + + default: + WARN_ON(&uacce->dev); + break; + } + + qfr = uacce_create_region(q, vma, type, flags); + if (IS_ERR(qfr)) { + ret = PTR_ERR(qfr); + goto out_with_lock; + } + q->qfrs[type] = qfr; + + mutex_unlock(&uacce_mutex); + + return 0; + +out_with_lock: + mutex_unlock(&uacce_mutex); + return ret; +} + +static __poll_t uacce_fops_poll(struct file *file, poll_table *wait) +{ + struct uacce_queue *q = file->private_data; + struct uacce_device *uacce = q->uacce; + + poll_wait(file, &q->wait, wait); + if (uacce->ops->is_q_updated && uacce->ops->is_q_updated(q)) + return EPOLLIN | EPOLLRDNORM; + + return 0; +} + +static const struct file_operations uacce_fops = { + .owner = THIS_MODULE, + .open = uacce_fops_open, + .release = uacce_fops_release, + .unlocked_ioctl = uacce_fops_unl_ioctl, +#ifdef CONFIG_COMPAT + .compat_ioctl = uacce_fops_compat_ioctl, +#endif + .mmap = uacce_fops_mmap, + .poll = uacce_fops_poll, +}; + +#define to_uacce_device(dev) container_of(dev, struct uacce_device, dev) + +static ssize_t id_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct uacce_device *uacce = to_uacce_device(dev); + + return sprintf(buf, "%d\n", uacce->dev_id); +} + +static ssize_t api_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct uacce_device *uacce = to_uacce_device(dev); + + return sprintf(buf, "%s\n", uacce->api_ver); +} + +static ssize_t numa_distance_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct uacce_device *uacce = to_uacce_device(dev); + int distance; + + distance = node_distance(smp_processor_id(), uacce->pdev->numa_node); + + return sprintf(buf, "%d\n", abs(distance)); +} + +static ssize_t node_id_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct uacce_device *uacce = to_uacce_device(dev); + int node_id; + + node_id = dev_to_node(uacce->pdev); + + return sprintf(buf, "%d\n", node_id); +} + +static ssize_t flags_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct uacce_device *uacce = to_uacce_device(dev); + + return sprintf(buf, "%u\n", uacce->flags); +} + +static ssize_t available_instances_show(struct device *dev, + struct device_attribute *attr, + char *buf) +{ + struct uacce_device *uacce = to_uacce_device(dev); + int val = 0; + + if (uacce->ops->get_available_instances) + val = uacce->ops->get_available_instances(uacce); + + return sprintf(buf, "%d\n", val); +} + +static ssize_t algorithms_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct uacce_device *uacce = to_uacce_device(dev); + + return sprintf(buf, "%s", uacce->algs); +} + +static ssize_t qfrt_mmio_size_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct uacce_device *uacce = to_uacce_device(dev); + + return sprintf(buf, "%lu\n", + uacce->qf_pg_size[UACCE_QFRT_MMIO] << PAGE_SHIFT); +} + +static ssize_t qfrt_dus_size_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct uacce_device *uacce = to_uacce_device(dev); + + return sprintf(buf, "%lu\n", + uacce->qf_pg_size[UACCE_QFRT_DUS] << PAGE_SHIFT); +} + +static DEVICE_ATTR_RO(id); +static DEVICE_ATTR_RO(api); +static DEVICE_ATTR_RO(numa_distance); +static DEVICE_ATTR_RO(node_id); +static DEVICE_ATTR_RO(flags); +static DEVICE_ATTR_RO(available_instances); +static DEVICE_ATTR_RO(algorithms); +static DEVICE_ATTR_RO(qfrt_mmio_size); +static DEVICE_ATTR_RO(qfrt_dus_size); + +static struct attribute *uacce_dev_attrs[] = { + &dev_attr_id.attr, + &dev_attr_api.attr, + &dev_attr_node_id.attr, + &dev_attr_numa_distance.attr, + &dev_attr_flags.attr, + &dev_attr_available_instances.attr, + &dev_attr_algorithms.attr, + &dev_attr_qfrt_mmio_size.attr, + &dev_attr_qfrt_dus_size.attr, + NULL, +}; +ATTRIBUTE_GROUPS(uacce_dev); + +static void uacce_release(struct device *dev) +{ + struct uacce_device *uacce = to_uacce_device(dev); + + kfree(uacce); +} + +/** + * uacce_register - register an accelerator + * @parent: pointer of uacce parent device + * @interface: pointer of uacce_interface for register + */ +struct uacce_device *uacce_register(struct device *parent, + struct uacce_interface *interface) +{ + unsigned int flags = interface->flags; + struct uacce_device *uacce; + int ret; + + uacce = kzalloc(sizeof(struct uacce_device), GFP_KERNEL); + if (!uacce) + return ERR_PTR(-ENOMEM); + + if (flags & UACCE_DEV_SVA) { + ret = iommu_dev_enable_feature(parent, IOMMU_DEV_FEAT_SVA); + if (ret) + flags &= ~UACCE_DEV_SVA; + } + + uacce->pdev = parent; + uacce->flags = flags; + uacce->ops = interface->ops; + + ret = xa_alloc(&uacce_xa, &uacce->dev_id, uacce, xa_limit_32b, + GFP_KERNEL); + if (ret < 0) + goto err_with_uacce; + + uacce->cdev = cdev_alloc(); + if (!uacce->cdev) { + ret = -ENOMEM; + goto err_with_xa; + } + + INIT_LIST_HEAD(&uacce->qs); + mutex_init(&uacce->q_lock); + uacce->cdev->ops = &uacce_fops; + uacce->cdev->owner = THIS_MODULE; + device_initialize(&uacce->dev); + uacce->dev.devt = MKDEV(MAJOR(uacce_devt), uacce->dev_id); + uacce->dev.class = uacce_class; + uacce->dev.groups = uacce_dev_groups; + uacce->dev.parent = uacce->pdev; + uacce->dev.release = uacce_release; + dev_set_name(&uacce->dev, "%s-%d", interface->name, uacce->dev_id); + ret = cdev_device_add(uacce->cdev, &uacce->dev); + if (ret) + goto err_with_xa; + + return uacce; + +err_with_xa: + if (uacce->cdev) + cdev_del(uacce->cdev); + xa_erase(&uacce_xa, uacce->dev_id); +err_with_uacce: + if (flags & UACCE_DEV_SVA) + iommu_dev_disable_feature(uacce->pdev, IOMMU_DEV_FEAT_SVA); + kfree(uacce); + return ERR_PTR(ret); +} +EXPORT_SYMBOL_GPL(uacce_register); + +/** + * uacce_unregister - unregisters an accelerator + * @uacce: the accelerator to unregister + */ +void uacce_unregister(struct uacce_device *uacce) +{ + if (!uacce) + return; + + mutex_lock(&uacce->q_lock); + if (!list_empty(&uacce->qs)) { + struct uacce_queue *q; + + list_for_each_entry(q, &uacce->qs, list) { + uacce_put_queue(q); + if (uacce->flags & UACCE_DEV_SVA) + iommu_sva_unbind_device(q->handle); + } + } + mutex_unlock(&uacce->q_lock); + + if (uacce->flags & UACCE_DEV_SVA) + iommu_dev_disable_feature(uacce->pdev, IOMMU_DEV_FEAT_SVA); + + cdev_device_del(uacce->cdev, &uacce->dev); + xa_erase(&uacce_xa, uacce->dev_id); + put_device(&uacce->dev); +} +EXPORT_SYMBOL_GPL(uacce_unregister); + +static int __init uacce_init(void) +{ + int ret; + + uacce_class = class_create(THIS_MODULE, UACCE_NAME); + if (IS_ERR(uacce_class)) + return PTR_ERR(uacce_class); + + ret = alloc_chrdev_region(&uacce_devt, 0, MINORMASK, UACCE_NAME); + if (ret) { + class_destroy(uacce_class); + return ret; + } + + return 0; +} + +static __exit void uacce_exit(void) +{ + unregister_chrdev_region(uacce_devt, MINORMASK); + class_destroy(uacce_class); +} + +subsys_initcall(uacce_init); +module_exit(uacce_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Hisilicon Tech. Co., Ltd."); +MODULE_DESCRIPTION("Accelerator interface for Userland applications"); diff --git a/include/linux/uacce.h b/include/linux/uacce.h new file mode 100644 index 0000000..04c8643 --- /dev/null +++ b/include/linux/uacce.h @@ -0,0 +1,163 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _LINUX_UACCE_H +#define _LINUX_UACCE_H + +#include +#include + +#define UACCE_NAME "uacce" +#define UACCE_QFRT_MAX 16 +#define UACCE_MAX_NAME_SIZE 64 + +struct uacce_queue; +struct uacce_device; + +/** + * enum uacce_qfr_flag: queue file flag: + * @UACCE_QFRF_SELFMT: self maintained qfr + */ +enum uacce_qfr_flag { + UACCE_QFRF_SELFMT = BIT(0), +}; + +/** + * struct uacce_qfile_region - structure of queue file region + * @type: type of the qfr + * @flags: flags of qfr + * @prot: qfr protection flag + */ +struct uacce_qfile_region { + enum uacce_qfrt type; + enum uacce_qfr_flag flags; + u32 prot; +}; + +/** + * struct uacce_ops - uacce device operations + * @get_available_instances: get available instances left of the device + * @get_queue: get a queue from the device + * @put_queue: free a queue to the device + * @start_queue: make the queue start work after get_queue + * @stop_queue: make the queue stop work before put_queue + * @is_q_updated: check whether the task is finished + * @mask_notify: mask the task irq of queue + * @mmap: mmap addresses of queue to user space + * @reset: reset the uacce device + * @reset_queue: reset the queue + * @ioctl: ioctl for user space users of the queue + */ +struct uacce_ops { + int (*get_available_instances)(struct uacce_device *uacce); + int (*get_queue)(struct uacce_device *uacce, unsigned long arg, + struct uacce_queue *q); + void (*put_queue)(struct uacce_queue *q); + int (*start_queue)(struct uacce_queue *q); + void (*stop_queue)(struct uacce_queue *q); + int (*is_q_updated)(struct uacce_queue *q); + void (*mask_notify)(struct uacce_queue *q, int event_mask); + int (*mmap)(struct uacce_queue *q, struct vm_area_struct *vma, + struct uacce_qfile_region *qfr); + int (*reset)(struct uacce_device *uacce); + int (*reset_queue)(struct uacce_queue *q); + long (*ioctl)(struct uacce_queue *q, unsigned int cmd, + unsigned long arg); +}; + +/** + * struct uacce_interface + * @name: the uacce device name. Will show up in sysfs + * @flags: uacce device attributes + * @ops: pointer to the struct uacce_ops + * + * This structure is used for the uacce_register() + */ +struct uacce_interface { + char name[UACCE_MAX_NAME_SIZE]; + enum uacce_dev_flag flags; + struct uacce_ops *ops; +}; + +enum uacce_q_state { + UACCE_Q_INIT, + UACCE_Q_STARTED, + UACCE_Q_ZOMBIE, +}; + +/** + * struct uacce_queue + * @uacce: pointer to uacce + * @priv: private pointer + * @wait: wait queue head + * @pasid: pasid of the queue + * @pid: pid of the process using the queue + * @handle: iommu_sva handle return from iommu_sva_bind_device + * @list: queue list + * @qfrs: pointer of qfr regions + * @state: queue state machine + */ +struct uacce_queue { + struct uacce_device *uacce; + void *priv; + wait_queue_head_t wait; + int pasid; + pid_t pid; + struct iommu_sva *handle; + struct list_head list; + struct uacce_qfile_region *qfrs[UACCE_QFRT_MAX]; + enum uacce_q_state state; +}; + +/** + * struct uacce_device + * @algs: supported algorithms + * @api_ver: api version + * @qf_pg_size: page size of the queue file regions + * @ops: pointer to the struct uacce_ops + * @pdev: pointer to the parent device + * @is_vf: whether virtual function + * @flags: uacce attributes + * @dev_id: id of the uacce device + * @prot: uacce protection flag + * @cdev: cdev of the uacce + * @dev: dev of the uacce + * @priv: private pointer of the uacce + * @qs: list head of queue->list + * @q_lock: lock for qs + */ +struct uacce_device { + const char *algs; + const char *api_ver; + unsigned long qf_pg_size[UACCE_QFRT_MAX]; + struct uacce_ops *ops; + struct device *pdev; + bool is_vf; + u32 flags; + u32 dev_id; + u32 prot; + struct cdev *cdev; + struct device dev; + void *priv; + struct list_head qs; + struct mutex q_lock; +}; + +#if IS_ENABLED(CONFIG_UACCE) + +struct uacce_device *uacce_register(struct device *parent, + struct uacce_interface *interface); +void uacce_unregister(struct uacce_device *uacce); + +#else /* CONFIG_UACCE */ + +static inline +struct uacce_device *uacce_register(struct device *parent, + struct uacce_interface *interface) +{ + return ERR_PTR(-ENODEV); +} + +static inline void uacce_unregister(struct uacce_device *uacce) {} + +#endif /* CONFIG_UACCE */ + +#endif /* _LINUX_UACCE_H */ diff --git a/include/uapi/misc/uacce/uacce.h b/include/uapi/misc/uacce/uacce.h new file mode 100644 index 0000000..a4f9378 --- /dev/null +++ b/include/uapi/misc/uacce/uacce.h @@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */ +#ifndef _UAPIUUACCE_H +#define _UAPIUUACCE_H + +#include +#include + +/* UACCE_CMD_START_Q: Start the queue */ +#define UACCE_CMD_START_Q _IO('W', 0) + +/** + * UACCE_CMD_PUT_Q: + * User actively stop queue and free queue resource immediately + * Optimization method since close fd may delay + */ +#define UACCE_CMD_PUT_Q _IO('W', 1) + +/** + * enum uacce_dev_flag: Device flags: + * @UACCE_DEV_SVA: Shared Virtual Addresses + * Support PASID + * Support device page faults (PCI PRI or SMMU Stall) + */ +enum uacce_dev_flag { + UACCE_DEV_SVA = BIT(0), +}; + +/** + * enum uacce_qfrt: qfrt type + * @UACCE_QFRT_MMIO: device mmio region + * @UACCE_QFRT_DUS: device user share region + */ +enum uacce_qfrt { + UACCE_QFRT_MMIO = 0, + UACCE_QFRT_DUS = 1, +}; + +#endif From patchwork Tue Oct 29 06:40:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhangfei Gao X-Patchwork-Id: 11217085 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 940141709 for ; Tue, 29 Oct 2019 06:41:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6E2E1217F9 for ; Tue, 29 Oct 2019 06:41:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="TxXdsa37" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732652AbfJ2Glj (ORCPT ); Tue, 29 Oct 2019 02:41:39 -0400 Received: from mail-pf1-f195.google.com ([209.85.210.195]:40226 "EHLO mail-pf1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732630AbfJ2Gli (ORCPT ); Tue, 29 Oct 2019 02:41:38 -0400 Received: by mail-pf1-f195.google.com with SMTP id r4so3154964pfl.7 for ; Mon, 28 Oct 2019 23:41:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=dekWozQpi8Yds4BtBBFvE6aNgm93yn3hRX+1DVnhSXA=; b=TxXdsa37VU48AW/7JNoNomAxHsjllorHtXzFiL55rXM8tizApaPdseiN4jRvWjh4hu 28njhgiqoINAZoT7iPK9J/uRgZsbDr9yfIp5ka2cak6LUudAOwPJgIcqmhv0F0hPzOkc P7DSdvH53puHBvbTXDDmMvOCq+RQRq+4QCeUPQdDUwkKQn5mOnGRux7Lw1+yJKxIMDAI qFTpQrsX/j0x4JMGE5Gz69GoeYz9rF0e1WZiBYJNfjbmsSlCSC2m5/x+cmuL7pLTYIB3 9nUPutB2IvHv4Y6Zq6aZIIV4/bp5l3/K0c3xrJc/1WLd8Ro6cYFJ4vIkJ7yrKtmPCckZ FhXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=dekWozQpi8Yds4BtBBFvE6aNgm93yn3hRX+1DVnhSXA=; b=egM7jlvL3F8ibhNAEfy1BRASzqRuGuyL7ntGFANdjE+VmkvIC2jebt9sQq3F2SPP2i PkBwcxwIiGmAvtWVxEh6Ep1dpbWGsQlaFaPOQ2auE+jsEIShPxdN5r5xtou5ge4Gp7NJ SpWso7M7M90OuRBvZXrDQEe6V0+FMGjIbRQgBnvEvWHU69/PToo0hWPA+gAyXLyclwBM 1kAcplIWwD+xO+3XUFfnqCUCqS2gHvZHy3KoWheUdEjL6RiEYz6QipZnmeVNgEUsMj6B AwBqudvS1Q3TmeJdpbQzNac4kXPCn0q1rNqDUhc+xrK4fAQRq8AwpC3TSTzyRf6BVNNj flJw== X-Gm-Message-State: APjAAAUaKYaDV9PMCinJbAt0vH2ydUcmnAKPPepuGNe8P0ng3WS8nFFC TaGwGFISzRKrJNsZBXZmfhBosQ== X-Google-Smtp-Source: APXvYqx2R8/rL2D0gYW7AXASpTKI2pP8bAJABqIYriX4186C/mAvdaw/Vwt8ezZm/1JbQYdtkAvY3Q== X-Received: by 2002:a17:90a:25e1:: with SMTP id k88mr4361121pje.14.1572331298008; Mon, 28 Oct 2019 23:41:38 -0700 (PDT) Received: from localhost.localdomain ([240e:362:4dc:3a00:892e:70f7:f486:8f02]) by smtp.gmail.com with ESMTPSA id e23sm13421834pgh.84.2019.10.28.23.41.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 28 Oct 2019 23:41:37 -0700 (PDT) From: Zhangfei Gao To: Greg Kroah-Hartman , Arnd Bergmann , Herbert Xu , jonathan.cameron@huawei.com, grant.likely@arm.com, jean-philippe , Jerome Glisse , ilias.apalodimas@linaro.org, francois.ozog@linaro.org, kenneth-lee-2012@foxmail.com, Wangzhou , "haojian . zhuang" , guodong.xu@linaro.org Cc: linux-accelerators@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, iommu@lists.linux-foundation.org, Zhangfei Gao Subject: [PATCH v7 3/3] crypto: hisilicon - register zip engine to uacce Date: Tue, 29 Oct 2019 14:40:16 +0800 Message-Id: <1572331216-9503-4-git-send-email-zhangfei.gao@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1572331216-9503-1-git-send-email-zhangfei.gao@linaro.org> References: <1572331216-9503-1-git-send-email-zhangfei.gao@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Register qm to uacce framework for user crypto driver Signed-off-by: Zhangfei Gao Signed-off-by: Zhou Wang --- drivers/crypto/hisilicon/qm.c | 253 ++++++++++++++++++++++++++++++-- drivers/crypto/hisilicon/qm.h | 13 +- drivers/crypto/hisilicon/zip/zip_main.c | 39 ++--- include/uapi/misc/uacce/qm.h | 23 +++ 4 files changed, 292 insertions(+), 36 deletions(-) create mode 100644 include/uapi/misc/uacce/qm.h diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c index a8ed6990..4b9cced 100644 --- a/drivers/crypto/hisilicon/qm.c +++ b/drivers/crypto/hisilicon/qm.c @@ -9,6 +9,9 @@ #include #include #include +#include +#include +#include #include "qm.h" /* eq/aeq irq enable */ @@ -465,17 +468,22 @@ static void qm_cq_head_update(struct hisi_qp *qp) static void qm_poll_qp(struct hisi_qp *qp, struct hisi_qm *qm) { - struct qm_cqe *cqe = qp->cqe + qp->qp_status.cq_head; - - if (qp->req_cb) { - while (QM_CQE_PHASE(cqe) == qp->qp_status.cqc_phase) { - dma_rmb(); - qp->req_cb(qp, qp->sqe + qm->sqe_size * cqe->sq_head); - qm_cq_head_update(qp); - cqe = qp->cqe + qp->qp_status.cq_head; - qm_db(qm, qp->qp_id, QM_DOORBELL_CMD_CQ, - qp->qp_status.cq_head, 0); - atomic_dec(&qp->qp_status.used); + struct qm_cqe *cqe; + + if (qp->event_cb) { + qp->event_cb(qp); + } else { + cqe = qp->cqe + qp->qp_status.cq_head; + + if (qp->req_cb) { + while (QM_CQE_PHASE(cqe) == qp->qp_status.cqc_phase) { + dma_rmb(); + qp->req_cb(qp, qp->sqe + qm->sqe_size * + cqe->sq_head); + qm_cq_head_update(qp); + cqe = qp->cqe + qp->qp_status.cq_head; + atomic_dec(&qp->qp_status.used); + } } /* set c_flag */ @@ -1397,6 +1405,220 @@ static void hisi_qm_cache_wb(struct hisi_qm *qm) } } +static void qm_qp_event_notifier(struct hisi_qp *qp) +{ + wake_up_interruptible(&qp->uacce_q->wait); +} + +static int hisi_qm_get_available_instances(struct uacce_device *uacce) +{ + int i, ret; + struct hisi_qm *qm = uacce->priv; + + read_lock(&qm->qps_lock); + for (i = 0, ret = 0; i < qm->qp_num; i++) + if (!qm->qp_array[i]) + ret++; + read_unlock(&qm->qps_lock); + + return ret; +} + +static int hisi_qm_uacce_get_queue(struct uacce_device *uacce, + unsigned long arg, + struct uacce_queue *q) +{ + struct hisi_qm *qm = uacce->priv; + struct hisi_qp *qp; + u8 alg_type = 0; + + qp = hisi_qm_create_qp(qm, alg_type); + if (IS_ERR(qp)) + return PTR_ERR(qp); + + q->priv = qp; + q->uacce = uacce; + qp->uacce_q = q; + qp->event_cb = qm_qp_event_notifier; + qp->pasid = arg; + + return 0; +} + +static void hisi_qm_uacce_put_queue(struct uacce_queue *q) +{ + struct hisi_qp *qp = q->priv; + + /* + * As put_queue is only called in uacce_mode=1, and only one queue can + * be used in this mode. we flush all sqc cache back in put queue. + */ + hisi_qm_cache_wb(qp->qm); + + /* need to stop hardware, but can not support in v1 */ + hisi_qm_release_qp(qp); +} + +/* map sq/cq/doorbell to user space */ +static int hisi_qm_uacce_mmap(struct uacce_queue *q, + struct vm_area_struct *vma, + struct uacce_qfile_region *qfr) +{ + struct hisi_qp *qp = q->priv; + struct hisi_qm *qm = qp->qm; + size_t sz = vma->vm_end - vma->vm_start; + struct pci_dev *pdev = qm->pdev; + struct device *dev = &pdev->dev; + unsigned long vm_pgoff; + int ret; + + switch (qfr->type) { + case UACCE_QFRT_MMIO: + if (qm->ver == QM_HW_V2) { + if (sz > PAGE_SIZE * (QM_DOORBELL_PAGE_NR + + QM_DOORBELL_SQ_CQ_BASE_V2 / PAGE_SIZE)) + return -EINVAL; + } else { + if (sz > PAGE_SIZE * QM_DOORBELL_PAGE_NR) + return -EINVAL; + } + + vma->vm_flags |= VM_IO; + + return remap_pfn_range(vma, vma->vm_start, + qm->phys_base >> PAGE_SHIFT, + sz, pgprot_noncached(vma->vm_page_prot)); + case UACCE_QFRT_DUS: + if (sz != qp->qdma.size) + return -EINVAL; + + /* dma_mmap_coherent() requires vm_pgoff as 0 + * restore vm_pfoff to initial value for mmap() + */ + vm_pgoff = vma->vm_pgoff; + vma->vm_pgoff = 0; + ret = dma_mmap_coherent(dev, vma, qp->qdma.va, + qp->qdma.dma, sz); + vma->vm_pgoff = vm_pgoff; + return ret; + + default: + return -EINVAL; + } +} + +static int hisi_qm_uacce_start_queue(struct uacce_queue *q) +{ + struct hisi_qp *qp = q->priv; + + return hisi_qm_start_qp(qp, qp->pasid); +} + +static void hisi_qm_uacce_stop_queue(struct uacce_queue *q) +{ + struct hisi_qp *qp = q->priv; + + hisi_qm_stop_qp(qp); +} + +static int qm_set_sqctype(struct uacce_queue *q, u16 type) +{ + struct hisi_qm *qm = q->uacce->priv; + struct hisi_qp *qp = q->priv; + + write_lock(&qm->qps_lock); + qp->alg_type = type; + write_unlock(&qm->qps_lock); + + return 0; +} + +static long hisi_qm_uacce_ioctl(struct uacce_queue *q, unsigned int cmd, + unsigned long arg) +{ + struct hisi_qp *qp = q->priv; + struct hisi_qp_ctx qp_ctx; + + if (cmd == UACCE_CMD_QM_SET_QP_CTX) { + if (copy_from_user(&qp_ctx, (void __user *)arg, + sizeof(struct hisi_qp_ctx))) + return -EFAULT; + + if (qp_ctx.qc_type != 0 && qp_ctx.qc_type != 1) + return -EINVAL; + + qm_set_sqctype(q, qp_ctx.qc_type); + qp_ctx.id = qp->qp_id; + + if (copy_to_user((void __user *)arg, &qp_ctx, + sizeof(struct hisi_qp_ctx))) + return -EFAULT; + } else { + return -EINVAL; + } + + return 0; +} + +static struct uacce_ops uacce_qm_ops = { + .get_available_instances = hisi_qm_get_available_instances, + .get_queue = hisi_qm_uacce_get_queue, + .put_queue = hisi_qm_uacce_put_queue, + .start_queue = hisi_qm_uacce_start_queue, + .stop_queue = hisi_qm_uacce_stop_queue, + .mmap = hisi_qm_uacce_mmap, + .ioctl = hisi_qm_uacce_ioctl, +}; + +static int qm_register_uacce(struct hisi_qm *qm) +{ + struct pci_dev *pdev = qm->pdev; + struct uacce_device *uacce; + unsigned long mmio_page_nr; + unsigned long dus_page_nr; + struct uacce_interface interface = { + .flags = UACCE_DEV_SVA, + .ops = &uacce_qm_ops, + }; + + strncpy(interface.name, pdev->driver->name, sizeof(interface.name)); + + uacce = uacce_register(&pdev->dev, &interface); + if (IS_ERR(uacce)) + return PTR_ERR(uacce); + + if (uacce->flags & UACCE_DEV_SVA) { + qm->use_sva = true; + } else { + /* only consider sva case */ + uacce_unregister(uacce); + return -EINVAL; + } + + uacce->is_vf = pdev->is_virtfn; + uacce->priv = qm; + uacce->algs = qm->algs; + + if (qm->ver == QM_HW_V1) { + mmio_page_nr = QM_DOORBELL_PAGE_NR; + uacce->api_ver = HISI_QM_API_VER_BASE; + } else { + mmio_page_nr = QM_DOORBELL_PAGE_NR + + QM_DOORBELL_SQ_CQ_BASE_V2 / PAGE_SIZE; + uacce->api_ver = HISI_QM_API_VER2_BASE; + } + + dus_page_nr = (PAGE_SIZE - 1 + qm->sqe_size * QM_Q_DEPTH + + sizeof(struct qm_cqe) * QM_Q_DEPTH) >> PAGE_SHIFT; + + uacce->qf_pg_size[UACCE_QFRT_MMIO] = mmio_page_nr; + uacce->qf_pg_size[UACCE_QFRT_DUS] = dus_page_nr; + + qm->uacce = uacce; + + return 0; +} + /** * hisi_qm_init() - Initialize configures about qm. * @qm: The qm needing init. @@ -1421,6 +1643,10 @@ int hisi_qm_init(struct hisi_qm *qm) return -EINVAL; } + ret = qm_register_uacce(qm); + if (ret < 0) + dev_warn(&pdev->dev, "fail to register uacce (%d)\n", ret); + ret = pci_enable_device_mem(pdev); if (ret < 0) { dev_err(&pdev->dev, "Failed to enable device mem!\n"); @@ -1433,6 +1659,8 @@ int hisi_qm_init(struct hisi_qm *qm) goto err_disable_pcidev; } + qm->phys_base = pci_resource_start(pdev, PCI_BAR_2); + qm->size = pci_resource_len(qm->pdev, PCI_BAR_2); qm->io_base = ioremap(pci_resource_start(pdev, PCI_BAR_2), pci_resource_len(qm->pdev, PCI_BAR_2)); if (!qm->io_base) { @@ -1504,6 +1732,9 @@ void hisi_qm_uninit(struct hisi_qm *qm) iounmap(qm->io_base); pci_release_mem_regions(pdev); pci_disable_device(pdev); + + if (qm->uacce) + uacce_unregister(qm->uacce); } EXPORT_SYMBOL_GPL(hisi_qm_uninit); diff --git a/drivers/crypto/hisilicon/qm.h b/drivers/crypto/hisilicon/qm.h index 103e2fd..84a3be9 100644 --- a/drivers/crypto/hisilicon/qm.h +++ b/drivers/crypto/hisilicon/qm.h @@ -77,6 +77,10 @@ #define HISI_ACC_SGL_SGE_NR_MAX 255 +/* page number for queue file region */ +#define QM_DOORBELL_PAGE_NR 1 + + enum qp_state { QP_STOP, }; @@ -161,7 +165,12 @@ struct hisi_qm { u32 error_mask; u32 msi_mask; + const char *algs; bool use_dma_api; + bool use_sva; + resource_size_t phys_base; + resource_size_t size; + struct uacce_device *uacce; }; struct hisi_qp_status { @@ -191,10 +200,12 @@ struct hisi_qp { struct hisi_qp_ops *hw_ops; void *qp_ctx; void (*req_cb)(struct hisi_qp *qp, void *data); + void (*event_cb)(struct hisi_qp *qp); struct work_struct work; struct workqueue_struct *wq; - struct hisi_qm *qm; + u16 pasid; + struct uacce_queue *uacce_q; }; int hisi_qm_init(struct hisi_qm *qm); diff --git a/drivers/crypto/hisilicon/zip/zip_main.c b/drivers/crypto/hisilicon/zip/zip_main.c index 1b2ee96..48860d2 100644 --- a/drivers/crypto/hisilicon/zip/zip_main.c +++ b/drivers/crypto/hisilicon/zip/zip_main.c @@ -316,8 +316,14 @@ static void hisi_zip_set_user_domain_and_cache(struct hisi_zip *hisi_zip) writel(AXUSER_BASE, base + HZIP_BD_RUSER_32_63); writel(AXUSER_BASE, base + HZIP_SGL_RUSER_32_63); writel(AXUSER_BASE, base + HZIP_BD_WUSER_32_63); - writel(AXUSER_BASE, base + HZIP_DATA_RUSER_32_63); - writel(AXUSER_BASE, base + HZIP_DATA_WUSER_32_63); + + if (hisi_zip->qm.use_sva) { + writel(AXUSER_BASE | AXUSER_SSV, base + HZIP_DATA_RUSER_32_63); + writel(AXUSER_BASE | AXUSER_SSV, base + HZIP_DATA_WUSER_32_63); + } else { + writel(AXUSER_BASE, base + HZIP_DATA_RUSER_32_63); + writel(AXUSER_BASE, base + HZIP_DATA_WUSER_32_63); + } /* let's open all compression/decompression cores */ writel(DECOMP_CHECK_ENABLE | ALL_COMP_DECOMP_EN, @@ -671,24 +677,12 @@ static int hisi_zip_probe(struct pci_dev *pdev, const struct pci_device_id *id) qm = &hisi_zip->qm; qm->pdev = pdev; qm->ver = rev_id; - + qm->use_dma_api = true; + qm->algs = "zlib\ngzip\n"; qm->sqe_size = HZIP_SQE_SIZE; qm->dev_name = hisi_zip_name; qm->fun_type = (pdev->device == PCI_DEVICE_ID_ZIP_PF) ? QM_HW_PF : QM_HW_VF; - switch (uacce_mode) { - case 0: - qm->use_dma_api = true; - break; - case 1: - qm->use_dma_api = false; - break; - case 2: - qm->use_dma_api = true; - break; - default: - return -EINVAL; - } ret = hisi_qm_init(qm); if (ret) { @@ -976,12 +970,10 @@ static int __init hisi_zip_init(void) goto err_pci; } - if (uacce_mode == 0 || uacce_mode == 2) { - ret = hisi_zip_register_to_crypto(); - if (ret < 0) { - pr_err("Failed to register driver to crypto.\n"); - goto err_crypto; - } + ret = hisi_zip_register_to_crypto(); + if (ret < 0) { + pr_err("Failed to register driver to crypto.\n"); + goto err_crypto; } return 0; @@ -996,8 +988,7 @@ static int __init hisi_zip_init(void) static void __exit hisi_zip_exit(void) { - if (uacce_mode == 0 || uacce_mode == 2) - hisi_zip_unregister_from_crypto(); + hisi_zip_unregister_from_crypto(); pci_unregister_driver(&hisi_zip_pci_driver); hisi_zip_unregister_debugfs(); } diff --git a/include/uapi/misc/uacce/qm.h b/include/uapi/misc/uacce/qm.h new file mode 100644 index 0000000..d79a8f2 --- /dev/null +++ b/include/uapi/misc/uacce/qm.h @@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */ +#ifndef HISI_QM_USR_IF_H +#define HISI_QM_USR_IF_H + +#include + +/** + * struct hisi_qp_ctx - User data for hisi qp. + * @id: Specifies which Turbo decode algorithm to use + * @qc_type: Accelerator algorithm type + */ +struct hisi_qp_ctx { + __u16 id; + __u16 qc_type; +}; + +#define HISI_QM_API_VER_BASE "hisi_qm_v1" +#define HISI_QM_API_VER2_BASE "hisi_qm_v2" + +/* UACCE_CMD_QM_SET_QP_CTX: Set qp algorithm type */ +#define UACCE_CMD_QM_SET_QP_CTX _IOWR('H', 10, struct hisi_qp_ctx) + +#endif