From patchwork Fri Feb 17 08:29:28 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dong Jia Shi X-Patchwork-Id: 9579163 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 403996042F for ; Fri, 17 Feb 2017 08:30:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 32F202868A for ; Fri, 17 Feb 2017 08:30:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 27B0A286AB; Fri, 17 Feb 2017 08:30:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6EC9E2868A for ; Fri, 17 Feb 2017 08:30:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932918AbdBQIaC (ORCPT ); Fri, 17 Feb 2017 03:30:02 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:33336 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932882AbdBQI36 (ORCPT ); Fri, 17 Feb 2017 03:29:58 -0500 Received: from pps.filterd (m0098413.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.20/8.16.0.20) with SMTP id v1H8T0Xf009678 for ; Fri, 17 Feb 2017 03:29:58 -0500 Received: from e17.ny.us.ibm.com (e17.ny.us.ibm.com [129.33.205.207]) by mx0b-001b2d01.pphosted.com with ESMTP id 28npy5w3cs-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Fri, 17 Feb 2017 03:29:57 -0500 Received: from localhost by e17.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 17 Feb 2017 03:29:57 -0500 Received: from d01dlp02.pok.ibm.com (9.56.250.167) by e17.ny.us.ibm.com (146.89.104.204) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Fri, 17 Feb 2017 03:29:54 -0500 Received: from b01cxnp23033.gho.pok.ibm.com (b01cxnp23033.gho.pok.ibm.com [9.57.198.28]) by d01dlp02.pok.ibm.com (Postfix) with ESMTP id 7ED1D6E801D; Fri, 17 Feb 2017 03:29:25 -0500 (EST) Received: from b01ledav03.gho.pok.ibm.com (b01ledav003.gho.pok.ibm.com [9.57.199.108]) by b01cxnp23033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id v1H8Trjh20251124; Fri, 17 Feb 2017 08:29:53 GMT Received: from b01ledav03.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9B8A5B2050; Fri, 17 Feb 2017 03:29:52 -0500 (EST) Received: from tuxmaker.boeblingen.de.ibm.com (unknown [9.152.85.9]) by b01ledav03.gho.pok.ibm.com (Postfix) with ESMTP id 5BE40B204D; Fri, 17 Feb 2017 03:29:51 -0500 (EST) From: Dong Jia Shi To: kvm@vger.kernel.org, linux-s390@vger.kernel.org, qemu-devel@nongnu.org Cc: bjsdjshi@linux.vnet.ibm.com, renxiaof@linux.vnet.ibm.com, cornelia.huck@de.ibm.com, borntraeger@de.ibm.com, agraf@suse.com, alex.williamson@redhat.com, pmorel@linux.vnet.ibm.com, pasic@linux.vnet.ibm.com, wkywang@linux.vnet.ibm.com Subject: [PATCH RFC v3 04/15] vfio: ccw: basic implementation for vfio_ccw driver Date: Fri, 17 Feb 2017 09:29:28 +0100 X-Mailer: git-send-email 2.8.4 In-Reply-To: <20170217082939.33208-1-bjsdjshi@linux.vnet.ibm.com> References: <20170217082939.33208-1-bjsdjshi@linux.vnet.ibm.com> X-TM-AS-GCONF: 00 X-Content-Scanned: Fidelis XPS MAILER x-cbid: 17021708-0040-0000-0000-000002A6C5D7 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00006631; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000203; SDB=6.00823158; UDB=6.00402782; IPR=6.00600604; BA=6.00005146; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00014319; XFM=3.00000011; UTC=2017-02-17 08:29:56 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17021708-0041-0000-0000-00000699E5EA Message-Id: <20170217082939.33208-5-bjsdjshi@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-02-17_07:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=2 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1612050000 definitions=main-1702170081 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP To make vfio support subchannel devices, we need a css driver for the vfio subchannels. This patch adds a basic vfio-ccw subchannel driver for this purpose. To enable VFIO for vfio-ccw, enable S390_CCW_IOMMU config option and configure VFIO as required. Signed-off-by: Dong Jia Shi Acked-by: Pierre Morel --- arch/s390/Kconfig | 10 ++ arch/s390/include/asm/isc.h | 1 + drivers/iommu/Kconfig | 8 ++ drivers/s390/cio/Makefile | 3 + drivers/s390/cio/vfio_ccw_drv.c | 262 ++++++++++++++++++++++++++++++++++++ drivers/s390/cio/vfio_ccw_private.h | 25 ++++ 6 files changed, 309 insertions(+) create mode 100644 drivers/s390/cio/vfio_ccw_drv.c create mode 100644 drivers/s390/cio/vfio_ccw_private.h diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index c6722112..b920df8 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -670,6 +670,16 @@ config EADM_SCH To compile this driver as a module, choose M here: the module will be called eadm_sch. +config VFIO_CCW + def_tristate n + prompt "Support for VFIO-CCW subchannels" + depends on S390_CCW_IOMMU && VFIO + help + This driver allows usage of VFIO-CCW subchannels. + + To compile this driver as a module, choose M here: the + module will be called vfio_ccw. + endmenu menu "Dump support" diff --git a/arch/s390/include/asm/isc.h b/arch/s390/include/asm/isc.h index 68d7d68..8a0b721 100644 --- a/arch/s390/include/asm/isc.h +++ b/arch/s390/include/asm/isc.h @@ -16,6 +16,7 @@ #define CONSOLE_ISC 1 /* console I/O subchannel */ #define EADM_SCH_ISC 4 /* EADM subchannels */ #define CHSC_SCH_ISC 7 /* CHSC subchannels */ +#define VFIO_CCW_ISC IO_SCH_ISC /* VFIO-CCW I/O subchannels */ /* Adapter interrupts. */ #define QDIO_AIRQ_ISC IO_SCH_ISC /* I/O subchannel in qdio mode */ #define PCI_ISC 2 /* PCI I/O subchannels */ diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index 8ee54d7..a5626d4 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -327,6 +327,14 @@ config S390_IOMMU help Support for the IOMMU API for s390 PCI devices. +config S390_CCW_IOMMU + bool "S390 CCW IOMMU Support" + depends on S390 && CCW + select IOMMU_API + help + Enables bits of IOMMU API required by VFIO. The iommu_ops + is not implemented as it is not necessary for VFIO. + config MTK_IOMMU bool "MTK IOMMU Support" depends on ARM || ARM64 diff --git a/drivers/s390/cio/Makefile b/drivers/s390/cio/Makefile index 3ab9aed..3d7390e 100644 --- a/drivers/s390/cio/Makefile +++ b/drivers/s390/cio/Makefile @@ -17,3 +17,6 @@ obj-$(CONFIG_CCWGROUP) += ccwgroup.o qdio-objs := qdio_main.o qdio_thinint.o qdio_debug.o qdio_setup.o obj-$(CONFIG_QDIO) += qdio.o + +vfio_ccw-objs += vfio_ccw_drv.o +obj-$(CONFIG_VFIO_CCW) += vfio_ccw.o diff --git a/drivers/s390/cio/vfio_ccw_drv.c b/drivers/s390/cio/vfio_ccw_drv.c new file mode 100644 index 0000000..b068207 --- /dev/null +++ b/drivers/s390/cio/vfio_ccw_drv.c @@ -0,0 +1,262 @@ +/* + * VFIO based Physical Subchannel device driver + * + * Copyright IBM Corp. 2017 + * + * Author(s): Dong Jia Shi + * Xiao Feng Ren + */ + +#include +#include +#include +#include + +#include + +#include "vfio_ccw_private.h" + +/* + * Helpers + */ +static int vfio_ccw_sch_quiesce(struct subchannel *sch) +{ + struct vfio_ccw_private *private = dev_get_drvdata(&sch->dev); + DECLARE_COMPLETION_ONSTACK(completion); + int iretry, ret = 0; + + spin_lock_irq(sch->lock); + if (!sch->schib.pmcw.ena) + goto out_unlock; + ret = cio_disable_subchannel(sch); + if (ret != -EBUSY) + goto out_unlock; + + do { + iretry = 255; + + ret = cio_cancel_halt_clear(sch, &iretry); + while (ret == -EBUSY) { + /* + * Flushing all I/O and wait the + * cancel/halt/clear completion. + */ + private->completion = &completion; + spin_unlock_irq(sch->lock); + + wait_for_completion(&completion); + + spin_lock_irq(sch->lock); + private->completion = NULL; + ret = cio_cancel_halt_clear(sch, &iretry); + }; + + ret = cio_disable_subchannel(sch); + } while (ret == -EBUSY); + +out_unlock: + spin_unlock_irq(sch->lock); + return ret; +} + +/* + * Sysfs interfaces + */ +static ssize_t chpids_show(struct device *dev, + struct device_attribute *attr, + char *buf) +{ + struct subchannel *sch = to_subchannel(dev); + struct chsc_ssd_info *ssd = &sch->ssd_info; + ssize_t ret = 0; + int chp; + int mask; + + for (chp = 0; chp < 8; chp++) { + mask = 0x80 >> chp; + if (ssd->path_mask & mask) + ret += sprintf(buf + ret, "%02x ", ssd->chpid[chp].id); + else + ret += sprintf(buf + ret, "00 "); + } + ret += sprintf(buf+ret, "\n"); + return ret; +} + +static ssize_t pimpampom_show(struct device *dev, + struct device_attribute *attr, + char *buf) +{ + struct subchannel *sch = to_subchannel(dev); + struct pmcw *pmcw = &sch->schib.pmcw; + + return sprintf(buf, "%02x %02x %02x\n", + pmcw->pim, pmcw->pam, pmcw->pom); +} + +static DEVICE_ATTR(chpids, 0444, chpids_show, NULL); +static DEVICE_ATTR(pimpampom, 0444, pimpampom_show, NULL); + +static struct attribute *vfio_subchannel_attrs[] = { + &dev_attr_chpids.attr, + &dev_attr_pimpampom.attr, + NULL, +}; + +static struct attribute_group vfio_subchannel_attr_group = { + .attrs = vfio_subchannel_attrs, +}; + +/* + * Css driver callbacks + */ +static void vfio_ccw_sch_irq(struct subchannel *sch) +{ + struct vfio_ccw_private *private = dev_get_drvdata(&sch->dev); + + inc_irq_stat(IRQIO_CIO); + + if (!private) + return; + + if (private->completion) + complete(private->completion); +} + +static int vfio_ccw_sch_probe(struct subchannel *sch) +{ + struct pmcw *pmcw = &sch->schib.pmcw; + struct vfio_ccw_private *private; + int ret; + + if (pmcw->qf) { + dev_warn(&sch->dev, "vfio: ccw: do not support QDIO: %s\n", + dev_name(&sch->dev)); + return -ENOTTY; + } + + private = kzalloc(sizeof(*private), GFP_KERNEL | GFP_DMA); + if (!private) + return -ENOMEM; + private->sch = sch; + dev_set_drvdata(&sch->dev, private); + + spin_lock_irq(sch->lock); + sch->isc = VFIO_CCW_ISC; + ret = cio_enable_subchannel(sch, (u32)(unsigned long)sch); + spin_unlock_irq(sch->lock); + if (ret) + goto out_free; + + ret = sysfs_create_group(&sch->dev.kobj, &vfio_subchannel_attr_group); + if (ret) + goto out_disable; + + return 0; + +out_disable: + cio_disable_subchannel(sch); +out_free: + dev_set_drvdata(&sch->dev, NULL); + kfree(private); + return ret; +} + +static int vfio_ccw_sch_remove(struct subchannel *sch) +{ + struct vfio_ccw_private *private = dev_get_drvdata(&sch->dev); + + vfio_ccw_sch_quiesce(sch); + + sysfs_remove_group(&sch->dev.kobj, &vfio_subchannel_attr_group); + + dev_set_drvdata(&sch->dev, NULL); + + kfree(private); + + return 0; +} + +static void vfio_ccw_sch_shutdown(struct subchannel *sch) +{ + vfio_ccw_sch_quiesce(sch); +} + +/** + * vfio_ccw_sch_event - process subchannel event + * @sch: subchannel + * @process: non-zero if function is called in process context + * + * An unspecified event occurred for this subchannel. Adjust data according + * to the current operational state of the subchannel. Return zero when the + * event has been handled sufficiently or -EAGAIN when this function should + * be called again in process context. + */ +static int vfio_ccw_sch_event(struct subchannel *sch, int process) +{ + unsigned long flags; + + spin_lock_irqsave(sch->lock, flags); + if (!device_is_registered(&sch->dev)) + goto out_unlock; + + if (work_pending(&sch->todo_work)) + goto out_unlock; + + if (cio_update_schib(sch)) { + /* Not operational. */ + css_sched_sch_todo(sch, SCH_TODO_UNREG); + + /* + * TODO: + * Probably we should send the machine check to the guest. + */ + goto out_unlock; + } + +out_unlock: + spin_unlock_irqrestore(sch->lock, flags); + + return 0; +} + +static struct css_device_id vfio_ccw_sch_ids[] = { + { .match_flags = 0x1, .type = SUBCHANNEL_TYPE_IO, }, + { /* end of list */ }, +}; +MODULE_DEVICE_TABLE(css, vfio_ccw_sch_ids); + +static struct css_driver vfio_ccw_sch_driver = { + .drv = { + .name = "vfio_ccw", + .owner = THIS_MODULE, + }, + .subchannel_type = vfio_ccw_sch_ids, + .irq = vfio_ccw_sch_irq, + .probe = vfio_ccw_sch_probe, + .remove = vfio_ccw_sch_remove, + .shutdown = vfio_ccw_sch_shutdown, + .sch_event = vfio_ccw_sch_event, +}; + +static int __init vfio_ccw_sch_init(void) +{ + int ret; + + isc_register(VFIO_CCW_ISC); + ret = css_driver_register(&vfio_ccw_sch_driver); + if (ret) + isc_unregister(VFIO_CCW_ISC); + + return ret; +} + +static void __exit vfio_ccw_sch_exit(void) +{ + css_driver_unregister(&vfio_ccw_sch_driver); + isc_unregister(VFIO_CCW_ISC); +} +module_init(vfio_ccw_sch_init); +module_exit(vfio_ccw_sch_exit); + +MODULE_LICENSE("GPL v2"); diff --git a/drivers/s390/cio/vfio_ccw_private.h b/drivers/s390/cio/vfio_ccw_private.h new file mode 100644 index 0000000..38d69a5 --- /dev/null +++ b/drivers/s390/cio/vfio_ccw_private.h @@ -0,0 +1,25 @@ +/* + * Private stuff for vfio_ccw driver + * + * Copyright IBM Corp. 2017 + * + * Author(s): Dong Jia Shi + * Xiao Feng Ren + */ + +#ifndef _VFIO_CCW_PRIVATE_H_ +#define _VFIO_CCW_PRIVATE_H_ + +#include "css.h" + +/** + * struct vfio_ccw_private + * @sch: pointer to the subchannel + * @completion: synchronization helper of the I/O completion + */ +struct vfio_ccw_private { + struct subchannel *sch; + struct completion *completion; +} __aligned(8); + +#endif