From patchwork Wed Mar 13 16:04:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pierre Morel X-Patchwork-Id: 10851445 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 65E52139A for ; Wed, 13 Mar 2019 16:05:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 50CF729B93 for ; Wed, 13 Mar 2019 16:05:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4BCCD2A0F4; Wed, 13 Mar 2019 16:05:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7F9542A0F4 for ; Wed, 13 Mar 2019 16:05:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726074AbfCMQFN (ORCPT ); Wed, 13 Mar 2019 12:05:13 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:35474 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725876AbfCMQFM (ORCPT ); Wed, 13 Mar 2019 12:05:12 -0400 Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x2DG4M2j066931 for ; Wed, 13 Mar 2019 12:05:11 -0400 Received: from e06smtp02.uk.ibm.com (e06smtp02.uk.ibm.com [195.75.94.98]) by mx0a-001b2d01.pphosted.com with ESMTP id 2r73mtw186-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 13 Mar 2019 12:05:11 -0400 Received: from localhost by e06smtp02.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 13 Mar 2019 16:05:08 -0000 Received: from b06cxnps3074.portsmouth.uk.ibm.com (9.149.109.194) by e06smtp02.uk.ibm.com (192.168.101.132) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Wed, 13 Mar 2019 16:05:06 -0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x2DG54D416318718 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 13 Mar 2019 16:05:05 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D9E0AA404D; Wed, 13 Mar 2019 16:05:04 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 77020A4040; Wed, 13 Mar 2019 16:05:04 +0000 (GMT) Received: from morel-ThinkPad-W530.boeblingen.de.ibm.com (unknown [9.152.224.140]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 13 Mar 2019 16:05:04 +0000 (GMT) From: Pierre Morel To: borntraeger@de.ibm.com Cc: alex.williamson@redhat.com, cohuck@redhat.com, linux-kernel@vger.kernel.org, linux-s390@vger.kernel.org, kvm@vger.kernel.org, frankja@linux.ibm.com, akrowiak@linux.ibm.com, pasic@linux.ibm.com, david@redhat.com, schwidefsky@de.ibm.com, heiko.carstens@de.ibm.com, freude@linux.ibm.com, mimu@linux.ibm.com Subject: [PATCH v5 0/7] vfio: ap: AP Queue Interrupt Control Date: Wed, 13 Mar 2019 17:04:57 +0100 X-Mailer: git-send-email 2.7.4 X-TM-AS-GCONF: 00 x-cbid: 19031316-0008-0000-0000-000002CC2DB8 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19031316-0009-0000-0000-000022385317 Message-Id: <1552493104-30510-1-git-send-email-pmorel@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-03-13_10:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=3 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=875 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1903130114 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch implement PQAP/AQIC interception in KVM. To implement this we need to add a new structure, vfio_ap_queue,to be able to retrieve the mediated device associated with a queue and specific values needed to register/unregister the interrupt structures: - APQN: to be able to issue the commands and search for queue structures - NIB : to unpin the NIB on clear IRQ - ISC : to unregister with the GIB interface - MATRIX: a pointer to the matrix mediated device - LIST: the list_head to handle the vfio_queue life cycle Having this structure and the list management greatly ease the handling of the AP queues and diminues the LOCs needed in the vfio_ap driver by more than 150 lines in comparison with the previous version. 0) Queues life cycle vfio_ap_queues are created on probe We define one bucket on the matrix device to store the free vfio_ap_queues, the queues not assign to any matrix mediated device. We define one bucket on each matrix mediated device to hold the vfio_ap_queues belonging to it. vfio_ap_queues are deleted on remove This makes the search for a queue easy and the detection of assignent incoherency obvious (the queue is not avilable) and simplifies assignment. 1) Phase 1, probe and remove from vfio_ap_queue The vfio_ap_queue structures are dynamically allocated and setup when a queue is probed by the ap_vfio_driver. The vfio_ap_queue is linked to the ap_queue device as the driver data. The new The vfio_ap_queue is put on a free_list belonging to the matrix device. The vfio_ap_queue are free during remove. 2) Phase 2, assignment of vfio_ap_queue to a mediated device When a APID is assigned we look for APQI already assigned to the matrix mediated device and associate all the queue with the APQN = (APID,APQI) to the mediated device by adding them to the mediated device queue list. We do the same when a APQI is assigned. If any queue with a matching APQN can not be found on the matrix device free list it means it is already associated to another matrix mediated device and no queue is added to the matrix mediated device. 3) Phase 3, starting the guest When the VFIO device is opened the PQAP callback and a pointer to the matrix mediated device are set inside KVM during the open callback. When the device is closed or if a queue is removed, the vfio_ap_queue is dissociated from the mediated device. 4) Phase 3 intercepting the PQAP/AQIC instruction On interception of the PQAP/AQIC instruction, the interception code makes sure the pqap_hook is initialized and allowed to be called and call it. Otherwise it reports the usual -EOPNOTSUPP return code to let QEMU handle the fault. the pqap callback search for the queue asociated with the APQN stored in the register 0, setting the code to "illegal APQN" if the vfio_ap_queue can not be found. Depending on the "i" bit of the register 1, the pqap callback setup or clear the interruption by calling the host format PQAP/AQIC instruction. When seting up the interruption it uses the NIB and the guest ISC provided by the guest and the host ISC provided by the registration to the GIB code, pin the NIB and also stores ISC and NIB inside the vfio_ap_queue structure. When clearing the interrupt it retrieves the host ISC to unregister with the GIB code and unpin the NIB. We take care when enabling GISA that the guest may have issued a reset and will not need to disable the interuptions before re-enabling interruptions. To make sure that the module holding the callback does not disapear we use a module reference counting in the structure containing the callback. 5) Phase 4 clean dissociation from the mediated device on remove On removing of the AP device the remove callback is called. To be sure that the guest will not access the queue anymore we clear the APID CRYCB bit. Cleaning the APID, over the APQI, is chosen because the architecture specifies that only the APID can be dynamically changed outside IPL. To be sure that the IRQ is cleared before the GISA is free we use the KVM reference counting, raise it in open, lower it on release. 6) Associated QEMU patch There is a QEMU patch which is needed to enable the PQAP/AQIC facility in the guest. Posted in qemu-devel@nongnu.org as: Message-Id: <1550146494-21085-1-git-send-email-pmorel@linux.ibm.com> Pierre Morel (7): s390: ap: kvm: add PQAP interception for AQIC s390: ap: new vfio_ap_queue structure vfio: ap: register IOMMU VFIO notifier s390: ap: setup relation betwen KVM and mediated device s390: ap: implement PAPQ AQIC interception in kernel s390: ap: Cleanup on removing the AP device s390: ap: kvm: Enable PQAP/AQIC facility for the guest arch/s390/include/asm/kvm_host.h | 8 + arch/s390/kvm/priv.c | 62 +++ arch/s390/tools/gen_facilities.c | 1 + drivers/s390/crypto/ap_bus.h | 1 + drivers/s390/crypto/vfio_ap_drv.c | 69 +++- drivers/s390/crypto/vfio_ap_ops.c | 727 +++++++++++++++++++++++----------- drivers/s390/crypto/vfio_ap_private.h | 20 + 7 files changed, 658 insertions(+), 230 deletions(-)